init
This commit is contained in:
commit
c6bb3cd1fc
|
@ -0,0 +1,4 @@
|
||||||
|
*.yaml
|
||||||
|
*.yml
|
||||||
|
dist/
|
||||||
|
rss-butt-plug
|
|
@ -0,0 +1,137 @@
|
||||||
|
# rss-butt-plug
|
||||||
|
|
||||||
|
> :peach: **e x p e r i m e n t a l** :peach:
|
||||||
|
|
||||||
|
A [SSB](https://scuttlebutt.nz) client which "plugs" a RSS feed into the Scuttleverse.
|
||||||
|
|
||||||
|
Created in the same spirit as [Lykin](https://git.coopcloud.tech/glyph/lykin),
|
||||||
|
for educational purposes and to help others dip a toe into the SSB development
|
||||||
|
ecosystem. It's mostly a demo and I don't necessarily intend to maintain it.
|
||||||
|
|
||||||
|
Here's a [short
|
||||||
|
screencast](http://vvvvvvaria.org/~decentral1se/ssb/rss-butt-plug/rbpc.mp4)
|
||||||
|
with a quick tour of the tool and some thoughts on it.
|
||||||
|
|
||||||
|
**Please** read the [Be Gentle](#be-gentle-monocle_face) section before using it on the mainnet.
|
||||||
|
|
||||||
|
## Scuttlin' :nerd_face:
|
||||||
|
|
||||||
|
Get a local copy of `rss-butt-plug`:
|
||||||
|
|
||||||
|
```
|
||||||
|
curl https://vvvvvvaria.org/~decentral1se/ssb/rss-butt-plug/rss-butt-plug_linux_amd64_v1/rss-butt-plug -o rss-butt-plug
|
||||||
|
chmod +x rss-butt-plug
|
||||||
|
```
|
||||||
|
|
||||||
|
If you're not running `amd64` arch, take a look in [vvvvvvaria.org/~decentral1se/ssb/rss-butt-plug](https://vvvvvvaria.org/~decentral1se/ssb/rss-butt-plug) for binaries which suit your system.
|
||||||
|
|
||||||
|
Create a `rss-butt-plug.yaml` in the same directory:
|
||||||
|
|
||||||
|
```yaml
|
||||||
|
---
|
||||||
|
# where all data will be stored, is a relative path to the current working directory
|
||||||
|
data-dir: .rss-butt-plug
|
||||||
|
|
||||||
|
# the RSS feed URL
|
||||||
|
feed: https://openrss.org/opencollective.com/secure-scuttlebutt-consortium/updates
|
||||||
|
|
||||||
|
# the RSS feed profile avatar URL (will be converted to blob)
|
||||||
|
avatar: https://images.opencollective.com/secure-scuttlebutt-consortium/676f245/logo/256.png
|
||||||
|
|
||||||
|
# the internal go-sbot configuration options
|
||||||
|
addr: localhost
|
||||||
|
port: 8008
|
||||||
|
ws-port: 8989
|
||||||
|
shs-cap: "1KHLiKZvAvjbY1ziZEHMXawbCEIM6qwjCDm3VYRan/s="
|
||||||
|
```
|
||||||
|
|
||||||
|
Run it:
|
||||||
|
|
||||||
|
```
|
||||||
|
./rss-butt-plug
|
||||||
|
```
|
||||||
|
|
||||||
|
Wire up a local Patchwork client (or something that supports *legacy*
|
||||||
|
replication, [see FAQ for
|
||||||
|
more](https://github.com/ssbc/go-ssb/blob/master/docs/faq.md#what-is-legacy-replication))
|
||||||
|
for reading & sharing with the broader SSB ecosystem.
|
||||||
|
|
||||||
|
`rss-butt-plug` generates an invite every time it runs which you can use to
|
||||||
|
invite clients with. Feeds will be polled every 5 minutes by default, you can
|
||||||
|
configure this with `-p`.
|
||||||
|
|
||||||
|
## Limitations :stop_sign:
|
||||||
|
|
||||||
|
* Multiple RSS feeds are not supported. It would be great to have but I don't
|
||||||
|
know how to do it (afaiu, `go-sbot` is one identity per-instance). You could
|
||||||
|
run multiple instances of `rss-butt-plug`, an instance per RSS feed. You'd
|
||||||
|
need to tweak the ports in the `rss-butt-plug.yaml` to not have conflicts but
|
||||||
|
it could work.
|
||||||
|
|
||||||
|
* The HTML -> Markdown might be a bit dodgy, so I would recommend doing some
|
||||||
|
testing on local throwaway Patchwork / `rss-butt-plug` identities before
|
||||||
|
doing any mainnet replication. You can test parse a feed by passing it as an
|
||||||
|
argument, e.g. `./rss-butt-plug https://laipower.xyz/rss` and the first post
|
||||||
|
will be shown.
|
||||||
|
|
||||||
|
* All `go-ssb` experimental caveats apply, see [the
|
||||||
|
FAQ](https://github.com/ssbc/go-ssb/blob/master/docs/faq.md) for more.
|
||||||
|
|
||||||
|
## Be gentle :monocle_face:
|
||||||
|
|
||||||
|
You may feel the sudden urge to plug loads of RSS feeds into the Scuttleverse.
|
||||||
|
I certainly did. However, with some testing, I quickly realised this isn't a
|
||||||
|
very good idea.
|
||||||
|
|
||||||
|
One example, a lot of great SSB people are also on the Fediverse but the RSS
|
||||||
|
feeds provided by Mastodon trim the titles and don't carry over the images in
|
||||||
|
the posts properly (have tested in a couple of feed readers).
|
||||||
|
|
||||||
|
Also, you can't `@` a plug identity on SSB because there are no humans behind
|
||||||
|
it. So, if you plug an RSS feed connected with other aspects of your online
|
||||||
|
life then how do people speak to you?
|
||||||
|
|
||||||
|
I think it's a question of going slow and waiting for people to test out the
|
||||||
|
tool and see what dynamic it produces. Please test your feeds first, see how it
|
||||||
|
looks, see what content the feed provides and ofc, see if it "fits" being
|
||||||
|
plugged into SSB.
|
||||||
|
|
||||||
|
## Internal features :computer:
|
||||||
|
|
||||||
|
Some internal things that might be nice to know / refer to if also hacking on
|
||||||
|
this or building your own stuff. I would guess that a lot of the following
|
||||||
|
ingredients could be generalised into other kinds of clients, scripts and
|
||||||
|
tools.
|
||||||
|
|
||||||
|
`rss-butt-plug` does the following...
|
||||||
|
|
||||||
|
* Internally manages a `go-sbot` instance, handy for emdedding magic scuttlin'
|
||||||
|
superpowers in your Go scripts.
|
||||||
|
|
||||||
|
* Creates a TCP client which makes requests to the managed `go-sbot` via the
|
||||||
|
MUXRPC interface. E.g. generates an invite.
|
||||||
|
|
||||||
|
* In general, uses some "direct" Go API bindings which work on the local log.
|
||||||
|
These APIs are more low-level and involved than the MUXRPC interface but are
|
||||||
|
more powerful.
|
||||||
|
|
||||||
|
* Image uploads. The HTML is parsed to look for images while converting it to
|
||||||
|
Markdown. When an image is found, it is uploaded as a blob and then the blob
|
||||||
|
ref replaces the traditional link. Clients like Patchwork then know how to
|
||||||
|
show the images in the renderer.
|
||||||
|
|
||||||
|
* Breaking up large posts into root + reply threads so that we do not go over
|
||||||
|
the length limit of a post. The implementation of this is quite a hack, so go
|
||||||
|
easy on me.
|
||||||
|
|
||||||
|
* Uses a `goreleaser` config to create cross-platform binaries.
|
||||||
|
|
||||||
|
## Inspo :sunflower:
|
||||||
|
|
||||||
|
* `%fzitwDLKcKgGL3RKv6BrBHzQRr1qC7QCeC6PpirMwwE=.sha256`
|
||||||
|
* `%JmVycMs1KqxH0LYzXDnbhNCKH9sCOtBZE/yDU+cBWoY=.sha256`
|
||||||
|
|
||||||
|
## ACK :+1:
|
||||||
|
|
||||||
|
* [`go-ssb`](https://github.com/ssbc/go-ssb)
|
||||||
|
* [`egonelbre/gophers`](https://github.com/egonelbre/gophers)
|
|
@ -0,0 +1,71 @@
|
||||||
|
module decentral1se/rss-butt-plug
|
||||||
|
|
||||||
|
go 1.18
|
||||||
|
|
||||||
|
require (
|
||||||
|
github.com/JohannesKaufmann/html-to-markdown v1.3.6
|
||||||
|
github.com/PuerkitoBio/goquery v1.8.0
|
||||||
|
github.com/mmcdole/gofeed v1.1.3
|
||||||
|
github.com/ssbc/go-luigi v0.3.7-0.20221019204020-324065b9a7c6
|
||||||
|
github.com/ssbc/go-ssb v0.2.2-0.20221114231348-43505cca26d4
|
||||||
|
github.com/ssbc/go-ssb-refs v0.5.2-0.20221019090322-8b558c2f31de
|
||||||
|
gopkg.in/yaml.v2 v2.4.0
|
||||||
|
)
|
||||||
|
|
||||||
|
require (
|
||||||
|
filippo.io/edwards25519 v1.0.0 // indirect
|
||||||
|
github.com/RoaringBitmap/roaring v1.2.1 // indirect
|
||||||
|
github.com/andybalholm/cascadia v1.3.1 // indirect
|
||||||
|
github.com/bits-and-blooms/bitset v1.3.3 // indirect
|
||||||
|
github.com/cespare/xxhash v1.1.0 // indirect
|
||||||
|
github.com/cespare/xxhash/v2 v2.1.2 // indirect
|
||||||
|
github.com/davecgh/go-spew v1.1.1 // indirect
|
||||||
|
github.com/dgraph-io/badger/v3 v3.2103.4 // indirect
|
||||||
|
github.com/dgraph-io/ristretto v0.1.1 // indirect
|
||||||
|
github.com/dgraph-io/sroar v0.0.0-20220527172339-b92b7eaaf6e0 // indirect
|
||||||
|
github.com/dustin/go-humanize v1.0.0 // indirect
|
||||||
|
github.com/go-kit/kit v0.12.0 // indirect
|
||||||
|
github.com/go-logfmt/logfmt v0.5.1 // indirect
|
||||||
|
github.com/gogo/protobuf v1.3.2 // indirect
|
||||||
|
github.com/golang/glog v1.0.0 // indirect
|
||||||
|
github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
|
||||||
|
github.com/golang/protobuf v1.5.2 // indirect
|
||||||
|
github.com/golang/snappy v0.0.4 // indirect
|
||||||
|
github.com/google/flatbuffers v22.10.26+incompatible // indirect
|
||||||
|
github.com/gorilla/websocket v1.5.0 // indirect
|
||||||
|
github.com/hashicorp/errwrap v1.1.0 // indirect
|
||||||
|
github.com/hashicorp/go-multierror v1.1.1 // indirect
|
||||||
|
github.com/json-iterator/go v1.1.12 // indirect
|
||||||
|
github.com/karrick/bufpool v1.2.0 // indirect
|
||||||
|
github.com/karrick/gopool v1.2.2 // indirect
|
||||||
|
github.com/keks/persist v0.0.0-20210520094901-9bdd97c1fad2 // indirect
|
||||||
|
github.com/klauspost/compress v1.15.12 // indirect
|
||||||
|
github.com/libp2p/go-reuseport v0.2.0 // indirect
|
||||||
|
github.com/machinebox/progress v0.2.0 // indirect
|
||||||
|
github.com/mmcdole/goxpp v0.0.0-20200921145534-2f3784f67354 // indirect
|
||||||
|
github.com/modern-go/concurrent v0.0.0-20180306012644-bacd9c7ef1dd // indirect
|
||||||
|
github.com/modern-go/reflect2 v1.0.2 // indirect
|
||||||
|
github.com/mschoch/smat v0.2.0 // indirect
|
||||||
|
github.com/pkg/errors v0.9.1 // indirect
|
||||||
|
github.com/rs/cors v1.8.2 // indirect
|
||||||
|
github.com/ssbc/go-gabbygrove v0.0.0-20221025092911-c274a44c3523 // indirect
|
||||||
|
github.com/ssbc/go-metafeed v1.1.3-0.20221019090205-458925e39156 // indirect
|
||||||
|
github.com/ssbc/go-muxrpc/v2 v2.0.14-0.20221111190521-10382533750c // indirect
|
||||||
|
github.com/ssbc/go-netwrap v0.1.5-0.20221019160355-cd323bb2e29d // indirect
|
||||||
|
github.com/ssbc/go-secretstream v1.2.11-0.20221111164233-4b41f899f844 // indirect
|
||||||
|
github.com/ssbc/go-ssb-multiserver v0.1.5-0.20221019203850-917ae0e23d57 // indirect
|
||||||
|
github.com/ssbc/margaret v0.4.4-0.20221101112304-4f5815095ef3 // indirect
|
||||||
|
github.com/ugorji/go/codec v1.2.7 // indirect
|
||||||
|
github.com/zeebo/bencode v1.0.0 // indirect
|
||||||
|
go.cryptoscope.co/nocomment v0.0.0-20210520094614-fb744e81f810 // indirect
|
||||||
|
go.mindeco.de v1.12.0 // indirect
|
||||||
|
go.opencensus.io v0.23.0 // indirect
|
||||||
|
golang.org/x/crypto v0.2.0 // indirect
|
||||||
|
golang.org/x/exp v0.0.0-20221025133541-111beb427cde // indirect
|
||||||
|
golang.org/x/net v0.2.0 // indirect
|
||||||
|
golang.org/x/sync v0.1.0 // indirect
|
||||||
|
golang.org/x/sys v0.2.0 // indirect
|
||||||
|
golang.org/x/text v0.4.0 // indirect
|
||||||
|
gonum.org/v1/gonum v0.12.0 // indirect
|
||||||
|
google.golang.org/protobuf v1.28.1 // indirect
|
||||||
|
)
|
|
@ -0,0 +1,622 @@
|
||||||
|
// package main is a SSB client which "plugs" a RSS feed into the Scuttleverse.
|
||||||
|
package main
|
||||||
|
|
||||||
|
import (
|
||||||
|
"bytes"
|
||||||
|
"context"
|
||||||
|
"encoding/json"
|
||||||
|
"flag"
|
||||||
|
"fmt"
|
||||||
|
"io"
|
||||||
|
"io/ioutil"
|
||||||
|
"log"
|
||||||
|
"net/http"
|
||||||
|
"os"
|
||||||
|
"os/signal"
|
||||||
|
"path/filepath"
|
||||||
|
"strings"
|
||||||
|
"syscall"
|
||||||
|
"time"
|
||||||
|
|
||||||
|
md "github.com/JohannesKaufmann/html-to-markdown"
|
||||||
|
"github.com/PuerkitoBio/goquery"
|
||||||
|
"github.com/mmcdole/gofeed"
|
||||||
|
"github.com/ssbc/go-luigi"
|
||||||
|
"github.com/ssbc/go-ssb"
|
||||||
|
refs "github.com/ssbc/go-ssb-refs"
|
||||||
|
ssbClient "github.com/ssbc/go-ssb/client"
|
||||||
|
"github.com/ssbc/go-ssb/message"
|
||||||
|
"github.com/ssbc/go-ssb/sbot"
|
||||||
|
"gopkg.in/yaml.v2"
|
||||||
|
)
|
||||||
|
|
||||||
|
// Config is a rss-butt-plug config file.
|
||||||
|
type Config struct {
|
||||||
|
DataDir string `yaml:"data-dir"`
|
||||||
|
Feed string `yaml:"feed"`
|
||||||
|
Addr string `yaml:"addr"`
|
||||||
|
Port string `yaml:"port"`
|
||||||
|
WsPort string `yaml:"ws-port"`
|
||||||
|
ShsCap string `yaml:"shs-cap"`
|
||||||
|
KeyPair ssb.KeyPair `yaml:"key-pair,omitempty"`
|
||||||
|
Avatar string `yaml:"avatar,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// Post is a ssb post message.
|
||||||
|
type Post struct {
|
||||||
|
Type string `json:"type"`
|
||||||
|
Link string `json:"link"`
|
||||||
|
Text string `json:"text"`
|
||||||
|
Root string `json:"root,omitempty"`
|
||||||
|
}
|
||||||
|
|
||||||
|
// help is the rss-butt-plug CLI help output.
|
||||||
|
const help = `rss-butt-plug [options] [<feed>]
|
||||||
|
|
||||||
|
A SSB client which "plugs" a RSS feed into the Scuttleverse.
|
||||||
|
|
||||||
|
An example configuration file:
|
||||||
|
|
||||||
|
---
|
||||||
|
data-dir: ~/.rss-butt-plug
|
||||||
|
feed: https://openrss.org/opencollective.com/secure-scuttlebutt-consortium/updates
|
||||||
|
addr: localhost
|
||||||
|
port: 8008
|
||||||
|
ws-port: 8989
|
||||||
|
shs-cap: "1KHLiKZvAvjbY1ziZEHMXawbCEIM6qwjCDm3VYRan/s="
|
||||||
|
avatar: https://images.opencollective.com/secure-scuttlebutt-consortium/676f245/logo/256.png
|
||||||
|
|
||||||
|
Arguments:
|
||||||
|
<feed> a feed to test parsing
|
||||||
|
|
||||||
|
Options:
|
||||||
|
-h output help
|
||||||
|
-c path to config file
|
||||||
|
-p feed poll frequency in minutes
|
||||||
|
`
|
||||||
|
|
||||||
|
// maxPostLength is a post limit set by rss-butt-plug which is smaller than the
|
||||||
|
// actual max post length of 8192, to allow some buffer. Any RSS post with a
|
||||||
|
// greater length will be split up into threads.
|
||||||
|
const maxPostLength = 7000
|
||||||
|
|
||||||
|
var helpFlag bool
|
||||||
|
var debugFlag bool
|
||||||
|
var configFlag string
|
||||||
|
var pollFrequencyFlag int
|
||||||
|
|
||||||
|
// handleCliFlags parses CLI flags.
|
||||||
|
func handleCliFlags() error {
|
||||||
|
flag.BoolVar(&helpFlag, "h", false, "output help")
|
||||||
|
flag.StringVar(&configFlag, "c", "rss-butt-plug.yaml", "config file")
|
||||||
|
flag.IntVar(&pollFrequencyFlag, "p", 5, "feed poll frequency in minutes")
|
||||||
|
flag.Parse()
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// parseRSSFeed parses an entire RSS feed into memory.
|
||||||
|
func parseRSSFeed(url string) (gofeed.Feed, error) {
|
||||||
|
ctx, cancel := context.WithTimeout(context.Background(), 60*time.Second)
|
||||||
|
defer cancel()
|
||||||
|
|
||||||
|
feedParser := gofeed.NewParser()
|
||||||
|
feed, err := feedParser.ParseURLWithContext(url, ctx)
|
||||||
|
if err != nil {
|
||||||
|
return gofeed.Feed{}, fmt.Errorf("unable to parse %s: %w", url, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return *feed, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// getImage retrieves an image from the internet.
|
||||||
|
func getImage(url string) (io.Reader, error) {
|
||||||
|
response, err := http.Get(url)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("getImage: unable to retrieve %s: %w", url, err)
|
||||||
|
}
|
||||||
|
defer response.Body.Close()
|
||||||
|
|
||||||
|
if response.StatusCode != 200 {
|
||||||
|
return nil, fmt.Errorf("getImage: unable to retrieve %s: HTTP %d", url, response.StatusCode)
|
||||||
|
}
|
||||||
|
|
||||||
|
body, error := ioutil.ReadAll(response.Body)
|
||||||
|
if error != nil {
|
||||||
|
return nil, fmt.Errorf("getImage: unable to read response body: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return bytes.NewReader(body), nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// htmlToMarkdown converts HTML to Markdown. Image links are processed into
|
||||||
|
// blob refs for SSB client readers.
|
||||||
|
func htmlToMarkdown(content string, pub *sbot.Sbot, postBlobs bool) (string, error) {
|
||||||
|
var markdown string
|
||||||
|
|
||||||
|
converter := md.NewConverter("", true, nil)
|
||||||
|
|
||||||
|
converter.AddRules(
|
||||||
|
md.Rule{
|
||||||
|
Filter: []string{"img"},
|
||||||
|
Replacement: func(content string, selec *goquery.Selection, opt *md.Options) *string {
|
||||||
|
if postBlobs {
|
||||||
|
src, _ := selec.Attr("src")
|
||||||
|
srcReader, err := getImage(src)
|
||||||
|
if err != nil {
|
||||||
|
log.Fatal(fmt.Errorf("htmlToMarkdown: %w", err))
|
||||||
|
}
|
||||||
|
|
||||||
|
ref, err := pub.BlobStore.Put(srcReader)
|
||||||
|
if err != nil {
|
||||||
|
log.Fatal(fmt.Errorf("htmlToMarkdown: %w", err))
|
||||||
|
}
|
||||||
|
|
||||||
|
log.Printf("htmlToMarkdown: successfully posted %s as blob", src)
|
||||||
|
|
||||||
|
return md.String("![](" + ref.String() + ")")
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
},
|
||||||
|
},
|
||||||
|
)
|
||||||
|
|
||||||
|
markdown, err := converter.ConvertString(content)
|
||||||
|
if err != nil {
|
||||||
|
return markdown, fmt.Errorf("htmlToMarkdown: unable to convert html to markdown: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return markdown, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// firstRSSPost retrieves the post content of the first message of a RSS feed.
|
||||||
|
func firstRSSPost(testFeed string, pub *sbot.Sbot) (string, error) {
|
||||||
|
var markdown string
|
||||||
|
|
||||||
|
feed, err := parseRSSFeed(testFeed)
|
||||||
|
if err != nil {
|
||||||
|
return markdown, fmt.Errorf("firstRSSPost: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, feed := range feed.Items {
|
||||||
|
content := feed.Content
|
||||||
|
if feed.Content == "" {
|
||||||
|
content = feed.Description
|
||||||
|
}
|
||||||
|
|
||||||
|
log.Printf("firstRSSPost: converting '%s' to markdown", feed.Title)
|
||||||
|
|
||||||
|
markdown, err = htmlToMarkdown(content, pub, false)
|
||||||
|
if err != nil {
|
||||||
|
return markdown, fmt.Errorf("firstRSSPost: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return markdown, err
|
||||||
|
}
|
||||||
|
|
||||||
|
return markdown, fmt.Errorf("firstRSSPost: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
// loadYAMLConfig loads a rss-butt-plug YAML user config.
|
||||||
|
func loadYAMLConfig() (Config, error) {
|
||||||
|
var cfg Config
|
||||||
|
|
||||||
|
configPath, err := filepath.Abs(configFlag)
|
||||||
|
if err != nil {
|
||||||
|
return Config{}, fmt.Errorf("loadYAMLConfig: unable to convert %s to an absolute path: %w", configFlag, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
conf, err := os.ReadFile(configPath)
|
||||||
|
if err != nil {
|
||||||
|
return Config{}, fmt.Errorf("loadYAMLConfig: unable to read %s: %w", configFlag, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
err = yaml.UnmarshalStrict(conf, &cfg)
|
||||||
|
if err != nil {
|
||||||
|
return Config{}, fmt.Errorf("loadYAMLConfig: unable to unmarshal %s: %w", string(conf), err)
|
||||||
|
}
|
||||||
|
|
||||||
|
return cfg, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// generatePublicInvite generates an invite by speaking to a local go-sbot instance. It
|
||||||
|
// uses a high "uses" value (666) so as to make the invite usable to more
|
||||||
|
// people. It's more a public share invite in that sense.
|
||||||
|
func generatePublicInvite(pub *sbot.Sbot) (string, error) {
|
||||||
|
var token string
|
||||||
|
|
||||||
|
client, err := ssbClient.NewTCP(pub.KeyPair, pub.Network.GetListenAddr())
|
||||||
|
if err != nil {
|
||||||
|
return token, fmt.Errorf("generatePublicInvite: unable to initalise TCP client: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
token, err = client.InviteCreate(message.InviteCreateArgs{Uses: 666})
|
||||||
|
if err != nil {
|
||||||
|
return token, fmt.Errorf("generatePublicInvite: unable to create invite: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := client.Close(); err != nil {
|
||||||
|
return token, fmt.Errorf("generatePublicInvite: unable to close TCP client: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if strings.Contains(token, "[::]") {
|
||||||
|
token = strings.Replace(token, "[::]", "localhost", 1)
|
||||||
|
}
|
||||||
|
|
||||||
|
return token, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// messagesFromLog retrieves all messages from the user log.
|
||||||
|
func messagesFromLog(pub *sbot.Sbot) ([]Post, error) {
|
||||||
|
var posts []Post
|
||||||
|
|
||||||
|
src, err := pub.ReceiveLog.Query()
|
||||||
|
if err != nil {
|
||||||
|
return posts, fmt.Errorf("messagesFromLog: unable to query log: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
for {
|
||||||
|
var post Post
|
||||||
|
|
||||||
|
v, err := src.Next(context.Background())
|
||||||
|
if luigi.IsEOS(err) {
|
||||||
|
break
|
||||||
|
}
|
||||||
|
|
||||||
|
message := v.(refs.Message)
|
||||||
|
content := message.ContentBytes()
|
||||||
|
if err = json.Unmarshal(content, &post); err != nil {
|
||||||
|
return posts, fmt.Errorf("messagesFromLog: unable to unmarshal %s: %w", string(content), err)
|
||||||
|
}
|
||||||
|
|
||||||
|
posts = append(posts, post)
|
||||||
|
}
|
||||||
|
|
||||||
|
return posts, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// newSbot instantiates a new go-sbot instance.
|
||||||
|
func newSbot(cfg Config) (*sbot.Sbot, error) {
|
||||||
|
dataDir, err := filepath.Abs(cfg.DataDir)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("newSbot: unable to convert %s to an absolute path: %w", cfg.DataDir, err)
|
||||||
|
}
|
||||||
|
|
||||||
|
sbotOpts := []sbot.Option{
|
||||||
|
sbot.EnableAdvertismentBroadcasts(true),
|
||||||
|
sbot.EnableAdvertismentDialing(true),
|
||||||
|
sbot.LateOption(sbot.WithUNIXSocket()),
|
||||||
|
sbot.WithHops(2),
|
||||||
|
sbot.WithListenAddr(fmt.Sprintf(":%s", cfg.Port)),
|
||||||
|
sbot.WithRepoPath(dataDir),
|
||||||
|
sbot.WithWebsocketAddress(fmt.Sprintf(":%s", cfg.WsPort)),
|
||||||
|
}
|
||||||
|
|
||||||
|
pub, err := sbot.New(sbotOpts...)
|
||||||
|
if err != nil {
|
||||||
|
return nil, fmt.Errorf("newSbot: unable to initialise sbot: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
c := make(chan os.Signal, 1)
|
||||||
|
signal.Notify(c, os.Interrupt, syscall.SIGTERM)
|
||||||
|
go func() {
|
||||||
|
<-c
|
||||||
|
|
||||||
|
pub.Shutdown()
|
||||||
|
time.Sleep(2 * time.Second)
|
||||||
|
|
||||||
|
if err := pub.Close(); err != nil {
|
||||||
|
log.Fatal(fmt.Errorf("newSbot: %w", err))
|
||||||
|
}
|
||||||
|
|
||||||
|
time.Sleep(2 * time.Second)
|
||||||
|
os.Exit(0)
|
||||||
|
}()
|
||||||
|
|
||||||
|
return pub, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// serveSbot serves a go-sbot over the network.
|
||||||
|
func serveSbot(pub *sbot.Sbot) {
|
||||||
|
for {
|
||||||
|
ctx := context.TODO()
|
||||||
|
err := pub.Network.Serve(ctx)
|
||||||
|
if err != nil {
|
||||||
|
log.Fatal(fmt.Errorf("serveSbot: %w", err))
|
||||||
|
}
|
||||||
|
|
||||||
|
time.Sleep(1 * time.Second)
|
||||||
|
|
||||||
|
select {
|
||||||
|
case <-ctx.Done():
|
||||||
|
err := pub.Close()
|
||||||
|
if err != nil {
|
||||||
|
log.Fatal(fmt.Errorf("serveSbot: %w", err))
|
||||||
|
}
|
||||||
|
default:
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
// getNewRSSPosts gathers new posts from a RSS feed.
|
||||||
|
func getNewRSSPosts(feed gofeed.Feed, posts []Post, pub *sbot.Sbot) ([]map[string]interface{}, error) {
|
||||||
|
var messages []map[string]interface{}
|
||||||
|
|
||||||
|
for idx := len(feed.Items) - 1; idx >= 0; idx-- {
|
||||||
|
feed := feed.Items[idx]
|
||||||
|
alreadyPosted := false
|
||||||
|
|
||||||
|
for _, post := range posts {
|
||||||
|
if feed.Link == post.Link {
|
||||||
|
alreadyPosted = true
|
||||||
|
break
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if alreadyPosted {
|
||||||
|
log.Printf("getNewRSSPosts: skipping %s, already posted", feed.Link)
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
feedContent := feed.Content
|
||||||
|
if feed.Content == "" {
|
||||||
|
feedContent = feed.Description
|
||||||
|
}
|
||||||
|
|
||||||
|
log.Printf("getNewRSSPosts: converting '%s' to markdown", feed.Title)
|
||||||
|
|
||||||
|
markdown, err := htmlToMarkdown(feedContent, pub, true)
|
||||||
|
if err != nil {
|
||||||
|
return messages, fmt.Errorf("getNewRSSPosts: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
content := fmt.Sprintf("# %s\n", feed.Title)
|
||||||
|
|
||||||
|
if feed.Image != nil {
|
||||||
|
srcReader, err := getImage(feed.Image.URL)
|
||||||
|
if err != nil {
|
||||||
|
return messages, fmt.Errorf("getNewRSSPosts: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
ref, err := pub.BlobStore.Put(srcReader)
|
||||||
|
if err != nil {
|
||||||
|
return messages, fmt.Errorf("getNewRSSPosts: unable to upload blob: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
content += "\n![](" + ref.String() + ")\n"
|
||||||
|
}
|
||||||
|
|
||||||
|
content += markdown
|
||||||
|
content += "\n---\n[Clearnet link](" + feed.Link + ")\n"
|
||||||
|
|
||||||
|
messages = append(messages, map[string]interface{}{
|
||||||
|
"type": "post",
|
||||||
|
"link": feed.Link,
|
||||||
|
"text": content,
|
||||||
|
})
|
||||||
|
}
|
||||||
|
|
||||||
|
return messages, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// createAboutMessage publishes an about message with accompanying avatar, if available in config).
|
||||||
|
func createAboutMessage(pub *sbot.Sbot, posts []Post, feed gofeed.Feed, cfg Config) (map[string]interface{}, bool, error) {
|
||||||
|
for _, post := range posts {
|
||||||
|
if post.Type == "about" {
|
||||||
|
log.Printf("createAboutMessage: skipping about message post, already done")
|
||||||
|
return nil, false, nil
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
message := map[string]interface{}{
|
||||||
|
"type": "about",
|
||||||
|
"about": pub.KeyPair.ID(),
|
||||||
|
"name": feed.Title,
|
||||||
|
}
|
||||||
|
|
||||||
|
if cfg.Avatar != "" {
|
||||||
|
srcReader, err := getImage(cfg.Avatar)
|
||||||
|
if err != nil {
|
||||||
|
return nil, false, fmt.Errorf("createAboutMessage: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
ref, err := pub.BlobStore.Put(srcReader)
|
||||||
|
if err != nil {
|
||||||
|
return nil, false, fmt.Errorf("createAboutMessage: unable to post blob: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
message["image"] = ref.String()
|
||||||
|
}
|
||||||
|
|
||||||
|
log.Printf("createAboutMessage: creating about message post")
|
||||||
|
|
||||||
|
return message, true, nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// chunkByLine chunks a full markdown converted RSS post into a thread.
|
||||||
|
// Meaning, a series of chunks which fit under the max post size of a ssb post.
|
||||||
|
func chunkByLine(content string) []string {
|
||||||
|
var chunks []string
|
||||||
|
|
||||||
|
toChunk := content
|
||||||
|
for toChunk != "" {
|
||||||
|
if len(toChunk) <= maxPostLength {
|
||||||
|
chunks = append(chunks, toChunk)
|
||||||
|
toChunk = ""
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
|
||||||
|
chunkIdx := maxPostLength
|
||||||
|
for toChunk[chunkIdx] != 10 {
|
||||||
|
chunkIdx--
|
||||||
|
}
|
||||||
|
|
||||||
|
chunks = append(chunks, toChunk[:chunkIdx])
|
||||||
|
toChunk = toChunk[chunkIdx:]
|
||||||
|
}
|
||||||
|
|
||||||
|
return chunks
|
||||||
|
}
|
||||||
|
|
||||||
|
// publishAsThread posts a message as a series of linked messages. This is
|
||||||
|
// useful when the content of the RSS post is too long.
|
||||||
|
func publishAsThread(publish ssb.Publisher, message map[string]interface{}) error {
|
||||||
|
chunks := chunkByLine(message["text"].(string))
|
||||||
|
|
||||||
|
root := map[string]interface{}{
|
||||||
|
"type": "post",
|
||||||
|
"link": message["link"],
|
||||||
|
"text": chunks[0],
|
||||||
|
}
|
||||||
|
|
||||||
|
ref, err := publish.Publish(root)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("publishAsThread: failed to publish: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, chunk := range chunks[1:] {
|
||||||
|
threadReply := map[string]interface{}{
|
||||||
|
"type": "post",
|
||||||
|
"link": message["link"],
|
||||||
|
"text": chunk,
|
||||||
|
"root": ref.Key().String(),
|
||||||
|
}
|
||||||
|
_, err := publish.Publish(threadReply)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("publishAsThread: failed to publish: %w", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// postMessagesToLog posts messages to the local user feed.
|
||||||
|
func postMessagesToLog(messages []map[string]interface{}, pub *sbot.Sbot) error {
|
||||||
|
publish, err := message.OpenPublishLog(pub.ReceiveLog, pub.Users, pub.KeyPair)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("postMessagesToLog: failed to open publish log: %w", err)
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, message := range messages {
|
||||||
|
if message["type"] == "post" {
|
||||||
|
log.Printf("postMessagesToLog: publishing %s to log", message["link"])
|
||||||
|
|
||||||
|
if len(message["text"].(string)) > maxPostLength {
|
||||||
|
log.Printf("postMessagesToLog: turning content of %s into thread, too long", message["link"])
|
||||||
|
if err := publishAsThread(publish, message); err != nil {
|
||||||
|
return fmt.Errorf("postMessagesToLog: unable to thread content for %s: %w", message["link"], err)
|
||||||
|
}
|
||||||
|
continue
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
_, err := publish.Publish(message)
|
||||||
|
if err != nil {
|
||||||
|
return fmt.Errorf("postMessagesToLog: failed to publish: %w", err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
return nil
|
||||||
|
}
|
||||||
|
|
||||||
|
// main is the main CLI entrypoint.
|
||||||
|
func main() {
|
||||||
|
handleCliFlags()
|
||||||
|
|
||||||
|
if helpFlag {
|
||||||
|
fmt.Printf(help)
|
||||||
|
os.Exit(0)
|
||||||
|
}
|
||||||
|
|
||||||
|
cfg, err := loadYAMLConfig()
|
||||||
|
if err != nil {
|
||||||
|
log.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
log.Printf("loaded %s", configFlag)
|
||||||
|
|
||||||
|
pub, err := newSbot(cfg)
|
||||||
|
if err != nil {
|
||||||
|
log.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
args := os.Args[1:]
|
||||||
|
if len(args) > 0 {
|
||||||
|
markdown, err := firstRSSPost(args[0], pub)
|
||||||
|
if err != nil {
|
||||||
|
log.Fatal(err)
|
||||||
|
}
|
||||||
|
fmt.Println(markdown)
|
||||||
|
return
|
||||||
|
}
|
||||||
|
|
||||||
|
go serveSbot(pub)
|
||||||
|
|
||||||
|
log.Print("main: bootstrapped internally managed go-sbot")
|
||||||
|
|
||||||
|
cfg.KeyPair = pub.KeyPair
|
||||||
|
feed, err := parseRSSFeed(cfg.Feed)
|
||||||
|
if err != nil {
|
||||||
|
log.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
log.Printf("main: parsed %s", cfg.Feed)
|
||||||
|
|
||||||
|
posts, err := messagesFromLog(pub)
|
||||||
|
if err != nil {
|
||||||
|
log.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
log.Printf("main: retrieved %d posts from log", len(posts))
|
||||||
|
|
||||||
|
var messages []map[string]interface{}
|
||||||
|
|
||||||
|
aboutMessage, posted, err := createAboutMessage(pub, posts, feed, cfg)
|
||||||
|
if err != nil {
|
||||||
|
log.Fatal(err)
|
||||||
|
}
|
||||||
|
if posted {
|
||||||
|
messages = append(messages, aboutMessage)
|
||||||
|
}
|
||||||
|
|
||||||
|
newRSSPosts, err := getNewRSSPosts(feed, posts, pub)
|
||||||
|
if err != nil {
|
||||||
|
log.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
for _, newRSSPost := range newRSSPosts {
|
||||||
|
messages = append(messages, newRSSPost)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := postMessagesToLog(messages, pub); err != nil {
|
||||||
|
log.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
token, err := generatePublicInvite(pub)
|
||||||
|
if err != nil {
|
||||||
|
log.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
log.Printf("main: pub invite: %s", token)
|
||||||
|
|
||||||
|
for {
|
||||||
|
log.Printf("main: going to sleep for %d minutes...", pollFrequencyFlag)
|
||||||
|
time.Sleep(time.Duration(pollFrequencyFlag) * time.Minute)
|
||||||
|
log.Printf("main: waking up to poll %s for new posts", cfg.Feed)
|
||||||
|
|
||||||
|
posts, err := messagesFromLog(pub)
|
||||||
|
if err != nil {
|
||||||
|
log.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
messages, err := getNewRSSPosts(feed, posts, pub)
|
||||||
|
if err != nil {
|
||||||
|
log.Fatal(err)
|
||||||
|
}
|
||||||
|
|
||||||
|
if err := postMessagesToLog(messages, pub); err != nil {
|
||||||
|
log.Fatal(err)
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
Loading…
Reference in New Issue