Solpunk Pub Failing To Connect To Patchwork #134

New Issue

2022-09-03T12:13:46Z

notplants commented

2022-09-03 12:13:46 +00:00

@abekonge.punk is running peachpub on yunohost on a raspberry pi 3 at solpunk (%/Xt6yV60kKuZHOk1jblAgY3yAAwqcSY9MYjaqjvug+s=.sha256)

the first couple days it worked, and now its running into some issues

from @abekonge.punk

I have it online, connected to it via punknet, I can access the yunohost web-iu, and the peachpub web-ui.

But it does not seem to connect to my patchwork:

It did show up shortly in local connections, but then disappeared
There is only 13 messages in the db (peachpub ui), I have much more on my patchwork (it is synced up with your pub, and my manyverse)
I’ve restarted sbot, and the whole thing
Any ideas? Or how to see some logs that might help me?

I’ve tried to send a pm from the peachpub to my patchwork profile, which is not received. So confirming non-connection somehow.

I don’t know if it helps. After I got it to show in the local connections at first, peachpub seemed to slow down, like it was working a lot … which makes sense, but it seems fine now, and does not connect still … any ideas?

I tried sudo shutdown on the pi, just to see if that would help. after restart the ssb-status in peachpub says db is unavailable …

I’m considering removing it, and installing peachpub again - would also be nice to try that process. What do you think?

@abekonge.punk is running peachpub on yunohost on a raspberry pi 3 at solpunk (%/Xt6yV60kKuZHOk1jblAgY3yAAwqcSY9MYjaqjvug+s=.sha256) the first couple days it worked, and now its running into some issues from @abekonge.punk ``` I have it online, connected to it via punknet, I can access the yunohost web-iu, and the peachpub web-ui. But it does not seem to connect to my patchwork: It did show up shortly in local connections, but then disappeared There is only 13 messages in the db (peachpub ui), I have much more on my patchwork (it is synced up with your pub, and my manyverse) I’ve restarted sbot, and the whole thing Any ideas? Or how to see some logs that might help me? I’ve tried to send a pm from the peachpub to my patchwork profile, which is not received. So confirming non-connection somehow. I don’t know if it helps. After I got it to show in the local connections at first, peachpub seemed to slow down, like it was working a lot … which makes sense, but it seems fine now, and does not connect still … any ideas? I tried sudo shutdown on the pi, just to see if that would help. after restart the ssb-status in peachpub says db is unavailable … I’m considering removing it, and installing peachpub again - would also be nice to try that process. What do you think? ```

notplants added the bug label 2022-09-03 12:13:46 +00:00

notplants commented

2022-09-03 12:15:13 +00:00

I wonder if this might be the issue https://github.com/ssbc/go-ssb/issues/81

I would try running this on the pi:

sudo su root
ln -s /home/yunohost.app/peachpub/.ssb-go /root/.ssb-go
/var/www/peachpub/sbotcli call whoami

the creation of the symbolic link above is necessary because currently there is no way to tell sbotcli where the .ssb-go folder is (https://github.com/ssbc/go-ssb/issues/151)

You can see all the commands you can run with sbotcli (a go cli tool packaged with go-ssb) via /var/www/peachpub/sbotcli -h (although they are not fully documented)

some other debugging commands to try:

systemctl status peachpub-go-sbot

… if peachpub-go-sbot is running, but peach-web says it can’t connect to the database

perhaps we can try the repair command:

systemctl stop peachpub-go-sbot
SSB_CONFIG_FILE=/home/yunohost.app/peachpub/.ssb-go/config.toml /var/www/peachpub/go-sbot -fsck sequences -repair
# wait for above to finish
# should see some log messages about repairing the feed
sudo chown -R peachpub:www-data /home/yunohost.app/peachpub/.ssb-go
systemctl start peachpub-go-sbot
systemctl status peachpub-go-sbot

I wonder if this might be the issue https://github.com/ssbc/go-ssb/issues/81 I would try running this on the pi: ``` sudo su root ln -s /home/yunohost.app/peachpub/.ssb-go /root/.ssb-go /var/www/peachpub/sbotcli call whoami ``` the creation of the symbolic link above is necessary because currently there is no way to tell sbotcli where the .ssb-go folder is (https://github.com/ssbc/go-ssb/issues/151) You can see all the commands you can run with sbotcli (a go cli tool packaged with go-ssb) via /var/www/peachpub/sbotcli -h (although they are not fully documented) some other debugging commands to try: systemctl status peachpub-go-sbot … if peachpub-go-sbot is running, but peach-web says it can’t connect to the database perhaps we can try the repair command: ``` systemctl stop peachpub-go-sbot SSB_CONFIG_FILE=/home/yunohost.app/peachpub/.ssb-go/config.toml /var/www/peachpub/go-sbot -fsck sequences -repair # wait for above to finish # should see some log messages about repairing the feed sudo chown -R peachpub:www-data /home/yunohost.app/peachpub/.ssb-go systemctl start peachpub-go-sbot systemctl status peachpub-go-sbot ```

notplants commented

2022-09-03 12:16:18 +00:00

cc @glyph , in case you have any additional ideas or just want to follow along ... unclear yet if the issue is from something in go-ssb, or peachpub or the yunohost packaging

👍 1

notplants commented

2022-09-03 12:23:56 +00:00

There is only 13 messages in the db (peachpub ui), I have much more on my patchwork (it is synced up with your pub, and my manyverse)

this itself is not necessarily a bug, as this number is actually the number of messages authored by the pub (including follow messages etc.) … but maybe this could be communicated more clearly in the UI… and a number which showed the total number of messages from all users also sounds very useful…

> There is only 13 messages in the db (peachpub ui), I have much more on my patchwork (it is synced up with your pub, and my manyverse) this itself is not necessarily a bug, as this number is actually the number of messages authored by the pub (including follow messages etc.) … but maybe this could be communicated more clearly in the UI… and a number which showed the total number of messages from all users also sounds very useful…

👍 1

glyph commented

2022-09-05 08:37:36 +00:00

Hey @notplants and @abekonge.punk, thanks for the ping.

Echoing the advice from @notplants, I would first check the status of the go-sbot:

systemctl status peachpub-go-sbot

Assuming there is an error which is preventing replication, I'd run the go-sbot with the repair flag enabled. I include the --repair flag as default whenever I run go-sbot (I do this in a systemd unit file, though I understand the YunoHost context is different); if there are no repairs to be done then it simply starts up as normal.

I wonder what the hops count is set to? It might help to set it to 1 initially so that the Pi doesn't have to work too hard. That being said, the hops count is not related to the replication / connection error - just adding that as an extra thought.

Hey @notplants and [@abekonge.punk](https://git.coopcloud.tech/abekonge.punk), thanks for the ping. Echoing the advice from @notplants, I would first check the status of the go-sbot: `systemctl status peachpub-go-sbot` Assuming there is an error which is preventing replication, I'd run the go-sbot with the repair flag enabled. I include the `--repair` flag as default whenever I run go-sbot (I do this in a systemd unit file, though I understand the YunoHost context is different); if there are no repairs to be done then it simply starts up as normal. ----- I wonder what the hops count is set to? It might help to set it to `1` initially so that the Pi doesn't have to work too hard. That being said, the hops count is not related to the replication / connection error - just adding that as an extra thought.

abekonge commented

2022-09-07 09:53:17 +00:00

I wonder if this might be the issue https://github.com/ssbc/go-ssb/issues/81

I would try running this on the pi:
sudo su root
ln -s /home/yunohost.app/peachpub/.ssb-go /root/.ssb-go
/var/www/peachpub/sbotcli call whoami
the creation of the symbolic link above is necessary because currently there is no way to tell sbotcli where the .ssb-go folder is (https://github.com/ssbc/go-ssb/issues/151)

You can see all the commands you can run with sbotcli (a go cli tool packaged with go-ssb) via /var/www/peachpub/sbotcli -h (although they are not fully documented)

@notplants @glyph

Okay, just to check where we were before linking, I checked the folders, and there was no .sbot-go link in /root/

And /var/www/peachpub/sbotcli call whoami gave

level=warn client="unix-path based init failed" err="ssbClient: failed to open unix path \"/root/.ssb-go/socket\""
level=error run-failure="open /root/.ssb-go/secret: no such file or directory"

After creating the link /var/www/peachpub/sbotcli call whoami gave:

level=warn client="unix-path based init failed" err="ssbClient: failed to open unix path \"/root/.ssb-go/socket\""
level=error run-failure="init: failed to connect to 127.0.0.1:8008|@oY8aLV/yEKs75d6TknxlwkhfiThB9HlN5FQY7WWTzho=.ed25519: error dialing: error wrapping connection: secrethandshake: failed during data transfer of receiving server auth: EOF"

Tried restarting sbot from peachpub ui, db still down, whoami had same error.

Tried rebooting the pi, same results

Some observations of weirdness:

There are no socket directory in the .ssb-go folder
The .ssb-go folder has a .ssb-go folder inside it, with some of the same files:

contents of .ssb-go:

root@mejeriet:/home/yunohost.app/peachpub/.ssb-go# ls -alt
total 44
-rw-r--r-- 1 peachpub peachpub  291 Aug 20 04:36 running-config.json
-rw------- 1 peachpub peachpub  366 Aug 20 04:35 config.toml
drwx------ 7 peachpub peachpub 4096 Aug 19 14:13 .ssb-go
drwxr-x--- 8 peachpub www-data 4096 Aug 19 14:08 .
drwx------ 2 peachpub peachpub 4096 Aug 19 14:08 ebt-state-matrix
drwx------ 3 peachpub peachpub 4096 Aug 17 12:38 indexes
drwx------ 3 peachpub peachpub 4096 Aug 17 11:03 sublogs
drwx------ 4 peachpub peachpub 4096 Aug 17 11:03 blobs
drwx------ 2 peachpub peachpub 4096 Aug 17 11:03 log
-r-------- 1 peachpub peachpub  254 Aug 17 11:03 secret
drwxr-x--- 4 peachpub www-data 4096 Aug 17 11:02 ..

contents of .ssb-go/.ssb-go/:

root@mejeriet:/home/yunohost.app/peachpub/.ssb-go/.ssb-go# ls -alt
total 32
drwx------ 2 peachpub peachpub 4096 Aug 20 04:36 ebt-state-matrix
drwx------ 7 peachpub peachpub 4096 Aug 19 14:13 .
drwx------ 3 peachpub peachpub 4096 Aug 19 14:13 indexes
drwx------ 3 peachpub peachpub 4096 Aug 19 14:08 sublogs
drwx------ 4 peachpub peachpub 4096 Aug 19 14:08 blobs
drwx------ 2 peachpub peachpub 4096 Aug 19 14:08 log
-r-------- 1 peachpub peachpub  254 Aug 19 14:08 secret
drwxr-x--- 8 peachpub www-data 4096 Aug 19 14:08 ..

> I wonder if this might be the issue https://github.com/ssbc/go-ssb/issues/81 > > I would try running this on the pi: > > ``` > sudo su root > ln -s /home/yunohost.app/peachpub/.ssb-go /root/.ssb-go > /var/www/peachpub/sbotcli call whoami > ``` > > the creation of the symbolic link above is necessary because currently there is no way to tell sbotcli where the .ssb-go folder is (https://github.com/ssbc/go-ssb/issues/151) > > You can see all the commands you can run with sbotcli (a go cli tool packaged with go-ssb) via /var/www/peachpub/sbotcli -h (although they are not fully documented) > @notplants @glyph Okay, just to check where we were before linking, I checked the folders, and there was no .sbot-go link in `/root/` And `/var/www/peachpub/sbotcli call whoami` gave ``` level=warn client="unix-path based init failed" err="ssbClient: failed to open unix path \"/root/.ssb-go/socket\"" level=error run-failure="open /root/.ssb-go/secret: no such file or directory" ``` After creating the link `/var/www/peachpub/sbotcli call whoami` gave: ``` level=warn client="unix-path based init failed" err="ssbClient: failed to open unix path \"/root/.ssb-go/socket\"" level=error run-failure="init: failed to connect to 127.0.0.1:8008|@oY8aLV/yEKs75d6TknxlwkhfiThB9HlN5FQY7WWTzho=.ed25519: error dialing: error wrapping connection: secrethandshake: failed during data transfer of receiving server auth: EOF" ``` Tried restarting sbot from peachpub ui, db still down, whoami had same error. Tried rebooting the pi, same results Some observations of weirdness: * There are no `socket` directory in the `.ssb-go` folder * The `.ssb-go` folder has a `.ssb-go` folder inside it, with some of the same files: *contents of .ssb-go*: ``` root@mejeriet:/home/yunohost.app/peachpub/.ssb-go# ls -alt total 44 -rw-r--r-- 1 peachpub peachpub 291 Aug 20 04:36 running-config.json -rw------- 1 peachpub peachpub 366 Aug 20 04:35 config.toml drwx------ 7 peachpub peachpub 4096 Aug 19 14:13 .ssb-go drwxr-x--- 8 peachpub www-data 4096 Aug 19 14:08 . drwx------ 2 peachpub peachpub 4096 Aug 19 14:08 ebt-state-matrix drwx------ 3 peachpub peachpub 4096 Aug 17 12:38 indexes drwx------ 3 peachpub peachpub 4096 Aug 17 11:03 sublogs drwx------ 4 peachpub peachpub 4096 Aug 17 11:03 blobs drwx------ 2 peachpub peachpub 4096 Aug 17 11:03 log -r-------- 1 peachpub peachpub 254 Aug 17 11:03 secret drwxr-x--- 4 peachpub www-data 4096 Aug 17 11:02 .. ``` *contents of .ssb-go/.ssb-go/:* ``` root@mejeriet:/home/yunohost.app/peachpub/.ssb-go/.ssb-go# ls -alt total 32 drwx------ 2 peachpub peachpub 4096 Aug 20 04:36 ebt-state-matrix drwx------ 7 peachpub peachpub 4096 Aug 19 14:13 . drwx------ 3 peachpub peachpub 4096 Aug 19 14:13 indexes drwx------ 3 peachpub peachpub 4096 Aug 19 14:08 sublogs drwx------ 4 peachpub peachpub 4096 Aug 19 14:08 blobs drwx------ 2 peachpub peachpub 4096 Aug 19 14:08 log -r-------- 1 peachpub peachpub 254 Aug 19 14:08 secret drwxr-x--- 8 peachpub www-data 4096 Aug 19 14:08 .. ```

abekonge commented

2022-09-07 09:54:07 +00:00

some other debugging commands to try:

systemctl status peachpub-go-sbot

… if peachpub-go-sbot is running, but peach-web says it can’t connect to the database

perhaps we can try the repair command:
systemctl stop peachpub-go-sbot
SSB_CONFIG_FILE=/home/yunohost.app/peachpub/.ssb-go/config.toml /var/www/peachpub/go-sbot -fsck sequences -repair
# wait for above to finish
# should see some log messages about repairing the feed
sudo chown -R peachpub:www-data /home/yunohost.app/peachpub/.ssb-go
systemctl start peachpub-go-sbot
systemctl status peachpub-go-sbot

systemctl status peachpub-go-sbot
gives

● peachpub-go-sbot.service - go-sbot service
     Loaded: loaded (/etc/systemd/system/peachpub-go-sbot.service; enabled; vendo>
     Active: active (running) since Sat 2022-08-20 04:36:10 BST; 10min ago
   Main PID: 517 (go-sbot)
      Tasks: 9 (limit: 780)
        CPU: 15.518s
     CGroup: /system.slice/peachpub-go-sbot.service
             └─517 /var/www/peachpub/go-sbot

The repair command does give an error, not sure if it is significant:

But after restart, systemctl status is all green

db is still unavailable though :(

> some other debugging commands to try: > > systemctl status peachpub-go-sbot > > … if peachpub-go-sbot is running, but peach-web says it can’t connect to the database > > perhaps we can try the repair command: > > > ``` > systemctl stop peachpub-go-sbot > SSB_CONFIG_FILE=/home/yunohost.app/peachpub/.ssb-go/config.toml /var/www/peachpub/go-sbot -fsck sequences -repair > # wait for above to finish > # should see some log messages about repairing the feed > sudo chown -R peachpub:www-data /home/yunohost.app/peachpub/.ssb-go > systemctl start peachpub-go-sbot > systemctl status peachpub-go-sbot > ``` ` systemctl status peachpub-go-sbot` gives ``` ● peachpub-go-sbot.service - go-sbot service Loaded: loaded (/etc/systemd/system/peachpub-go-sbot.service; enabled; vendo> Active: active (running) since Sat 2022-08-20 04:36:10 BST; 10min ago Main PID: 517 (go-sbot) Tasks: 9 (limit: 780) CPU: 15.518s CGroup: /system.slice/peachpub-go-sbot.service └─517 /var/www/peachpub/go-sbot ``` The repair command does give an error, not sure if it is significant: ![image](/attachments/6b78b216-069e-43db-9f5a-b26ae95c26b0) But after restart, systemctl status is all green db is still unavailable though :( ![image](/attachments/525715de-f885-42c4-aff0-acbf81e4dcb2)

image.png

499 KiB

image.png

232 KiB

abekonge commented

2022-09-07 09:56:20 +00:00

Ok, so .ssb-go was owned by root, after fixing that, the repair went better:

Still has that error, and db is still offline

Ok, so .ssb-go was owned by root, after fixing that, the repair went better: ![image](/attachments/692f5aea-8deb-413e-96bc-b2914d484283) Still has that error, and db is still offline

image.png

1.0 MiB

abekonge commented

2022-09-07 09:57:33 +00:00

Hey @notplants and @abekonge.punk, thanks for the ping.

Echoing the advice from @notplants, I would first check the status of the go-sbot:

systemctl status peachpub-go-sbot

Assuming there is an error which is preventing replication, I'd run the go-sbot with the repair flag enabled. I include the --repair flag as default whenever I run go-sbot (I do this in a systemd unit file, though I understand the YunoHost context is different); if there are no repairs to be done then it simply starts up as normal.

I wonder what the hops count is set to? It might help to set it to 1 initially so that the Pi doesn't have to work too hard. That being said, the hops count is not related to the replication / connection error - just adding that as an extra thought.

Thanks for looking into it. I tried setting hops to 1, repairing, and restarting, still no dice though.

> Hey @notplants and [@abekonge.punk](https://git.coopcloud.tech/abekonge.punk), thanks for the ping. > > Echoing the advice from @notplants, I would first check the status of the go-sbot: > > `systemctl status peachpub-go-sbot` > > Assuming there is an error which is preventing replication, I'd run the go-sbot with the repair flag enabled. I include the `--repair` flag as default whenever I run go-sbot (I do this in a systemd unit file, though I understand the YunoHost context is different); if there are no repairs to be done then it simply starts up as normal. > > ----- > > I wonder what the hops count is set to? It might help to set it to `1` initially so that the Pi doesn't have to work too hard. That being said, the hops count is not related to the replication / connection error - just adding that as an extra thought. Thanks for looking into it. I tried setting hops to 1, repairing, and restarting, still no dice though.

glyph commented

2022-09-07 10:50:20 +00:00

@abekonge

Thanks for the additional info.

Some observations of weirdness:

There are no socket directory in the .ssb-go folder

The .ssb-go folder has a .ssb-go folder inside it, with some of the same files:

This is indeed very weird, especially the nested .ssb-go directory. @notplants is the expert when it comes to YunoHost configuration of PeachPub. @notplants, can you think of any reason why the nested directory might have been created?

I hesitate to recommend this without further investigation but it might be worth deleting the inner .ssb-go directory (so that only the top-level directory remains). But let's wait until @notplants has a chance to respond.

Other thoughts:

It's a promising sign that the go-sbot service is active and running. The problem seems to be the communication between the peach application and the go-sbot. The DATABASE UNAVAILABLE text appears if the application fails in an attempt to get the latest sequence number for the local feed from the go-sbot. It calls the following methods to do that:

		peach-workspace/peach-web/src/utils/sbot.rs
		Line 126 in c83a22461d
	
				pub fn latest_sequence_number() -> Result<u64, Box<dyn Error>> {

So a failure to execute any of those sbot methods will result in DATABASE UNAVAILABLE.

I'm going to give this some more thought and try to come up with some debugging advice.

@abekonge Thanks for the additional info. > Some observations of weirdness: > > - There are no socket directory in the .ssb-go folder > - The .ssb-go folder has a .ssb-go folder inside it, with some of the same files: This is indeed very weird, especially the nested `.ssb-go` directory. @notplants is the expert when it comes to YunoHost configuration of PeachPub. @notplants, can you think of any reason why the nested directory might have been created? I hesitate to recommend this without further investigation but it might be worth deleting the inner `.ssb-go` directory (so that only the top-level directory remains). But let's wait until @notplants has a chance to respond. Other thoughts: It's a promising sign that the `go-sbot` service is active and running. The problem seems to be the communication between the peach application and the go-sbot. The `DATABASE UNAVAILABLE` text appears if the application fails in an attempt to get the latest sequence number for the local feed from the go-sbot. It calls the following methods to do that: https://git.coopcloud.tech/PeachCloud/peach-workspace/src/commit/c83a22461da6c9ce18d19e59b30a483e22fa9dee/peach-web/src/utils/sbot.rs#L126 So a failure to execute any of those sbot methods will result in `DATABASE UNAVAILABLE`. I'm going to give this some more thought and try to come up with some debugging advice.

glyph referenced this issue

2022-09-07 10:52:00 +00:00

Add detailed logging & log export #90

glyph commented

2022-09-07 10:53:53 +00:00

@abekonge

What do you see when you go to the Profile page of the PeachCloud UI?

@abekonge What do you see when you go to the Profile page of the PeachCloud UI?

abekonge commented

2022-09-07 11:15:58 +00:00

@glyph Thanks for looking into it!

@abekonge

What do you see when you go to the Profile page of the PeachCloud UI?

@glyph Thanks for looking into it! > @abekonge > > What do you see when you go to the Profile page of the PeachCloud UI? ![image](/attachments/a24af9e7-a96b-4c35-aec0-d28a439c3000)

image.png

46 KiB

glyph commented

2022-09-07 12:24:34 +00:00

@abekonge

I have a hypothesis:

The nested .ssb-go directories mean that there are two secret files, each one with a different public-private keypair. The go-sbot is running with one keypair and PeachCloud is trying to connect with the other keypair. This could explain why you received an error when trying to run sbotcli whoami: the keypair being used to make the connection is not authorised to call that method.

There is further evidence in the logs you shared:

After creating the link /var/www/peachpub/sbotcli call whoami gave:

level=warn client="unix-path based init failed" err="ssbClient: failed to open unix path "/root/.ssb-go/socket""
level=error run-failure="init: failed to connect to 127.0.0.1:8008|@oY8aLV/yEKs75d6TknxlwkhfiThB9HlN5FQY7WWTzho=.ed25519: error dialing: error wrapping connection: secrethandshake: failed during data transfer of receiving server auth: EOF"

Notice the public key: @oY8aLV/yEKs75d6TknxlwkhfiThB9HlN5FQY7WWTzho=.ed25519

Later, when you ran the go-sbot -fsck sequences -repair command, a different public key appeared in the output:

saved identity @Uzcv1AqCgXB... to .ssb-go/secret

You can confirm this by comparing the contents of .ssb-go/secret and .ssb-go/.ssb-go/secret. Are they the same or different?

Important: make sure NOT to post the contents here! We don't want to reveal your private key(s) to the world.

@abekonge I have a hypothesis: The nested `.ssb-go` directories mean that there are two `secret` files, each one with a different public-private keypair. The go-sbot is running with one keypair and PeachCloud is trying to connect with the other keypair. This could explain why you received an error when trying to run `sbotcli whoami`: the keypair being used to make the connection is not authorised to call that method. There is further evidence in the logs you shared: >After creating the link /var/www/peachpub/sbotcli call whoami gave: > >level=warn client="unix-path based init failed" err="ssbClient: failed to open unix path \"/root/.ssb-go/socket\"" level=error run-failure="init: failed to connect to 127.0.0.1:8008|@oY8aLV/yEKs75d6TknxlwkhfiThB9HlN5FQY7WWTzho=.ed25519: error dialing: error wrapping connection: secrethandshake: failed during data transfer of receiving server auth: EOF" Notice the public key: `@oY8aLV/yEKs75d6TknxlwkhfiThB9HlN5FQY7WWTzho=.ed25519` Later, when you ran the `go-sbot -fsck sequences -repair` command, a different public key appeared in the output: `saved identity @Uzcv1AqCgXB... to .ssb-go/secret` You can confirm this by comparing the contents of `.ssb-go/secret` and `.ssb-go/.ssb-go/secret`. Are they the same or different? **Important: make sure NOT to post the contents here! We don't want to reveal your private key(s) to the world.**

notplants commented

2022-09-07 13:16:32 +00:00

I messaged this to @abekonge on ssb, but switching to here, so its all in this thread.

yup I would echo what glyph said to compare .ssb-go/secret and .ssb-go/.ssb-go/secret and see which has the pub key of the pub. If the inner one is a different pub key than the pub, then I would delete the inner folder.

After that, I would confirm

systemctl restart peachpub-go-sbot 
systemctl status peachpub-go-sbot (still green?)

... if not green, I would try

systemctl stop peachpub-go-sbot
SSB_CONFIG_FILE=/home/yunohost.app/peachpub/.ssb-go/config.toml /var/www/peachpub/go-sbot

... and see what logs you get there by running go-sbot directly
... and running /var/www/peachpub/sbotcli call whoami in a separate terminal

... if peachpub-go-sbot is green, then I would repeate the test again (now that there are not two .ssb-go folders)

/var/www/peachpub/sbotcli call whoami

That whoami call is run as root, that's why we needed to make the symbolic link before. Currently sbotcli can't be configured to tell it where .ssb-go, and just looks in ~/.ssb-go, so we trick it with the symbolic link.
ln -s /home/yunohost.app/peachpub/.ssb-go /root/.ssb-go

... I would also confirm that the symbolic link goes to the outer .ssb-go and not to the inner (I'm not totally sure what could have created the inner)

... if peachpub-go-sbot is green, there is no inner .ssb-go folder, and whoami is still giving an error... we still have a mystery

ps: sorry we don't have more streamlined tools for log reporting and debugging, but thanks for the patience. also still up for hopping on a call if this gets too messy

I messaged this to @abekonge on ssb, but switching to here, so its all in this thread. yup I would echo what glyph said to compare .ssb-go/secret and .ssb-go/.ssb-go/secret and see which has the pub key of the pub. If the inner one is a different pub key than the pub, then I would delete the inner folder. After that, I would confirm ``` systemctl restart peachpub-go-sbot systemctl status peachpub-go-sbot (still green?) ``` ... if not green, I would try ``` systemctl stop peachpub-go-sbot SSB_CONFIG_FILE=/home/yunohost.app/peachpub/.ssb-go/config.toml /var/www/peachpub/go-sbot ``` ... and see what logs you get there by running go-sbot directly ... and running /var/www/peachpub/sbotcli call whoami in a separate terminal ... if peachpub-go-sbot is green, then I would repeate the test again (now that there are not two .ssb-go folders) ``` /var/www/peachpub/sbotcli call whoami ``` That whoami call is run as root, that's why we needed to make the symbolic link before. Currently sbotcli can't be configured to tell it where .ssb-go, and just looks in ~/.ssb-go, so we trick it with the symbolic link. ```ln -s /home/yunohost.app/peachpub/.ssb-go /root/.ssb-go``` ... I would also confirm that the symbolic link goes to the outer .ssb-go and not to the inner (I'm not totally sure what could have created the inner) ... if peachpub-go-sbot is green, there is no inner .ssb-go folder, and whoami is still giving an error... we still have a mystery __________ ps: sorry we don't have more streamlined tools for log reporting and debugging, but thanks for the patience. also still up for hopping on a call if this gets too messy

abekonge commented

2022-09-07 15:59:33 +00:00

@notplants They are different!

I'm not sure how to see which public key is the pub's? The ui can't see it itself, in the profile view at least .... and I do not yet have a mental picture of what the difference between sbot-go and peachcloud pub is .... I get it that the pub is a ssb-profile like anyother ... and sbot-go is the ssb-server doing all the things, but where is the peachpubs ".ssb" folder or equivalent?

@notplants They are different! I'm not sure how to see which public key is the pub's? The ui can't see it itself, in the profile view at least .... and I do not yet have a mental picture of what the difference between sbot-go and peachcloud pub is .... I get it that the pub is a ssb-profile like anyother ... and sbot-go is the ssb-server doing all the things, but where is the peachpubs ".ssb" folder or equivalent?

👍 1

glyph commented

2022-09-07 16:11:56 +00:00

I'm not sure how to see which public key is the pub's?

This is an interesting problem. PeachCloud usually queries the sbot to find out the key and then displays it in the web interface. In this case, since the communication between PeachCloud and the sbot is not working, it can't display the key 😅

I looked over on Patchwork and there is an identity named solpunk pub with the following public key:

@oY8aLV/yEKs75d6TknxlwkhfiThB9HlN5FQY7WWTzho=.ed25519

One request I have: @abekonge, could you open the homepage of the PeachCloud interface and then go Settings -> Scuttlebutt -> Configure Sbot and tell me what the DATABASE DIRECTORY input contains?

> I'm not sure how to see which public key is the pub's? This is an interesting problem. PeachCloud usually queries the sbot to find out the key and then displays it in the web interface. In this case, since the communication between PeachCloud and the sbot is not working, it can't display the key 😅 I looked over on Patchwork and there is an identity named `solpunk pub` with the following public key: `@oY8aLV/yEKs75d6TknxlwkhfiThB9HlN5FQY7WWTzho=.ed25519` ----- One request I have: @abekonge, could you open the homepage of the PeachCloud interface and then go Settings -> Scuttlebutt -> Configure Sbot and tell me what the `DATABASE DIRECTORY` input contains?

abekonge commented

2022-09-07 16:30:47 +00:00

EDIT: I figured it out, see here: #134 (comment) if you do not want to follow my process minutely :)

I'm not sure how to see which public key is the pub's?

This is an interesting problem. PeachCloud usually queries the sbot to find out the key and then displays it in the web interface. In this case, since the communication between PeachCloud and the sbot is not working, it can't display the key 😅

I looked over on Patchwork and there is an identity named solpunk pub with the following public key:

@oY8aLV/yEKs75d6TknxlwkhfiThB9HlN5FQY7WWTzho=.ed25519

Of course, through patchwork! Ok the outer .ssb-go secret is the same as solpunk pub.

One request I have: @abekonge, could you open the homepage of the PeachCloud interface and then go Settings -> Scuttlebutt -> Configure Sbot and tell me what the DATABASE DIRECTORY input contains?

db dir is .ssb-go

EDIT: I figured it out, see here: https://git.coopcloud.tech/PeachCloud/peach-workspace/issues/134#issuecomment-14002 if you do not want to follow my process minutely :) > > I'm not sure how to see which public key is the pub's? > > This is an interesting problem. PeachCloud usually queries the sbot to find out the key and then displays it in the web interface. In this case, since the communication between PeachCloud and the sbot is not working, it can't display the key 😅 > > I looked over on Patchwork and there is an identity named `solpunk pub` with the following public key: > > `@oY8aLV/yEKs75d6TknxlwkhfiThB9HlN5FQY7WWTzho=.ed25519` Of course, through patchwork! Ok the outer .ssb-go secret is the same as solpunk pub. > > ----- > > One request I have: @abekonge, could you open the homepage of the PeachCloud interface and then go Settings -> Scuttlebutt -> Configure Sbot and tell me what the `DATABASE DIRECTORY` input contains? db dir is `.ssb-go`

👍 1

abekonge commented

2022-09-07 16:46:58 +00:00

I messaged this to @abekonge on ssb, but switching to here, so its all in this thread.

yup I would echo what glyph said to compare .ssb-go/secret and .ssb-go/.ssb-go/secret and see which has the pub key of the pub. If the inner one is a different pub key than the pub, then I would delete the inner folder.

After that, I would confirm
systemctl restart peachpub-go-sbot 
systemctl status peachpub-go-sbot (still green?)

This took quite a while, like 3 minutes, but afterwards it was green

... if not green, I would try
systemctl stop peachpub-go-sbot
SSB_CONFIG_FILE=/home/yunohost.app/peachpub/.ssb-go/config.toml /var/www/peachpub/go-sbot
... and see what logs you get there by running go-sbot directly
... and running /var/www/peachpub/sbotcli call whoami in a separate terminal

... if peachpub-go-sbot is green, then I would repeate the test again (now that there are not two .ssb-go folders)
/var/www/peachpub/sbotcli call whoami

this still gives the warn and error:

root@yunohost:/home/yunohost.app/peachpub/.ssb-go# /var/www/peachpub/sbotcli call whoami
level=warn client="unix-path based init failed" err="ssbClient: failed to open unix path \"/root/.ssb-go/socket\""
level=error run-failure="init: failed to connect to 127.0.0.1:8008|@oY8aLV/yEKs75d6TknxlwkhfiThB9HlN5FQY7WWTzho=.ed25519: error dialing: error wrapping connection: secrethandshake: failed during data transfer of receiving server auth: EOF"

That whoami call is run as root, that's why we needed to make the symbolic link before. Currently sbotcli can't be configured to tell it where .ssb-go, and just looks in ~/.ssb-go, so we trick it with the symbolic link.
ln -s /home/yunohost.app/peachpub/.ssb-go /root/.ssb-go

... I would also confirm that the symbolic link goes to the outer .ssb-go and not to the inner (I'm not totally sure what could have created the inner)

... if peachpub-go-sbot is green, there is no inner .ssb-go folder, and whoami is still giving an error... we still have a mystery

ps: sorry we don't have more streamlined tools for log reporting and debugging, but thanks for the patience. also still up for hopping on a call if this gets too messy

> I messaged this to @abekonge on ssb, but switching to here, so its all in this thread. > > yup I would echo what glyph said to compare .ssb-go/secret and .ssb-go/.ssb-go/secret and see which has the pub key of the pub. If the inner one is a different pub key than the pub, then I would delete the inner folder. > > After that, I would confirm > ``` > systemctl restart peachpub-go-sbot > systemctl status peachpub-go-sbot (still green?) > ``` > This took quite a while, like 3 minutes, but afterwards it was green > ... if not green, I would try > ``` > systemctl stop peachpub-go-sbot > SSB_CONFIG_FILE=/home/yunohost.app/peachpub/.ssb-go/config.toml /var/www/peachpub/go-sbot > ``` > ... and see what logs you get there by running go-sbot directly > ... and running /var/www/peachpub/sbotcli call whoami in a separate terminal > > > ... if peachpub-go-sbot is green, then I would repeate the test again (now that there are not two .ssb-go folders) > ``` > /var/www/peachpub/sbotcli call whoami > ``` this still gives the warn and error: ``` root@yunohost:/home/yunohost.app/peachpub/.ssb-go# /var/www/peachpub/sbotcli call whoami level=warn client="unix-path based init failed" err="ssbClient: failed to open unix path \"/root/.ssb-go/socket\"" level=error run-failure="init: failed to connect to 127.0.0.1:8008|@oY8aLV/yEKs75d6TknxlwkhfiThB9HlN5FQY7WWTzho=.ed25519: error dialing: error wrapping connection: secrethandshake: failed during data transfer of receiving server auth: EOF" ``` > > That whoami call is run as root, that's why we needed to make the symbolic link before. Currently sbotcli can't be configured to tell it where .ssb-go, and just looks in ~/.ssb-go, so we trick it with the symbolic link. > ```ln -s /home/yunohost.app/peachpub/.ssb-go /root/.ssb-go``` > > ... I would also confirm that the symbolic link goes to the outer .ssb-go and not to the inner (I'm not totally sure what could have created the inner) > > ... if peachpub-go-sbot is green, there is no inner .ssb-go folder, and whoami is still giving an error... we still have a mystery > > __________ > > ps: sorry we don't have more streamlined tools for log reporting and debugging, but thanks for the patience. also still up for hopping on a call if this gets too messy

abekonge commented

2022-09-07 16:48:16 +00:00

Okay, I tried repair and fsck again, and now I can't start it, and inner .ssb-go IS BACK.

running just sbot-go gives

root@yunohost:/home/yunohost.app/peachpub/.ssb-go# SSB_CONFIG_FILE=/home/yunohost.app/peachpub/.ssb-go/config.toml /var/www/peachpub/go-sbot
level=info t=393.376µs event="read config" msg="config detected" path=/home/yunohost.app/peachpub/.ssb-go/config.toml
level=info t=3.706778ms event="write running-config.json" msg="active config and env vars have been persisted" path=/home/yunohost.app/peachpub/.ssb-go/running-config.json
level=info t=3.979792ms event="set repo" path=/home/yunohost.app/peachpub/.ssb-go/.ssb-go
t=5.254654ms starting=metrics addr=localhost:6078
level=info t=61.042404ms event="waiting for indexes to catch up"
level=info t=61.760562ms event="repo open" feeds=0 msgs=0
level=info t=61.971182ms event=serving ID="@Z7gQhlBKzEunBQM+PTQaD1yxtEbb4Q6T+Dzu4fsmFOE=.ed25519" addr=:8008 version=snapshot build=

I'll try to find out which thing makes the evil inner .ssb-go :)

Okay, I tried repair and fsck again, and now I can't start it, and inner .ssb-go IS BACK. running just sbot-go gives ``` root@yunohost:/home/yunohost.app/peachpub/.ssb-go# SSB_CONFIG_FILE=/home/yunohost.app/peachpub/.ssb-go/config.toml /var/www/peachpub/go-sbot level=info t=393.376µs event="read config" msg="config detected" path=/home/yunohost.app/peachpub/.ssb-go/config.toml level=info t=3.706778ms event="write running-config.json" msg="active config and env vars have been persisted" path=/home/yunohost.app/peachpub/.ssb-go/running-config.json level=info t=3.979792ms event="set repo" path=/home/yunohost.app/peachpub/.ssb-go/.ssb-go t=5.254654ms starting=metrics addr=localhost:6078 level=info t=61.042404ms event="waiting for indexes to catch up" level=info t=61.760562ms event="repo open" feeds=0 msgs=0 level=info t=61.971182ms event=serving ID="@Z7gQhlBKzEunBQM+PTQaD1yxtEbb4Q6T+Dzu4fsmFOE=.ed25519" addr=:8008 version=snapshot build= ``` I'll try to find out which thing makes the evil inner .ssb-go :)

abekonge commented

2022-09-07 16:52:05 +00:00

Ok, deleting new new inner .ssb-go, and running

SSB_CONFIG_FILE=/home/yunohost.app/peachpub/.ssb-go/config.toml /var/www/peachpub/go-sbot

creates a new inner .ssb-go ?!?

Ok, deleting new new inner .ssb-go, and running ``` SSB_CONFIG_FILE=/home/yunohost.app/peachpub/.ssb-go/config.toml /var/www/peachpub/go-sbot ``` creates a new inner .ssb-go ?!?

abekonge commented

2022-09-07 16:53:31 +00:00

ps: sorry we don't have more streamlined tools for log reporting and debugging, but thanks for the patience. also still up for hopping on a call if this gets too messy

no worries, I love bug-fixing with expert support. And async fits me well right now, with my family and a bit under the weather - but if we get stuck, sure lets make a call ...

> ps: sorry we don't have more streamlined tools for log reporting and debugging, but thanks for the patience. also still up for hopping on a call if this gets too messy no worries, I love bug-fixing with expert support. And async fits me well right now, with my family and a bit under the weather - but if we get stuck, sure lets make a call ...

abekonge commented

2022-09-07 16:58:10 +00:00

I wonder if the creation is an artifact of me running

SSB_CONFIG_FILE=/home/yunohost.app/peachpub/.ssb-go/config.toml /var/www/peachpub/go-sbot

inside the /home/yunohost.app/peachpub/.ssb-go/ folder.

In the config.toml the repo is:

repo = ".ssb-go"

So maybe it looks for .ssb-go, and not finding one in the relative context, creates a new one ... I'll test it by running it from the parent folder /home/yunohost.app/peachpub/

I wonder if the creation is an artifact of me running ``` SSB_CONFIG_FILE=/home/yunohost.app/peachpub/.ssb-go/config.toml /var/www/peachpub/go-sbot ``` inside the `/home/yunohost.app/peachpub/.ssb-go/` folder. In the config.toml the repo is: ``` repo = ".ssb-go" ``` So maybe it looks for .ssb-go, and not finding one in the relative context, creates a new one ... I'll test it by running it from the parent folder `/home/yunohost.app/peachpub/`

abekonge commented

2022-09-07 17:05:58 +00:00

Ok that worked, now go-sbot is crunching!

So it seems that what happened was that when trying to run the repair command from the .ssb-go folder, it looked for a .ssb-go subfolder, did not find one and went right ahead and created one :)

Now I'll try to see if I can get it to run as system service and see how it goes.

Ok that worked, now go-sbot is crunching! So it seems that what happened was that when trying to run the repair command from the .ssb-go folder, it looked for a .ssb-go subfolder, did not find one and went right ahead and created one :) Now I'll try to see if I can get it to run as system service and see how it goes.

👍 1 🎉 1

abekonge commented

2022-09-07 17:39:04 +00:00

ok after 30 minutes, my pipe was broken, and the go-sbot was stopped along with my ssh session.

now going back I can start the service, and get green status, but db is still unavailable and profile gives.

Stopping and trying the repair command gives a new error:

root@yunohost:/home/yunohost.app/peachpub# SSB_CONFIG_FILE=/home/yunohost.app/peachpub/.ssb-go/config.toml /var/www/peachpub/go-sbot -fsck sequences -repair
level=info t=463.79µs event="read config" msg="config detected" path=/home/yunohost.app/peachpub/.ssb-go/config.toml
level=info t=5.25757ms event="write running-config.json" msg="active config and env vars have been persisted" path=/home/yunohost.app/peachpub/.ssb-go/running-config.json
level=info t=5.611467ms event="set repo" path=/home/yunohost.app/peachpub/.ssb-go
t=7.473033ms starting=metrics addr=localhost:6078
go-sbot: failed to instantiate ssb server: while opening memtables error: while opening fid: 1 error: while updating skiplist error: mremap size mismatch: requested: 3007282 got: 134217728

ok after 30 minutes, my pipe was broken, and the go-sbot was stopped along with my ssh session. now going back I can start the service, and get green status, but db is still unavailable and profile gives. ![image](/attachments/35edf8a5-b3ac-4c2c-9e2a-af6058e87610) Stopping and trying the repair command gives a new error: ``` root@yunohost:/home/yunohost.app/peachpub# SSB_CONFIG_FILE=/home/yunohost.app/peachpub/.ssb-go/config.toml /var/www/peachpub/go-sbot -fsck sequences -repair level=info t=463.79µs event="read config" msg="config detected" path=/home/yunohost.app/peachpub/.ssb-go/config.toml level=info t=5.25757ms event="write running-config.json" msg="active config and env vars have been persisted" path=/home/yunohost.app/peachpub/.ssb-go/running-config.json level=info t=5.611467ms event="set repo" path=/home/yunohost.app/peachpub/.ssb-go t=7.473033ms starting=metrics addr=localhost:6078 go-sbot: failed to instantiate ssb server: while opening memtables error: while opening fid: 1 error: while updating skiplist error: mremap size mismatch: requested: 3007282 got: 134217728 ```

image.png

40 KiB

abekonge commented

2022-09-07 17:48:59 +00:00

Ok, running go-sbot again, quitting, running the repair again, seemed to work

But db is still unavail. etc.

Ok, running go-sbot again, quitting, running the repair again, seemed to work ![image](/attachments/41a587eb-12cd-4d1a-bf31-89c4fa557031) But db is still unavail. etc.

image.png

441 KiB

abekonge commented

2022-09-07 18:29:07 +00:00

ok `systemctl start peachpub-go-sbot´ creates an inner .ssb-go ...

running go-sbot after repair works, and whoami works:

root@yunohost:/home/admin# /var/www/peachpub/sbotcli call whoami
level=warn client="unix-path based init failed" err="ssbClient: failed to open unix path \"/root/.ssb-go/socket\""
level=info client=connected method=tcp
level=debug event="call reply"
{
  "id": "@oY8aLV/yEKs75d6TknxlwkhfiThB9HlN5FQY7WWTzho=.ed25519"
}

ok `systemctl start peachpub-go-sbot´ creates an inner .ssb-go ... running go-sbot after repair works, and whoami works: ``` root@yunohost:/home/admin# /var/www/peachpub/sbotcli call whoami level=warn client="unix-path based init failed" err="ssbClient: failed to open unix path \"/root/.ssb-go/socket\"" level=info client=connected method=tcp level=debug event="call reply" { "id": "@oY8aLV/yEKs75d6TknxlwkhfiThB9HlN5FQY7WWTzho=.ed25519" } ```

abekonge commented

2022-09-07 19:19:40 +00:00

Ok I tried changing the peachoub ui DATABASE DIRECTORY to the absolute path of /home/yunohost.app/peachpub/.ssb-go

but that fails to start with systemctl

also relative paths like ./ and ../.ssb-go fails to start. Systemctl logs says

Aug 20 11:07:05 yunohost.local systemd[1]: peachpub-go-sbot.service: Main process exited, code=exited, status=1/FAILURE
Aug 20 11:07:05 yunohost.local systemd[1]: peachpub-go-sbot.service: Failed with result 'exit-code'.

I like the exit code: exit code, hahaha

I feel like I'm missing something:

so go-sbot works, whoami works.

But when I start the service, that peachpub ui depends on, it creates a new inner .ssb-go which has another secret ... and things does not work .... not sure what to do next - do you have ideas?

Ok I tried changing the peachoub ui `DATABASE DIRECTORY` to the absolute path of `/home/yunohost.app/peachpub/.ssb-go` but that fails to start with systemctl also relative paths like `./` and `../.ssb-go` fails to start. Systemctl logs says ``` Aug 20 11:07:05 yunohost.local systemd[1]: peachpub-go-sbot.service: Main process exited, code=exited, status=1/FAILURE Aug 20 11:07:05 yunohost.local systemd[1]: peachpub-go-sbot.service: Failed with result 'exit-code'. ``` I like the exit code: `exit code`, hahaha I feel like I'm missing something: so go-sbot works, whoami works. But when I start the service, that peachpub ui depends on, it creates a new inner .ssb-go which has another secret ... and things does not work .... not sure what to do next - do you have ideas?

abekonge commented

2022-09-07 20:32:29 +00:00

ahahahahaha, it works!

Ok, so I found out that go-ssb would make a new .ssb-go in any location it was run, so I figured that the systemd settings would point to the wrong dir somehow, so I went to the systemd unit file: /etc/systemd/system/peachpub-go-sbot.service and changed the following

WorkingDirectory=/home/yunohost.app/peachpub/.ssb-go

to

WorkingDirectory=/home/yunohost.app/peachpub

I also found the log files, which was helpfull, because when I tried to start, it failed as it did earlier when I changed the path to the db to be absolute: StandardOutput=append:/var/log/peachpub/peachpub-go-sbot.log

Now when starting and failing, it logged this:

go-sbot: failed to instantiate ssb server: open .ssb-go/ebt-state-matrix/0000a18f1a2d5ff210ab3be5de93927c65c2485f893841f4794de45418ed6593ce1a: permission denied

chowning that, it now gave

go-sbot: failed to instantiate ssb server: Cannot write pid file "/home/yunohost.app/peachpub/.ssb-go/sublogs/shared-badger/LOCK" error: open /home/yunohost.app/peachpub/.ssb-go/sublogs/shared-badger/LOCK: permission denied

chowning that as well, and it starts, and peachpub ui runs - yay!

ahahahahaha, it works! Ok, so I found out that go-ssb would make a new .ssb-go in any location it was run, so I figured that the systemd settings would point to the wrong dir somehow, so I went to the systemd unit file: `/etc/systemd/system/peachpub-go-sbot.service` and changed the following ``` WorkingDirectory=/home/yunohost.app/peachpub/.ssb-go ``` to ``` WorkingDirectory=/home/yunohost.app/peachpub ``` I also found the log files, which was helpfull, because when I tried to start, it failed as it did earlier when I changed the path to the db to be absolute: `StandardOutput=append:/var/log/peachpub/peachpub-go-sbot.log` Now when starting and failing, it logged this: ``` go-sbot: failed to instantiate ssb server: open .ssb-go/ebt-state-matrix/0000a18f1a2d5ff210ab3be5de93927c65c2485f893841f4794de45418ed6593ce1a: permission denied ``` chowning that, it now gave ``` go-sbot: failed to instantiate ssb server: Cannot write pid file "/home/yunohost.app/peachpub/.ssb-go/sublogs/shared-badger/LOCK" error: open /home/yunohost.app/peachpub/.ssb-go/sublogs/shared-badger/LOCK: permission denied ``` chowning that as well, and it starts, and peachpub ui runs - yay! ![image](/attachments/dfabc1f0-fe7f-4692-b4d2-09a60b59e9c3)

image.png

82 KiB

🎉 1

abekonge commented

2022-09-07 20:38:44 +00:00

Now as to how these files both got root:root, I have no idea.

And whether the working dir should be changed, I'm not sure.

Also, should go-ssb create a new database when it does not find one?

If there is something I can do to help this not be a problem in the future, I'll be willing to help - this was fun!

Now as to how these files both got root:root, I have no idea. And whether the working dir should be changed, I'm not sure. Also, should go-ssb create a new database when it does not find one? --- If there is something I can do to help this not be a problem in the future, I'll be willing to help - this was fun!

❤️ 1

abekonge commented

2022-09-08 07:16:36 +00:00

Okay, now I have hit the original problem maybe.

So after everything works, I open up patchwork on my machine, solpunk pub shows up as a connection, and it start indexing, the little green circle on the solpunk pub connection fills out.

And then it disappears from patchwork, and the pi is now unresponsive, the green led is on, all the time. I left it overnight, and it was still green, and unresponsive, so had to restart the pi.

Afterwards, I checked that peachpub was up in the ui, and that worked, but not showing up in patchwork.

I stopped go-sbot, repaired the feed, started again manually, and it connected to patchwork and started the indexing and froze yet again.

The following is the output from go-sbot when going into freeze:


root@yunohost:/home/yunohost.app/peachpub# SSB_CONFIG_FILE=/home/yunohost.app/peachpub/.ssb-go/config.toml /var/www/peachpub/go-sbot -repair
level=info t=1.389235ms event="read config" msg="config detected" path=/home/yunohost.app/peachpub/.ssb-go/config.toml
level=info t=6.99216ms event="write running-config.json" msg="active config and env vars have been persisted" path=/home/yunohost.app/peachpub/.ssb-go/running-config.json
level=info t=7.18153ms event="set repo" path=/home/yunohost.app/peachpub/.ssb-go
t=19.259962ms starting=metrics addr=localhost:6078
2022/08/20 22:40:51 adverstiser using local address :8008
level=info t=435.373904ms event="waiting for indexes to catch up"
level=debug t=4.409008521s event=update-replicate feed-want-count=236 hops=1 took=976.404662ms
level=info t=7.595335321s event="repo open" feeds=2432 msgs=424830
level=info t=7.596287223s event=serving ID="@oY8aLV/yEKs75d6TknxlwkhfiThB9HlN5FQY7WWTzho=.ed25519" addr=:8008 version=snapshot build=
level=debug t=23.604332544s plugin=gossip event=gossiprx fr=<@F/Lw.ed25519> starting=1756 received=3 took=869.682002ms
level=debug t=23.976643525s plugin=gossip event=gossiprx fr=<@G98X.ed25519> starting=5093 received=8 took=1.241666273s
level=debug t=49.012404794s plugin=gossip event=gossiprx fr=<@uVCh.ed25519> starting=0 received=154 took=26.273536343s
level=debug t=49.4751798s plugin=gossip event=gossiprx fr=<@J/Xd.ed25519> starting=0 received=135 took=26.738857947s
level=debug t=54.193177946s plugin=gossip event=gossiprx fr=<@n0d6.ed25519> starting=0 received=122 took=31.455566544s
level=debug t=55.047802481s plugin=gossip event=gossiprx fr=<@is/9.ed25519> starting=0 received=172 took=32.30981838s
level=debug t=55.85434078s plugin=gossip event=gossiprx fr=<@Hr9q.ed25519> starting=0 received=166 took=33.118095384s
level=debug t=1m3.57353633s plugin=gossip event=gossiprx fr=<@uHed.ed25519> starting=0 received=155 took=40.834709178s
level=debug t=1m5.477842399s plugin=gossip event=gossiprx fr=<@T+Qc.ed25519> starting=0 received=162 took=42.738261257s
level=debug t=1m9.391207977s plugin=gossip event=gossiprx fr=<@MwnJ.ed25519> starting=0 received=170 took=46.65244697s
level=debug t=1m9.69797688s plugin=gossip event=gossiprx fr=<@Bg62.ed25519> starting=0 received=206 took=46.959143688s
level=debug t=1m11.15390444s plugin=gossip event=gossiprx fr=<@kvye.ed25519> starting=0 received=226 took=48.41799216s
level=debug t=1m17.276766595s plugin=gossip event=gossiprx fr=<@uNGj.ed25519> starting=0 received=244 took=54.534669373s

So somehow, when trying to sync with my patchwork go-sbot dies or goes into an infinite loop or something?

Okay, now I have hit the original problem maybe. So after everything works, I open up patchwork on my machine, solpunk pub shows up as a connection, and it start indexing, the little green circle on the solpunk pub connection fills out. And then it disappears from patchwork, and the pi is now unresponsive, the green led is on, all the time. I left it overnight, and it was still green, and unresponsive, so had to restart the pi. Afterwards, I checked that peachpub was up in the ui, and that worked, but not showing up in patchwork. I stopped go-sbot, repaired the feed, started again manually, and it connected to patchwork and started the indexing and froze yet again. The following is the output from go-sbot when going into freeze: ``` root@yunohost:/home/yunohost.app/peachpub# SSB_CONFIG_FILE=/home/yunohost.app/peachpub/.ssb-go/config.toml /var/www/peachpub/go-sbot -repair level=info t=1.389235ms event="read config" msg="config detected" path=/home/yunohost.app/peachpub/.ssb-go/config.toml level=info t=6.99216ms event="write running-config.json" msg="active config and env vars have been persisted" path=/home/yunohost.app/peachpub/.ssb-go/running-config.json level=info t=7.18153ms event="set repo" path=/home/yunohost.app/peachpub/.ssb-go t=19.259962ms starting=metrics addr=localhost:6078 2022/08/20 22:40:51 adverstiser using local address :8008 level=info t=435.373904ms event="waiting for indexes to catch up" level=debug t=4.409008521s event=update-replicate feed-want-count=236 hops=1 took=976.404662ms level=info t=7.595335321s event="repo open" feeds=2432 msgs=424830 level=info t=7.596287223s event=serving ID="@oY8aLV/yEKs75d6TknxlwkhfiThB9HlN5FQY7WWTzho=.ed25519" addr=:8008 version=snapshot build= level=debug t=23.604332544s plugin=gossip event=gossiprx fr=<@F/Lw.ed25519> starting=1756 received=3 took=869.682002ms level=debug t=23.976643525s plugin=gossip event=gossiprx fr=<@G98X.ed25519> starting=5093 received=8 took=1.241666273s level=debug t=49.012404794s plugin=gossip event=gossiprx fr=<@uVCh.ed25519> starting=0 received=154 took=26.273536343s level=debug t=49.4751798s plugin=gossip event=gossiprx fr=<@J/Xd.ed25519> starting=0 received=135 took=26.738857947s level=debug t=54.193177946s plugin=gossip event=gossiprx fr=<@n0d6.ed25519> starting=0 received=122 took=31.455566544s level=debug t=55.047802481s plugin=gossip event=gossiprx fr=<@is/9.ed25519> starting=0 received=172 took=32.30981838s level=debug t=55.85434078s plugin=gossip event=gossiprx fr=<@Hr9q.ed25519> starting=0 received=166 took=33.118095384s level=debug t=1m3.57353633s plugin=gossip event=gossiprx fr=<@uHed.ed25519> starting=0 received=155 took=40.834709178s level=debug t=1m5.477842399s plugin=gossip event=gossiprx fr=<@T+Qc.ed25519> starting=0 received=162 took=42.738261257s level=debug t=1m9.391207977s plugin=gossip event=gossiprx fr=<@MwnJ.ed25519> starting=0 received=170 took=46.65244697s level=debug t=1m9.69797688s plugin=gossip event=gossiprx fr=<@Bg62.ed25519> starting=0 received=206 took=46.959143688s level=debug t=1m11.15390444s plugin=gossip event=gossiprx fr=<@kvye.ed25519> starting=0 received=226 took=48.41799216s level=debug t=1m17.276766595s plugin=gossip event=gossiprx fr=<@uNGj.ed25519> starting=0 received=244 took=54.534669373s ``` So somehow, when trying to sync with my patchwork go-sbot dies or goes into an infinite loop or something?

glyph commented

2022-09-08 07:50:11 +00:00

Wow, thanks @abekonge, this is some A-grade detective work! I really appreciate you tackling the problem methodically.

Seems like there are a few issues at play. One is essentially a configuration issue: an incorrect WorkingDirectory path in the unit file and incorrect permissions for some of the files in .ssb-go.

The second issue, the one which is crashing the go-sbot during replication, appears to be related to memory usage. See this issue comment:

https://github.com/ssbc/go-ssb/issues/124#issuecomment-1078357758

I noticed the memtables error in one of the logs you posted:

go-sbot: failed to instantiate ssb server: while opening memtables error: while opening fid: 1 error: while updating skiplist error: mremap size mismatch: requested: 3007282 got: 134217728

It seems that @boreq has a fix for the issue but we'll have to investigate further.

Wow, thanks @abekonge, this is some A-grade detective work! I really appreciate you tackling the problem methodically. Seems like there are a few issues at play. One is essentially a configuration issue: an incorrect `WorkingDirectory` path in the unit file and incorrect permissions for some of the files in `.ssb-go`. The second issue, the one which is crashing the go-sbot during replication, appears to be related to memory usage. See this issue comment: https://github.com/ssbc/go-ssb/issues/124#issuecomment-1078357758 I noticed the `memtables` error in one of the logs you posted: `go-sbot: failed to instantiate ssb server: while opening memtables error: while opening fid: 1 error: while updating skiplist error: mremap size mismatch: requested: 3007282 got: 134217728` It seems that `@boreq` has a fix for the issue but we'll have to investigate further.

❤️ 1

glyph commented

2022-09-08 09:23:59 +00:00

Now as to how these files both got root:root, I have no idea.

My current thinking is that the files became root:root when the symlink was created and go-sbot was run as root. Any files created by the go-sbot would then have root:root permissions. So I think this particular issue was a side-effect of the debugging process.

And whether the working dir should be changed, I'm not sure.

I think @notplants should have some insight on this one. Based on your experience, it seems the working dir should indeed be changed.

Also, should go-ssb create a new database when it does not find one?

This is the default behaviour and I don't think it's possible to disable it "from the outside". The best we can do is look out for this dual-directory issue in the future.

One thing which puzzles me: the device was originally working and replicating with Patchwork. I suspect the sbot then encountered the memtables error and crashed (no longer showing up in local connections of Patchwork). Then, somehow, things got all mixed up. I think part of this was the symlink + root execution, but I'm not sure why the working directory was initially fine but then later it was not?

> Now as to how these files both got root:root, I have no idea. My current thinking is that the files became `root:root` when the symlink was created and `go-sbot` was run as `root`. Any files created by the `go-sbot` would then have `root:root` permissions. So I think this particular issue was a side-effect of the debugging process. > And whether the working dir should be changed, I'm not sure. I think @notplants should have some insight on this one. Based on your experience, it seems the working dir should indeed be changed. > Also, should go-ssb create a new database when it does not find one? This is the default behaviour and I don't think it's possible to disable it "from the outside". The best we can do is look out for this dual-directory issue in the future. ----- One thing which puzzles me: the device was originally working and replicating with Patchwork. I suspect the sbot then encountered the `memtables` error and crashed (no longer showing up in local connections of Patchwork). Then, somehow, things got all mixed up. I think part of this was the symlink + root execution, but I'm not sure why the working directory was initially fine but then later it was not?

glyph commented

2022-09-08 09:46:31 +00:00

Are you willing and able to share the logfile?

/var/log/peachpub/peachpub-go-sbot.log

Then I can look through the full sequence of events.

Are you willing and able to share the logfile? `/var/log/peachpub/peachpub-go-sbot.log` Then I can look through the full sequence of events.

abekonge commented

2022-09-08 10:13:34 +00:00

Are you willing and able to share the logfile?

/var/log/peachpub/peachpub-go-sbot.log

Then I can look through the full sequence of events.

Sure, how do you want it?

> Are you willing and able to share the logfile? > > `/var/log/peachpub/peachpub-go-sbot.log` > > Then I can look through the full sequence of events. Sure, how do you want it?

glyph commented

2022-09-08 10:22:37 +00:00

You could upload it here if it's not too large?

abekonge commented

2022-09-08 10:31:31 +00:00

here you go

peachpub-go-sbot.log

428 KiB

👍 1

notplants commented

2022-09-08 10:41:48 +00:00

@abekong cool to read, was a suspensefull report. I was right there with you with your excitement when you figured it out and got it working again

I reach the same conclusions as @glyph.

two issues, the original issue, which seems to be an issue with go-ssb, and we might not be able to fix without some help from our go friends.

the second issue, which was a side-effect of the debugging process. I'm not totally sure how the WorkingDirectory + database path configuration could have been originally working, and then switched to something not working.

in my peachpub running on yunohost on a digital ocean droplet,
/home/yunohost.app/peachpub/.ssb-go/config.toml
has an absolute path to the .ssb-go folder
repo = "/home/yunohost.app/peachpub/.ssb-go"

this absolute path is also whats initially configured in the yunohost package install (https://github.com/YunoHost-Apps/peachpub_ynh/blob/master/conf/config.toml),

so my best guess right now is that when you changed the hops to 1, it also inadvertently changed the repo= to be a relative path of just ".ssb-go", which didn't work with the rest of the configuration settings, and thus required that WorkingDirectory change to accomodate

I will try to confirm this, and possibly see if the go-sbot configuration page might not be reading the already set values and supplying its own default, or something like that

and then the root:root issue that is just that while debugging we ran as root. so after the debugging process, we always need to run chown -R peachpub:peachpub of relevant directories (and this is useful to remember for any future debugging)

@abekong cool to read, was a suspensefull report. I was right there with you with your excitement when you figured it out and got it working again I reach the same conclusions as @glyph. two issues, the original issue, which seems to be an issue with go-ssb, and we might not be able to fix without some help from our go friends. the second issue, which was a side-effect of the debugging process. I'm not totally sure how the WorkingDirectory + database path configuration could have been originally working, and then switched to something not working. in my peachpub running on yunohost on a digital ocean droplet, /home/yunohost.app/peachpub/.ssb-go/config.toml has an absolute path to the .ssb-go folder repo = "/home/yunohost.app/peachpub/.ssb-go" this absolute path is also whats initially configured in the yunohost package install (https://github.com/YunoHost-Apps/peachpub_ynh/blob/master/conf/config.toml), so my best guess right now is that when you changed the hops to 1, it also inadvertently changed the repo= to be a relative path of just ".ssb-go", which didn't work with the rest of the configuration settings, and thus required that WorkingDirectory change to accomodate I will try to confirm this, and possibly see if the go-sbot configuration page might not be reading the already set values and supplying its own default, or something like that and then the root:root issue that is just that while debugging we ran as root. so after the debugging process, we always need to run chown -R peachpub:peachpub of relevant directories (and this is useful to remember for any future debugging)

❤️ 1

notplants commented

2022-09-08 10:50:03 +00:00

so in good news, assuming my guess is correct, through the debugging process we uncovered one specific bug we can fix in peachpub

but the root issue may not be immediately solveable. still really appreciate your trying it out and such detailed debugging. this is also another useful datapoint for plan making

maybe beyond the scope of this thread, but the possible paths I imagine,

see if anyone can upstream some of boreqs fixes from scuttlego into go-ssb (I talked to @decentralize yesterday and they were not sure if the projects were too different or not for that to be possible. ideally both for this issue, and for the ebt-manyverse-compatability issue)
switch peachpub to depend on js sbot
switch peachpub to depend on scuttlego

so in good news, assuming my guess is correct, through the debugging process we uncovered one specific bug we can fix in peachpub but the root issue may not be immediately solveable. still really appreciate your trying it out and such detailed debugging. this is also another useful datapoint for plan making maybe beyond the scope of this thread, but the possible paths I imagine, - see if anyone can upstream some of boreqs fixes from scuttlego into go-ssb (I talked to @decentralize yesterday and they were not sure if the projects were too different or not for that to be possible. ideally both for this issue, and for the ebt-manyverse-compatability issue) - switch peachpub to depend on js sbot - switch peachpub to depend on scuttlego

notplants commented

2022-09-08 11:15:14 +00:00

... * I guess because you were seeing the inner .ssb-go before changing the hops, that might not have been when the value of repo was changed to be relative ... but maybe at some point that value was changed to be relative for some reason ...

glyph commented

2022-09-08 11:17:18 +00:00

@notplants

Thanks for your assistance with this. I certainly learned a lot from this exercise and it's great that we have a couple of targeted areas to improve on when it comes to PeachCloud.

so my best guess right now is that when you changed the hops to 1, it also inadvertently changed the repo= to be a relative path of just ".ssb-go", which didn't work with the rest of the configuration settings, and thus required that WorkingDirectory change to accomodate

Good thinking! I hadn't thought of that.

Do you know when you (or I) compiled the version of go-sbot that is being used in the YunoHost application? I notice that @cryptix merged a change to go-sbot from @boreq on 12 April which may improve the memory situation slightly (PR link).

maybe beyond the scope of this thread, but the possible paths I imagine,

Would love to have a call dedicated to exploring this together. I'm almost finished writing the lykin tutorial series and then I'll have time to work on PeachCloud again.

@notplants Thanks for your assistance with this. I certainly learned a lot from this exercise and it's great that we have a couple of targeted areas to improve on when it comes to PeachCloud. > so my best guess right now is that when you changed the hops to 1, it also inadvertently changed the repo= to be a relative path of just ".ssb-go", which didn't work with the rest of the configuration settings, and thus required that WorkingDirectory change to accomodate Good thinking! I hadn't thought of that. Do you know when you (or I) compiled the version of `go-sbot` that is being used in the YunoHost application? I notice that `@cryptix` merged a change to `go-sbot` from `@boreq` on 12 April which may improve the memory situation slightly ([PR link](https://github.com/ssbc/go-ssb/pull/147)). > maybe beyond the scope of this thread, but the possible paths I imagine, Would love to have a call dedicated to exploring this together. I'm almost finished writing the lykin tutorial series and then I'll have time to work on PeachCloud again.

👍 1

abekonge commented

2022-09-08 17:39:31 +00:00

Cool, happy that this exercise had some value :)

and then the root:root issue that is just that while debugging we ran as root. so after the debugging process, we always need to run chown -R peachpub:peachpub of relevant directories (and this is useful to remember for any future debugging)

I noticed that the other files had peachpub:www-data, is that significant?

And do you have ideas to how I can get this pub working again?

I'm up for resetting/reinstalling, if that is what it takes...

Cool, happy that this exercise had some value :) > and then the root:root issue that is just that while debugging we ran as root. so after the debugging process, we always need to run chown -R peachpub:peachpub of relevant directories (and this is useful to remember for any future debugging) I noticed that the other files had peachpub:www-data, is that significant? And do you have ideas to how I can get this pub working again? I'm up for resetting/reinstalling, if that is what it takes...

notplants commented

2022-09-08 18:23:15 +00:00

@abekonge, I think both peachpub:www-data or peachpub:peachpub would work as permissions for the files. go-sbot and peach-web both run as peachpub user in the yunohost configuration, so its not so important (I believe)

to get the pub working again,
one could uninstall peachpub via yunohost and then reinstall it -- I think this will work, and just give you a fresh install (with a new public key)

however, based on my understanding of this bug, unfortunately I guess that it will likely run into the same error condition at some point again

could reinstall and try again if you have the spoons and see if it reaches the same error condition... but may need to wait for an update to the underlying sbot for a real/stable working fix ... once the update comes though, I should be able to publish a new package to yunohost with the new binary, and then you should be able to simply update peachpub through the admin UI to get the new working code ... I will keep you updated... but from my perspective I can't say if that could be a few weeks, or a few months...

I will also look into what glyph suggested, and try to confirm the current peachpub package includes the latest go-sbot code at the very least

@abekonge, I think both peachpub:www-data or peachpub:peachpub would work as permissions for the files. go-sbot and peach-web both run as peachpub user in the yunohost configuration, so its not so important (I believe) to get the pub working again, one could uninstall peachpub via yunohost and then reinstall it -- I think this will work, and just give you a fresh install (with a new public key) however, based on my understanding of this bug, unfortunately I guess that it will likely run into the same error condition at some point again could reinstall and try again if you have the spoons and see if it reaches the same error condition... but may need to wait for an update to the underlying sbot for a real/stable working fix ... once the update comes though, I should be able to publish a new package to yunohost with the new binary, and then you should be able to simply update peachpub through the admin UI to get the new working code ... I will keep you updated... but from my perspective I can't say if that could be a few weeks, or a few months... I will also look into what glyph suggested, and try to confirm the current peachpub package includes the latest go-sbot code at the very least

decentral1se commented

2022-09-09 18:13:20 +00:00

Epic reporting and back & forth debugging folks. I'm in awe.

decentral1se commented

2022-09-09 18:19:09 +00:00

switch peachpub to depend on scuttlego

Asking in golang ssb general (https://matrix.to/#/#golang-ssb-general:autonomic.zone) about builds for scuttlego.

Interesting idea...

> switch peachpub to depend on scuttlego Asking in `golang ssb general` (https://matrix.to/#/#golang-ssb-general:autonomic.zone) about builds for scuttlego. Interesting idea...

decentral1se commented

2022-09-09 18:47:24 +00:00

Asking in golang ssb general (https://matrix.to/#/#golang-ssb-general:autonomic.zone) about builds for scuttlego.

Summary of scuttlego pub possibilities right now is that there is now CLI code in scuttlego that allows us to run it as a pub. boreq is saying there would need to be some things changed inside the lib to make that possible and it is not clear how to do it right now. I've asked to be let known when things might change or if it is possible for me to take a stab at writing one... more in the matrix chat as it comes in i guess...

> Asking in golang ssb general (https://matrix.to/#/#golang-ssb-general:autonomic.zone) about builds for scuttlego. Summary of scuttlego pub possibilities right now is that there is now CLI code in scuttlego that allows us to run it as a pub. boreq is saying there would need to be some things changed inside the lib to make that possible and it is not clear how to do it right now. I've asked to be let known when things might change or if it is possible for me to take a stab at writing one... more in the matrix chat as it comes in i guess...

👀 1

notplants commented

2022-09-10 10:22:39 +00:00

nice recon, seems like a promising direction ~ https://github.com/planetary-social/scuttlego/issues/54

abekonge commented

2022-09-17 08:12:45 +00:00

to get the pub working again,
one could uninstall peachpub via yunohost and then reinstall it -- I think this will work, and just give you a fresh install (with a new public key)

however, based on my understanding of this bug, unfortunately I guess that it will likely run into the same error condition at some point again

could reinstall and try again if you have the spoons and see if it reaches the same error condition... but may need to wait for an update to the underlying sbot for a real/stable working fix ... once the update comes though, I should be able to publish a new package to yunohost with the new binary, and then you should be able to simply update peachpub through the admin UI to get the new working code ... I will keep you updated... but from my perspective I can't say if that could be a few weeks, or a few months...

Ok I'll try resetting. A question: the peachpub @notplants run, you do not run into this issue?

ASIDE: Hanging as hardware-installation protocol is teh best

> > to get the pub working again, > one could uninstall peachpub via yunohost and then reinstall it -- I think this will work, and just give you a fresh install (with a new public key) > > however, based on my understanding of this bug, unfortunately I guess that it will likely run into the same error condition at some point again > > could reinstall and try again if you have the spoons and see if it reaches the same error condition... but may need to wait for an update to the underlying sbot for a real/stable working fix ... once the update comes though, I should be able to publish a new package to yunohost with the new binary, and then you should be able to simply update peachpub through the admin UI to get the new working code ... I will keep you updated... but from my perspective I can't say if that could be a few weeks, or a few months... > Ok I'll try resetting. A question: the peachpub @notplants run, you do not run into this issue? --- ASIDE: Hanging as hardware-installation protocol is teh best ![image](/attachments/a74fdc95-5c3d-4fe4-9834-241c1c6a6f3f)

image.png

3.0 MiB

notplants commented

2022-09-19 11:39:01 +00:00

@abekonge the peachpub I've been running is on a digital ocean droplet, and it hasn't run into this issue, probably because its more powerful hardware

my peachpub on digital ocean droplet still has the issue of not connecting with manyverse, but not this issue your seeing, and has otherwise been working

I had only tested running on a raspberry pi before with test accounts, and not a full amount of data, and naivelly assumed it was working, so I hadn't run into this issue myself yet

but I'm hopeful one way or another we'll be able to get a stable version at some point, and then share the update... so this may be currently more of a taster/speculative-motivator, than the real thing

@abekonge the peachpub I've been running is on a digital ocean droplet, and it hasn't run into this issue, probably because its more powerful hardware my peachpub on digital ocean droplet still has the issue of not connecting with manyverse, but not this issue your seeing, and has otherwise been working I had only tested running on a raspberry pi before with test accounts, and not a full amount of data, and naivelly assumed it was working, so I hadn't run into this issue myself yet ___ but I'm hopeful one way or another we'll be able to get a stable version at some point, and then share the update... so this may be currently more of a taster/speculative-motivator, than the real thing

Sign in to join this conversation.

4 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: PeachCloud/peach-workspace#134