Backupbot Revolution #23

2023-09-07T13:42:29Z

moritz commented

2023-09-07 13:42:29 +00:00

No description provided.

🎉 1

moritz added 12 commits 2023-09-07 13:42:29 +00:00

Backup volumes from host instead of copying paths 29bbd5dedd

* Backupbot will now copy all volumes from a service with
  backupbot.enabled = 'true' label from the /var/lib/docker/volumes/
  path directly. This reduces the resource overhead of copying
  stuff from one volume to another.
  Recipes need to be adjustet that db-dumps are saved into a volume
  now!
* Remove the Dockerfile and move stuff into a entrypoint. This
  simplifies the whole versioning thing and makes this "just"
  a recipe

Co-authored-by: Moritz < moritz.m@local-it.org>

mount volume ro 4d6371eba7

fix shellcheck prevent globbing 9f93a55c09

remove unused traefik labels 9b9cfc6e17

initial rewrite 1633c3e856

mount volumes read/write to restore backups ece8fd514c

use latest snapshot as default 4dbb7ada8d

change repo per option 56eafa714e

choos specific restore target 89487cc219

README 31e746696d

update README 085a8a5ae3

update README

continuous-integration/drone/pr Build is failing

Details

9dfdb61ba1

moritz added this to the backupbot revolution project 2023-09-07 13:42:56 +00:00

moritz self-assigned this 2023-09-07 13:43:02 +00:00

moritz referenced this pull request

2023-09-07 13:44:43 +00:00

Support saving multiple local backups #7

moritz referenced this pull request

2023-09-07 14:07:04 +00:00

layout a backupbot (drone-like) interface #20

moritz added 2 commits 2023-09-07 23:17:25 +00:00

add sftp storage a64ae10aa1

add healthcheck

continuous-integration/drone/pr Build is failing

Details

55cbae63a8

moritz added 1 commit 2023-09-07 23:30:58 +00:00

specify program versions to prevent future breakage

continuous-integration/drone/pr Build is failing

Details

789cc7972b

moritz added 1 commit 2023-09-07 23:48:20 +00:00

fix entrypoint

continuous-integration/drone/pr Build is failing

Details

77e8792949

decentral1se reviewed 2023-09-08 09:14:00 +00:00

decentral1se left a comment

First quick pass, looking great! Great work!

README.md Outdated

						
				@@ -45,0 +80,4 @@

				Add your `SSH_KEY` as docker secret:

				```

				abra app secret insert <app_name> ssh_key v1 """$(cat backupkey)

				"""

decentral1se commented

2023-09-08 09:08:50 +00:00

Unwanted newline?

moritz commented

2023-09-21 13:46:28 +00:00

Unfortunately this newline-hack is required, else the ssh-key is not accepted.

👍 1

moritz marked this conversation as resolved

moritz added 2 commits 2023-09-21 22:54:33 +00:00

fix restic version, timeout and cron default timer b37b3d13d6

feat: Backup Secrets #28

continuous-integration/drone/pr Build is failing

Details

2838a36d43

moritz added 3 commits 2023-09-22 14:39:51 +00:00

Revert "feat: Backup Secrets #28 " f61666b6a6

This reverts commit 2838a36d43.

small fixes b90ed19023

feat: Backup Secrets (copy secrets) #28

continuous-integration/drone/pr Build is failing

Details

ef9fbda7d0

moritz added 1 commit 2023-09-22 14:50:51 +00:00

formatting

continuous-integration/drone/pr Build is failing

Details

e81131668f

moritz added 1 commit 2023-09-22 16:34:56 +00:00

fix error messages

continuous-integration/drone/pr Build is failing

Details

c6276274c6

kawaiipunk commented

2023-09-26 16:08:24 +00:00

Will it break anything on our existing backup setups if we merge this? Is a migration path needed? @moritz

👀 1

decentral1se commented

2023-10-01 06:22:34 +00:00

@moritz ran into this error while trying to set things up:

docker -c foo.com exec -it bbt.foo.com_app.1.h66skv7b40r7chqkhbrslt365 backup ls
File "/usr/bin/backup", line 106
if (app_name in apps and
^^^^^^^^^^^^^^^^^^^^
SyntaxError: cannot use assignment expressions with expression

Would it also be an idea to try and set up some tests for this while we're here? I can imagine we'll be changing the code as we go. Idk if you'd like to do that in Python or we could try something simple like the integration suite for abra (manually for the time being...).

@moritz ran into this error while trying to set things up: > docker -c foo.com exec -it bbt.foo.com_app.1.h66skv7b40r7chqkhbrslt365 backup ls > File "/usr/bin/backup", line 106 > if (app_name in apps and > ^^^^^^^^^^^^^^^^^^^^ > SyntaxError: cannot use assignment expressions with expression Would it also be an idea to try and set up some tests for this while we're here? I can imagine we'll be changing the code as we go. Idk if you'd like to do that in Python or we could try something simple like the integration suite for `abra` (manually for the time being...).

decentral1se referenced this pull request from toolshed/abra

2023-10-01 06:24:12 +00:00

feat: backup revolution #361

moritz added 1 commit 2023-10-03 16:19:54 +00:00

fix typo

continuous-integration/drone/pr Build is failing

Details

2fd0ac16bc

moritz commented

2023-10-03 16:23:52 +00:00

@moritz ran into this error while trying to set things up:

docker -c foo.com exec -it bbt.foo.com_app.1.h66skv7b40r7chqkhbrslt365 backup ls
File "/usr/bin/backup", line 106
if (app_name in apps and
^^^^^^^^^^^^^^^^^^^^
SyntaxError: cannot use assignment expressions with expression

Would it also be an idea to try and set up some tests for this while we're here? I can imagine we'll be changing the code as we go. Idk if you'd like to do that in Python or we could try something simple like the integration suite for abra (manually for the time being...).

I broke it through some silly code formatting. Yeah some tests would prevent this.

> @moritz ran into this error while trying to set things up: > > > docker -c foo.com exec -it bbt.foo.com_app.1.h66skv7b40r7chqkhbrslt365 backup ls > > File "/usr/bin/backup", line 106 > > if (app_name in apps and > > ^^^^^^^^^^^^^^^^^^^^ > > SyntaxError: cannot use assignment expressions with expression > > Would it also be an idea to try and set up some tests for this while we're here? I can imagine we'll be changing the code as we go. Idk if you'd like to do that in Python or we could try something simple like the integration suite for `abra` (manually for the time being...). I broke it through some silly code formatting. Yeah some tests would prevent this.

moritz added 1 commit 2023-10-03 16:41:26 +00:00

release note for migration

continuous-integration/drone/pr Build is failing

Details

a2aec9b59e

moritz commented

2023-10-03 16:50:04 +00:00

Will it break anything on our existing backup setups if we merge this? Is a migration path needed? @moritz

For migration the repository variables need to be merged, I pushed a release note: a2aec9b59e
This way the backup script is independent of the repository type.

> Will it break anything on our existing backup setups if we merge this? Is a migration path needed? @moritz For migration the repository variables need to be merged, I pushed a release note: https://git.coopcloud.tech/coop-cloud/backup-bot-two/commit/a2aec9b59e3a5ac3c91ed14d6551f8859274fb89 This way the backup script is independent of the repository type.

moritz added 1 commit 2023-10-03 20:04:37 +00:00

Prompt before restore

continuous-integration/drone/pr Build is failing

Details

06724428d3

moritz added 1 commit 2023-10-03 21:45:32 +00:00

restic_repo as secret option #31

continuous-integration/drone/pr Build is failing

Details

f6e062a9c3

decentral1se commented

2023-10-04 09:36:24 +00:00

@moritz any chance you could resolve the merge conflicts? Unsure I've got the right changeset resolving myself while testing / implementing the abra side of things. Thanks!

@moritz any chance you could resolve the merge conflicts? Unsure I've got the right changeset resolving myself while testing / implementing the `abra` side of things. Thanks!

decentral1se commented

2023-10-04 09:38:06 +00:00

Oh right, they're diverging 🤔

CONFLICT (content): Merge conflict in .env.sample
CONFLICT (modify/delete): backup.sh deleted in HEAD and modified in 83a3d82 (More HTTPS fixes). Version 83a3d82 (More HTTPS fixes) of backup.sh left in tree.

I think you need to undelete backup.sh for backwards compat while we're still transitioning?

Oh right, they're diverging 🤔 > CONFLICT (content): Merge conflict in .env.sample > CONFLICT (modify/delete): backup.sh deleted in HEAD and modified in 83a3d82 (More HTTPS fixes). Version 83a3d82 (More HTTPS fixes) of backup.sh left in tree. I think you need to undelete `backup.sh` for backwards compat while we're still transitioning?

moritz added 1 commit 2023-10-04 11:59:33 +00:00

remove package versions, to avoid conflicts

continuous-integration/drone/pr Build is failing

Details

d47ff6caac

moritz added 1 commit 2023-10-04 15:12:53 +00:00

extend download to download the secrets or all app volumes at once

continuous-integration/drone/pr Build is failing

Details

328adffdc1

moritz added 1 commit 2023-10-04 16:48:14 +00:00

fix secret name

continuous-integration/drone/pr Build is failing

Details

0cc9c3598d

moritz commented

2023-10-04 17:04:20 +00:00

Oh right, they're diverging 🤔

CONFLICT (content): Merge conflict in .env.sample
CONFLICT (modify/delete): backup.sh deleted in HEAD and modified in 83a3d82 (More HTTPS fixes). Version 83a3d82 (More HTTPS fixes) of backup.sh left in tree.

I think you need to undelete backup.sh for backwards compat while we're still transitioning?

Should the cronjob then also only run backup.sh? At the moment I don't see any issues for backwards compat apart the migration steps inside release/next. Maybe we shouldn't release the next version until it's not tested well.

> Oh right, they're diverging 🤔 > > > CONFLICT (content): Merge conflict in .env.sample > > CONFLICT (modify/delete): backup.sh deleted in HEAD and modified in 83a3d82 (More HTTPS fixes). Version 83a3d82 (More HTTPS fixes) of backup.sh left in tree. > > I think you need to undelete `backup.sh` for backwards compat while we're still transitioning? Should the cronjob then also only run backup.sh? At the moment I don't see any issues for backwards compat apart the migration steps inside `release/next`. Maybe we shouldn't release the next version until it's not tested well.

moritz force-pushed backupbot_revolution from 0cc9c3598d to bb1237f9ad

2023-10-04 17:08:58 +00:00

Compare

moritz commented

2023-10-04 17:09:33 +00:00