WIP: Backupbot Specification #258

Draft
p4u1 wants to merge 8 commits from p4u1/docs.coopcloud.tech:backupbottwo-spec into main
Member

For coop-cloud/organising#594

This is an initial draft for the backupbot-two specification. Note that this is the first time, that I tried to write a somewhat formal description on how something should work :D
Also note, that this does not reflect what is implemented, but what we want, as discussed in the following issues and meeting:

TODO:

  • specification

  • For Maintainers

    • tutorial
    • examples
  • For Operators

    • deployment
  • check when sh -c 'command' is necessary

  • decide where files should go

Open Questions

  • should the post hook be executed when the backup/restore fails?
  • should abra implement backup and restore or only provide an integration?
  • should we add a specification version?
For https://git.coopcloud.tech/coop-cloud/organising/issues/594 This is an initial draft for the backupbot-two specification. Note that this is the first time, that I tried to write a somewhat formal description on how something should work :D Also note, that this does not reflect what is implemented, but what we want, as discussed in the following issues and meeting: - https://pad.riseup.net/p/UEC2JUPGb6tmRCZ7RX9X-keep - https://git.coopcloud.tech/coop-cloud/backup-bot-two/issues/41#issuecomment-19094 TODO: - specification - For Maintainers - [ ] tutorial - [ ] examples - For Operators - [ ] deployment - [ ] check when `sh -c 'command'` is necessary - [ ] decide where files should go Open Questions - [ ] should the post hook be executed when the backup/restore fails? - [ ] should abra implement backup and restore or only provide an integration? - [ ] should we add a specification version?
p4u1 added 2 commits 2024-04-08 11:24:53 +00:00
Owner

@3wordchant @nicksellen @moritz @yksflip (am I forgetting anyone?) as backup-bot-two hackers, your input would be great!

@3wordchant @nicksellen @moritz @yksflip (am I forgetting anyone?) as `backup-bot-two` hackers, your input would be great!
decentral1se reviewed 2024-04-08 12:56:35 +00:00
decentral1se left a comment
Owner

Incredible work @p4u1! Thanks for diving into it 🤯

One bikeshed, on the dreaded naming topic: should we pick a name for it now we're going "official" with a spec? backup-bot-two could be something more generalised? This also might make sense in the context of how general this sepc is written in, referring to stacks/docker/etc. and not Co-op Cloud specifics. The labels are very Co-op Cloud specific. Also backupbot vs. backup-bot-two. No strong feelings but feels like maybe the time to bring it up...

For hooks, I know borgmatic has specific hooks for error scenarios: https://torsion.org/borgmatic/docs/how-to/add-preparation-and-cleanup-steps-to-backups/#error-hooks 🤔

Unsure what "should abra implement backup and restore or only provide an integration?" means? What we have now is that abra implements a docker exec-based "pass args" approach to trigger backup-bot-two on demand... maybe that should be documented also? E.g. 004cd70aed/cli/app/backup.go (L153) I think I followed coop-cloud/backup-bot-two#20 (comment)

Incredible work @p4u1! Thanks for diving into it 🤯 One bikeshed, on the dreaded naming topic: should we pick a name for it now we're going "official" with a spec? `backup-bot-two` could be something more generalised? This also might make sense in the context of how general this sepc is written in, referring to stacks/docker/etc. and *not* Co-op Cloud specifics. The labels are very Co-op Cloud specific. Also `backupbot` vs. `backup-bot-two`. No strong feelings but feels like maybe the time to bring it up... For hooks, I know `borgmatic` has specific hooks for error scenarios: https://torsion.org/borgmatic/docs/how-to/add-preparation-and-cleanup-steps-to-backups/#error-hooks 🤔 Unsure what "should abra implement backup and restore or only provide an integration?" means? What we have now is that `abra` implements a `docker exec`-based "pass args" approach to trigger `backup-bot-two` on demand... maybe that should be documented also? E.g. https://git.coopcloud.tech/coop-cloud/abra/src/commit/004cd70aed835cc2c71ba0ef447b93ad2cd6f1b4/cli/app/backup.go#L153 I think I followed https://git.coopcloud.tech/coop-cloud/backup-bot-two/issues/20#issuecomment-17748
Specification.md Outdated
@ -0,0 +6,4 @@
## Requirements
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this specification are to be interpreted as described in [RFC-2119].
Owner

[RFC-2119] missing link?

`[RFC-2119]` missing link?
p4u1 marked this conversation as resolved
@ -0,0 +45,4 @@
**Type:** boolean
**Default:** false
**Description:**
Enables backupbot for this compose stack. The labe should be added to the main service of the compose stack.
Owner

%s/labe/label
%s/backupbot/backup-bot-two

`%s/labe/label` `%s/backupbot/backup-bot-two`
Author
Member

Did %s/backupbot/backups to keep it generic

Did `%s/backupbot/backups` to keep it generic
Author
Member

Thanks for the feedback! Note that I'm still working on the docs for maintainers and operators, but feedback on the spec part is very appreciated :)

I'm all in for a different name, but not so creative atm.

"should abra implement backup and restore or only provide an integration?" means, that before 575f9905f1 it was possible to create a backup or do a restore using abra without having a backupbot-two deployed which could be nice for testing purposes and maybe other use cases

Thanks for the feedback! Note that I'm still working on the docs for maintainers and operators, but feedback on the spec part is very appreciated :) I'm all in for a different name, but not so creative atm. "should abra implement backup and restore or only provide an integration?" means, that before https://git.coopcloud.tech/coop-cloud/abra/commit/575f9905f1e26359bcd3ac2bc1a01e77062ca4a2 it was possible to create a backup or do a restore using `abra` without having a `backupbot-two` deployed which could be nice for testing purposes and maybe other use cases
p4u1 added 1 commit 2024-04-08 15:20:09 +00:00
3wordchant approved these changes 2024-04-09 20:11:01 +00:00
3wordchant left a comment
Owner

Looks fantastic, amazing work @p4u1 👏

Looks fantastic, amazing work @p4u1 👏
@ -0,0 +8,4 @@
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this specification are to be interpreted as described in [RFC-2119].
When backing up a docker stack you MUST first check if the `backupbot.backup`. It MUST be set to true, for the backup to happen.
Owner

Maybe remove this, seems the same as line 15 "To enable backups"

Maybe remove this, seems the same as line 15 "To enable backups"
p4u1 marked this conversation as resolved
Specification.md Outdated
@ -0,0 +12,4 @@
## Backup
To enable backups for a docker stack, the `backupbot.backup=true` label MUST be on any of its services. It SHOULD be declared on the first service.
Owner

Is the behaviour defined if one service sets backupbot.backup=true and another sets it to backupbot.backup=false?

Is the behaviour defined if one service sets `backupbot.backup=true` and another sets it to `backupbot.backup=false`?
@ -0,0 +16,4 @@
### Volumes and paths
By default all volumes MUST be backed up. A volume MAY be excluded from the backup when `backupbot.backup.volumes.{volume_name}=false` is set, where `{volume_name}` is the name of the volume.
Owner

So the spec allows backing up a volume even if ...{volume_name}=false? Is there any harm in changing this to "MUST"?

So the spec allows backing up a volume even if `...{volume_name}=false`? Is there any harm in changing this to "MUST"?
Author
Member

Changed it to "MUST"

Changed it to "MUST"
3wordchant marked this conversation as resolved
Specification.md Outdated
@ -0,0 +69,4 @@
**Type:** string
**Default:** ""
**Description:**
A comma seperated list of paths. When one or more paths are set, it only backups up those on the given volume instead of the whole volume.
Owner

seperated -> separated

backups up -> backs up

seperated -> separated backups up -> backs up
p4u1 marked this conversation as resolved
@ -0,0 +95,4 @@
**Type:** string
**Default:** ""
**Description:**
A command, that gets executed after the files are backuped.
Owner

backuped -> backed up

backuped -> backed up
p4u1 marked this conversation as resolved
Owner

One bikeshed, on the dreaded naming topic: should we pick a name for it now we're going "official" with a spec? backup-bot-two could be something more generalised?

I'm all in for a different name, but not so creative atm.

I like how self explanatory backup-bot is. @decentral1se if you're down to fully retire backup-bot we could maybe drop the -two.

If we wanted a more bRaNdEd name, how about:

  • chancey (following "definitely not Pokémon" tool naming convention, something something takes care of eggs, heals things)
  • guarda (following "spanish imperative" accidental tool naming convention, means "save!", unfortunately a name collision with a crypto wallet tho)
  • pocus (following "magic words" abra naming convention)
> One bikeshed, on the dreaded naming topic: should we pick a name for it now we're going "official" with a spec? `backup-bot-two` could be something more generalised? > I'm all in for a different name, but not so creative atm. I like how self explanatory `backup-bot` is. @decentral1se if you're down to fully retire [`backup-bot`](https://git.coopcloud.tech/coop-cloud/backup-bot) we could maybe drop the `-two`. If we wanted a more bRaNdEd name, how about: - `chancey` (following "definitely not Pokémon" tool naming convention, something something takes care of eggs, heals things) - `guarda` (following "spanish imperative" accidental tool naming convention, means "save!", unfortunately a name collision with a crypto wallet tho) - `pocus` (following "magic words" abra naming convention)
p4u1 added 1 commit 2024-04-14 10:29:51 +00:00
moritz reviewed 2024-04-16 12:02:13 +00:00
@ -0,0 +24,4 @@
### Output
A backup implementation SHOULD provide the backup of one or multiple stacks in a `.tar.gz` format. In that case each volume MUST be in `/var/lib/docker/volumes/{volume_name}`, where `{volume_name}` is the name of each volume that got backed up.
Member

I think {volume_name} can be confusing in this case. Because in backupbot.backup.volumes.{volume_name}.path the {volume_name} would be for example assets, while in the path /var/lib/docker/volumes/{volume_name} it would be appname_example_com_assets.
Maybe writing /var/lib/docker/volumes/{stack_name}_{volume_name} would clearer.

I think `{volume_name}` can be confusing in this case. Because in `backupbot.backup.volumes.{volume_name}.path` the `{volume_name}` would be for example `assets`, while in the path `/var/lib/docker/volumes/{volume_name}` it would be `appname_example_com_assets`. Maybe writing `/var/lib/docker/volumes/{stack_name}_{volume_name}` would clearer.
Member

Thank you @p4u1 great work 🚀

Thank you @p4u1 great work 🚀
p4u1 added 1 commit 2024-04-17 12:42:34 +00:00
p4u1 added 1 commit 2024-04-17 12:49:17 +00:00
p4u1 added 1 commit 2024-04-17 12:59:27 +00:00
p4u1 added 1 commit 2024-04-26 14:46:37 +00:00
This pull request is marked as a work in progress.
This branch is out-of-date with the base branch
You can also view command line instructions.

Step 1:

From your project repository, check out a new branch and test the changes.
git checkout -b p4u1-backupbottwo-spec main
git pull backupbottwo-spec

Step 2:

Merge the changes and update on Gitea.
git checkout main
git merge --no-ff p4u1-backupbottwo-spec
git push origin main
Sign in to join this conversation.
No reviewers
No Label
No Milestone
No Assignees
4 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: coop-cloud/docs.coopcloud.tech#258
No description provided.