WIP: Backupbot Specification #258
|
@ -0,0 +1,166 @@
|
|||
# For Maintainers
|
||||
|
||||
From the perspective of the recipe maintainer, backup/restore is just more
|
||||
`deploy: ...` labels. Tools can read these labels and then perform the
|
||||
backup/restore logic.
|
||||
|
||||
## Tools
|
||||
|
||||
Two of the current "blessed" options are, which both implement the [backupbot specification](link to spec)
|
||||
|
||||
- [`backup-bot-two`](https://git.coopcloud.tech/coop-cloud/backup-bot-two)
|
||||
- [`abra`](https://git.coopcloud.tech/coop-cloud/abra)
|
||||
|
||||
### `backup-bot-two`
|
||||
|
||||
`backup-bot-two` is a recipe which gets deployed on the server, it can perform automatic backups and uses restic.
|
||||
Please see the [`README.md`](https://git.coopcloud.tech/coop-cloud/backup-bot-two#backupbot-ii) for the full docs.
|
||||
|
||||
### `abra`
|
||||
|
||||
`abra` will read labels and store backups in `~/.abra/backups/...` .
|
||||
It also provides an integration for `backup-bot-two`.
|
||||
|
||||
## Backup
|
||||
|
||||
### How to Configure backups
|
||||
|
||||
Unless otherwise stated all labels should be added to the main service (which should be named `app`).
|
||||
|
||||
1. Enable backups for the recipe:
|
||||
You need to enable backups for the recipe by adding the following deploy label:
|
||||
|
||||
```
|
||||
backupbot.backup=true
|
||||
```
|
||||
|
||||
2. Decide wich volumes should be backed up:
|
||||
By default all volumes will be backed up. To disable a certain volume you can add the following deploy label:
|
||||
|
||||
```
|
||||
backupbot.backup.volumes.{volume_name}=false
|
||||
```
|
||||
|
||||
3. Decide which path should be backed up on each volume
|
||||
By default all files get backed up for a volume. To only include certain paths you can add the following deploy label:
|
||||
|
||||
```
|
||||
backupbot.backup.volumes.{volume_name}.path=/mypath1/foo,/mypath2/bar
|
||||
```
|
||||
|
||||
Note: You can include multiple paths by providing a comma seperated list
|
||||
Note: All paths are specified relativ to the volume root
|
||||
|
||||
4. Run commands before the backup
|
||||
For certain services like a database it is not reccomend to just backup files, because the backup might end up in a corrupted state. Instead it is reccomended to make a database dump. You can run arbitrary commands in any container before the files are backed up.
|
||||
To do this add the following deploy label to the service on which you want the command being run:
|
||||
|
||||
```
|
||||
backupbot.backup.pre-hook=mysqldump -u root -pghost ghost --tab /var/lib/foo
|
||||
```
|
||||
|
||||
5. Run commands after the backup
|
||||
Sometimes you want to clean up after the backup. You can run arbitrary commands in any container after the files were backed up.
|
||||
To do this add the following deploy label to the service on which you want the command being run:
|
||||
|
||||
```
|
||||
backupbot.backup.post-hook=rm -rf /var/lib/mysql-files/*
|
||||
```
|
||||
|
||||
### Testing the backup
|
||||
|
||||
To test that your backup is configured correctly you can deploy the recipe you are working on in a test app either [locally](link to local server deployment) or on a test server.
|
||||
|
||||
After the deployment is succesfull run the backup and inspect its content
|
||||
|
||||
```
|
||||
abra app backup myrecipe.example.com
|
||||
tar -tf ~/.abra/backups/mybackup
|
||||
```
|
||||
|
||||
TODO: this is not complete yet
|
||||
|
||||
## Restore
|
||||
|
||||
When restoring an app, it takes the files from a backup and copies them to their correct location.
|
||||
In the case of restoring database tables, you can use the `pre-hook` & `post-hook` commands to run the insertion logic.
|
||||
|
||||
## Pre and Post hooks
|
||||
|
||||
To back up some services correctly it involves more than just copying a few files from one location to another. Some services already have specific backup tools that allow taking a coherent snapshot of its data like `mysqldump`.
|
||||
The pre and post hooks can be used to prepare the files which should get backed up and clean up afterwards.
|
||||
|
||||
Here are some examples:
|
||||
|
||||
### Example 1: Execute simple command
|
||||
|
||||
```
|
||||
backupbot.backup.pre-hook: "echo 'foo' > /path/to/volume/bar.txt
|
||||
```
|
||||
|
||||
### Example 2: Access environment variable
|
||||
|
||||
```
|
||||
backupbot.backup.pre-hook: "cat $${POSTGRES_PASSWORD_FILE}"
|
||||
```
|
||||
|
||||
### Example 3: Access secret
|
||||
|
||||
```
|
||||
backupbot.backup.pre-hook: "cat /var/run/secrets/mysupersecret"
|
||||
```
|
||||
|
||||
```
|
||||
backupbot.backup.pre-hook: 'mysqldump -p"$$(cat /run/secrets/mysupersecret)" mydatabase'
|
||||
```
|
||||
|
||||
### Example 4: Complex script
|
||||
|
||||
Sometimes the logic to backup up a service can get quite complex. In that case it might be easier to add a script (via mount or config) inside the container and call that from the pre and post hook:
|
||||
|
||||
```
|
||||
backupbot.backup.pre-hook: "/scripts/my-pre-backup-scripts"
|
||||
backupbot.backup.post-hook: "/scripts/my-post-backup-scripts"
|
||||
```
|
||||
|
||||
## Configuration Examples
|
||||
|
||||
### Mariadb
|
||||
|
||||
```
|
||||
services:
|
||||
db:
|
||||
image: mariadb
|
||||
volumes:
|
||||
- "mariadb:/var/lib/mysql"
|
||||
deploy:
|
||||
labels:
|
||||
backupbot.backup: "true"
|
||||
backupbot.backup.pre-hook: "sh -c 'mariadb-dump --single-transaction -u root -p\"$$(cat /run/secrets/db_root_password)\" wordpress | gzip > /var/lib/mysql/dump.sql.gz'"
|
||||
backupbot.backup.volume.mariadb.path: "dump.sql.gz"
|
||||
backupbot.backup.post-hook: "rm -f /var/lib/mysql/dump.sql.gz"
|
||||
backupbot.restore.post-hook: "sh -c 'gzip -d /var/lib/mysql/dump.sql.gz && mariadb -u root -p\"$$(cat /run/secrets/db_root_password)\" wordpress < /var/lib/mysql/dump.sql && rm -f /var/lib/mysql/dump.sql'"
|
||||
```
|
||||
|
||||
### Postgres
|
||||
|
||||
```
|
||||
version: '3.8'
|
||||
|
||||
services:
|
||||
db:
|
||||
image: "postgres"
|
||||
volumes:
|
||||
- "postgres:/var/lib/postgresql/data"
|
||||
secrets:
|
||||
- db_password
|
||||
deploy:
|
||||
labels:
|
||||
backupbot.backup: "true"
|
||||
backupbot.backup.pre-hook: "PGPASSWORD=$$(cat $${POSTGRES_PASSWORD_FILE}) pg_dump -U $${POSTGRES_USER} $${POSTGRES_DB} > /var/lib/postgresql/data/backup.sql"
|
||||
backupbot.backup.post-hook: "rm -rf /var/lib/postgresql/data/backup.sql"
|
||||
backupbot.backup.volume.postgres.path: "backup.sql"
|
||||
|
||||
volumes:
|
||||
postgres:
|
||||
```
|
|
@ -0,0 +1,136 @@
|
|||
# Specification
|
||||
|
||||
## Summary
|
||||
|
||||
Creating automated backups of docker swarm services is an often needed task. This specification describes how backups can be configured via [service labels](https://docs.docker.com/compose/compose-file/compose-file-v3/#labels-1) in a standardised way.
|
||||
|
||||
## Requirements
|
||||
|
||||
The key words "MUST", "MUST NOT", "REQUIRED", "SHALL", "SHALL NOT", "SHOULD", "SHOULD NOT", "RECOMMENDED", "MAY", and "OPTIONAL" in this specification are to be interpreted as described in [RFC-2119](https://datatracker.ietf.org/doc/html/rfc2119).
|
||||
p4u1 marked this conversation as resolved
Outdated
|
||||
|
||||
## Backup
|
||||
p4u1 marked this conversation as resolved
3wordchant
commented
Maybe remove this, seems the same as line 15 "To enable backups" Maybe remove this, seems the same as line 15 "To enable backups"
|
||||
|
||||
To enable backups for a docker stack, the `backupbot.backup=true` label MUST be set on one of its services. The label MUST NOT be set multiple times for a docker stack. Otherwise the implementation MUST show an error. The label SHOULD be declared on the main service.
|
||||
|
||||
### Volumes and paths
|
||||
3wordchant
commented
Is the behaviour defined if one service sets Is the behaviour defined if one service sets `backupbot.backup=true` and another sets it to `backupbot.backup=false`?
|
||||
|
||||
By default all volumes MUST be backed up. A volume MUST be excluded from the backup when `backupbot.backup.volumes.{volume_name}=false` is set, where `{volume_name}` is the name of the volume.
|
||||
By default all files MUST be backed up on a volume. `backupbot.backup.volumes.{volume_name}.path` MAY be set to limit the paths for that volume. The value MUST be a valid path relative to the volume root. It MAY contain multiple paths which get separated by a comma. When the label is set only the given paths MUST be backed up.
|
||||
|
||||
3wordchant marked this conversation as resolved
3wordchant
commented
So the spec allows backing up a volume even if So the spec allows backing up a volume even if `...{volume_name}=false`? Is there any harm in changing this to "MUST"?
p4u1
commented
Changed it to "MUST" Changed it to "MUST"
|
||||
### Pre/Post Hooks
|
||||
|
||||
A `backupbot.backup.pre-hook` and `backupbot.backup.post-hook` MAY be set on a service. When set the command MUST be executed inside the running container of the service before/after backing up files.
|
||||
There is no guaranteed order in which different hooks MUST be executed.
|
||||
|
||||
TODO: escaping
|
||||
|
||||
### Output
|
||||
moritz
commented
I think I think `{volume_name}` can be confusing in this case. Because in `backupbot.backup.volumes.{volume_name}.path` the `{volume_name}` would be for example `assets`, while in the path `/var/lib/docker/volumes/{volume_name}` it would be `appname_example_com_assets`.
Maybe writing `/var/lib/docker/volumes/{stack_name}_{volume_name}` would clearer.
|
||||
|
||||
A backup implementation SHOULD provide the backup of one or multiple stacks in a `.tar.gz` format. In that case each volume MUST be in `/var/lib/docker/volumes/{stack_name}_{volume_name}`, where `{stack_name}` is the name of the docker stack and `{volume_name}` is the name of each volume that got backed up.
|
||||
|
||||
## Restore
|
||||
|
||||
By default all files MUST be restored into their volume. A volume or path MAY be excluded from restoring. When restoring a backup from a `.tar.gz` it expects the directory layout as described in the [backup output](#output) section.
|
||||
|
||||
### Pre/Post Hooks
|
||||
|
||||
A `backupbot.restore.pre-hook` and `backupbot.restore.post-hook` MAY be set on a service. When set the command MUST be executed inside the running container of the service before/after restoring the files.
|
||||
There is no guaranteed order in which different hooks MUST be executed.
|
||||
|
||||
## Labels
|
||||
|
||||
### `backupbot.backup`
|
||||
|
||||
**Type:** boolean
|
||||
**Default:** false
|
||||
**Description:**
|
||||
Enables backups for this compose stack. The label should be added to the main service of the compose stack.
|
||||
|
||||
decentral1se
commented
`%s/labe/label`
`%s/backupbot/backup-bot-two`
p4u1
commented
Did Did `%s/backupbot/backups` to keep it generic
|
||||
**Example:**
|
||||
|
||||
```
|
||||
backupbot.backup: true
|
||||
```
|
||||
|
||||
### `backupbot.backup.volumes.{volume_name}`
|
||||
|
||||
**Type:** boolean
|
||||
**Default:** true
|
||||
**Description:** When set to false the volume is excluded from backups.
|
||||
|
||||
**Example:**
|
||||
|
||||
```
|
||||
backupbot.backup.volumes.{volume_name}: false
|
||||
```
|
||||
|
||||
### `backupbot.backup.volumes.{volume_name}.path`
|
||||
|
||||
**Type:** string
|
||||
**Default:** ""
|
||||
**Description:**
|
||||
A comma seperated list of paths. When one or more paths are set, it only backs up those on the given volume instead of the whole volume.
|
||||
p4u1 marked this conversation as resolved
Outdated
3wordchant
commented
seperated -> separated backups up -> backs up seperated -> separated
backups up -> backs up
|
||||
|
||||
**Example 1:**
|
||||
|
||||
```
|
||||
backupbot.backup.volumes.{volume_name}.path: '/var/lib/mariadb/dump.sql.gz'
|
||||
```
|
||||
|
||||
**Example 2:**
|
||||
```
|
||||
backupbot.backup.volumes.{volume_name}.path: '/var/lib/myapp/foo,/var/lib/myapp/bar'
|
||||
```
|
||||
|
||||
### `backupbot.backup.pre-hook`
|
||||
|
||||
**Type:** string
|
||||
**Default:** ""
|
||||
**Description:**
|
||||
A command, that gets executed before the files are backed up.
|
||||
|
||||
**Example:**
|
||||
|
||||
```
|
||||
backupbot.backup.pre-hook: 'mysqldump -u root -p"$(cat /run/secrets/db_root_password)" -f /volume_path/dump.db'
|
||||
```
|
||||
|
||||
### `backupbot.backup.post-hook`
|
||||
p4u1 marked this conversation as resolved
3wordchant
commented
backuped -> backed up backuped -> backed up
|
||||
|
||||
**Type:** string
|
||||
**Default:** ""
|
||||
**Description:**
|
||||
A command, that gets executed after the files are backed up.
|
||||
|
||||
**Example:**
|
||||
|
||||
```
|
||||
backupbot.backup.post-hook: "rm -rf /volume_path/dump.db"
|
||||
```
|
||||
|
||||
### `backupbot.restore.pre-hook`
|
||||
|
||||
**Type:** string
|
||||
**Default:** ""
|
||||
**Description:**
|
||||
A command, that gets executed before the files are restored.
|
||||
Note, that there is no guaranteed order in which multiple hooks get executed.
|
||||
|
||||
**Example:**
|
||||
|
||||
```
|
||||
backupbot.restore.pre-hook: "lock db"
|
||||
```
|
||||
|
||||
### `backupbot.restore.post-hook`
|
||||
|
||||
**Type:** string
|
||||
**Default:** ""
|
||||
**Description:**
|
||||
A command, that gets executed after the files are restored.
|
||||
|
||||
**Example:**
|
||||
|
||||
```
|
||||
backupbot.restore.post-hook: "sqldump dump.sql && unlock db && rm dump.sql"
|
||||
```
|
[RFC-2119]
missing link?