coop-cloud/backup-bot-two
6
0
Fork 4
Code Issues 9 Pull Requests Packages Projects Releases Wiki Activity

Remove unnecessary files and configurations

Browse Source
This commit is contained in:
Christian Galo
2024-04-09 22:32:35 +00:00
parent 7865907811
commit 0588a06a97
15 changed files with 76 additions and 303 deletions
Download Patch File Download Diff File Expand all files Collapse all files

204
README.md
View File

@ -1,106 +1,27 @@
# Backupbot II
[![Build Status](https://build.coopcloud.tech/api/badges/coop-cloud/backup-bot-two/status.svg)](https://build.coopcloud.tech/coop-cloud/backup-bot-two)
_This Time, It's Easily Configurable_
Automatically take backups from all volumes of running Docker Swarm services and runs pre- and post commands.
<!-- metadata -->
* **Category**: Utilities
* **Status**: 0, work-in-progress
* **Image**: [`thecoopcloud/backup-bot-two`](https://hub.docker.com/r/thecoopcloud/backup-bot-two), 4, upstream
* **Healthcheck**: No
* **Backups**: N/A
* **Email**: N/A
* **Tests**: No
* **SSO**: N/A
<!-- endmetadata -->
Wiki Cafe's configuration for a Backupbot II deployment. Originally slimmed down from an `abra` [recipe](https://git.coopcloud.tech/coop-cloud/backup-bot-two) by [Co-op Cloud](https://coopcloud.tech/).
## Background
## Deploying the app with Docker Swarm
There are lots of Docker volume backup systems; all of them have one or both of these limitations:
- You need to define all the volumes to back up in the configuration system
- Backups require services to be stopped to take consistent copies
Set the environment variables from the .env file during the shell session.
Backupbot II tries to help, by
1. **letting you define backups using Docker labels**, so you can **easily collect your backups for use with another system** like docker-volume-backup.
2. **running pre- and post-commands** before and after backups, for example to use database tools to take a backup from a running service.
## Deployment
### With Co-op Cloud
* `abra app new backup-bot-two`
* `abra app config <app-name>`
- set storage options. Either configure `CRON_SCHEDULE`, or set up `swarm-cronjob`
* `abra app secret generate -a <backupbot_name>`
* `abra app deploy <app-name>`
## Configuration
Per default Backupbot stores the backups locally in the repository `/backups/restic`, which is accessible as volume at `/var/lib/docker/volumes/<backupbot_name>_backups/_data/restic/`
The backup location can be changed using the `RESTIC_REPOSITORY` env variable.
### S3 Storage
To use S3 storage as backup location set the following envs:
```
RESTIC_REPOSITORY=s3:<S3-SERVICE-URL>/<BUCKET-NAME>
SECRET_AWS_SECRET_ACCESS_KEY_VERSION=v1
AWS_ACCESS_KEY_ID=<MY_ACCESS_KEY>
COMPOSE_FILE="$COMPOSE_FILE:compose.s3.yml"
set -a && source .env && set +a
```
and add your `<SECRET_ACCESS_KEY>` as docker secret:
`abra app secret insert <backupbot_name> aws_secret_access_key v1 <SECRET_ACCESS_KEY>`
See [restic s3 docs](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#amazon-s3) for more information.
Set the secrets.
### SFTP Storage
> With sftp it is not possible to prevent the backupbot from deleting backups in case of a compromised machine. Therefore we recommend to use S3, REST or rclone server without delete permissions.
To use SFTP storage as backup location set the following envs:
```
RESTIC_REPOSITORY=sftp:user@host:/restic-repo-path
SECRET_SSH_KEY_VERSION=v1
SSH_HOST_KEY="hostname ssh-rsa AAAAB3...
COMPOSE_FILE="$COMPOSE_FILE:compose.ssh.yml"
printf "SECRET_HERE" | docker secret create SECRET_NAME -
```
To get the `SSH_HOST_KEY` run the following command `ssh-keyscan <hostname>`
Generate an ssh keypair: `ssh-keygen -t ed25519 -f backupkey -P ''`
Add the key to your `authorized_keys`:
`ssh-copy-id -i backupkey <user>@<hostname>`
Add your `SSH_KEY` as docker secret:
```
abra app secret insert <backupbot_name> ssh_key v1 """$(cat backupkey)
"""
```
> Attention: This command needs to be executed exactly as stated above, because it places a trailing newline at the end, if this is missing you will get the following error: `Load key "/run/secrets/ssh_key": error in libcrypto`
Deploy using the `-c` flag to specify one or multiple compose files.
### Restic REST server Storage
You can simply set the `RESTIC_REPOSITORY` variable to your REST server URL `rest:http://host:8000/`.
If you access the REST server with a password `rest:https://user:pass@host:8000/` you should hide the whole URL containing the password inside a secret.
Uncomment these lines:
```
SECRET_RESTIC_REPO_VERSION=v1
COMPOSE_FILE="$COMPOSE_FILE:compose.secret.yml"
docker stack deploy backup-bot-two -c compose.yaml
```
Add your REST server url as secret:
```
`abra app secret insert <backupbot_name> restic_repo v1 "rest:https://user:pass@host:8000/"`
```
The secret will overwrite the `RESTIC_REPOSITORY` variable.
See [restic REST docs](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#rest-server) for more information.
## Push notifications
@ -113,71 +34,82 @@ PUSH_URL_SUCCESS=https://status.example.com/api/push/xxxxxxxxxx?status=up&msg=OK
PUSH_URL_FAIL=https://status.example.com/api/push/xxxxxxxxxx?status=down&msg=fail
```
## Commands
## Usage
Run the cronjob that creates a backup, including the push notifications and docker logging:
`abra app cmd <backupbot_name> app run_cron`
- Find the ID or name of the backup container:
```
docker ps --filter "name=backup-bot-two_app"
```
Create a backup of all apps:
2. Run the desired command using `docker exec`:
```
docker exec -it <container_id_or_name> backup <command> [options]
```
Replace `<container_id_or_name>` with the ID or name of the backup container.
`abra app run <backupbot_name> app -- backup create`
Available commands:
- `create`: Initiate the backup process.
- `restore`: Restore a specific snapshot to a target directory.
- `snapshots`: List available snapshots.
- `ls`: List files in a specific snapshot.
- `download`: Download specific files, volumes, or secrets from a snapshot.
> The apps to backup up need to be deployed
Options:
- `--host`, `-h`: Specify the service name (e.g., `app`).
- `--repo`, `-r`: Specify the Restic repository location (e.g., `/run/secrets/restic_repo`).
- `--log`, `-l`: Set the log level (e.g., `debug`, `info`, `warning`, `error`).
- `--machine-logs`, `-m`: Enable machine-readable JSON logging.
Create an individual backup:
## Examples
`abra app run <backupbot_name> app -- backup --host <target_app_name> create`
Create a backup to a local repository:
`abra app run <backupbot_name> app -- backup create -r /backups/restic`
> It is recommended to shutdown/undeploy an app before restoring the data
Restore the latest snapshot of all including apps:
`abra app run <backupbot_name> app -- backup restore`
Restore a specific snapshot of an individual app:
`abra app run <backupbot_name> app -- backup --host <target_app_name> restore --snapshot <snapshot_id>`
Show all snapshots:
`abra app run <backupbot_name> app -- backup snapshots`
Show all snapshots containing a specific app:
`abra app run <backupbot_name> app -- backup --host <target_app_name> snapshots`
Show all files inside the latest snapshot (can be very verbose):
`abra app run <backupbot_name> app -- backup ls`
Show specific files inside a selected snapshot:
`abra app run <backupbot_name> app -- backup ls --snapshot <snapshot_id> --path /var/lib/docker/volumes/`
Download files from a snapshot:
Create a backup:
```
filename=$(abra app run <backupbot_name> app -- backup download --snapshot <snapshot_id> --path <absolute_path>)
abra app cp <backupbot_name> app:$filename .
docker exec -it <container_id_or_name> backup create --host app
```
## Run restic
Restore a snapshot:
```
abra app run <backupbot_name> app bash
export AWS_SECRET_ACCESS_KEY=$(cat $AWS_SECRET_ACCESS_KEY_FILE)
export RESTIC_PASSWORD=$(cat $RESTIC_PASSWORD_FILE)
restic snapshots
docker exec -it <container_id_or_name> backup restore --snapshot <snapshot_id> --target /path/to/restore
```
List snapshots:
```
docker exec -it <container_id_or_name> backup snapshots
```
List files in a snapshot:
```
docker exec -it <container_id_or_name> backup ls --snapshot <snapshot_id> --path /path/to/directory
```
Download files, volumes, or secrets from a snapshot:
```
docker exec -it <container_id_or_name> backup download --snapshot <snapshot_id> [--path /path/to/file] [--volumes] [--secrets]
```
Note: Make sure to replace `<container_id_or_name>` and `<snapshot_id>` with the appropriate values for your setup.
Remember to review and adjust the Docker Compose file and environment variables according to your specific requirements before running the backup commands.
When using `docker exec`, you don't need to specify the volume mounts or the Restic repository location as command-line arguments because they are already defined in the Docker Compose file and are available within the running container.
If you need to access the downloaded files, volumes, or secrets from the backup, you can use `docker cp` to copy them from the container to the host machine:
```
docker cp <container_id_or_name>:/path/to/backup/file /path/on/host
```
This allows you to retrieve the backed-up data from the container.
## Recipe Configuration
Like Traefik, or `swarm-cronjob`, Backupbot II uses access to the Docker socket to read labels from running Docker Swarm services:
Backupbot II uses access to the Docker socket to read labels from running Docker Swarm services:
```
services:
@ -194,5 +126,3 @@ services:
- `backupbot.backup.post-hook` -- command to run after copying files (optional)
As in the above example, you can reference Docker Secrets, e.g. for looking up database passwords, by reading the files in `/run/secrets` directly.
[abra]: https://git.autonomic.zone/autonomic-cooperative/abra