wiki-cafe
/
backup-bot-two
forked from coop-cloud/backup-bot-two
0
0
Fork 0
Code Pull Requests Activity

Remove unnecessary files and configurations

This commit is contained in:
Christian Galo 2024-04-09 22:32:35 +00:00
parent 7865907811
commit 0588a06a97
15 changed files with 76 additions and 303 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

12
.drone.yml
View File

@ -1,12 +0,0 @@
---
kind: pipeline
name: linters
steps:
- name: run shellcheck
image: koalaman/shellcheck-alpine
commands:
- shellcheck backup.sh
trigger:
branch:
- main

28
.env.sample
View File

@ -1,34 +1,10 @@
TYPE=backup-bot-two
SECRET_RESTIC_PASSWORD_VERSION=v1
COMPOSE_FILE=compose.yml
STACK_NAME=backup-bot-two
RESTIC_REPOSITORY=/backups/restic
CRON_SCHEDULE='30 3 * * *'
# Push Notifiactions
# Push Notifications
#PUSH_URL_START=https://status.example.com/api/push/xxxxxxxxxx?status=up&msg=start
#PUSH_URL_SUCCESS=https://status.example.com/api/push/xxxxxxxxxx?status=up&msg=OK
#PUSH_URL_FAIL=https://status.example.com/api/push/xxxxxxxxxx?status=down&msg=fail
# swarm-cronjob, instead of built-in cron
#COMPOSE_FILE="$COMPOSE_FILE:compose.swarm-cronjob.yml"
# SSH storage
#SECRET_SSH_KEY_VERSION=v1
#SSH_HOST_KEY="hostname ssh-rsa AAAAB3...
#COMPOSE_FILE="$COMPOSE_FILE:compose.ssh.yml"
# S3 storage
#SECRET_AWS_SECRET_ACCESS_KEY_VERSION=v1
#AWS_ACCESS_KEY_ID=something-secret
#COMPOSE_FILE="$COMPOSE_FILE:compose.s3.yml"
# Secret restic repository
# use a secret to store the RESTIC_REPOSITORY if the repository location contains a secret value
# i.E rest:https://user:SECRET_PASSWORD@host:8000/
# it overwrites the RESTIC_REPOSITORY variable
#SECRET_RESTIC_REPO_VERSION=v1
#COMPOSE_FILE="$COMPOSE_FILE:compose.secret.yml"

17
.envrc.sample
View File

@ -1,17 +0,0 @@
export RESTIC_HOST="user@domain.tld"
export RESTIC_PASSWORD_FILE=/run/secrets/restic-password
export BACKUP_DEST=/backups
export SERVER_NAME=domain.tld
export DOCKER_CONTEXT=$SERVER_NAME
# uncomment either this:
#export SSH_KEY_FILE=~/.ssh/id_rsa
# or this:
#export AWS_SECRET_ACCESS_KEY_FILE=s3
#export AWS_ACCESS_KEY_ID=easter-october-emphatic-tug-urgent-customer
# or this:
#export HTTPS_PASSWORD_FILE=/run/secrets/https_password
# optionally limit subset of services for testing
#export SERVICES_OVERRIDE="ghost_domain_tld_app ghost_domain_tld_db"

2
.gitignore vendored
View File

@ -1 +1 @@
/testing
.env

204
README.md
View File

@ -1,106 +1,27 @@
# Backupbot II
[![Build Status](https://build.coopcloud.tech/api/badges/coop-cloud/backup-bot-two/status.svg)](https://build.coopcloud.tech/coop-cloud/backup-bot-two)
_This Time, It's Easily Configurable_
Automatically take backups from all volumes of running Docker Swarm services and runs pre- and post commands.
<!-- metadata -->
* **Category**: Utilities
* **Status**: 0, work-in-progress
* **Image**: [`thecoopcloud/backup-bot-two`](https://hub.docker.com/r/thecoopcloud/backup-bot-two), 4, upstream
* **Healthcheck**: No
* **Backups**: N/A
* **Email**: N/A
* **Tests**: No
* **SSO**: N/A
<!-- endmetadata -->
Wiki Cafe's configuration for a Backupbot II deployment. Originally slimmed down from an `abra` [recipe](https://git.coopcloud.tech/coop-cloud/backup-bot-two) by [Co-op Cloud](https://coopcloud.tech/).
## Background
## Deploying the app with Docker Swarm
There are lots of Docker volume backup systems; all of them have one or both of these limitations:
- You need to define all the volumes to back up in the configuration system
- Backups require services to be stopped to take consistent copies
Set the environment variables from the .env file during the shell session.
Backupbot II tries to help, by
1. **letting you define backups using Docker labels**, so you can **easily collect your backups for use with another system** like docker-volume-backup.
2. **running pre- and post-commands** before and after backups, for example to use database tools to take a backup from a running service.
## Deployment
### With Co-op Cloud
* `abra app new backup-bot-two`
* `abra app config <app-name>`
- set storage options. Either configure `CRON_SCHEDULE`, or set up `swarm-cronjob`
* `abra app secret generate -a <backupbot_name>`
* `abra app deploy <app-name>`
## Configuration
Per default Backupbot stores the backups locally in the repository `/backups/restic`, which is accessible as volume at `/var/lib/docker/volumes/<backupbot_name>_backups/_data/restic/`
The backup location can be changed using the `RESTIC_REPOSITORY` env variable.
### S3 Storage
To use S3 storage as backup location set the following envs:
```
RESTIC_REPOSITORY=s3:<S3-SERVICE-URL>/<BUCKET-NAME>
SECRET_AWS_SECRET_ACCESS_KEY_VERSION=v1
AWS_ACCESS_KEY_ID=<MY_ACCESS_KEY>
COMPOSE_FILE="$COMPOSE_FILE:compose.s3.yml"
set -a && source .env && set +a
```
and add your `<SECRET_ACCESS_KEY>` as docker secret:
`abra app secret insert <backupbot_name> aws_secret_access_key v1 <SECRET_ACCESS_KEY>`
See [restic s3 docs](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#amazon-s3) for more information.
Set the secrets.
### SFTP Storage
> With sftp it is not possible to prevent the backupbot from deleting backups in case of a compromised machine. Therefore we recommend to use S3, REST or rclone server without delete permissions.
To use SFTP storage as backup location set the following envs:
```
RESTIC_REPOSITORY=sftp:user@host:/restic-repo-path
SECRET_SSH_KEY_VERSION=v1
SSH_HOST_KEY="hostname ssh-rsa AAAAB3...
COMPOSE_FILE="$COMPOSE_FILE:compose.ssh.yml"
printf "SECRET_HERE" | docker secret create SECRET_NAME -
```
To get the `SSH_HOST_KEY` run the following command `ssh-keyscan <hostname>`
Generate an ssh keypair: `ssh-keygen -t ed25519 -f backupkey -P ''`
Add the key to your `authorized_keys`:
`ssh-copy-id -i backupkey <user>@<hostname>`
Add your `SSH_KEY` as docker secret:
```
abra app secret insert <backupbot_name> ssh_key v1 """$(cat backupkey)
"""
```
> Attention: This command needs to be executed exactly as stated above, because it places a trailing newline at the end, if this is missing you will get the following error: `Load key "/run/secrets/ssh_key": error in libcrypto`
Deploy using the `-c` flag to specify one or multiple compose files.
### Restic REST server Storage
You can simply set the `RESTIC_REPOSITORY` variable to your REST server URL `rest:http://host:8000/`.
If you access the REST server with a password `rest:https://user:pass@host:8000/` you should hide the whole URL containing the password inside a secret.
Uncomment these lines:
```
SECRET_RESTIC_REPO_VERSION=v1
COMPOSE_FILE="$COMPOSE_FILE:compose.secret.yml"
docker stack deploy backup-bot-two -c compose.yaml
```
Add your REST server url as secret:
```
`abra app secret insert <backupbot_name> restic_repo v1 "rest:https://user:pass@host:8000/"`
```
The secret will overwrite the `RESTIC_REPOSITORY` variable.
See [restic REST docs](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#rest-server) for more information.
## Push notifications
@ -113,71 +34,82 @@ PUSH_URL_SUCCESS=https://status.example.com/api/push/xxxxxxxxxx?status=up&msg=OK
PUSH_URL_FAIL=https://status.example.com/api/push/xxxxxxxxxx?status=down&msg=fail
```
## Commands
## Usage
Run the cronjob that creates a backup, including the push notifications and docker logging:
`abra app cmd <backupbot_name> app run_cron`
- Find the ID or name of the backup container:
```
docker ps --filter "name=backup-bot-two_app"
```
Create a backup of all apps:
2. Run the desired command using `docker exec`:
```
docker exec -it <container_id_or_name> backup <command> [options]
```
Replace `<container_id_or_name>` with the ID or name of the backup container.
`abra app run <backupbot_name> app -- backup create`
Available commands:
- `create`: Initiate the backup process.
- `restore`: Restore a specific snapshot to a target directory.
- `snapshots`: List available snapshots.
- `ls`: List files in a specific snapshot.
- `download`: Download specific files, volumes, or secrets from a snapshot.
> The apps to backup up need to be deployed
Options:
- `--host`, `-h`: Specify the service name (e.g., `app`).
- `--repo`, `-r`: Specify the Restic repository location (e.g., `/run/secrets/restic_repo`).
- `--log`, `-l`: Set the log level (e.g., `debug`, `info`, `warning`, `error`).
- `--machine-logs`, `-m`: Enable machine-readable JSON logging.
Create an individual backup:
## Examples
`abra app run <backupbot_name> app -- backup --host <target_app_name> create`
Create a backup to a local repository:
`abra app run <backupbot_name> app -- backup create -r /backups/restic`
> It is recommended to shutdown/undeploy an app before restoring the data
Restore the latest snapshot of all including apps:
`abra app run <backupbot_name> app -- backup restore`
Restore a specific snapshot of an individual app:
`abra app run <backupbot_name> app -- backup --host <target_app_name> restore --snapshot <snapshot_id>`
Show all snapshots:
`abra app run <backupbot_name> app -- backup snapshots`
Show all snapshots containing a specific app:
`abra app run <backupbot_name> app -- backup --host <target_app_name> snapshots`
Show all files inside the latest snapshot (can be very verbose):
`abra app run <backupbot_name> app -- backup ls`
Show specific files inside a selected snapshot:
`abra app run <backupbot_name> app -- backup ls --snapshot <snapshot_id> --path /var/lib/docker/volumes/`
Download files from a snapshot:
Create a backup:
```
filename=$(abra app run <backupbot_name> app -- backup download --snapshot <snapshot_id> --path <absolute_path>)
abra app cp <backupbot_name> app:$filename .
docker exec -it <container_id_or_name> backup create --host app
```
## Run restic
Restore a snapshot:
```
abra app run <backupbot_name> app bash
export AWS_SECRET_ACCESS_KEY=$(cat $AWS_SECRET_ACCESS_KEY_FILE)
export RESTIC_PASSWORD=$(cat $RESTIC_PASSWORD_FILE)
restic snapshots
docker exec -it <container_id_or_name> backup restore --snapshot <snapshot_id> --target /path/to/restore
```
List snapshots:
```
docker exec -it <container_id_or_name> backup snapshots
```
List files in a snapshot:
```
docker exec -it <container_id_or_name> backup ls --snapshot <snapshot_id> --path /path/to/directory
```
Download files, volumes, or secrets from a snapshot:
```
docker exec -it <container_id_or_name> backup download --snapshot <snapshot_id> [--path /path/to/file] [--volumes] [--secrets]
```
Note: Make sure to replace `<container_id_or_name>` and `<snapshot_id>` with the appropriate values for your setup.
Remember to review and adjust the Docker Compose file and environment variables according to your specific requirements before running the backup commands.
When using `docker exec`, you don't need to specify the volume mounts or the Restic repository location as command-line arguments because they are already defined in the Docker Compose file and are available within the running container.
If you need to access the downloaded files, volumes, or secrets from the backup, you can use `docker cp` to copy them from the container to the host machine:
```
docker cp <container_id_or_name>:/path/to/backup/file /path/on/host
```
This allows you to retrieve the backed-up data from the container.
## Recipe Configuration
Like Traefik, or `swarm-cronjob`, Backupbot II uses access to the Docker socket to read labels from running Docker Swarm services:
Backupbot II uses access to the Docker socket to read labels from running Docker Swarm services:
```
services:
@ -194,5 +126,3 @@ services:
- `backupbot.backup.post-hook` -- command to run after copying files (optional)
As in the above example, you can reference Docker Secrets, e.g. for looking up database passwords, by reading the files in `/run/secrets` directly.
[abra]: https://git.autonomic.zone/autonomic-cooperative/abra

11
abra.sh
View File

@ -1,11 +0,0 @@
export ENTRYPOINT_VERSION=v1
export BACKUPBOT_VERSION=v1
export SSH_CONFIG_VERSION=v1
run_cron () {
schedule="$(crontab -l | tr -s " " | cut -d ' ' -f-5)"
rm -f /tmp/backup.log
echo "* * * * * $(crontab -l | tr -s " " | cut -d ' ' -f6-)" | crontab -
while [ ! -f /tmp/backup.log ]; do sleep 1; done
echo "$schedule $(crontab -l | tr -s " " | cut -d ' ' -f6-)" | crontab -
}

14
compose.s3.yml
View File

@ -1,14 +0,0 @@
---
version: "3.8"
services:
app:
environment:
- AWS_ACCESS_KEY_ID
- AWS_SECRET_ACCESS_KEY_FILE=/run/secrets/aws_secret_access_key
secrets:
- aws_secret_access_key
secrets:
aws_secret_access_key:
external: true
name: ${STACK_NAME}_aws_secret_access_key_${SECRET_AWS_SECRET_ACCESS_KEY_VERSION}

13
compose.secret.yml
View File

@ -1,13 +0,0 @@
---
version: "3.8"
services:
app:
environment:
- RESTIC_REPOSITORY_FILE=/run/secrets/restic_repo
secrets:
- restic_repo
secrets:
restic_repo:
external: true
name: ${STACK_NAME}_restic_repo_${SECRET_RESTIC_REPO_VERSION}

23
compose.ssh.yml
View File

@ -1,23 +0,0 @@
---
version: "3.8"
services:
app:
environment:
- SSH_KEY_FILE=/run/secrets/ssh_key
- SSH_HOST_KEY
secrets:
- source: ssh_key
mode: 0400
configs:
- source: ssh_config
target: /root/.ssh/config
secrets:
ssh_key:
external: true
name: ${STACK_NAME}_ssh_key_${SECRET_SSH_KEY_VERSION}
configs:
ssh_config:
name: ${STACK_NAME}_ssh_config_${SSH_CONFIG_VERSION}
file: ssh_config

15
compose.swarm-cronjob.yml
View File

@ -1,15 +0,0 @@
---
version: "3.8"
services:
app:
deploy:
mode: replicated
replicas: 0
labels:
- "swarm.cronjob.enable=true"
# Note(3wc): every 5m, testing
- "swarm.cronjob.schedule=*/5 * * * *"
# Note(3wc): blank label to be picked up by `abra recipe sync`
restart_policy:
condition: none
entrypoint: [ "/usr/bin/backup.sh" ]

25
compose.yml → compose.yaml
View File

@ -1,5 +1,3 @@
---
version: "3.8"
services:
app:
image: docker:24.0.7-dind
@ -7,18 +5,11 @@ services:
- "/var/run/docker.sock:/var/run/docker.sock"
- "/var/lib/docker/volumes/:/var/lib/docker/volumes/"
- "/var/lib/docker/containers/:/var/lib/docker/containers/:ro"
- backups:/backups
environment:
- CRON_SCHEDULE
- RESTIC_REPOSITORY
- RESTIC_PASSWORD_FILE=/run/secrets/restic_password
- RESTIC_REPOSITORY_FILE=/run/secrets/restic_repo
secrets:
- restic_password
deploy:
labels:
- coop-cloud.${STACK_NAME}.version=0.1.0+latest
- coop-cloud.${STACK_NAME}.timeout=${TIMEOUT:-300}
- coop-cloud.backupbot.enabled=true
- restic_repo
configs:
- source: entrypoint
target: /entrypoint.sh
@ -27,7 +18,6 @@ services:
target: /usr/bin/backup
mode: 0555
entrypoint: ['/entrypoint.sh']
#entrypoint: ['tail', '-f','/dev/null']
healthcheck:
test: "pgrep crond"
interval: 30s
@ -36,17 +26,14 @@ services:
start_period: 5m
secrets:
restic_password:
restic_repo:
external: true
name: ${STACK_NAME}_restic_password_${SECRET_RESTIC_PASSWORD_VERSION}
volumes:
backups:
name: ${STACK_NAME}_restic_repo
configs:
entrypoint:
name: ${STACK_NAME}_entrypoint_${ENTRYPOINT_VERSION}
name: ${STACK_NAME}_entrypoint
file: entrypoint.sh
backupbot:
name: ${STACK_NAME}_backupbot_${BACKUPBOT_VERSION}
name: ${STACK_NAME}_backupbot
file: backupbot.py

5
entrypoint.sh
View File

@ -7,11 +7,6 @@ apk add --upgrade --no-cache restic bash python3 py3-pip py3-click py3-docker-py
# Todo use requirements file with specific versions
pip install --break-system-packages resticpy==1.0.2
if [ -n "$SSH_HOST_KEY" ]
then
echo "$SSH_HOST_KEY" > /root/.ssh/known_hosts
fi
cron_schedule="${CRON_SCHEDULE:?CRON_SCHEDULE not set}"
if [ -n "$PUSH_URL_START" ]

3
release/1.0.0+latest
View File

@ -1,3 +0,0 @@
Breaking Change: the variables `SERVER_NAME` and `RESTIC_HOST` are merged into `RESTIC_REPOSITORY`. The format can be looked up here: https://restic.readthedocs.io/en/stable/030_preparing_a_new_repo.html
ssh/sftp: `sftp:user@host:/repo-path`
S3: `s3:https://s3.example.com/bucket_name`

3
renovate.json
View File

@ -1,3 +0,0 @@
{
"$schema": "https://docs.renovatebot.com/renovate-schema.json"
}

4
ssh_config
View File

@ -1,4 +0,0 @@
Host *
IdentityFile /run/secrets/ssh_key
ServerAliveInterval 60
ServerAliveCountMax 240