Update backupbot.py and compose.yaml

Reviewed-on: #40

.drone.yml

@@ -1,12 +0,0 @@
----
-kind: pipeline
-name: linters
-steps:
-  - name: run shellcheck
-    image: koalaman/shellcheck-alpine
-    commands:
-      - shellcheck backup.sh
-
-trigger:
-  branch:
-    - main

.env.sample

@@ -1,29 +1,10 @@
-TYPE=backup-bot-two
+STACK_NAME=backup-bot-two
 
-SECRET_RESTIC_PASSWORD_VERSION=v1
+RESTIC_REPOSITORY=/backups/restic
-
-COMPOSE_FILE=compose.yml
-
-RESTIC_REPO=/backups/restic
 
 CRON_SCHEDULE='30 3 * * *'
 
-# swarm-cronjob, instead of built-in cron
+# Push Notifications
-#COMPOSE_FILE="$COMPOSE_FILE:compose.swarm-cronjob.yml"
+#PUSH_URL_START=https://status.example.com/api/push/xxxxxxxxxx?status=up&msg=start
-
+#PUSH_URL_SUCCESS=https://status.example.com/api/push/xxxxxxxxxx?status=up&msg=OK
-# SSH storage
+#PUSH_URL_FAIL=https://status.example.com/api/push/xxxxxxxxxx?status=down&msg=fail
-#SECRET_SSH_KEY_VERSION=v1
-#SSH_HOST_KEY="hostname ssh-rsa AAAAB3...
-#COMPOSE_FILE="$COMPOSE_FILE:compose.ssh.yml"
-
-# S3 storage
-#SECRET_AWS_SECRET_ACCESS_KEY_VERSION=v1
-#AWS_ACCESS_KEY_ID=something-secret
-#COMPOSE_FILE="$COMPOSE_FILE:compose.s3.yml"
-
-# Secret restic repository
-# use a secret to store the RESTIC_REPO if the repository location contains a secret value
-# i.E rest:https://user:SECRET_PASSWORD@host:8000/
-# it overwrites the RESTIC_REPO variable
-#SECRET_RESTIC_REPO_VERSION=v1
-#COMPOSE_FILE="$COMPOSE_FILE:compose.secret.yml"

.envrc.sample

@@ -1,17 +0,0 @@
-export RESTIC_HOST="user@domain.tld"
-export RESTIC_PASSWORD_FILE=/run/secrets/restic-password
-export BACKUP_DEST=/backups
-
-export SERVER_NAME=domain.tld
-export DOCKER_CONTEXT=$SERVER_NAME
-
-# uncomment either this:
-#export SSH_KEY_FILE=~/.ssh/id_rsa
-# or this:
-#export AWS_SECRET_ACCESS_KEY_FILE=s3
-#export AWS_ACCESS_KEY_ID=easter-october-emphatic-tug-urgent-customer
-# or this:
-#export HTTPS_PASSWORD_FILE=/run/secrets/https_password
-
-# optionally limit subset of services for testing
-#export SERVICES_OVERRIDE="ghost_domain_tld_app ghost_domain_tld_db"

.gitignore

@@ -1 +1 @@
-/testing
+.env

README.md

@@ -1,160 +1,115 @@
 # Backupbot II
 
-[![Build Status](https://build.coopcloud.tech/api/badges/coop-cloud/backup-bot-two/status.svg)](https://build.coopcloud.tech/coop-cloud/backup-bot-two)
+Wiki Cafe's configuration for a Backupbot II deployment. Originally slimmed down from an `abra` [recipe](https://git.coopcloud.tech/coop-cloud/backup-bot-two) by [Co-op Cloud](https://coopcloud.tech/).
-
-_This Time, It's Easily Configurable_
-
-Automatically take backups from all volumes of running Docker Swarm services and runs pre- and post commands.
-
-<!-- metadata -->
-
-* **Category**: Utilities
-* **Status**: 0, work-in-progress
-* **Image**: [`thecoopcloud/backup-bot-two`](https://hub.docker.com/r/thecoopcloud/backup-bot-two), 4, upstream
-* **Healthcheck**: No
-* **Backups**: N/A
-* **Email**: N/A
-* **Tests**: No
-* **SSO**: N/A
-
-<!-- endmetadata -->
 
 
-## Background
+## Deploying the app with Docker Swarm
 
-There are lots of Docker volume backup systems; all of them have one or both of these limitations:
+Set the environment variables from the .env file during the shell session.
- - You need to define all the volumes to back up in the configuration system
- - Backups require services to be stopped to take consistent copies
-
-Backupbot II tries to help, by
-1. **letting you define backups using Docker labels**, so you can **easily collect your backups for use with another system** like docker-volume-backup.
-2. **running pre- and post-commands** before and after backups, for example to use database tools to take a backup from a running service.
-
-## Deployment
-
-### With Co-op Cloud
-
-
-* `abra app new backup-bot-two`
-* `abra app config <app-name>`
-    - set storage options. Either configure `CRON_SCHEDULE`, or set up `swarm-cronjob`
-* `abra app secret generate -a <app_name>`
-* `abra app deploy <app-name>`
-
-## Configuration
-
-Per default Backupbot stores the backups locally in the repository `/backups/restic`, which is accessible as volume at `/var/lib/docker/volumes/<app_name>_backups/_data/restic/`
-
-The backup location can be changed using the `RESTIC_REPO` env variable.
-
-### S3 Storage
-
-To use S3 storage as backup location set the following envs:
-```
-RESTIC_REPO=s3:<S3-SERVICE-URL>/<BUCKET-NAME>
-SECRET_AWS_SECRET_ACCESS_KEY_VERSION=v1
-AWS_ACCESS_KEY_ID=<MY_ACCESS_KEY>
-COMPOSE_FILE="$COMPOSE_FILE:compose.s3.yml"
-```
-and add your `<SECRET_ACCESS_KEY>` as docker secret:
-`abra app secret insert <app_name> aws_secret_access_key v1 <SECRET_ACCESS_KEY>`
-
-See [restic s3 docs](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#amazon-s3) for more information.
-
-### SFTP Storage
-
-> With sftp it is not possible to prevent the backupbot from deleting backups in case of a compromised machine. Therefore we recommend to use S3, REST or rclone server without delete permissions.
-
-To use SFTP storage as backup location set the following envs:
-```
-RESTIC_REPO=sftp:user@host:/restic-repo-path
-SECRET_SSH_KEY_VERSION=v1
-SSH_HOST_KEY="hostname ssh-rsa AAAAB3...
-COMPOSE_FILE="$COMPOSE_FILE:compose.ssh.yml"
-```
-To get the `SSH_HOST_KEY` run the following command `ssh-keyscan <hostname>`
-
-Generate an ssh keypair: `ssh-keygen -t ed25519 -f backupkey -P ''`
-Add the key to your `authorized_keys`:
-`ssh-copy-id -i backupkey <user>@<hostname>`
-Add your `SSH_KEY` as docker secret:
-```
-abra app secret insert <app_name> ssh_key v1 """$(cat backupkey)
-"""
-```
-
-### Restic REST server Storage
-
-You can simply set the `RESTIC_REPO` variable to your REST server URL `rest:http://host:8000/`.
-If you access the REST server with a password `rest:https://user:pass@host:8000/` you should hide the whole URL containing the password inside a secret.
-Uncomment these lines:
-```
-SECRET_RESTIC_REPO_VERSION=v1
-COMPOSE_FILE="$COMPOSE_FILE:compose.secret.yml"
-```
-Add your REST server url as secret:
-```
-`abra app secret insert <app_name> restic_repo v1 "rest:https://user:pass@host:8000/"`
-```
-The secret will overwrite the `RESTIC_REPO` variable.
-
-
-See [restic REST docs](https://restic.readthedocs.io/en/latest/030_preparing_a_new_repo.html#rest-server) for more information.
-
-## Usage
-
-
-Create a backup of all apps:
-
-`abra app run <app_name> app -- backup create`
-
-> The apps to backup up need to be deployed
-
-Create an individual backup:
-
-`abra app run <app_name> app -- backup --host <target_app_name> create`
-
-Create a backup to a local repository:
-
-`abra app run <app_name> app -- backup create -r /backups/restic`
-
-> It is recommended to shutdown/undeploy an app before restoring the data
-
-Restore the latest snapshot of all including apps:
-
-`abra app run <app_name> app -- backup restore`
-
-Restore a specific snapshot of an individual app:
-
-`abra app run <app_name> app -- backup --host <target_app_name> restore --snapshot <snapshot_id>`
-
-Show all snapshots:
-
-`abra app run <app_name> app -- backup snapshots`
-
-Show all snapshots containing a specific app:
-
-`abra app run <app_name> app -- backup --host <target_app_name> snapshots`
-
-Show all files inside the latest snapshot (can be very verbose):
-
-`abra app run <app_name> app -- backup ls`
-
-Show specific files inside a selected snapshot:
-
-`abra app run <app_name> app -- backup ls --snapshot <snapshot_id> --path /var/lib/docker/volumes/`
-
-Download files from a snapshot:
 
 ```
-filename=$(abra app run <app_name> app -- backup download --snapshot <snapshot_id> --path <absolute_path>)
+set -a && source .env && set +a
-abra app cp <app_name> app:$filename .
 ```
 
+Set the secrets.
+
+```
+printf "SECRET_HERE" | docker secret create SECRET_NAME -
+```
+
+Deploy using the `-c` flag to specify one or multiple compose files.
+
+```
+docker stack deploy backup-bot-two -c compose.yaml
+```
+
+## Push notifications
+
+The following env variables can be used to setup push notifications for backups. `PUSH_URL_START` is requested just before the backups starts, `PUSH_URL_SUCCESS` is only requested if the backup was successful and if the backup fails `PUSH_URL_FAIL` will be requested.
+Each variable is optional and independent of the other.
+```
+
+PUSH_URL_START=https://status.example.com/api/push/xxxxxxxxxx?status=up&msg=start
+PUSH_URL_SUCCESS=https://status.example.com/api/push/xxxxxxxxxx?status=up&msg=OK
+PUSH_URL_FAIL=https://status.example.com/api/push/xxxxxxxxxx?status=down&msg=fail
+```
+
+## Commands
+
+
+- Find the ID or name of the backup container:
+   ```
+   docker ps --filter "name=backup-bot-two_app"
+   ```
+
+2. Run the desired command using `docker exec`:
+   ```
+   docker exec -it <container_id_or_name> backup <command> [options]
+   ```
+   Replace `<container_id_or_name>` with the ID or name of the backup container.
+
+   Available commands:
+   - `create`: Initiate the backup process.
+   - `restore`: Restore a specific snapshot to a target directory.
+   - `snapshots`: List available snapshots.
+   - `ls`: List files in a specific snapshot.
+   - `download`: Download specific files, volumes, or secrets from a snapshot.
+
+   Options:
+   - `--host`, `-h`: Specify the service name (e.g., `app`).
+   - `--repo`, `-r`: Specify the Restic repository location (e.g., `/run/secrets/restic_repo`).
+   - `--log`, `-l`: Set the log level (e.g., `debug`, `info`, `warning`, `error`).
+   - `--machine-logs`, `-m`: Enable machine-readable JSON logging.
+
+## Examples
+
+Create a backup:
+
+```
+docker exec -it <container_id_or_name> backup create --host app
+```
+
+Restore a snapshot:
+
+```
+docker exec -it <container_id_or_name> backup restore --snapshot <snapshot_id> --target /path/to/restore
+```
+
+List snapshots:
+
+  ```
+  docker exec -it <container_id_or_name> backup snapshots
+  ```
+
+List files in a snapshot:
+
+```
+docker exec -it <container_id_or_name> backup ls --snapshot <snapshot_id> --path /path/to/directory
+```
+
+Download files, volumes, or secrets from a snapshot:
+
+```
+docker exec -it <container_id_or_name> backup download --snapshot <snapshot_id> [--path /path/to/file] [--volumes] [--secrets]
+```
+
+Note: Make sure to replace `<container_id_or_name>` and `<snapshot_id>` with the appropriate values for your setup.
+
+Remember to review and adjust the Docker Compose file and environment variables according to your specific requirements before running the backup commands.
+
+When using `docker exec`, you don't need to specify the volume mounts or the Restic repository location as command-line arguments because they are already defined in the Docker Compose file and are available within the running container.
+
+If you need to access the downloaded files, volumes, or secrets from the backup, you can use `docker cp` to copy them from the container to the host machine:
+
+```
+docker cp <container_id_or_name>:/path/to/backup/file /path/on/host
+```
+
+This allows you to retrieve the backed-up data from the container.
 
 ## Recipe Configuration
 
-Like Traefik, or `swarm-cronjob`, Backupbot II uses access to the Docker socket to read labels from running Docker Swarm services:
+Backupbot II uses access to the Docker socket to read labels from running Docker Swarm services:
 
 ```
 services:
@@ -171,5 +126,3 @@ services:
 - `backupbot.backup.post-hook` -- command to run after copying files (optional)
 
 As in the above example, you can reference Docker Secrets, e.g. for looking up database passwords, by reading the files in `/run/secrets` directly.
-
-[abra]: https://git.autonomic.zone/autonomic-cooperative/abra

abra.sh

@@ -1,3 +0,0 @@
-export ENTRYPOINT_VERSION=v1
-export BACKUPBOT_VERSION=v1
-export SSH_CONFIG_VERSION=v1

backupbot.py

@@ -1,45 +1,73 @@
 #!/usr/bin/python3
 
 import os
+import sys
 import click
 import json
 import subprocess
 import logging
 import docker
 import restic
+import tarfile
+import io
+from pythonjsonlogger import jsonlogger
 from datetime import datetime, timezone
 from restic.errors import ResticFailedError
 from pathlib import Path
 from shutil import copyfile, rmtree
-# logging.basicConfig(level=logging.INFO)
 
 VOLUME_PATH = "/var/lib/docker/volumes/"
 SECRET_PATH = '/secrets/'
 SERVICE = None
 
+logger = logging.getLogger("backupbot")
+logging.addLevelName(55, 'SUMMARY')
+setattr(logging, 'SUMMARY', 55)
+setattr(logger, 'summary', lambda message, *args, **
+        kwargs: logger.log(55, message, *args, **kwargs))
+
+
+def handle_exception(exc_type, exc_value, exc_traceback):
+    if issubclass(exc_type, KeyboardInterrupt):
+        sys.__excepthook__(exc_type, exc_value, exc_traceback)
+        return
+    logger.critical("Uncaught exception", exc_info=(
+        exc_type, exc_value, exc_traceback))
+
+
+sys.excepthook = handle_exception
+
 
 @click.group()
 @click.option('-l', '--log', 'loglevel')
+@click.option('-m', '--machine-logs', 'machine_logs', is_flag=True)
 @click.option('service', '--host', '-h', envvar='SERVICE')
-@click.option('repository', '--repo', '-r', envvar='RESTIC_REPO', required=True)
+@click.option('repository', '--repo', '-r', envvar='RESTIC_REPOSITORY')
-def cli(loglevel, service, repository):
+def cli(loglevel, service, repository, machine_logs):
     global SERVICE
     if service:
         SERVICE = service.replace('.', '_')
     if repository:
-        os.environ['RESTIC_REPO'] = repository
+        os.environ['RESTIC_REPOSITORY'] = repository
     if loglevel:
         numeric_level = getattr(logging, loglevel.upper(), None)
         if not isinstance(numeric_level, int):
             raise ValueError('Invalid log level: %s' % loglevel)
-        logging.basicConfig(level=numeric_level)
+        logger.setLevel(numeric_level)
+    if machine_logs:
+        logHandler = logging.StreamHandler()
+        formatter = jsonlogger.JsonFormatter(
+            "%(levelname)s %(filename)s %(lineno)s %(process)d %(message)s", rename_fields={"levelname": "message_type"})
+        logHandler.setFormatter(formatter)
+        logger.addHandler(logHandler)
+
     export_secrets()
     init_repo()
 
 
 def init_repo():
-    repo = os.environ['RESTIC_REPO']
+    repo = os.environ['RESTIC_REPOSITORY']
-    logging.debug(f"set restic repository location: {repo}")
+    logger.debug(f"set restic repository location: {repo}")
     restic.repository = repo
     restic.password_file = '/var/run/secrets/restic_password'
     try:
@@ -47,7 +75,7 @@ def init_repo():
     except ResticFailedError as error:
         if 'unable to open config file' in str(error):
             result = restic.init()
-            logging.info(f"Initialized restic repo: {result}")
+            logger.info(f"Initialized restic repo: {result}")
         else:
             raise error
 
@@ -55,20 +83,25 @@ def init_repo():
 def export_secrets():
     for env in os.environ:
         if env.endswith('FILE') and not "COMPOSE_FILE" in env:
-            logging.debug(f"exported secret: {env}")
+            logger.debug(f"exported secret: {env}")
             with open(os.environ[env]) as file:
-                secret =  file.read()
+                secret = file.read()
                 os.environ[env.removesuffix('_FILE')] = secret
-                # logging.debug(f"Read secret value: {secret}")
+
+            if env == 'RESTIC_REPOSITORY_FILE':
+                # RESTIC_REPOSITORY_FILE and RESTIC_REPOSITORY are mutually exclusive
+                logger.info("RESTIC_REPOSITORY set to RESTIC_REPOSITORY_FILE. Unsetting RESTIC_REPOSITORY_FILE.")
+                del os.environ['RESTIC_REPOSITORY_FILE']
 
 
 @cli.command()
-def create():
+@click.option('retries', '--retries', '-r', envvar='RETRIES', default=1)
+def create(retries):
     pre_commands, post_commands, backup_paths, apps = get_backup_cmds()
     copy_secrets(apps)
     backup_paths.append(SECRET_PATH)
     run_commands(pre_commands)
-    backup_volumes(backup_paths, apps)
+    backup_volumes(backup_paths, apps, int(retries))
     run_commands(post_commands)
 
 
@@ -84,24 +117,28 @@ def get_backup_cmds():
     for s in services:
         labels = s.attrs['Spec']['Labels']
         if (backup := labels.get('backupbot.backup')) and bool(backup):
+            # volumes: s.attrs['Spec']['TaskTemplate']['ContainerSpec']['Mounts'][0]['Source']
             stack_name = labels['com.docker.stack.namespace']
-            if SERVICE and SERVICE != stack_name:
+            # Remove this lines to backup only a specific service
-                continue
+            # This will unfortenately decrease restice performance
+            # if SERVICE and SERVICE != stack_name:
+            #     continue
             backup_apps.add(stack_name)
-            container = container_by_service.get(s.name)
+            backup_paths = backup_paths.union(
-            if not container:
+                Path(VOLUME_PATH).glob(f"{stack_name}_*"))
-                logging.error(
+            if not (container := container_by_service.get(s.name)):
+                logger.error(
                     f"Container {s.name} is not running, hooks can not be executed")
+                continue
             if prehook := labels.get('backupbot.backup.pre-hook'):
                 pre_commands[container] = prehook
             if posthook := labels.get('backupbot.backup.post-hook'):
                 post_commands[container] = posthook
-            backup_paths = backup_paths.union(
-                Path(VOLUME_PATH).glob(f"{stack_name}_*"))
     return pre_commands, post_commands, list(backup_paths), list(backup_apps)
 
 
 def copy_secrets(apps):
+    # TODO: check if it is deployed
     rmtree(SECRET_PATH, ignore_errors=True)
     os.mkdir(SECRET_PATH)
     client = docker.from_env()
@@ -113,12 +150,16 @@ def copy_secrets(apps):
         if (app_name in apps and
                 (app_secs := s.attrs['Spec']['TaskTemplate']['ContainerSpec'].get('Secrets'))):
             if not container_by_service.get(s.name):
-                logging.error(
+                logger.error(
                     f"Container {s.name} is not running, secrets can not be copied.")
                 continue
             container_id = container_by_service[s.name].id
             for sec in app_secs:
                 src = f'/var/lib/docker/containers/{container_id}/mounts/secrets/{sec["SecretID"]}'
+                if not Path(src).exists():
+                    logger.error(
+                        f"For the secret {sec['SecretName']} the file {src} does not exist for {s.name}")
+                    continue
                 dst = SECRET_PATH + sec['SecretName']
                 copyfile(src, dst)
 
@@ -127,30 +168,43 @@ def run_commands(commands):
     for container, command in commands.items():
         if not command:
             continue
+        # Remove bash/sh wrapping
+        command = command.removeprefix('bash -c').removeprefix('sh -c').removeprefix(' ')
+        # Remove quotes surrounding the command
+        if (len(command) >= 2 and command[0] == command[-1] and (command[0] == "'" or command[0] == '"')):
+            command = command[1:-1]
         # Use bash's pipefail to return exit codes inside a pipe to prevent silent failure
-        command = command.removeprefix('bash -c \'').removeprefix('sh -c \'')
-        command = command.removesuffix('\'')
         command = f"bash -c 'set -o pipefail;{command}'"
+        logger.info(f"run command in {container.name}:")
+        logger.info(command)
         result = container.exec_run(command)
-        logging.info(f"run command in {container.name}")
-        logging.info(command)
         if result.exit_code:
-            logging.error(
+            logger.error(
                 f"Failed to run command {command} in {container.name}: {result.output.decode()}")
         else:
-            logging.info(result.output.decode())
+            logger.info(result.output.decode())
 
 
-def backup_volumes(backup_paths, apps, dry_run=False):
+def backup_volumes(backup_paths, apps, retries, dry_run=False):
-    result = restic.backup(backup_paths, dry_run=dry_run, tags=apps)
+    while True:
-    print(result)
+        try:
-    logging.info(result)
+            result = restic.backup(backup_paths, dry_run=dry_run, tags=apps)
+            logger.summary("backup finished", extra=result)
+            return
+        except ResticFailedError as error:
+            logger.error(
+                f"Backup failed for {apps}. Could not Backup these paths: {backup_paths}")
+            logger.error(error, exc_info=True)
+            if retries > 0:
+                retries -= 1
+            else:
+                exit(1)
 
 
 @cli.command()
 @click.option('snapshot', '--snapshot', '-s', envvar='SNAPSHOT', default='latest')
 @click.option('target', '--target', '-t', envvar='TARGET', default='/')
-@click.option('noninteractive', '--noninteractive', envvar='NONINTERACTIVE', default=False)
+@click.option('noninteractive', '--noninteractive', envvar='NONINTERACTIVE', is_flag=True)
 def restore(snapshot, target, noninteractive):
     # Todo: recommend to shutdown the container
     service_paths = VOLUME_PATH
@@ -158,30 +212,41 @@ def restore(snapshot, target, noninteractive):
         service_paths = service_paths + f'{SERVICE}_*'
     snapshots = restic.snapshots(snapshot_id=snapshot)
     if not snapshot:
-        logging.error("No Snapshots with ID {snapshots}")
+        logger.error("No Snapshots with ID {snapshots}")
         exit(1)
     if not noninteractive:
         snapshot_date = datetime.fromisoformat(snapshots[0]['time'])
         delta = datetime.now(tz=timezone.utc) - snapshot_date
-        print(f"You are going to restore Snapshot {snapshot} of {service_paths} at {target}")
+        print(
+            f"You are going to restore Snapshot {snapshot} of {service_paths} at {target}")
         print(f"This snapshot is {delta} old")
-        print(f"THIS COMMAND WILL IRREVERSIBLY OVERWRITES {target}{service_paths.removeprefix('/')}")
+        print(
+            f"THIS COMMAND WILL IRREVERSIBLY OVERWRITES {target}{service_paths.removeprefix('/')}")
         prompt = input("Type YES (uppercase) to continue: ")
         if prompt != 'YES':
-            logging.error("Restore aborted")
+            logger.error("Restore aborted")
             exit(1)
     print(f"Restoring Snapshot {snapshot} of {service_paths} at {target}")
+    # TODO: use tags if no snapshot is selected, to use a snapshot including SERVICE
     result = restic.restore(snapshot_id=snapshot,
                             include=service_paths, target_dir=target)
-    logging.debug(result)
+    logger.debug(result)
 
 
 @cli.command()
 def snapshots():
     snapshots = restic.snapshots()
+    no_snapshots = True
     for snap in snapshots:
         if not SERVICE or (tags := snap.get('tags')) and SERVICE in tags:
             print(snap['time'], snap['id'])
+            no_snapshots = False
+    if no_snapshots:
+        err_msg = "No Snapshots found"
+        if SERVICE:
+            service_name = SERVICE.replace('_', '.')
+            err_msg += f' for app {service_name}'
+        logger.warning(err_msg)
 
 
 @cli.command()
@@ -201,7 +266,17 @@ def list_files(snapshot, path):
     cmd.append(snapshot)
     if path:
         cmd.append(path)
-    output = restic.internal.command_executor.execute(cmd)
+    try:
+        output = restic.internal.command_executor.execute(cmd)
+    except ResticFailedError as error:
+        if 'no snapshot found' in str(error):
+            err_msg = f'There is no snapshot "{snapshot}"'
+            if SERVICE:
+                err_msg += f' for the app "{SERVICE}"'
+            logger.error(err_msg)
+            exit(1)
+        else:
+            raise error
     output = output.replace('}\n{', '}|{')
     results = list(map(json.loads, output.split('|')))
     return results
@@ -210,62 +285,82 @@ def list_files(snapshot, path):
 @cli.command()
 @click.option('snapshot', '--snapshot', '-s', envvar='SNAPSHOT', default='latest')
 @click.option('path', '--path', '-p', envvar='INCLUDE_PATH')
-@click.option('volumes', '--volumes', '-v', is_flag=True)
+@click.option('volumes', '--volumes', '-v', envvar='VOLUMES')
-@click.option('secrets', '--secrets', '-c', is_flag=True)
+@click.option('secrets', '--secrets', '-c', is_flag=True, envvar='SECRETS')
 def download(snapshot, path, volumes, secrets):
-    if sum(map(bool, [path, volumes, secrets])) != 1:
+    file_dumps = []
-        logging.error("Please specify exactly one of '--path', '--volumes', '--secrets'")
+    if not any([path, volumes, secrets]):
-        exit(1)
+        volumes = secrets = True
     if path:
         path = path.removesuffix('/')
+        binary_output = dump(snapshot, path)
         files = list_files(snapshot, path)
         filetype = [f.get('type') for f in files if f.get('path') == path][0]
-        filename = "/tmp/" + Path(path).name
+        filename = Path(path).name
         if filetype == 'dir':
             filename = filename + ".tar"
-        output = dump(snapshot, path)
+        tarinfo = tarfile.TarInfo(name=filename)
-        with open(filename, "wb") as file:
+        tarinfo.size = len(binary_output)
-            file.write(output)
+        file_dumps.append((binary_output, tarinfo))
-        print(filename)
+    if volumes:
-    elif volumes:
         if not SERVICE:
-            logging.error("Please specify '--host' when using '--volumes'")
+            logger.error("Please specify '--host' when using '--volumes'")
             exit(1)
-        filename = f"/tmp/{SERVICE}.tar"
         files = list_files(snapshot, VOLUME_PATH)
         for f in files[1:]:
-            path = f[ 'path' ]
+            path = f['path']
-            if SERVICE in path and f['type'] == 'dir':
+            if Path(path).name.startswith(SERVICE) and f['type'] == 'dir':
-                content = dump(snapshot, path)
+                binary_output = dump(snapshot, path)
-                # Concatenate tar files (extract with tar -xi)
+                filename = f"{Path(path).name}.tar"
-                with open(filename, "ab") as file:
+                tarinfo = tarfile.TarInfo(name=filename)
-                    file.write(content)
+                tarinfo.size = len(binary_output)
-    elif secrets:
+                file_dumps.append((binary_output, tarinfo))
+    if secrets:
         if not SERVICE:
-            logging.error("Please specify '--host' when using '--secrets'")
+            logger.error("Please specify '--host' when using '--secrets'")
             exit(1)
-        filename = f"/tmp/SECRETS_{SERVICE}.json"
+        filename = f"{SERVICE}.json"
         files = list_files(snapshot, SECRET_PATH)
         secrets = {}
         for f in files[1:]:
-            path = f[ 'path' ]
+            path = f['path']
-            if SERVICE in path and f['type'] == 'file':
+            if Path(path).name.startswith(SERVICE) and f['type'] == 'file':
                 secret = dump(snapshot, path).decode()
                 secret_name = path.removeprefix(f'{SECRET_PATH}{SERVICE}_')
                 secrets[secret_name] = secret
-        with open(filename, "w") as file:
+        binary_output = json.dumps(secrets).encode()
-            json.dump(secrets, file)
+        tarinfo = tarfile.TarInfo(name=filename)
-        print(filename)
+        tarinfo.size = len(binary_output)
+        file_dumps.append((binary_output, tarinfo))
+    with tarfile.open('/tmp/backup.tar.gz', "w:gz") as tar:
+        print(f"Writing files to /tmp/backup.tar.gz...")
+        for binary_output, tarinfo in file_dumps:
+            tar.addfile(tarinfo, fileobj=io.BytesIO(binary_output))
+    size = get_formatted_size('/tmp/backup.tar.gz')
+    print(
+        f"Backup has been written to /tmp/backup.tar.gz with a size of {size}")
+
+
+def get_formatted_size(file_path):
+    file_size = os.path.getsize(file_path)
+    units = ['Bytes', 'KB', 'MB', 'GB', 'TB']
+    for unit in units:
+        if file_size < 1024:
+            return f"{round(file_size, 3)} {unit}"
+        file_size /= 1024
+    return f"{round(file_size, 3)} {units[-1]}"
+
 
 def dump(snapshot, path):
     cmd = restic.cat.base_command() + ['dump']
     if SERVICE:
         cmd = cmd + ['--tag', SERVICE]
-    cmd = cmd +[snapshot, path]
+    cmd = cmd + [snapshot, path]
-    logging.debug(f"Dumping {path} from snapshot '{snapshot}'")
+    print(f"Dumping {path} from snapshot '{snapshot}'")
     output = subprocess.run(cmd, capture_output=True)
     if output.returncode:
-        logging.error(f"error while dumping {path} from snapshot '{snapshot}': {output.stderr}")
+        logger.error(
+            f"error while dumping {path} from snapshot '{snapshot}': {output.stderr}")
         exit(1)
     return output.stdout
 

compose.s3.yml

@@ -1,14 +0,0 @@
----
-version: "3.8"
-services:
-  app:
-    environment:
-      - AWS_ACCESS_KEY_ID
-      - AWS_SECRET_ACCESS_KEY_FILE=/run/secrets/aws_secret_access_key
-    secrets:
-      - aws_secret_access_key
-
-secrets:
-  aws_secret_access_key:
-    external: true
-    name: ${STACK_NAME}_aws_secret_access_key_${SECRET_AWS_SECRET_ACCESS_KEY_VERSION}

compose.secret.yml

@@ -1,13 +0,0 @@
----
-version: "3.8"
-services:
-  app:
-    environment:
-      - RESTIC_REPO_FILE=/run/secrets/restic_repo
-    secrets:
-      - restic_repo
-
-secrets:
-  restic_repo:
-    external: true
-    name: ${STACK_NAME}_restic_repo_${SECRET_RESTIC_REPO_VERSION}

compose.ssh.yml

@@ -1,23 +0,0 @@
----
-version: "3.8"
-services:
-  app:
-    environment:
-      - SSH_KEY_FILE=/run/secrets/ssh_key
-      - SSH_HOST_KEY
-    secrets:
-      - source: ssh_key
-        mode: 0400
-    configs:
-      - source: ssh_config
-        target: /root/.ssh/config
-
-secrets:
-  ssh_key:
-    external: true
-    name: ${STACK_NAME}_ssh_key_${SECRET_SSH_KEY_VERSION}
-
-configs:
-  ssh_config:
-    name: ${STACK_NAME}_ssh_config_${SSH_CONFIG_VERSION}
-    file: ssh_config

compose.swarm-cronjob.yml

@@ -1,15 +0,0 @@
----
-version: "3.8"
-services:
-  app:
-    deploy:
-      mode: replicated
-      replicas: 0
-      labels:
-        - "swarm.cronjob.enable=true"
-        # Note(3wc): every 5m, testing
-        - "swarm.cronjob.schedule=*/5 * * * *"
-        # Note(3wc): blank label to be picked up by `abra recipe sync`
-      restart_policy:
-        condition: none
-    entrypoint: [ "/usr/bin/backup.sh" ]

compose.yaml

@@ -1,24 +1,17 @@
----
-version: "3.8"
 services:
   app:
-    image: docker:24.0.2-dind
+    image: docker:24.0.7-dind
     volumes:
       - "/var/run/docker.sock:/var/run/docker.sock"
       - "/var/lib/docker/volumes/:/var/lib/docker/volumes/"
       - "/var/lib/docker/containers/:/var/lib/docker/containers/:ro"
-      - backups:/backups
     environment:
       - CRON_SCHEDULE
-      - RESTIC_REPO
+      - RESTIC_REPOSITORY_FILE=/run/secrets/restic_repo
       - RESTIC_PASSWORD_FILE=/run/secrets/restic_password
     secrets:
+      - restic_repo
       - restic_password
-    deploy:
-      labels:
-        - coop-cloud.${STACK_NAME}.version=0.1.0+latest
-        - coop-cloud.${STACK_NAME}.timeout=${TIMEOUT:-300}
-        - coop-cloud.backupbot.enabled=true
     configs:
       - source: entrypoint
         target: /entrypoint.sh
@@ -35,17 +28,17 @@ services:
       start_period: 5m
 
 secrets:
+  restic_repo:
+    external: true
+    name: ${STACK_NAME}_restic_repo
   restic_password:
     external: true
-    name: ${STACK_NAME}_restic_password_${SECRET_RESTIC_PASSWORD_VERSION}
+    name: ${STACK_NAME}_restic_password
-    
-volumes:
-  backups:
 
 configs:
   entrypoint:
-    name: ${STACK_NAME}_entrypoint_${ENTRYPOINT_VERSION}
+    name: ${STACK_NAME}_entrypoint
     file: entrypoint.sh
   backupbot:
-    name: ${STACK_NAME}_backupbot_${BACKUPBOT_VERSION}
+    name: ${STACK_NAME}_backupbot
     file: backupbot.py

entrypoint.sh

@@ -2,19 +2,29 @@
 
 set -e -o pipefail
 
-apk add --upgrade --no-cache restic bash python3 py3-pip
+apk add --upgrade --no-cache restic bash python3 py3-pip py3-click py3-docker-py py3-json-logger curl
 
 # Todo use requirements file with specific versions
-pip install click==8.1.7 docker==6.1.3 resticpy==1.0.2
+pip install --break-system-packages resticpy==1.0.2
-
-if [ -n "$SSH_HOST_KEY" ]
-then
-    echo "$SSH_HOST_KEY" > /root/.ssh/known_hosts
-fi
 
 cron_schedule="${CRON_SCHEDULE:?CRON_SCHEDULE not set}"
 
-echo "$cron_schedule backup create" | crontab -
+if [ -n "$PUSH_URL_START" ]
+then
+    push_start_notification="curl -s '$PUSH_URL_START' &&"
+fi
+
+if [ -n "$PUSH_URL_FAIL" ]
+then
+    push_fail_notification="|| curl -s '$PUSH_URL_FAIL'"
+fi
+
+if [ -n "$PUSH_URL_SUCCESS" ]
+then
+    push_notification=" && (grep -q 'backup finished' /tmp/backup.log && curl -s '$PUSH_URL_SUCCESS' $push_fail_notification)"
+fi
+
+echo "$cron_schedule $push_start_notification backup --machine-logs create  2>&1 | tee /tmp/backup.log $push_notification" | crontab -
 crontab -l
 
 crond -f -d8 -L /dev/stdout

release/1.0.0+latest

@@ -1,3 +0,0 @@
-Breaking Change: the variables `SERVER_NAME` and `RESTIC_HOST` are merged into `RESTIC_REPO`. The format can be looked up here: https://restic.readthedocs.io/en/stable/030_preparing_a_new_repo.html
-ssh/sftp: `sftp:user@host:/repo-path`
-S3:  `s3:https://s3.example.com/bucket_name`

renovate.json

@@ -1,3 +0,0 @@
-{
-  "$schema": "https://docs.renovatebot.com/renovate-schema.json"
-}

ssh_config

@@ -1,4 +0,0 @@
-Host *
-    IdentityFile    /run/secrets/ssh_key
-    ServerAliveInterval 60
-    ServerAliveCountMax 240