move rm up, to keep the latest backup in the volume

[ci skip]

.drone.yml

@@ -0,0 +1,28 @@
+---
+kind: pipeline
+name: linters
+steps:
+  - name: run shellcheck
+    image: koalaman/shellcheck-alpine
+    commands:
+      - shellcheck backup.sh
+
+  - name: publish image
+    image: plugins/docker
+    settings:
+      auto_tag: true
+      username: thecoopcloud
+      password:
+        from_secret: thecoopcloud_password
+      repo: thecoopcloud/backup-bot-two
+      tags: latest
+    depends_on:
+      - run shellcheck
+    when:
+      event:
+        exclude:
+          - pull_request
+
+trigger:
+  branch:
+    - main

.env.sample

@@ -0,0 +1,23 @@
+TYPE=backup-bot-two
+
+SECRET_RESTIC_PASSWORD_VERSION=v1
+
+COMPOSE_FILE=compose.yml
+
+SERVER_NAME=example.com
+RESTIC_HOST=minio.example.com
+
+CRON_SCHEDULE='*/5 * * * *'
+
+# swarm-cronjob, instead of built-in cron
+#COMPOSE_FILE="$COMPOSE_FILE:compose.swarm-cronjob.yml"
+
+# SSH storage
+#SECRET_SSH_KEY_VERSION=v1
+#SSH_HOST_KEY="hostname ssh-rsa AAAAB3...
+#COMPOSE_FILE="$COMPOSE_FILE:compose.ssh.yml"
+
+# S3 storage
+#SECRET_AWS_SECRET_ACCESS_KEY_VERSION=v1
+#AWS_ACCESS_KEY_ID=something-secret
+#COMPOSE_FILE="$COMPOSE_FILE:compose.s3.yml"

.envrc.sample

@@ -0,0 +1,15 @@
+export RESTIC_HOST="user@domain.tld"
+export RESTIC_PASSWORD_FILE=/run/secrets/restic-password
+export BACKUP_DEST=/backups
+
+export SERVER_NAME=domain.tld
+export DOCKER_CONTEXT=$SERVER_NAME
+
+# uncomment either this:
+#export SSH_KEY_FILE=~/.ssh/id_rsa
+# or this:
+#export AWS_SECRET_ACCESS_KEY_FILE=s3
+#export AWS_ACCESS_KEY_ID=easter-october-emphatic-tug-urgent-customer
+
+# optionally limit subset of services for testing
+#export SERVICES_OVERRIDE="ghost_domain_tld_app ghost_domain_tld_db"

.gitignore

@@ -0,0 +1 @@
+/testing

Dockerfile

@@ -0,0 +1,13 @@
+FROM docker:19.03.13-dind
+
+RUN apk add --upgrade --no-cache \
+  bash \
+  curl \
+  jq \
+  restic
+
+COPY backup.sh /usr/bin/backup.sh
+COPY setup-cron.sh /usr/bin/setup-cron.sh
+RUN chmod +x /usr/bin/backup.sh /usr/bin/setup-cron.sh
+
+ENTRYPOINT [ "/usr/bin/setup-cron.sh" ]

README.md

@@ -1,45 +1,72 @@
-# Backupbot II: This Time It's Easily Configurable
+# Backupbot II
 
-Automatically backup files from running Docker Swarm services based on labels.
+[![Build Status](https://build.coopcloud.tech/api/badges/coop-cloud/backup-bot-two/status.svg)](https://build.coopcloud.tech/coop-cloud/backup-bot-two)
 
-## TODO
+_This Time, It's Easily Configurable_
 
-- [ ] Make a Docker image of this
-- [ ] Rip out or improve Restic stuff
-- [ ] Add secret handling for database backups
-- [ ] Continuous linting with shellcheck
+Automatically take backups from running Docker Swarm services into a volume.
 
-## Label format
+## Background
 
-(Haven't done secrets yet, here are two options)
+There are lots of Docker volume backup systems; all of them have one or both of these limitations:
+ - You need to define all the volumes to back up in the configuration system
+ - Backups require services to be stopped to take consistent copies
+
+Backupbot II tries to help, by
+1. **letting you define backups using Docker labels**, so you can **easily collect your backups for use with another system** like docker-volume-backup.
+2. **running pre- and post-commands** before and after backups, for example to use database tools to take a backup from a running service.
+
+## Deployment
+
+### With Co-op Cloud
+
+1. Set up Docker Swarm and [`abra`][abra]
+2. `abra app new backup-bot-two`
+3. `abra app config <your-app-name>`, and set storage options. Either configure `CRON_SCHEDULE`, or set up `swarm-cronjob`
+4. `abra app secret generate <your-app-name> restic-password v1`, optionally with `--pass` before `<your-app-name>` to save the generated secret in `pass`.
+5. `abra app secret insert <your-app-name> ssh-key v1 ...` or similar, to load required secrets.
+4. `abra app deploy <your-app-name>`
+
+<!-- metadata -->
+
+* **Category**: Utilities
+* **Status**: 0, work-in-progress
+* **Image**: [`thecoopcloud/backup-bot-two`](https://hub.docker.com/r/thecoopcloud/backup-bot-two), 4, upstream
+* **Healthcheck**: No
+* **Backups**: N/A
+* **Email**: N/A
+* **Tests**: No
+* **SSO**: N/A
+
+<!-- endmetadata -->
+
+## Configuration
+
+Like Traefik, or `swarm-cronjob`, Backupbot II uses access to the Docker socket to read labels from running Docker Swarm services:
 
-v1:
 ```
+services:
+  db:
     deploy:
       labels:
         backupbot.backup: "true"
-        backupbot.backup.repos: "$some_thing"
-        backupbot.backup.at: "* * * * *"
         backupbot.backup.pre-hook: 'mysqldump -u root -p"$(cat /run/secrets/db_root_password)" -f /tmp/dump/dump.db'
         backupbot.backup.post-hook: "rm -rf /tmp/dump/dump.db"
-        backupbot.backup.path: "/tmp/dump/"
-```
-v2:
-```
-    deploy:
-      labels:
-        backupbot.backup: "true"
-        backupbot.backup.repos: "$some_thing"
-        backupbot.backup.at: "* * * * *"
-        backupbot.backup.post-hook: "rm -rf /tmp/dump/dump.db"
-        backupbot.backup.secrets": "db_root_password",
-        backupbot.backup.pre-hook: 'mysqldump -u root -p"$DB_ROOT_PASSWORD" -f /tmp/dump/dump.db'
+        backupbot.backup.path: "/tmp/dump/,/etc/foo/"
 ```
 
-## Questions:
+- `backupbot.backup` -- set to `true` to back up this service (REQUIRED)
+- `backupbot.backup.path` -- comma separated list of file paths within the service to copy (REQUIRED)
+- `backupbot.backup.pre-hook` -- command to run before copying files (optional)
+- `backupbot.backup.post-hook` -- command to run after copying files (optional)
 
-- Should frequency be configurable per service, centrally, or both?
+As in the above example, you can reference Docker Secrets, e.g. for looking up database passwords, by reading the files in `/run/secrets` directly.
 
-```
-- "backupbot.backup.at: "* * * * *"
-```
+## Development
+
+1. Install `direnv`
+2. `cp .envrc.sample .envrc`
+3. Edit `.envrc` as appropriate, including setting `DOCKER_CONTEXT` to a remote Docker context, if you're not running a swarm server locally.
+4. Run `./backup.sh` -- you can add the `--skip-backup` or `--skip-upload` options if you just want to test one other step
+
+[abra]: https://git.autonomic.zone/autonomic-cooperative/abra

backup.sh

@@ -1,50 +1,126 @@
 #!/bin/bash
 
-# FIXME: just for testing
-backup_path=backups
+server_name="${SERVER_NAME:?SERVER_NAME not set}"
 
-# FIXME: just for testing
-export DOCKER_CONTEXT=demo.coopcloud.tech
+restic_password_file="${RESTIC_PASSWORD_FILE:?RESTIC_PASSWORD_FILE not set}"
 
-mapfile -t services < <(docker service ls --format '{{ .Name }}')
+restic_host="${RESTIC_HOST:?RESTIC_HOST not set}"
 
-# FIXME: just for testing
-services=( "ghost_demo_app" "ghost_demo_db" )
+backup_path="${BACKUP_DEST:?BACKUP_DEST not set}"
 
-for service in "${services[@]}"; do
-	echo "service: $service"
-	details=$(docker service inspect "$service" --format "{{ json .Spec.Labels }}")
-	if echo "$details" | jq -r '.["backupbot.backup"]' | grep -q 'true'; then
-		pre=$(echo "$details" | jq -r '.["backupbot.backup.pre-hook"]')
-		post=$(echo "$details" | jq -r '.["backupbot.backup.post-hook"]')
-		path=$(echo "$details" | jq -r '.["backupbot.backup.path"]')
-		
-		if [ "$path" = "null" ]; then
-			echo "ERROR: missing 'path' for $service"
-			continue  # or maybe exit?
-		fi
+# shellcheck disable=SC2153
+ssh_key_file="${SSH_KEY_FILE}"
+s3_key_file="${AWS_SECRET_ACCESS_KEY_FILE}"
 
-		container=$(docker container ls -f "name=$service" --format '{{ .ID }}')
-		
-		echo "backing up $service"
-		test -d "$backup_path/$service" || mkdir "$backup_path/$service"
-		
-		if [ "$pre" != "null" ]; then
-			# run the precommand
-			# shellcheck disable=SC2086
-			docker exec "$container" $pre
-		fi
+restic_repo=
+restic_extra_options=
 
-		# run the backup
-		docker cp "$container:$path" "$backup_path/$service"
+if [ -n "$ssh_key_file" ] && [ -f "$ssh_key_file" ]; then
+	restic_repo="sftp:$restic_host:/$server_name"
 
-		if [ "$post" != "null" ]; then
-			# run the postcommand
-			# shellcheck disable=SC2086
-			docker exec "$container" $post
-		fi
+	# Only check server against provided SSH_HOST_KEY, if set
+	if [ -n "$SSH_HOST_KEY" ]; then
+		tmpfile=$(mktemp)
+		echo "$SSH_HOST_KEY" >>"$tmpfile"
+		echo "using host key $SSH_HOST_KEY"
+		ssh_options="-o 'UserKnownHostsFile $tmpfile'"
+	elif [ "$SSH_HOST_KEY_DISABLE" = "1" ]; then
+		echo "disabling SSH host key checking"
+		ssh_options="-o 'StrictHostKeyChecking=No'"
+	else
+		echo "neither SSH_HOST_KEY nor SSH_HOST_KEY_DISABLE set"
 	fi
-	restic -p restic-password \
-		backup --quiet -r sftp:u272979@u272979.your-storagebox.de:/demo.coopcloud.tech \
-		--tag coop-cloud "$backup_path"
-done
+	restic_extra_options="sftp.command=ssh $ssh_options -i $ssh_key_file $restic_host -s sftp"
+fi
+
+if [ -n "$s3_key_file" ] && [ -f "$s3_key_file" ] && [ -n "$AWS_ACCESS_KEY_ID" ]; then
+	AWS_SECRET_ACCESS_KEY="$(cat "${s3_key_file}")"
+	export AWS_SECRET_ACCESS_KEY
+	restic_repo="s3:$restic_host:/$server_name"
+fi
+
+if [ -z "$restic_repo" ]; then
+	echo "you must configure either SFTP or S3 storage, see README"
+	exit 1
+fi
+
+echo "restic_repo: $restic_repo"
+
+# Pre-bake-in some default restic options
+_restic() {
+	if [ -z "$restic_extra_options" ]; then
+		# shellcheck disable=SC2068
+		restic -p "$restic_password_file" \
+			--quiet -r "$restic_repo" \
+			$@
+	else
+		# shellcheck disable=SC2068
+		restic -p "$restic_password_file" \
+			--quiet -r "$restic_repo" \
+			-o "$restic_extra_options" \
+			$@
+	fi
+}
+
+if [ -n "$SERVICES_OVERRIDE" ]; then
+	# this is fine because docker service names should never include spaces or
+	# glob characters
+	# shellcheck disable=SC2206
+	services=($SERVICES_OVERRIDE)
+else
+	mapfile -t services < <(docker service ls --format '{{ .Name }}')
+fi
+
+if [[ \ $*\  != *\ --skip-backup\ * ]]; then
+    rm -rf "${backup_path}"
+
+	for service in "${services[@]}"; do
+		echo "service: $service"
+		details=$(docker service inspect "$service" --format "{{ json .Spec.Labels }}")
+		if echo "$details" | jq -r '.["backupbot.backup"]' | grep -q 'true'; then
+			pre=$(echo "$details" | jq -r '.["backupbot.backup.pre-hook"]')
+			post=$(echo "$details" | jq -r '.["backupbot.backup.post-hook"]')
+			path=$(echo "$details" | jq -r '.["backupbot.backup.path"]')
+
+			if [ "$path" = "null" ]; then
+				echo "ERROR: missing 'path' for $service"
+				continue # or maybe exit?
+			fi
+
+			container=$(docker container ls -f "name=$service" --format '{{ .ID }}')
+
+			echo "backing up $service"
+
+			if [ "$pre" != "null" ]; then
+				# run the precommand
+				# shellcheck disable=SC2086
+				docker exec "$container" sh -c "$pre"
+			fi
+
+			# run the backup
+			for p in ${path//,/ }; do
+				# creates the parent folder, so `docker cp` has reliable behaviour no matter if $p ends with `/` or `/.`
+				dir=$backup_path/$service/$(dirname "$p")
+				test -d "$dir" || mkdir -p "$dir"
+				docker cp -a "$container:$p" "$dir/$(basename "$p")"
+			done
+
+			if [ "$post" != "null" ]; then
+				# run the postcommand
+				# shellcheck disable=SC2086
+				docker exec "$container" sh -c "$post"
+			fi
+		fi
+	done
+
+	# check if restic repo exists, initialise if not
+	if [ -z "$(_restic cat config)" ] 2>/dev/null; then
+		echo "initializing restic repo"
+		_restic init
+	fi
+fi
+
+if [[ \ $*\  != *\ --skip-upload\ * ]]; then
+	_restic backup --host "$server_name" --tag coop-cloud "$backup_path"
+fi
+

compose.s3.yml

@@ -0,0 +1,14 @@
+---
+version: "3.8"
+services:
+  app:
+    environment:
+      - AWS_ACCESS_KEY_ID
+      - AWS_SECRET_ACCESS_KEY_FILE=/run/secrets/aws_secret_access_key
+    secrets:
+      - aws_secret_access_key
+
+secrets:
+  aws_secret_access_key:
+    external: true
+    name: ${STACK_NAME}_aws_secret_access_key_${SECRET_AWS_SECRET_ACCESS_KEY_VERSION}

compose.ssh.yml

@@ -0,0 +1,16 @@
+---
+version: "3.8"
+services:
+  app:
+    environment:
+      - SSH_KEY_FILE=/run/secrets/ssh_key
+      - SSH_HOST_KEY
+      - SSH_HOST_KEY_DISABLE
+    secrets:
+      - source: ssh_key
+        mode: 0400
+
+secrets:
+  ssh_key:
+    external: true
+    name: ${STACK_NAME}_ssh_key_${SECRET_SSH_KEY_VERSION}

compose.swarm-cronjob.yml

@@ -0,0 +1,15 @@
+---
+version: "3.8"
+services:
+  app:
+    deploy:
+      mode: replicated
+      replicas: 0
+      labels:
+        - "swarm.cronjob.enable=true"
+        # Note(3wc): every 5m, testing
+        - "swarm.cronjob.schedule=*/5 * * * *"
+        # Note(3wc): blank label to be picked up by `abra recipe sync`
+      restart_policy:
+        condition: none
+    entrypoint: [ "/usr/bin/backup.sh" ]

compose.yml

@@ -0,0 +1,29 @@
+---
+version: "3.8"
+services:
+  app:
+    image: thecoopcloud/backup-bot-two:latest
+    build: .
+    volumes:
+      - "/var/run/docker.sock:/var/run/docker.sock"
+      - "backups:/backups"
+    environment:
+      - CRON_SCHEDULE
+      - RESTIC_REPO
+      - RESTIC_PASSWORD_FILE=/run/secrets/restic_password
+      - BACKUP_DEST=/backups
+      - RESTIC_HOST
+      - SERVER_NAME
+    secrets:
+      - restic_password
+    deploy:
+      labels:
+        - coop-cloud.${STACK_NAME}.app.version=
+
+volumes:
+  backups:
+
+secrets:
+  restic_password:
+    external: true
+    name: ${STACK_NAME}_restic_password_${SECRET_RESTIC_PASSWORD_VERSION}

setup-cron.sh

@@ -0,0 +1,11 @@
+#!/bin/bash
+
+set -e
+set -o pipefail
+
+cron_schedule="${CRON_SCHEDULE:?CRON_SCHEDULE not set}"
+
+echo "$cron_schedule /usr/bin/backup.sh" | crontab -
+crontab -l
+
+crond -f -d8 -L /dev/stdout