forked from coop-cloud/backup-bot-two
Compare commits
51 Commits
main
...
2.2.0+2.2.
Author | SHA1 | Date | |
---|---|---|---|
79cdec6705 | |||
2bc9400807 | |||
9b141a5185 | |||
6ff2312090 | |||
8b66b80332 | |||
c9b04db7a0 | |||
333b7ec16d | |||
aeccd605ee | |||
f877186a57 | |||
9dc239b199 | |||
43548273fe | |||
5a0467dbdd | |||
3aefae61c0 | |||
ac7c5fb50d | |||
cc59087b8c | |||
eb7c35d4cd | |||
249772ec03 | |||
45af6e8b5e | |||
f7207cdf36 | |||
241fe3ce92 | |||
b8d61d01cd | |||
6ac781c7e6 | |||
197cabf564 | |||
fe35f1ede8 | |||
f254a365f2 | |||
0d15765673 | |||
e09e1b476c | |||
72688dc42c | |||
10e460ff2d | |||
f2d0b92fa3 | |||
cc049b858b | |||
b7bc8ed58f | |||
68e37f5c23 | |||
4d39d84733 | |||
e5b9bc0446 | |||
ec4c4509dc | |||
26162a9e38 | |||
bd581fd8d7 | |||
e77432e3ab | |||
001a654e37 | |||
c5574edc54 | |||
50e4d68717 | |||
c7830ceb6f | |||
b6f859efbb | |||
7f14698824 | |||
2a9a98172f | |||
282215cf9c | |||
ae7a14b6f1 | |||
8acdb20e5b | |||
5582744073 | |||
84d606fa80 |
21
.drone.yml
21
.drone.yml
@ -2,11 +2,16 @@
|
||||
kind: pipeline
|
||||
name: linters
|
||||
steps:
|
||||
- name: run shellcheck
|
||||
image: koalaman/shellcheck-alpine
|
||||
commands:
|
||||
- shellcheck backup.sh
|
||||
|
||||
trigger:
|
||||
branch:
|
||||
- main
|
||||
- name: publish image
|
||||
image: plugins/docker
|
||||
settings:
|
||||
username: 3wordchant
|
||||
password:
|
||||
from_secret: git_coopcloud_tech_token_3wc
|
||||
repo: git.coopcloud.tech/coop-cloud/backup-bot-two
|
||||
tags: ${DRONE_SEMVER_BUILD}
|
||||
registry: git.coopcloud.tech
|
||||
when:
|
||||
event:
|
||||
include:
|
||||
- tag
|
||||
|
6
CHANGELOG.md
Normal file
6
CHANGELOG.md
Normal file
@ -0,0 +1,6 @@
|
||||
# Change log
|
||||
|
||||
## 2.0.0 (unreleased)
|
||||
|
||||
- Rewrite from Bash to Python
|
||||
- Add support for push notifications (#24)
|
11
Dockerfile
Normal file
11
Dockerfile
Normal file
@ -0,0 +1,11 @@
|
||||
FROM docker:24.0.7-dind
|
||||
|
||||
RUN apk add --upgrade --no-cache restic bash python3 py3-pip py3-click py3-docker-py py3-json-logger curl
|
||||
|
||||
# Todo use requirements file with specific versions
|
||||
RUN pip install --break-system-packages resticpy==1.0.2
|
||||
|
||||
COPY backupbot.py /usr/bin/backup
|
||||
COPY entrypoint.sh /entrypoint.sh
|
||||
|
||||
ENTRYPOINT /entrypoint.sh
|
33
README.md
33
README.md
@ -10,7 +10,7 @@ Automatically take backups from all volumes of running Docker Swarm services and
|
||||
|
||||
* **Category**: Utilities
|
||||
* **Status**: 0, work-in-progress
|
||||
* **Image**: [`thecoopcloud/backup-bot-two`](https://hub.docker.com/r/thecoopcloud/backup-bot-two), 4, upstream
|
||||
* **Image**: [`git.coopcloud.tech/coop-cloud/backup-bot-two`](https://git.coopcloud.tech/coop-cloud/-/packages/container/backup-bot-two), 4, upstream
|
||||
* **Healthcheck**: No
|
||||
* **Backups**: N/A
|
||||
* **Email**: N/A
|
||||
@ -95,7 +95,7 @@ COMPOSE_FILE="$COMPOSE_FILE:compose.secret.yml"
|
||||
```
|
||||
Add your REST server url as secret:
|
||||
```
|
||||
`abra app secret insert <backupbot_name> restic_repo v1 "rest:https://user:pass@host:8000/"`
|
||||
abra app secret insert <backupbot_name> restic_repo v1 "rest:https://user:pass@host:8000/"
|
||||
```
|
||||
The secret will overwrite the `RESTIC_REPOSITORY` variable.
|
||||
|
||||
@ -196,3 +196,32 @@ services:
|
||||
As in the above example, you can reference Docker Secrets, e.g. for looking up database passwords, by reading the files in `/run/secrets` directly.
|
||||
|
||||
[abra]: https://git.autonomic.zone/autonomic-cooperative/abra
|
||||
|
||||
## Backupbot Development
|
||||
|
||||
1. Copy modified backupbot.py into the container:
|
||||
|
||||
```
|
||||
cp backupbot.py /tmp/backupbot.py; git stash; abra app cp <backupbot_name> /tmp/backupbot.py app:/usr/bin/backupbot.py; git checkout main; git stash pop
|
||||
```
|
||||
|
||||
2. Testing stuff with the python interpreter inside the container:
|
||||
|
||||
```
|
||||
abra app run <backupbot_name> app bash
|
||||
cd /usr/bin/
|
||||
python
|
||||
from backupbot import *
|
||||
```
|
||||
|
||||
### Versioning
|
||||
|
||||
- App version: changes to `backup.py` (build a new image)
|
||||
- Co-op Cloud package version: changes to recipe.
|
||||
|
||||
For example, starting with 1.0.0+2.0.0:
|
||||
"patch" change to recipe: 1.0.1+2.0.0
|
||||
"patch" change to backup.py: increment both, so 1.1.0+2.0.1
|
||||
because bumping the image version would result in a minor recipe release
|
||||
|
||||
https://git.coopcloud.tech/coop-cloud/backup-bot-two/issues/4
|
||||
|
1
abra.sh
1
abra.sh
@ -1,4 +1,3 @@
|
||||
export ENTRYPOINT_VERSION=v1
|
||||
export BACKUPBOT_VERSION=v1
|
||||
export SSH_CONFIG_VERSION=v1
|
||||
|
||||
|
339
backupbot.py
339
backupbot.py
@ -18,7 +18,7 @@ from shutil import copyfile, rmtree
|
||||
|
||||
VOLUME_PATH = "/var/lib/docker/volumes/"
|
||||
SECRET_PATH = '/secrets/'
|
||||
SERVICE = None
|
||||
SERVICE = 'ALL'
|
||||
|
||||
logger = logging.getLogger("backupbot")
|
||||
logging.addLevelName(55, 'SUMMARY')
|
||||
@ -42,7 +42,7 @@ sys.excepthook = handle_exception
|
||||
@click.option('-l', '--log', 'loglevel')
|
||||
@click.option('-m', '--machine-logs', 'machine_logs', is_flag=True)
|
||||
@click.option('service', '--host', '-h', envvar='SERVICE')
|
||||
@click.option('repository', '--repo', '-r', envvar='RESTIC_REPOSITORY', required=True)
|
||||
@click.option('repository', '--repo', '-r', envvar='RESTIC_REPOSITORY')
|
||||
def cli(loglevel, service, repository, machine_logs):
|
||||
global SERVICE
|
||||
if service:
|
||||
@ -54,21 +54,25 @@ def cli(loglevel, service, repository, machine_logs):
|
||||
if not isinstance(numeric_level, int):
|
||||
raise ValueError('Invalid log level: %s' % loglevel)
|
||||
logger.setLevel(numeric_level)
|
||||
logHandler = logging.StreamHandler()
|
||||
if machine_logs:
|
||||
logHandler = logging.StreamHandler()
|
||||
formatter = jsonlogger.JsonFormatter(
|
||||
"%(levelname)s %(filename)s %(lineno)s %(process)d %(message)s", rename_fields={"levelname": "message_type"})
|
||||
logHandler.setFormatter(formatter)
|
||||
logger.addHandler(logHandler)
|
||||
logger.addHandler(logHandler)
|
||||
|
||||
export_secrets()
|
||||
init_repo()
|
||||
|
||||
|
||||
def init_repo():
|
||||
repo = os.environ['RESTIC_REPOSITORY']
|
||||
if repo:= os.environ.get('RESTIC_REPOSITORY_FILE'):
|
||||
# RESTIC_REPOSITORY_FILE and RESTIC_REPOSITORY are mutually exclusive
|
||||
del os.environ['RESTIC_REPOSITORY']
|
||||
else:
|
||||
repo = os.environ['RESTIC_REPOSITORY']
|
||||
restic.repository = repo
|
||||
logger.debug(f"set restic repository location: {repo}")
|
||||
restic.repository = repo
|
||||
restic.password_file = '/var/run/secrets/restic_password'
|
||||
try:
|
||||
restic.cat.config()
|
||||
@ -93,44 +97,214 @@ def export_secrets():
|
||||
@cli.command()
|
||||
@click.option('retries', '--retries', '-r', envvar='RETRIES', default=1)
|
||||
def create(retries):
|
||||
pre_commands, post_commands, backup_paths, apps = get_backup_cmds()
|
||||
copy_secrets(apps)
|
||||
backup_paths.append(SECRET_PATH)
|
||||
app_settings = parse_backup_labels()
|
||||
pre_commands, post_commands, backup_paths, apps_versions = get_backup_details(app_settings)
|
||||
copy_secrets(apps_versions)
|
||||
backup_paths.append(Path(SECRET_PATH))
|
||||
run_commands(pre_commands)
|
||||
backup_volumes(backup_paths, apps, int(retries))
|
||||
backup_volumes(backup_paths, apps_versions, int(retries))
|
||||
run_commands(post_commands)
|
||||
|
||||
|
||||
def get_backup_cmds():
|
||||
@cli.command()
|
||||
@click.option('snapshot', '--snapshot', '-s', envvar='SNAPSHOT', default='latest')
|
||||
@click.option('target', '--target', '-t', envvar='TARGET', default='/')
|
||||
@click.option('noninteractive', '--noninteractive', envvar='NONINTERACTIVE', is_flag=True)
|
||||
@click.option('volumes', '--volumes', '-v', envvar='VOLUMES', multiple=True)
|
||||
@click.option('container', '--container', '-c', envvar='CONTAINER', multiple=True)
|
||||
@click.option('no_commands', '--no-commands', envvar='NO_COMMANDS', is_flag=True)
|
||||
def restore(snapshot, target, noninteractive, volumes, container, no_commands):
|
||||
app_settings = parse_backup_labels('restore', container)
|
||||
if SERVICE != 'ALL':
|
||||
app_settings = {SERVICE: app_settings[SERVICE]}
|
||||
pre_commands, post_commands, backup_paths, apps_versions = get_backup_details(app_settings, volumes)
|
||||
snapshots = get_snapshots(snapshot_id=snapshot)
|
||||
if not snapshot:
|
||||
logger.error(f"No Snapshots with ID {snapshots} for {apps_versions.keys()} found.")
|
||||
exit(1)
|
||||
if not noninteractive:
|
||||
print(f"Snapshot to restore: \t{snapshot}")
|
||||
restore_app_versions = app_versions_from_tags(snapshots[0].get('tags'))
|
||||
print("Apps:")
|
||||
for app, version in apps_versions.items():
|
||||
restore_version = restore_app_versions.get(app)
|
||||
print(f"\t{app} \t {restore_version}")
|
||||
if version != restore_version:
|
||||
print(f"WARNING!!! The running app is deployed with version {version}")
|
||||
print("The following volume paths will be restored:")
|
||||
for p in backup_paths:
|
||||
print(f'\t{p}')
|
||||
if not no_commands:
|
||||
print("The following commands will be executed:")
|
||||
for container, cmd in list(pre_commands.items()) + list(post_commands.items()):
|
||||
print(f"\t{container.labels['com.docker.swarm.service.name']}:\t{cmd}")
|
||||
snapshot_date = datetime.fromisoformat(snapshots[0]['time'])
|
||||
delta = datetime.now(tz=timezone.utc) - snapshot_date
|
||||
print(f"This snapshot is {delta} old")
|
||||
print("\nTHIS COMMAND WILL IRREVERSIBLY OVERWRITES FILES")
|
||||
prompt = input("Type YES (uppercase) to continue: ")
|
||||
if prompt != 'YES':
|
||||
logger.error("Restore aborted")
|
||||
exit(1)
|
||||
print(f"Restoring Snapshot {snapshot} at {target}")
|
||||
if not no_commands and pre_commands:
|
||||
print(f"Run pre commands.")
|
||||
run_commands(pre_commands)
|
||||
result = restic_restore(snapshot_id=snapshot, include=backup_paths, target_dir=target)
|
||||
if not no_commands and post_commands:
|
||||
print(f"Run post commands.")
|
||||
run_commands(post_commands)
|
||||
logger.debug(result)
|
||||
|
||||
|
||||
def restic_restore(snapshot_id='latest', include=[], target_dir=None):
|
||||
cmd = restic.cat.base_command() + ['restore', snapshot_id]
|
||||
for path in include:
|
||||
cmd.extend(['--include', path])
|
||||
if target_dir:
|
||||
cmd.extend(['--target', target_dir])
|
||||
return restic.internal.command_executor.execute(cmd)
|
||||
|
||||
|
||||
def get_snapshots(snapshot_id=None):
|
||||
if snapshot_id and snapshot_id != 'latest':
|
||||
snapshots = restic.snapshots(snapshot_id=snapshot_id)
|
||||
if not SERVICE in app_versions_from_tags(snapshots[0].get('tags')):
|
||||
logger.error(f'Snapshot with ID {snapshot_id} does not contain {SERVICE}')
|
||||
exit(1)
|
||||
else:
|
||||
snapshots = restic.snapshots()
|
||||
snapshots = list(filter(lambda x: SERVICE in app_versions_from_tags(x.get('tags')), snapshots))
|
||||
if snapshot_id == 'latest':
|
||||
return snapshots[-1:]
|
||||
else:
|
||||
return snapshots
|
||||
|
||||
|
||||
def app_versions_from_tags(tags):
|
||||
if tags:
|
||||
app_versions = map(lambda x: x.split(':'), tags)
|
||||
return {i[0]: i[1] if len(i) > 1 else None for i in app_versions}
|
||||
else:
|
||||
return {}
|
||||
|
||||
def parse_backup_labels(hook_type='backup', selected_container=[]):
|
||||
client = docker.from_env()
|
||||
container_by_service = {
|
||||
c.labels['com.docker.swarm.service.name']: c for c in client.containers.list()}
|
||||
backup_paths = set()
|
||||
backup_apps = set()
|
||||
pre_commands = {}
|
||||
post_commands = {}
|
||||
c.labels.get('com.docker.swarm.service.name'): c for c in client.containers.list()}
|
||||
services = client.services.list()
|
||||
app_settings = {}
|
||||
for s in services:
|
||||
labels = s.attrs['Spec']['Labels']
|
||||
specs = s.attrs['Spec']
|
||||
labels = specs['Labels']
|
||||
stack_name = labels['com.docker.stack.namespace']
|
||||
container_name = s.name.removeprefix(f"{stack_name}_")
|
||||
version = labels.get(f'coop-cloud.{stack_name}.version')
|
||||
settings = app_settings[stack_name] = app_settings.get(stack_name) or {}
|
||||
if (backup := labels.get('backupbot.backup')) and bool(backup):
|
||||
# volumes: s.attrs['Spec']['TaskTemplate']['ContainerSpec']['Mounts'][0]['Source']
|
||||
stack_name = labels['com.docker.stack.namespace']
|
||||
# Remove this lines to backup only a specific service
|
||||
# This will unfortenately decrease restice performance
|
||||
# if SERVICE and SERVICE != stack_name:
|
||||
# continue
|
||||
backup_apps.add(stack_name)
|
||||
backup_paths = backup_paths.union(
|
||||
Path(VOLUME_PATH).glob(f"{stack_name}_*"))
|
||||
if not (container := container_by_service.get(s.name)):
|
||||
logger.error(
|
||||
f"Container {s.name} is not running, hooks can not be executed")
|
||||
settings['enabled'] = True
|
||||
if version:
|
||||
settings['version'] = version
|
||||
if selected_container and container_name not in selected_container:
|
||||
logger.debug(f"Skipping {s.name} because it's not a selected container")
|
||||
continue
|
||||
if mounts:= specs['TaskTemplate']['ContainerSpec'].get('Mounts'):
|
||||
volumes = parse_volumes(stack_name, mounts)
|
||||
volumes.update(settings.get('volumes') or {})
|
||||
settings['volumes'] = volumes
|
||||
excluded_volumes, included_volume_paths = parse_excludes_includes(labels)
|
||||
settings['excluded_volumes'] = excluded_volumes.union(settings.get('excluded_volumes') or set())
|
||||
settings['included_volume_paths'] = included_volume_paths.union(settings.get('included_volume_paths') or set())
|
||||
if container := container_by_service.get(s.name):
|
||||
if command := labels.get(f'backupbot.{hook_type}.pre-hook'):
|
||||
if not (pre_hooks:= settings.get('pre_hooks')):
|
||||
pre_hooks = settings['pre_hooks'] = {}
|
||||
pre_hooks[container] = command
|
||||
if command := labels.get(f'backupbot.{hook_type}.post-hook'):
|
||||
if not (post_hooks:= settings.get('post_hooks')):
|
||||
post_hooks = settings['post_hooks'] = {}
|
||||
post_hooks[container] = command
|
||||
else:
|
||||
logger.debug(f"Container {s.name} is not running.")
|
||||
if labels.get(f'backupbot.{hook_type}.pre-hook') or labels.get(f'backupbot.{hook_type}.post-hook'):
|
||||
logger.error(f"Container {s.name} contain hooks but it's not running")
|
||||
return app_settings
|
||||
|
||||
|
||||
def get_backup_details(app_settings, volumes=[]):
|
||||
backup_paths = set()
|
||||
backup_apps_versions = {}
|
||||
pre_hooks= {}
|
||||
post_hooks = {}
|
||||
for app, settings in app_settings.items():
|
||||
if settings.get('enabled'):
|
||||
if SERVICE != 'ALL' and SERVICE != app:
|
||||
continue
|
||||
if prehook := labels.get('backupbot.backup.pre-hook'):
|
||||
pre_commands[container] = prehook
|
||||
if posthook := labels.get('backupbot.backup.post-hook'):
|
||||
post_commands[container] = posthook
|
||||
return pre_commands, post_commands, list(backup_paths), list(backup_apps)
|
||||
backup_apps_versions[app] = settings.get('version')
|
||||
add_backup_paths(backup_paths, settings, app, volumes)
|
||||
if hooks:= settings.get('pre_hooks'):
|
||||
pre_hooks.update(hooks)
|
||||
if hooks:= settings.get('post_hooks'):
|
||||
post_hooks.update(hooks)
|
||||
return pre_hooks, post_hooks, list(backup_paths), backup_apps_versions
|
||||
|
||||
|
||||
def add_backup_paths(backup_paths, settings, app, selected_volumes):
|
||||
if (volumes := settings.get('volumes')):
|
||||
if includes:= settings.get('included_volume_paths'):
|
||||
included_volumes = list(zip(*includes))[0]
|
||||
for volume, rel_paths in includes:
|
||||
if not (volume_path:= volumes.get(volume)):
|
||||
logger.error(f'Can not find volume with the name {volume}')
|
||||
continue
|
||||
if selected_volumes and volume not in selected_volumes:
|
||||
logger.debug(f'Skipping {volume}:{rel_paths} because the volume is not selected')
|
||||
continue
|
||||
for p in rel_paths:
|
||||
absolute_path = Path(f"{volume_path}/{p}")
|
||||
backup_paths.add(absolute_path)
|
||||
else:
|
||||
included_volumes = []
|
||||
excluded_volumes = settings.get('excluded_volumes') or []
|
||||
for name, path in volumes.items():
|
||||
if selected_volumes and name not in selected_volumes:
|
||||
logger.debug(f'Skipping volume: {name} because the volume is not selected')
|
||||
continue
|
||||
if name in excluded_volumes:
|
||||
logger.debug(f'Skipping volume: {name} because the volume is excluded')
|
||||
continue
|
||||
if name in included_volumes:
|
||||
logger.debug(f'Skipping volume: {name} because a path is selected')
|
||||
continue
|
||||
backup_paths.add(path)
|
||||
else:
|
||||
logger.warning(f"{app} does not contain any volumes")
|
||||
|
||||
|
||||
def parse_volumes(stack_name, mounts):
|
||||
volumes = {}
|
||||
for m in mounts:
|
||||
if m['Type'] != 'volume':
|
||||
continue
|
||||
relative_path = m['Source']
|
||||
name = relative_path.removeprefix(stack_name + '_')
|
||||
absolute_path = Path(f"{VOLUME_PATH}{relative_path}/_data/")
|
||||
volumes[name] = absolute_path
|
||||
return volumes
|
||||
|
||||
|
||||
def parse_excludes_includes(labels):
|
||||
excluded_volumes = set()
|
||||
included_volume_paths = set()
|
||||
for label, value in labels.items():
|
||||
if label.startswith('backupbot.backup.volumes.'):
|
||||
volume_name = label.removeprefix('backupbot.backup.volumes.').removesuffix('.path')
|
||||
if label.endswith('path'):
|
||||
relative_paths = tuple(value.split(','))
|
||||
included_volume_paths.add((volume_name, relative_paths))
|
||||
elif bool(value):
|
||||
excluded_volumes.add(volume_name)
|
||||
return excluded_volumes, included_volume_paths
|
||||
|
||||
|
||||
def copy_secrets(apps):
|
||||
@ -139,14 +313,14 @@ def copy_secrets(apps):
|
||||
os.mkdir(SECRET_PATH)
|
||||
client = docker.from_env()
|
||||
container_by_service = {
|
||||
c.labels['com.docker.swarm.service.name']: c for c in client.containers.list()}
|
||||
c.labels.get('com.docker.swarm.service.name'): c for c in client.containers.list()}
|
||||
services = client.services.list()
|
||||
for s in services:
|
||||
app_name = s.attrs['Spec']['Labels']['com.docker.stack.namespace']
|
||||
if (app_name in apps and
|
||||
(app_secs := s.attrs['Spec']['TaskTemplate']['ContainerSpec'].get('Secrets'))):
|
||||
if not container_by_service.get(s.name):
|
||||
logger.error(
|
||||
logger.warning(
|
||||
f"Container {s.name} is not running, secrets can not be copied.")
|
||||
continue
|
||||
container_id = container_by_service[s.name].id
|
||||
@ -157,6 +331,7 @@ def copy_secrets(apps):
|
||||
f"For the secret {sec['SecretName']} the file {src} does not exist for {s.name}")
|
||||
continue
|
||||
dst = SECRET_PATH + sec['SecretName']
|
||||
logger.debug(f"Copy Secret {sec['SecretName']}")
|
||||
copyfile(src, dst)
|
||||
|
||||
|
||||
@ -181,15 +356,26 @@ def run_commands(commands):
|
||||
logger.info(result.output.decode())
|
||||
|
||||
|
||||
def backup_volumes(backup_paths, apps, retries, dry_run=False):
|
||||
def backup_volumes(backup_paths, apps_versions, retries, dry_run=False):
|
||||
while True:
|
||||
try:
|
||||
result = restic.backup(backup_paths, dry_run=dry_run, tags=apps)
|
||||
logger.info("Backup these paths:")
|
||||
logger.debug("\n".join(map(str, backup_paths)))
|
||||
backup_paths = list(filter(path_exists, backup_paths))
|
||||
cmd = restic.cat.base_command()
|
||||
parent = get_snapshots('latest')
|
||||
if parent:
|
||||
# https://restic.readthedocs.io/en/stable/040_backup.html#file-change-detection
|
||||
cmd.extend(['--parent', parent[0]['short_id']])
|
||||
tags = [f"{app}:{version}" for app,version in apps_versions.items()]
|
||||
if SERVICE == 'ALL':
|
||||
tags.append(SERVICE)
|
||||
logger.info("Start volume backup")
|
||||
result = restic.internal.backup.run(cmd, backup_paths, dry_run=dry_run, tags=tags)
|
||||
logger.summary("backup finished", extra=result)
|
||||
return
|
||||
except ResticFailedError as error:
|
||||
logger.error(
|
||||
f"Backup failed for {apps}. Could not Backup these paths: {backup_paths}")
|
||||
logger.error(f"Backup failed for {SERVICE}.")
|
||||
logger.error(error, exc_info=True)
|
||||
if retries > 0:
|
||||
retries -= 1
|
||||
@ -197,49 +383,25 @@ def backup_volumes(backup_paths, apps, retries, dry_run=False):
|
||||
exit(1)
|
||||
|
||||
|
||||
@cli.command()
|
||||
@click.option('snapshot', '--snapshot', '-s', envvar='SNAPSHOT', default='latest')
|
||||
@click.option('target', '--target', '-t', envvar='TARGET', default='/')
|
||||
@click.option('noninteractive', '--noninteractive', envvar='NONINTERACTIVE', is_flag=True)
|
||||
def restore(snapshot, target, noninteractive):
|
||||
# Todo: recommend to shutdown the container
|
||||
service_paths = VOLUME_PATH
|
||||
if SERVICE:
|
||||
service_paths = service_paths + f'{SERVICE}_*'
|
||||
snapshots = restic.snapshots(snapshot_id=snapshot)
|
||||
if not snapshot:
|
||||
logger.error("No Snapshots with ID {snapshots}")
|
||||
exit(1)
|
||||
if not noninteractive:
|
||||
snapshot_date = datetime.fromisoformat(snapshots[0]['time'])
|
||||
delta = datetime.now(tz=timezone.utc) - snapshot_date
|
||||
print(
|
||||
f"You are going to restore Snapshot {snapshot} of {service_paths} at {target}")
|
||||
print(f"This snapshot is {delta} old")
|
||||
print(
|
||||
f"THIS COMMAND WILL IRREVERSIBLY OVERWRITES {target}{service_paths.removeprefix('/')}")
|
||||
prompt = input("Type YES (uppercase) to continue: ")
|
||||
if prompt != 'YES':
|
||||
logger.error("Restore aborted")
|
||||
exit(1)
|
||||
print(f"Restoring Snapshot {snapshot} of {service_paths} at {target}")
|
||||
# TODO: use tags if no snapshot is selected, to use a snapshot including SERVICE
|
||||
result = restic.restore(snapshot_id=snapshot,
|
||||
include=service_paths, target_dir=target)
|
||||
logger.debug(result)
|
||||
def path_exists(path):
|
||||
if not path.exists():
|
||||
logger.error(f'{path} does not exist')
|
||||
return path.exists()
|
||||
|
||||
|
||||
@cli.command()
|
||||
def snapshots():
|
||||
snapshots = restic.snapshots()
|
||||
no_snapshots = True
|
||||
snapshots = get_snapshots()
|
||||
for snap in snapshots:
|
||||
if not SERVICE or (tags := snap.get('tags')) and SERVICE in tags:
|
||||
print(snap['time'], snap['id'])
|
||||
no_snapshots = False
|
||||
if no_snapshots:
|
||||
output = [snap['time'], snap['id']]
|
||||
if tags:= snap.get('tags'):
|
||||
app_versions = app_versions_from_tags(tags)
|
||||
if version:= app_versions.get(SERVICE):
|
||||
output.append(version)
|
||||
print(*output)
|
||||
if not snapshots:
|
||||
err_msg = "No Snapshots found"
|
||||
if SERVICE:
|
||||
if SERVICE != 'ALL':
|
||||
service_name = SERVICE.replace('_', '.')
|
||||
err_msg += f' for app {service_name}'
|
||||
logger.warning(err_msg)
|
||||
@ -257,8 +419,12 @@ def ls(snapshot, path):
|
||||
|
||||
def list_files(snapshot, path):
|
||||
cmd = restic.cat.base_command() + ['ls']
|
||||
if SERVICE:
|
||||
cmd = cmd + ['--tag', SERVICE]
|
||||
if snapshot == 'latest':
|
||||
latest_snapshot = get_snapshots('latest')
|
||||
if not latest_snapshot:
|
||||
logger.error(f"There is no latest snapshot for {SERVICE}")
|
||||
exit(1)
|
||||
snapshot = latest_snapshot[0]['short_id']
|
||||
cmd.append(snapshot)
|
||||
if path:
|
||||
cmd.append(path)
|
||||
@ -267,7 +433,7 @@ def list_files(snapshot, path):
|
||||
except ResticFailedError as error:
|
||||
if 'no snapshot found' in str(error):
|
||||
err_msg = f'There is no snapshot "{snapshot}"'
|
||||
if SERVICE:
|
||||
if SERVICE != 'ALL':
|
||||
err_msg += f' for the app "{SERVICE}"'
|
||||
logger.error(err_msg)
|
||||
exit(1)
|
||||
@ -299,7 +465,7 @@ def download(snapshot, path, volumes, secrets):
|
||||
tarinfo.size = len(binary_output)
|
||||
file_dumps.append((binary_output, tarinfo))
|
||||
if volumes:
|
||||
if not SERVICE:
|
||||
if SERVICE == 'ALL':
|
||||
logger.error("Please specify '--host' when using '--volumes'")
|
||||
exit(1)
|
||||
files = list_files(snapshot, VOLUME_PATH)
|
||||
@ -312,7 +478,7 @@ def download(snapshot, path, volumes, secrets):
|
||||
tarinfo.size = len(binary_output)
|
||||
file_dumps.append((binary_output, tarinfo))
|
||||
if secrets:
|
||||
if not SERVICE:
|
||||
if SERVICE == 'ALL':
|
||||
logger.error("Please specify '--host' when using '--secrets'")
|
||||
exit(1)
|
||||
filename = f"{SERVICE}.json"
|
||||
@ -349,8 +515,13 @@ def get_formatted_size(file_path):
|
||||
|
||||
def dump(snapshot, path):
|
||||
cmd = restic.cat.base_command() + ['dump']
|
||||
if SERVICE:
|
||||
cmd = cmd + ['--tag', SERVICE]
|
||||
if snapshot == 'latest':
|
||||
latest_snapshot = get_snapshots('latest')
|
||||
if not latest_snapshot:
|
||||
logger.error(f"There is no latest snapshot for {SERVICE}")
|
||||
exit(1)
|
||||
snapshot = latest_snapshot[0]['short_id']
|
||||
cmd.append(snapshot)
|
||||
cmd = cmd + [snapshot, path]
|
||||
print(f"Dumping {path} from snapshot '{snapshot}'")
|
||||
output = subprocess.run(cmd, capture_output=True)
|
||||
|
22
compose.yml
22
compose.yml
@ -2,7 +2,7 @@
|
||||
version: "3.8"
|
||||
services:
|
||||
app:
|
||||
image: docker:24.0.7-dind
|
||||
image: git.coopcloud.tech/coop-cloud/backup-bot-two:2.2.1-beta
|
||||
volumes:
|
||||
- "/var/run/docker.sock:/var/run/docker.sock"
|
||||
- "/var/lib/docker/volumes/:/var/lib/docker/volumes/"
|
||||
@ -16,18 +16,10 @@ services:
|
||||
- restic_password
|
||||
deploy:
|
||||
labels:
|
||||
- coop-cloud.${STACK_NAME}.version=0.1.0+latest
|
||||
- coop-cloud.${STACK_NAME}.version=2.2.0+2.2.1-beta
|
||||
- coop-cloud.${STACK_NAME}.timeout=${TIMEOUT:-300}
|
||||
- coop-cloud.backupbot.enabled=true
|
||||
configs:
|
||||
- source: entrypoint
|
||||
target: /entrypoint.sh
|
||||
mode: 0555
|
||||
- source: backupbot
|
||||
target: /usr/bin/backup
|
||||
mode: 0555
|
||||
entrypoint: ['/entrypoint.sh']
|
||||
#entrypoint: ['tail', '-f','/dev/null']
|
||||
#entrypoint: ['tail', '-f','/dev/null']
|
||||
healthcheck:
|
||||
test: "pgrep crond"
|
||||
interval: 30s
|
||||
@ -42,11 +34,3 @@ secrets:
|
||||
|
||||
volumes:
|
||||
backups:
|
||||
|
||||
configs:
|
||||
entrypoint:
|
||||
name: ${STACK_NAME}_entrypoint_${ENTRYPOINT_VERSION}
|
||||
file: entrypoint.sh
|
||||
backupbot:
|
||||
name: ${STACK_NAME}_backupbot_${BACKUPBOT_VERSION}
|
||||
file: backupbot.py
|
||||
|
7
entrypoint.sh
Normal file → Executable file
7
entrypoint.sh
Normal file → Executable file
@ -1,11 +1,6 @@
|
||||
#!/bin/sh
|
||||
|
||||
set -e -o pipefail
|
||||
|
||||
apk add --upgrade --no-cache restic bash python3 py3-pip py3-click py3-docker-py py3-json-logger curl
|
||||
|
||||
# Todo use requirements file with specific versions
|
||||
pip install --break-system-packages resticpy==1.0.2
|
||||
set -e
|
||||
|
||||
if [ -n "$SSH_HOST_KEY" ]
|
||||
then
|
||||
|
1
release/1.0.0+2.0.0-beta
Normal file
1
release/1.0.0+2.0.0-beta
Normal file
@ -0,0 +1 @@
|
||||
This is the first beta release of the new backup-bot-two rewrite in python. Be aware when updating, it can break. Please read the readme and update your config according to it.
|
Reference in New Issue
Block a user