fix: download without buffering volumes in RAM #92

2026-07-10T13:11:00Z

dannygroenewegen commented

2026-07-10 13:11:00 +00:00

backup download dumps every volume with subprocess.run(capture_output=True) and keeps all dumps in memory at the same time before writing /tmp/backup.tar.gz. For apps with large volumes, this exhausts the host's RAM.

This PR fixes the download function to fetch each path with restic restore into a temp directory, pack it into a <volume>.tar file, and compress everything into the final backup.tar.gz. By using restic restore instead of streaming restic dump, memory usage stays low and it's also much faster: restore downloads with parallel workers, while dump fetches blobs sequentially. Measured against an S3 repo: restore ~22MiB/s vs dump ~0.8MiB/s.

The final archive layout is unchanged. Verified by diffing old vs new output for the same snapshot: names, inner tar listings (permissions, sizes, mtimes) and extracted contents are identical.

`backup download` dumps every volume with `subprocess.run(capture_output=True)` and keeps all dumps in memory at the same time before writing `/tmp/backup.tar.gz`. For apps with large volumes, this exhausts the host's RAM. This PR fixes the download function to fetch each path with `restic restore` into a temp directory, pack it into a `<volume>.tar` file, and compress everything into the final backup.tar.gz. By using `restic restore` instead of streaming `restic dump`, memory usage stays low and it's also much faster: restore downloads with parallel workers, while dump fetches blobs sequentially. Measured against an S3 repo: restore ~22MiB/s vs dump ~0.8MiB/s. The final archive layout is unchanged. Verified by diffing old vs new output for the same snapshot: names, inner tar listings (permissions, sizes, mtimes) and extracted contents are identical.

❤️ 2

dannygroenewegen added 1 commit 2026-07-10 13:11:01 +00:00

fix: download without buffering volumes in RAM

continuous-integration/drone/pr Build is passing

Details

fc5c6989e8

Fetch each path with 'restic restore' into a temp dir, instead of
buffering whole volume dumps in memory, which crashed on large volumes.
Restore is also much faster than the sequential 'restic dump'.
The archive layout is unchanged.

continuous-integration/drone/pr Build is passing

Details

This pull request can be merged automatically.

You are not authorized to merge this pull request.

View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.

git fetch -u https://git.coopcloud.tech/eCommons/backup-bot-two fix-download-oom:eCommons-fix-download-oom

git checkout eCommons-fix-download-oom

Sign in to join this conversation.

1 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: coop-cloud/backup-bot-two#92

Allow edits from maintainers