support multi-hdd config #5

New Issue

2026-04-02T16:36:57Z

fauno commented

2026-04-02 16:36:57 +00:00

how could something like this could be achieved via .env and garage.toml.templ? or include/concatenate an external file with custom configurations?

https://garagehq.deuxfleurs.fr/documentation/operations/multi-hdd/

data_dir = [
    { path = "/path/to/old_data", read_only = true },
    { path = "/path/to/new_hdd1", capacity = "2T" },
    { path = "/path/to/new_hdd2", capacity = "4T" },
]

how could something like this could be achieved via `.env` and `garage.toml.templ`? or include/concatenate an external file with custom configurations? https://garagehq.deuxfleurs.fr/documentation/operations/multi-hdd/ ``` data_dir = [ { path = "/path/to/old_data", read_only = true }, { path = "/path/to/new_hdd1", capacity = "2T" }, { path = "/path/to/new_hdd2", capacity = "4T" }, ] ```

decentral1se commented

2026-04-02 18:39:08 +00:00

If you put data_dir = [{{ env "UGLY_DATA_DIR_DUMP" }}] and then your .tmpl file and then just dump a super ugly '{ path = "/path/to/old_data", read_only = true }, { path = "/path/to/new_hdd1", capacity = "2T" }, { path = "/path/to/new_hdd2", capacity = "4T" },' then you're done? This might be guaranteed to cause further headaches 😬 I can't quite think of a recipe that deals with a complex config setup like this 🤔

If you put `data_dir = [{{ env "UGLY_DATA_DIR_DUMP" }}]` and then your `.tmpl` file and then just dump a super ugly `'{ path = "/path/to/old_data", read_only = true }, { path = "/path/to/new_hdd1", capacity = "2T" }, { path = "/path/to/new_hdd2", capacity = "4T" },'` then you're done? This might be guaranteed to cause further headaches 😬 I can't quite think of a recipe that deals with a complex config setup like this 🤔

fauno commented

2026-04-02 18:45:47 +00:00

thanks! but yeah we'll be dealing with quote-escaping issues then, no?

👍 1

decentral1se commented

2026-04-02 18:47:00 +00:00

Yes, you'd need to check the usage of ' and ". You could potentially escape \" on the .env side. There might be some obscure functionality in Golang templating to help you out here but I wouldn't know. Good luck!

Yes, you'd need to check the usage of `'` and `"`. You could potentially escape `\"` on the `.env` side. There might be some obscure functionality in Golang templating to help you out here but I wouldn't know. Good luck!

fauno commented

2026-04-02 19:02:34 +00:00

@decentral1se what's the templating called? i seem to find a lot of these for golang and never the official reference

decentral1se commented

2026-04-02 20:22:34 +00:00

It's a disaster in terms of docs: https://pkg.go.dev/text/template#hdr-Actions I just search online and read what other people did. Looking at other recipes... there are even features which are only implemented in swarms version of the templating which aren't even supported in the compose spec. Dog bless us all.

fauno commented

2026-04-03 15:00:00 +00:00

besides templating issues, i'm thinking disks could be mounted on $LOCAL_FOLDER_DATA/diskX so they example config would be like this:

data_dir = [
    { path = "/var/lib/garage/data/disk1", read_only = true },
    { path = "/var/lib/garage/data/disk2", capacity = "2T" },
    { path = "/var/lib/garage/data/disk3", capacity = "4T" },
]

but i'm not sure if docker will show the nested mount points to the container.

reference:

https://git.coopcloud.tech/coop-cloud/garage/src/branch/main/compose.yml#L33

besides templating issues, i'm thinking disks could be mounted on `$LOCAL_FOLDER_DATA/diskX` so they example config would be like this: ``` data_dir = [ { path = "/var/lib/garage/data/disk1", read_only = true }, { path = "/var/lib/garage/data/disk2", capacity = "2T" }, { path = "/var/lib/garage/data/disk3", capacity = "4T" }, ] ``` but i'm not sure if docker will show the nested mount points to the container. reference: https://git.coopcloud.tech/coop-cloud/garage/src/branch/main/compose.yml#L33

fauno commented

2026-04-03 15:40:26 +00:00

looks like it would work, so no changes needed to compose.yml:

mkdir local_folder_data

for i in 1 2; do
  dd if=/dev/zero of=disk$i.img bs=1M count=100
  losetup /dev/loop$i disk$i.img
  mkfs.ext4 /dev/loop$i
  mkdir -p local_folder_data/disk$i
  mount /dev/loop$i local_folder_data/disk$i
done

docker run -v ./local_folder_data:/srv -it --rm alpine:latest sh

# now inside the container
touch /srv/disk1/hi
touch /srv/disk2/hi
exit

# back to host
ls -hal local_folder_data/disk*

for i in 1 2; do
  umount local_folder_data/disk$i
  losetup -d /dev/loop$i
done

looks like it would work, so no changes needed to `compose.yml`: ```sh mkdir local_folder_data for i in 1 2; do dd if=/dev/zero of=disk$i.img bs=1M count=100 losetup /dev/loop$i disk$i.img mkfs.ext4 /dev/loop$i mkdir -p local_folder_data/disk$i mount /dev/loop$i local_folder_data/disk$i done docker run -v ./local_folder_data:/srv -it --rm alpine:latest sh # now inside the container touch /srv/disk1/hi touch /srv/disk2/hi exit # back to host ls -hal local_folder_data/disk* for i in 1 2; do umount local_folder_data/disk$i losetup -d /dev/loop$i done ```

Brooke added this to the Issue tracking project 2026-05-08 19:29:39 +00:00

Brooke added the Kind/Enhancement

Priority

Low

4

labels 2026-05-15 20:19:05 +00:00

Brooke commented

2026-06-17 22:24:53 +00:00

looks like it would work, so no changes needed to compose.yml:

mkdir local_folder_data

for i in 1 2; do
  dd if=/dev/zero of=disk$i.img bs=1M count=100
  losetup /dev/loop$i disk$i.img
  mkfs.ext4 /dev/loop$i
  mkdir -p local_folder_data/disk$i
  mount /dev/loop$i local_folder_data/disk$i
done

docker run -v ./local_folder_data:/srv -it --rm alpine:latest sh

# now inside the container
touch /srv/disk1/hi
touch /srv/disk2/hi
exit

# back to host
ls -hal local_folder_data/disk*

for i in 1 2; do
  umount local_folder_data/disk$i
  losetup -d /dev/loop$i
done

I'm confused about the purpose of this script, what is the suggested change to the recipe to support multi-hdd?

> looks like it would work, so no changes needed to `compose.yml`: > > ```sh > mkdir local_folder_data > > for i in 1 2; do > dd if=/dev/zero of=disk$i.img bs=1M count=100 > losetup /dev/loop$i disk$i.img > mkfs.ext4 /dev/loop$i > mkdir -p local_folder_data/disk$i > mount /dev/loop$i local_folder_data/disk$i > done > > docker run -v ./local_folder_data:/srv -it --rm alpine:latest sh > > # now inside the container > touch /srv/disk1/hi > touch /srv/disk2/hi > exit > > # back to host > ls -hal local_folder_data/disk* > > for i in 1 2; do > umount local_folder_data/disk$i > losetup -d /dev/loop$i > done > ``` I'm confused about the purpose of this script, what is the suggested change to the recipe to support multi-hdd?

Brooke commented

2026-06-17 23:16:34 +00:00

I was thinking through this, what do people think about a syntax like:

MULTI_DISKS_ROOT=/mnt
MULTI_DISKS="disk1:2T:false,disk2:4T:false,disk3:1T:true,disk4:8T:false"

Where disks are set with a defined format and assumed to be relative to the set root, this way there's not a ton of complex changes to compose.yml

I'm still working on it but I believe that I have a working tmpl file that can ingest this syntax.

I was thinking through this, what do people think about a syntax like: ``` MULTI_DISKS_ROOT=/mnt MULTI_DISKS="disk1:2T:false,disk2:4T:false,disk3:1T:true,disk4:8T:false" ``` Where disks are set with a defined format and assumed to be relative to the set root, this way there's not a ton of complex changes to compose.yml I'm still working on it but I believe that I have a working tmpl file that can ingest this syntax.

fauno commented

2026-06-18 15:50:25 +00:00

looks great @Brooke! the script was an experiment to see if bind mounting partitions into a volume would make them available to the container, which seemed to be the case and you can render a template like this #5 (comment) without changes to the compose file

looks great @Brooke! the script was an experiment to see if bind mounting partitions into a volume would make them available to the container, which seemed to be the case and you can render a template like this https://git.coopcloud.tech/coop-cloud/garage/issues/5#issuecomment-32693 without changes to the compose file

👍 1

fauno commented

2026-10-17 16:27:12 +00:00

@Brooke could you push your changes? we have a garage node ready to test this!

Brooke commented

2026-10-21 14:58:03 +00:00

Sorry I will have to check in on this later, currently traveling. The thing I was working on wasn't totally working but will certainly put it near the top of my todo for next week.

Brooke commented

2026-10-28 07:13:57 +00:00

Ok so going through my notes the issue I was running into was the lack of split in the built-in golang interpreter for docker swarm. This makes it impossible(?) to parse a syntax like I was suggesting.

I had thought that it might be possible to setup an entrypoint that handles it instead but I forgot that garage uses a scratch container so there's no bash.

The truly cursed solution is a helper container that mounts the same volumes available to garage but readonly. This would eleviate the distroless issue #1

Ok so going through my notes the issue I was running into was the lack of split in the built-in golang interpreter for docker swarm. This makes it impossible(?) to parse a syntax like I was suggesting. I had thought that it might be possible to setup an entrypoint that handles it instead but I forgot that garage uses a scratch container so there's no bash. The truly cursed solution is a helper container that mounts the same volumes available to garage but readonly. This would eleviate the distroless issue https://git.coopcloud.tech/coop-cloud/garage/issues/1

Brooke commented

2026-10-28 07:14:50 +00:00

I'll see if I can get a working branch done today that just has a second alpine container for running backups and setting up multi-hdd but I'm unsure how I feel about it as a long term solution.

fauno commented

2026-10-28 12:14:53 +00:00

have you seen this? maybe it helps with attaching an alpine container to the same garage container? i really think we should be able to split and traverse arrays/lists on templates, ping @decentral1se ?

https://garagehq.deuxfleurs.fr/documentation/operations/debugging/

have you seen this? maybe it helps with attaching an alpine container to the same garage container? i really think we should be able to split and traverse arrays/lists on templates, ping @decentral1se ? https://garagehq.deuxfleurs.fr/documentation/operations/debugging/

Brooke commented

2026-10-29 05:49:43 +00:00

I agree re: split, it's extremely useful for this exact purpose when we're confining users to only editing .env.

I chose the alpine container cause I was following the debugging page from when you suggested it previously, I think the only issue is ensuring that depends_on properly stalls the garage container from starting. I've read conflicting posts abt depends_on with swarm, but it seems like it works?

I agree re: split, it's extremely useful for this exact purpose when we're confining users to only editing .env. I chose the alpine container cause I was following the debugging page from when you suggested it previously, I think the only issue is ensuring that depends_on properly stalls the garage container from starting. I've read conflicting posts abt depends_on with swarm, but it seems like it works?

Brooke commented

2026-10-30 04:18:34 +00:00

Ok this may be blocked by missing split. the configuration is handled as a config within swarm/stack so obviously you can't actually touch it until the container has started and once it's started we risk causing a restart of the container or just launching garage in a degraded state by not having a consistent value for data_dirs.

Unless we change the config after the container starts and trigger the daemon to restart? But at this point it's very hacky. I'm still going to try and get backupbot working with an alpine helper container.

Ok this may be blocked by missing split. the configuration is handled as a config within swarm/stack so obviously you can't actually touch it until the container has started and once it's started we risk causing a restart of the container or just launching garage in a degraded state by not having a consistent value for `data_dirs`. Unless we change the config after the container starts and trigger the daemon to restart? But at this point it's very hacky. I'm still going to try and get backupbot working with an alpine helper container.

Brooke moved this to Waiting/Blocked in Issue tracking on 2026-10-30 04:27:44 +00:00

fauno commented

2026-10-30 13:18:07 +00:00

i'm assigning @decentral1se to get their input on the possibility of adding split support to abra or how would it work

decentral1se was assigned by fauno

2026-10-30 13:18:18 +00:00

fauno commented

2026-10-30 13:19:22 +00:00

ps: i think we need an abra-developers team so we don't ping individual people

👍 1

fauno commented

2026-10-30 17:48:53 +00:00

i can't find any official documentation on what the template engine for docker swarm supports or not. another case for abra rendering templates rather than docker :)

decentral1se commented

2026-10-31 17:11:30 +00:00

There was no templating support implemented explicitly in the .env to maintain compatibility with other tools that consume the .env format. This has been debated over the years and even there is some movement around wanting a reciipe.yaml per recipe instead of an .env.example. I don't know where this will finish but I think adding some special case parsing implementation into the .env file would be controversial. I won't be working on this any time soon (unless funding miracle), so I would say the best option is to request support in the Abra Hackers channel or create an issue on the abra repo with a proposal so it is tracked. Good luck!

There was no templating support implemented explicitly in the `.env` to maintain compatibility with other tools that consume the `.env` format. This has been debated over the years and even there is some movement around wanting a `reciipe.yaml` per recipe instead of an `.env.example`. I don't know where this will finish but I think adding some special case parsing implementation into the `.env` file would be controversial. I won't be working on this any time soon (unless funding miracle), so I would say the best option is to request support in the Abra Hackers channel or create an issue on the abra repo with a proposal so it is tracked. Good luck!

decentral1se removed their assignment 2026-10-31 17:12:14 +00:00

decentral1se commented

2026-10-31 17:12:50 +00:00

i can't find any official documentation on what the template engine for docker swarm supports or not.

#5 (comment)

> i can't find any official documentation on what the template engine for docker swarm supports or not. https://git.coopcloud.tech/coop-cloud/garage/issues/5#issuecomment-32674

fauno referenced this issue from toolshed/organising

2026-11-02 13:14:55 +00:00

The `recipe.yaml` idea #695

fauno commented

2026-11-02 13:15:43 +00:00