forked from toolshed/docs.coopcloud.tech
Compare commits
1
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
 Amras
|
0c804a382b |
+1
-1
@@ -45,7 +45,7 @@ use it! We could really use your input.
|
||||
|
||||
| Feature | Explanation |
|
||||
| ----------- | ----------- |
|
||||
| Multi-node | It is possible but it doesn't seem like anyone in our community is really doing this? Please report in to `#coop-cloud-tech-future-brain:autonomic.zone` to discuss your usage if you are using multi-node swarm! We believe the majority of Co-op Cloud installs are single node. There is also a lack of [CSI](https://github.com/olljanat/csi-plugins-for-docker-swarm?tab=readme-ov-file) support for coordinating storage across multiple hosts when using Swarm mode. This means we kind of throw out [a bunch](https://docs.docker.com/engine/swarm/#feature-highlights) of the features of Swarm mode. |
|
||||
| Multi-node | The vast majority of Co-op Cloud installs are single node. There is a lack of [CSI](https://github.com/olljanat/csi-plugins-for-docker-swarm?tab=readme-ov-file) support for coordinating storage across multiple hosts when using Swarm mode, which means we kind of throw out [a bunch](https://docs.docker.com/engine/swarm/#feature-highlights) of the features of Swarm mode. However, there is growing interest in multinode setups and a push to improve our understanding. [Here's what we know so far, and some hints on getting started.](/operators/multinode) |
|
||||
|
||||
## Limitations
|
||||
|
||||
|
||||
@@ -4,26 +4,27 @@ title: Membership
|
||||
|
||||
> Are you also interested in joining the federation? Please check [our process](/federation/join/) on how to join. If you have any questions, [drop us a line](/intro/contact/) or come for a chat.
|
||||
|
||||
| Name | Dues Paid | Notes | Contact | Status |
|
||||
| --------- | --------- | -------- |-------- | -------- |
|
||||
| [Agaric](https://agaric.coop/) | ⭕ | - | `@wolcen:matrix.org` | Paused |
|
||||
| [Autonomic](https://autonomic.zone) | ⭕ | - | `@kawaiipunk:autonomic.zone` | Active |
|
||||
| [Bonfire](https://bonfirenetworks.org) | ✅ | - | `@mayel:matrix.org`) | Active |
|
||||
| [Bunk Computer Cooperative](https://bunk.computer) | ✅ | - | `@maren@bunk.computer` | Active |
|
||||
| [CoQuest - IT Coop Stuttgart](https://coquest.coop/en) | ✅ | voting rights waived | `hallo@coquest.coop` | Active |
|
||||
| [Democratic Tech Fund](https://democratictech.fund/) | ✅ | - | `@mike_hales:matrix.org` | Active |
|
||||
| [Doop.coop](https://doop.coop) | ⭕ | - | `@yusf:gottsnack.net` | Active |
|
||||
| [eCommons](https://ecommons.nl) | ✅ | - | `@dannygroenewegen:matrix.org` | Active |
|
||||
| [EOTL](https://eotl.supply) | ✅ | - | `@basebuilder:pub.solar` | Paused |
|
||||
| [Karrot](https://karrot.world) | ⭕ | - | `@nicksellen:matrix.org` | Active |
|
||||
| [Klasse & Methode](https://klasse-methode.it) | Waiver | - | `@p4u1_f4u1:matrix.org` | Active |
|
||||
| [Local IT](https://local-it.org/) | ✅ | - | `@simon_sth:matrix.org`| Active |
|
||||
| [Merri-bek tech](https://www.merri-bek.tech/) | ✅ | - | `@sarah@merri-bek.tech`| Active |
|
||||
| [MIR](https://mirnet.org/) | ✅ | - | `@sixsmith:matrix.org` | Active |
|
||||
| [Red Abya Yala](https://abyayala.sutty.nl/) | ⭕ | - | `@fauno:sutty.nl` | Active |
|
||||
| [RTM](https://resisttechmonopolies.online) | ✅ | - | `@ammaratef45:matrix.org`| Active |
|
||||
| [ruangrupa](https://ruangrupa.id) | ⭕ | - | `@babystepper:matrix.org` | Paused |
|
||||
| [UTAW](https://utaw.tech) | ⭕ | - | `@javielico:matrix.org` | Paused |
|
||||
| Amras | Waiver | - | `coop-cloud@joinmeonmy.quest` | Active |
|
||||
| Mirsal ™ | ✅ | - | `@mirsal:1312.media` | Active |
|
||||
| Name | Dues Paid | Notes | Contact |
|
||||
| --------- | --------- | -------- |-------- |
|
||||
| [Agaric](https://agaric.coop/) | ⭕ | - | `@wolcen:matrix.org` |
|
||||
| [Autonomic](https://autonomic.zone) | ⭕ | - | `@3wc`, `@cas`, `@knoflook`, `@travvy`, `@aadil` |
|
||||
| [Bonfire](https://bonfirenetworks.org) | ✅ | - | `@mayel:matrix.org` + Ivan (`@cambriale:matrix.org`) |
|
||||
| [Bunk Computer Cooperative](https://bunk.computer) | ✅ | - | `@maren@bunk.computer` + `@sorrel@bunk.computer` |
|
||||
| [CoQuest - IT Coop Stuttgart](https://coquest.coop/en) | ✅ | voting rights waived | `hallo@coquest.coop` |
|
||||
| [Democratic Tech Fund](https://democratictech.fund/) | ✅ | - | `@wtebbens` + `@mikemh` |
|
||||
| [Doop.coop](https://doop.coop) | ⭕ | - | `@yusf:gottsnack.net` |
|
||||
| [eCommons](https://ecommons.nl) | ✅ | - | `@dannygroenewegen:matrix.org` |
|
||||
| [EOTL](https://eotl.supply) | ✅ | - | `@basebuilder:pub.solar` |
|
||||
| [Karrot](https://karrot.world) | ⭕ | - | `@nicksellen:matrix.org` |
|
||||
| [Klasse & Methode](https://klasse-methode.it) | Waiver | - | `@p4u1_f4u1:matrix.org` |
|
||||
| [Local IT](https://local-it.org/) | ✅ | - | `@moritz:matrix.local-it.org` + `@simon_sth:matrix.org`|
|
||||
| [Merri-bek tech](https://www.merri-bek.tech/) | ✅ | - | `coop-cloud-delegate@merri-bek.tech`|
|
||||
| [MIR](https://mirnet.org/) | ✅ | - | `@sixsmith:matrix.org` |
|
||||
| [Red Abya Yala](https://abyayala.sutty.nl/) | ⭕ | - | `@fauno:sutty.nl` |
|
||||
| [RTM](https://resisttechmonopolies.online) | ✅ | - | `@ammaratef45:matrix.org` + `@linnealovespie:matrix.org`|
|
||||
| [ruangrupa](https://ruangrupa.id) | ⭕ | - | Henry `@babystepper:matrix.org` |
|
||||
| [UTAW](https://utaw.tech) | ⭕ | - | `@javielico:matrix.org` |
|
||||
| Amras | Waiver | - | `coop-cloud@joinmeonmy.quest` |
|
||||
| `@decentral1se` | Waiver | - | `@decentral1se` |
|
||||
| Mirsal ™ | ✅ | - | `@mirsal:1312.media` |
|
||||
|
||||
|
||||
@@ -1,32 +0,0 @@
|
||||
---
|
||||
title: "Resolution 046: Restore servers.coop as a Coop Cloud project"
|
||||
---
|
||||
|
||||
- Topic: Restore servers.coop as a Coop Cloud project
|
||||
- Date: TBD
|
||||
- Deadline: 2 weeks after voting starts
|
||||
- Size: medium
|
||||
|
||||
### History
|
||||
|
||||
[servers.coop](https://servers.coop) was launched as a project of Coop Cloud, intended to cooperatise lower down the stack than the abra toolshed. Similarly to how abra gives small cooperatives the tools to easily and collaboratively deploy services on a cloud, servers.coop is meant to support small cooperatives in deploying VPSs on hardware owned or managed by them.
|
||||
|
||||
The project was left on the backburner for some years. Recently, we were motivated by rising data center costs and concerns about digital sovereignty to start working on the project again. A working group of four-ish people from the Coop Cloud community is currently working on defining and deploying an MVP.
|
||||
|
||||
Servers.coop goals are closely aligned with those of Coop Cloud as a whole, but its usecases and userbase may end up diverging from those of abra. In the long term, servers.coop ought to become an independent collective, with its own governance, membership, budget, and infrastructure - similarly to how Coop Cloud grew from an Autonomic project into a distinct federation. In the short term, servers.coop needs the support of the Coop Cloud federation.
|
||||
|
||||
### Proposal
|
||||
|
||||
Formally, we treat servers.coop as a project of the Coop cloud federation.
|
||||
|
||||
This grants the servers.coop project access to Coop Cloud's resources. For instance:
|
||||
|
||||
- Coop Cloud server infrastructure can be used to organize and manage the project, including:
|
||||
- conferencing or notetaking services provided by members for Coop Cloud use
|
||||
- git.coopcloud.tech
|
||||
- docs.coopcloud.tech
|
||||
- Grants written on behalf of Coop Cloud can include servers.coop in their goals and budget.
|
||||
- Coop Cloud federation members can propose and vote on resolutions regarding servers.coop governance, infrastructure, and budget.
|
||||
- Coop Cloud budget can be allocated to servers.coop goals, through resolutions.
|
||||
|
||||
This support is intended to be temporary. Over the course of several years, the group working on/with servers.coop ought to move its resources onto dedicated instances and establish its own structure of governance and financing. When this happens, a new resolution should be passed to formally establish the new collective and change how Coop Cloud participates in the project.
|
||||
@@ -1,18 +0,0 @@
|
||||
---
|
||||
title: "Resolution 046: 3wc to join the federation as an individual member"
|
||||
---
|
||||
|
||||
- Topic: Resolution 046: 3wc to join the federation as an individual member
|
||||
- Date: 2026-09-30
|
||||
- Deadline: 2026-10-15
|
||||
- Size: large
|
||||
|
||||
## Summary
|
||||
|
||||
3wc to join the federation as an individual member
|
||||
|
||||
## Details
|
||||
|
||||
@3wc:autonomic.zone is an active member of our community and supports with some of the ongoing work. they were a member of one of our cooperative members (autonomic) but they are no longer part of the cooperative but would like to stay in the cooperative as an individual member.
|
||||
|
||||
@ammaratef45:matrix.org from [RTM](https://resisttechmonopolies.online/) is happy to vouch.
|
||||
@@ -1,28 +0,0 @@
|
||||
---
|
||||
title: "Resolution 047: Kolektivo Sennacieca Cifereca to join the federation as a member"
|
||||
---
|
||||
|
||||
- Topic: Resolution 047: Kolektivo Sennacieca Cifereca to join the federation as a member
|
||||
- Date: 2026-10-06
|
||||
- Deadline: 2026-10-20
|
||||
- Size: large
|
||||
|
||||
## Summary
|
||||
|
||||
[Kolektivo Sennacieca Cifereca](https://sekuro.nl) is a security and organizational consultancy responsible for the proof of concept [Hestia](https://git.coopcloud.tech/hex/hestia) build quality and security analyzer and [CCSATool](https://git.coopcloud.tech/hex/CCSATool) security advisory tool. KSC is focused on the improving the security of the platform.
|
||||
|
||||
## Details
|
||||
|
||||
At Kolektivo Sennacieca Cifereca we provide security consulting, security reviews, security program guidance, and penetration testing services. We would like to use and recommend services provided by Co-op Cloud member organizations, and thus have a vested interest in driving security improvements within the federation.
|
||||
|
||||
We have worked closely with eCommons, providing security guidance and identifying opportunities for improvement at the federation level (recipe scanning via Hestia, and security advisory announcement via CCSATool).
|
||||
|
||||
We agree with eCommon's assessment:
|
||||
|
||||
> Co-op Cloud's federated model, tooling maintained collectively by the co-ops that use it, matches how we think infrastructure should be built and sustained.
|
||||
|
||||
Further, we believe that distributed systems may present additional challenges related to pushing forward security initiatives, but they provide increased resilience against state-level threat actors with the goal of shutting down infrastructure by geographical region or ideological alignment.
|
||||
|
||||
@dannygroenewegen:matrix.org from eCommons has agreed to vouch for this organization.
|
||||
|
||||
Kolektivo Sennacieca Cifereca's representative to the federation will be hex. He can be reached at `@hex:matrix.org` on Matrix or `biz@sekuro.nl` by email.
|
||||
@@ -365,42 +365,39 @@ If we run `abra recipe upgrade wordpress` (at time of running), we end up with a
|
||||
|
||||
```
|
||||
$ abra recipe upgrade wordpress
|
||||
? upgrade to which tag? (service: app, image: wordpress, tag: 7.1.2) 7.1.3
|
||||
INFO tag upgraded and pinned from 7.1.2 to 7.1.3@sha256:8ecd8d8e374167ea22235acf30f80d1165b82f07ffa5bcef90c1cad8f7be251f for wordpress
|
||||
INFO no new versions available for mariadb, assuming 13.0 is the latest (use -a/--all-tags to see all anyway)
|
||||
INFO tag upgraded and pinned from 13.0 to 13.0@sha256:f1bba652ba57bea3099ca2fe1af692af537c27d96e0bcde39dce29e2ba1ec4f3 for mariadb
|
||||
WARN unable to parse atmoz/sftp, error was: version alpine is not supported (matched: ^[a-zA-Z]+$), skipping upgrade for ftp
|
||||
? upgrade to which tag? (service: app, image: wordpress, tag: 6.9.0) 6.9.1
|
||||
? upgrade to which tag? (service: db, image: mariadb, tag: 12.1) 12.2
|
||||
WARN unable to read tag for image atmoz/sftp, is it missing? skipping upgrade for ftp
|
||||
INFO wordpress currently has these unstaged changes 👇
|
||||
diff --git a/compose.yml b/compose.yml
|
||||
index 591d157..99d5eb0 100644
|
||||
index b3c3871..07d0a54 100644
|
||||
--- a/compose.yml
|
||||
+++ b/compose.yml
|
||||
@@ -3,7 +3,7 @@ version: "3.8"
|
||||
|
||||
services:
|
||||
app:
|
||||
- image: "wordpress:7.1.2"
|
||||
+ image: "wordpress:7.1.3@sha256:8ecd8d8e374167ea22235acf30f80d1165b82f07ffa5bcef90c1cad8f7be251f"
|
||||
- image: "wordpress:6.9.0"
|
||||
+ image: "wordpress:6.9.1"
|
||||
volumes:
|
||||
- "wordpress_content:/var/www/html/wp-content/"
|
||||
networks:
|
||||
@@ -66,7 +66,7 @@ services:
|
||||
- "coop-cloud.${STACK_NAME}.version=4.1.0+7.1.2"
|
||||
@@ -65,7 +65,7 @@ services:
|
||||
- "coop-cloud.${STACK_NAME}.version=2.17.0+6.9.0"
|
||||
|
||||
db:
|
||||
- image: "mariadb:13.0"
|
||||
+ image: "mariadb:13.0@sha256:f1bba652ba57bea3099ca2fe1af692af537c27d96e0bcde39dce29e2ba1ec4f3"
|
||||
- image: "mariadb:12.1"
|
||||
+ image: "mariadb:12.2"
|
||||
volumes:
|
||||
- "mariadb:/var/lib/mysql"
|
||||
networks:
|
||||
? commit changes? (Y/n)
|
||||
? commit changes? Yes
|
||||
INFO committed changes as 'chore: update image tags'
|
||||
```
|
||||
|
||||
Here is a rundown of what happened:
|
||||
|
||||
- `abra` queried the upstream container repositories of all the images listed in the Wordpress recipe configuration and checked if there are new tags available
|
||||
- `abra` also pulled the current manifest sha256 digest of the images at the new tags for that container
|
||||
- Once you make some choices on the prompt, `abra` will update the recipe configurations and show you the diff
|
||||
- If you choose to, a new commit containing the changes will be created
|
||||
|
||||
|
||||
@@ -74,24 +74,6 @@ version.
|
||||
`abra` will also warn Co-op Cloud operators about missing environment variables
|
||||
that are present in the recipe `.env.sample` but not in their app `.env` file.
|
||||
|
||||
## Upstream versioning
|
||||
|
||||
When upgrading a recipe automatically, abra will attempt to pin to a specific version
|
||||
of an upstream container image. The versions is a multi-architecture digest that is
|
||||
essentially immutable: if the upstream publishes a new image to that semantic version,
|
||||
abra won't use it until the recipe is upgraded again. This is a security feature: if a
|
||||
maintainer publishes a new image with new contents, recipes won't use it unless
|
||||
explicitly requested.
|
||||
|
||||
Manifest images can be inspected manually:
|
||||
```
|
||||
docker buildx imagetools inspect 'gitlab/gitlab-ce:19.4.0-ce.0'
|
||||
|
||||
Name: ...
|
||||
MediaType: ...
|
||||
Digest: sha256:... <- This should go after gitlab-ce:19.4.0-ce.0@...
|
||||
```
|
||||
|
||||
## Creating new release notes
|
||||
|
||||
Help other Co-op Cloud operators understand what they need to take care of when
|
||||
|
||||
@@ -264,6 +264,10 @@ At time of writing (Jan 2022), we think there is a limitation in our design whic
|
||||
|
||||
This may be possible to overcome if someone really needs it, we encourage people to investigate. We've found that often there are limitations in the actual software which don't support this anyway and several of the current operators simply use a new domain per app.
|
||||
|
||||
## Can I add worker nodes to my docker swarm?
|
||||
|
||||
At time of writing (Sep 2026), we officially only support single-node swarms. However, we are working on improving our tools and understanding to support multi-node in the future. If you'd like to get involved, [here is what we know so far](/operators/multinode).
|
||||
|
||||
## How do I bootstrap a server for running Co-op Cloud apps?
|
||||
|
||||
The requirements are:
|
||||
|
||||
@@ -0,0 +1,272 @@
|
||||
---
|
||||
title: Multinode Best Practices
|
||||
---
|
||||
|
||||
## Who This How-To Guide is For
|
||||
|
||||
Configuring coop cloud on multiple nodes poses some difficult problems, and we have not yet streamlined this process. For the sake of security and sanity, you should have a good understanding of the following before you attempt this:
|
||||
|
||||
- VPN configuration, understanding how authentication and encryption are handled.
|
||||
- Firewalls, and `iptables` in particular.
|
||||
- Managing and synchronizing files on a network (e.g. sftp, nfs, rsync)
|
||||
|
||||
What follows is our best knowledge to date. Tread carefully, for here be dragons.
|
||||
|
||||
## Before You Start
|
||||
|
||||
**1. *On every node*, make sure docker is installed and running.
|
||||
|
||||
```bash
|
||||
docker run hello-world
|
||||
```
|
||||
|
||||
**2. *On every node*, ensure kernel modules `ip_vs`[^ip_vs] and `br_netfilter`[^br_netfilter] are loaded.
|
||||
|
||||
```bash
|
||||
lsmod | grep -E "^ip_vs |^br_netfilter "
|
||||
```
|
||||
|
||||
??? Why?
|
||||
|
||||
- `ip_vs` enables multiple Linux kernels to act as *one* virtual server, coordinating request handling and processing with eachother over IP. [kernelconfig.io](https://www.kernelconfig.io/config_ip_vs)
|
||||
- `br_netfilter` enables the kernelspace netfilter capability for bridge network interfaces. Without this kernel module, docker needs to route swarm packets via a slow and insecure userland proxy to communicate with the other docker networks. [Serverfault answer](https://serverfault.com/a/964491), [kernelconfig.io](https://www.kernelconfig.io/CONFIG_BRIDGE_NETFILTER?q=br_netfilter&kernelversion=7.2.4&arch=x86)
|
||||
|
||||
**3. Choose one node to be the manager node.
|
||||
|
||||
Ensure you can connect to this node with `ssh`.
|
||||
|
||||
!!! info "If your server isn't reachable"
|
||||
|
||||
If your manager node is not your main traefik proxy, `abra` may complain about the server not being reachable from the Internet. You can use the `-D` flag to ignore this warning.
|
||||
|
||||
??? warning "Do you need multiple manager nodes?"
|
||||
|
||||
It's easiest to start your setup by choosing only one node in your swarm to be the manager (the others will be worker nodes). If you need multiple manager nodes, e.g. to improve uptime, [read this first](https://docs.docker.com/engine/swarm/how-swarm-mode-works/nodes/#manager-nodes).
|
||||
|
||||
**4. establish a trusted network connection between each worker node and the manager.
|
||||
|
||||
This can be
|
||||
|
||||
- A standard "hub-and-spokes" VPN like strongswan, where worker nodes are connected to the manager node,
|
||||
- Or a mesh VPN like tailscale, where all nodes are connected together.
|
||||
|
||||
??? info "Use a route-based VPN"
|
||||
|
||||
There are two types of VPN implementations: policy-based and route-based. They differ in that route-based VPNs create a distinct network interface for trusted traffic. We'll be using this interface in the next step.
|
||||
|
||||
**5. *On every node*, using `iptables`[^iptables], block ingress on ports `2377/tcp`, `4789/udp`, `7946/udp`, `7946/tcp` except when it comes from a trusted interface. [Docker docs](https://docs.docker.com/engine/swarm/swarm-tutorial/#open-protocols-and-ports-between-the-hosts).
|
||||
|
||||
E.g., if trusted packets arrive on `vpn0`:
|
||||
|
||||
```bash
|
||||
iptables -A INPUT -p tcp --dport 2377 -i !vpn0 -j DROP
|
||||
iptables -A INPUT -p udp --dport 4789 -i !vpn0 -j DROP
|
||||
iptables -A INPUT -p udp --dport 7946 -i !vpn0 -j DROP
|
||||
iptables -A INPUT -p tcp --dport 7946 -i !vpn0 -j DROP
|
||||
```
|
||||
|
||||
??? warning "Use iptables, not nftables"
|
||||
|
||||
All hosts must use `iptables` as the firewall backend, not `nftables`. Nftables support in Docker is experimental, and docker swarm mode hasn't been migrated to support nftables yet. [docker docs](https://docs.docker.com/engine/network/firewall-nftables).
|
||||
|
||||
## Configuring the Swarm and Abra
|
||||
|
||||
**1. In `abra`, create a server using the manager node. (Refer to the [New Operators' Tutorial](https://docs.coopcloud.tech/operators/tutorial/) for a refresher on how to do this.)
|
||||
|
||||
When calling `docker swarm init`, include `--advertise-addr` and point to the trusted interface of your VPN.
|
||||
|
||||
**2. On the *manager node*, call
|
||||
|
||||
```bash
|
||||
docker swarm join-token worker
|
||||
```
|
||||
|
||||
**3. On a *worker node*, call the command provided, e.g.
|
||||
|
||||
```bash
|
||||
docker swarm join --token MYTOKEN M.Y.I.P:2377
|
||||
```
|
||||
|
||||
**4. On the *manager node*, verify the node is in the swarm:
|
||||
|
||||
```bash
|
||||
docker node ls
|
||||
```
|
||||
|
||||
## Volume Duplication
|
||||
|
||||
!!! warning "Docker swarm does not synchronize volume data between nodes."
|
||||
|
||||
- If services on the same recipe deploy on different nodes, they will not have access to any volumes shared between them.
|
||||
- If a service dies or is stopped, and docker swarm deploys it to a different node, it will not have access to its existing volume data.
|
||||
|
||||
You must choose one of these strategies:
|
||||
|
||||
**1. Using placement constraints, restrict apps to always run on specific nodes.
|
||||
|
||||
**2. Choose a networked filesystem (e.g. nfs, rclone) and use its *docker volume plugin* to synchronize volume data between worker nodes.
|
||||
|
||||
Both strategies are described below.
|
||||
|
||||
### How to restrict apps to specific nodes
|
||||
|
||||
**1. In your *abra config*, create a new compose file and set appropriate deploy.placement.constraints for each service. For example:
|
||||
|
||||
`~/.abra/compose/compose.restrict-nextcloud.yml`
|
||||
|
||||
```yml
|
||||
---
|
||||
version: "3.8"
|
||||
|
||||
services:
|
||||
web:
|
||||
deploy:
|
||||
placement:
|
||||
constraints:
|
||||
- node.labels.nextcloud_node == true
|
||||
app:
|
||||
deploy:
|
||||
placement:
|
||||
constraints:
|
||||
- node.labels.nextcloud_node == true
|
||||
cron:
|
||||
deploy:
|
||||
placement:
|
||||
constraints:
|
||||
- node.labels.nextcloud_node == true
|
||||
cache:
|
||||
deploy:
|
||||
placement:
|
||||
constraints:
|
||||
- node.labels.redis_node == true
|
||||
# nb: redis does not share volumes with other services,
|
||||
# so it could be deployed on a separate node.
|
||||
```
|
||||
|
||||
**2. On your *worker node*, assign the appropriate labels:
|
||||
|
||||
```bash
|
||||
docker node update --label-add nextcloud_node=true --label-add redis_node=true <worker_hostname>
|
||||
```
|
||||
|
||||
**3. On the *abra server*, Add your compose file to your app's config. E.g.:
|
||||
|
||||
```bash
|
||||
abra app config my.app
|
||||
```
|
||||
|
||||
```yml
|
||||
...
|
||||
COMPOSE_FILE="compose.yml"
|
||||
COMPOSE_FILE="$COMPOSE_FILE:../../compose/compose.restrict-nextcloud.yml"
|
||||
...
|
||||
```
|
||||
|
||||
**4. Deploy the app.
|
||||
|
||||
**5. Confirm all the services were correctly assigned:
|
||||
|
||||
```bash
|
||||
docker node ps <worker_hostname> | grep "Running"
|
||||
```
|
||||
|
||||
??? info "You can make much more complex setups with placement constraints and node configurations"
|
||||
|
||||
- [Placement constraints (about)](https://docs.docker.com/engine/swarm/services/#control-service-placement)
|
||||
- [Placement constraints (compose file reference)](https://docs.docker.com/reference/compose-file/deploy/#placement)
|
||||
- [Node labels (how to)](https://docs.docker.com/engine/swarm/manage-nodes/#add-or-remove-label-metadata)
|
||||
|
||||
### How to Use docker volume plugins
|
||||
|
||||
!!! warning "Do not use this strategy to synchronize database volumes."
|
||||
|
||||
Instead:
|
||||
- optionally configure a distributed database across your nodes (unknown what tools are best).
|
||||
- be careful when using `db` services in your recipes. Ignore them in favor of a dedicated database instance, or restrict the services to a dedicated node.
|
||||
- in your app configuration, use `DB_HOST` and similar environment variables to point to your network's database host.
|
||||
|
||||
!!! warning "Use the S3 protocol wherever a recipe allows it."
|
||||
|
||||
S3 is much more efficient at synchronizing large amounts of data. We recommend [garage](https://git.coopcloud.tech/coop-cloud/garage).
|
||||
On the other hand, avoid using `s3fs` to store docker volumes - this can cause race conditions because S3 does not allow file locking.
|
||||
|
||||
**1. Choose a machine to be your dedicated volume data store.
|
||||
|
||||
**2. On your *data storage machine*, choose and install a network-accessible file store. Just about anything can be made to work (with tradeoffs): ssh, nfs, NextCloud, WebDav, ProtonDrive, etc.
|
||||
|
||||
**3. Ensure your file store can be securely accessed from every worker node.
|
||||
|
||||
**4. *On every node in your swarm*, install a dedicated docker volume plugin for your file store, or use a generic middleware like [Rclone](https://rclone.org/docker/).
|
||||
|
||||
**5. In your *abra config*, create a custom compose file and configure the plugin on all of your recipe's volumes. E.g.:
|
||||
|
||||
`~/.abra/compose/compose.nextcloud-rclone.yml`
|
||||
|
||||
```yml
|
||||
---
|
||||
version: "3.8"
|
||||
|
||||
volumes:
|
||||
nextcloud:
|
||||
driver: rclone
|
||||
driver_opts:
|
||||
...
|
||||
nextapps:
|
||||
driver: rclone
|
||||
driver_opts:
|
||||
...
|
||||
nextdata: # note: you should use S3 for the majority of your data.
|
||||
driver: rclone
|
||||
driver_opts:
|
||||
...
|
||||
nextconfig:
|
||||
driver: rclone
|
||||
driver_opts:
|
||||
...
|
||||
# here we skip the redis volume, because its cache data
|
||||
# doesn't need to be persisted between nodes
|
||||
```
|
||||
|
||||
**6. Include the compose file in your instance's config. E.g.
|
||||
|
||||
```bash
|
||||
abra app config my.app
|
||||
```
|
||||
|
||||
```yml
|
||||
...
|
||||
COMPOSE_FILE="compose.yml"
|
||||
COMPOSE_FILE="$COMPOSE_FILE:../../compose/compose.nextcloud-rclone.yml"
|
||||
...
|
||||
```
|
||||
|
||||
**7. Deploy the app
|
||||
|
||||
**8. Optionally, verify the volumes are no longer on disk:
|
||||
|
||||
```bash
|
||||
docker volume ls
|
||||
sudo ls /var/lib/docker/volumes/
|
||||
```
|
||||
|
||||
**9. Optionally, destroy any volumes that were previously created:
|
||||
|
||||
```bash
|
||||
docker volume prune -a
|
||||
```
|
||||
|
||||
## Additional Notes
|
||||
|
||||
- Manager nodes should be placed on your most reliable machines, since without them workers cannot deploy services. Because of this, you may want to constrain critical apps to run only on managers: `node.role == manager`.
|
||||
- If an app needs certain resources to be available (e.g. 4 GiB RAM), you can set a resource constraint, which ensures the app can only be scheduled on nodes with that resource available. [See docker docs](https://docs.docker.com/reference/compose-file/deploy/#resources)
|
||||
- A Docker swarm cluster should have an odd number of manager nodes. 1 manager is sufficient, and 3 managers is the minimum for redundancy. [See docker docs](https://docs.docker.com/engine/swarm/how-swarm-mode-works/nodes/#manager-nodes)
|
||||
- When using a swarm with worker nodes, some information will be invisible to abra. In particular, `abra app logs` and `abra app ps` will (at time of writing) give incomplete information. Here are some workarounds:
|
||||
- Instead of `abra app logs example.com app`, call `docker service logs example_com_app` on a manager node.
|
||||
- In addition to `abra app ps example.com`, try: `docker node ps <node_name> | grep example_com`, `docker service ps example_com_app`, `docker service ls -f name=example_com`, `docker stats`
|
||||
- Abra can only interact with manager nodes; worker nodes lack permissions to run most of abra's commands. When creating an `abra server`, remember to connect to a manager node.
|
||||
|
||||
### Additional Resources
|
||||
|
||||
- [nix-config by papiris](https://codeberg.org/papiris/nix-config/src/branch/main/overlays/virtualisation/docker.nix) (multi-node Co-op Cloud, docker within systemd-nspawn)
|
||||
- [SweHarris blog post](https://www.sweharris.org/post/2017-07-30-docker-placement/) about docker swarm placement
|
||||
- [OneUpTime article](https://oneuptime.com/blog/post/2026-03-20-portainer-service-placement-constraints/view) by @nawazdhandala about docker swarm placement
|
||||
@@ -149,8 +149,6 @@ nav:
|
||||
- federation/resolutions/stalled/030.md
|
||||
- "In Progress":
|
||||
- federation/resolutions/in-progress/045.md
|
||||
- federation/resolutions/in-progress/046.md
|
||||
- federation/resolutions/in-progress/047.md
|
||||
- federation/resolutions/index.md
|
||||
- "Minutes":
|
||||
- federation/minutes/index.md
|
||||
|
||||
Reference in New Issue
Block a user