docs.coopcloud.tech/docs/operators/multinode.md

title

title
Multinode Best Practices

Who This How-To Guide is For

Configuring coop cloud on multiple nodes poses some difficult problems, and we have not yet streamlined this process. For the sake of security and sanity, you should have a good understanding of the following before you attempt this:

• VPN configuration, understanding how authentication and encryption are handled.
• Firewalls, and iptables in particular.
• Managing and synchronizing files on a network (e.g. sftp, nfs, rsync)

What follows is our best knowledge to date. Tread carefully, for here be dragons.

Before You Start

**1. On every node, make sure docker is installed and running.

docker run hello-world

**2. On every node, ensure kernel modules ip_vs[^ip_vs] and br_netfilter[^br_netfilter] are loaded.

lsmod | grep -E "^ip_vs |^br_netfilter "

??? Why?

- `ip_vs` enables multiple Linux kernels to act as *one* virtual server, coordinating request handling and processing with eachother over IP. [kernelconfig.io](https://www.kernelconfig.io/config_ip_vs)  
- `br_netfilter` enables the kernelspace netfilter capability for bridge network interfaces. Without this kernel module, docker needs to route swarm packets via a slow and insecure userland proxy to communicate with the other docker networks. [Serverfault answer](https://serverfault.com/a/964491), [kernelconfig.io](https://www.kernelconfig.io/CONFIG_BRIDGE_NETFILTER?q=br_netfilter&kernelversion=7.2.4&arch=x86)

**3. Choose one node to be the manager node.

Ensure you can connect to this node with ssh.

!!! info "If your server isn't reachable"

If your manager node is not your main traefik proxy, `abra` may complain about the server not being reachable from the Internet. You can use the `-D` flag to ignore this warning.

??? warning "Do you need multiple manager nodes?"

It's easiest to start your setup by choosing only one node in your swarm to be the manager (the others will be worker nodes). If you need multiple manager nodes, e.g. to improve uptime, [read this first](https://docs.docker.com/engine/swarm/how-swarm-mode-works/nodes/#manager-nodes).

**4. establish a trusted network connection between each worker node and the manager.

This can be

• A standard "hub-and-spokes" VPN like strongswan, where worker nodes are connected to the manager node,
• Or a mesh VPN like tailscale, where all nodes are connected together.

??? info "Use a route-based VPN"

There are two types of VPN implementations: policy-based and route-based. They differ in that route-based VPNs create a distinct network interface for trusted traffic. We'll be using this interface in the next step.

**5. On every node, using iptables[^iptables], block ingress on ports 2377/tcp, 4789/udp, 7946/udp, 7946/tcp except when it comes from a trusted interface. Docker docs.

E.g., if trusted packets arrive on vpn0:

iptables -A INPUT -p tcp --dport 2377 -i !vpn0 -j DROP
iptables -A INPUT -p udp --dport 4789 -i !vpn0 -j DROP
iptables -A INPUT -p udp --dport 7946 -i !vpn0 -j DROP
iptables -A INPUT -p tcp --dport 7946 -i !vpn0 -j DROP

??? warning "Use iptables, not nftables"

All hosts must use `iptables` as the firewall backend, not `nftables`. Nftables support in Docker is experimental, and docker swarm mode hasn't been migrated to support nftables yet. [docker docs](https://docs.docker.com/engine/network/firewall-nftables).

Configuring the Swarm and Abra

**1. In abra, create a server using the manager node. (Refer to the New Operators' Tutorial for a refresher on how to do this.)

When calling docker swarm init, include --advertise-addr and point to the trusted interface of your VPN.

**2. On the manager node, call

docker swarm join-token worker

**3. On a worker node, call the command provided, e.g.

docker swarm join --token MYTOKEN M.Y.I.P:2377

**4. On the manager node, verify the node is in the swarm:

docker node ls

Volume Duplication

!!! warning "Docker swarm does not synchronize volume data between nodes."

- If services on the same recipe deploy on different nodes, they will not have access to any volumes shared between them.
- If a service dies or is stopped, and docker swarm deploys it to a different node, it will not have access to its existing volume data.

You must choose one of these strategies:

**1. Using placement constraints, restrict apps to always run on specific nodes.

**2. Choose a networked filesystem (e.g. nfs, rclone) and use its docker volume plugin to synchronize volume data between worker nodes.

Both strategies are described below.

How to restrict apps to specific nodes

**1. In your abra config, create a new compose file and set appropriate deploy.placement.constraints for each service. For example:

~/.abra/compose/compose.restrict-nextcloud.yml

---
version: "3.8"

services:
  web:
    deploy:
      placement:
        constraints:
          - node.labels.nextcloud_node == true
  app:
    deploy:
      placement:
        constraints:
          - node.labels.nextcloud_node == true
  cron:
    deploy:
      placement:
        constraints:
          - node.labels.nextcloud_node == true
  cache:
    deploy:
      placement:
        constraints:
          - node.labels.redis_node == true
          # nb: redis does not share volumes with other services,
          # so it could be deployed on a separate node.

**2. On your worker node, assign the appropriate labels:

docker node update --label-add nextcloud_node=true --label-add redis_node=true <worker_hostname>

**3. On the abra server, Add your compose file to your app's config. E.g.:

abra app config my.app

...
COMPOSE_FILE="compose.yml"
COMPOSE_FILE="$COMPOSE_FILE:../../compose/compose.restrict-nextcloud.yml"
...

**4. Deploy the app.

**5. Confirm all the services were correctly assigned:

docker node ps <worker_hostname> | grep "Running"

??? info "You can make much more complex setups with placement constraints and node configurations"

- [Placement constraints (about)](https://docs.docker.com/engine/swarm/services/#control-service-placement)
- [Placement constraints (compose file reference)](https://docs.docker.com/reference/compose-file/deploy/#placement)
- [Node labels (how to)](https://docs.docker.com/engine/swarm/manage-nodes/#add-or-remove-label-metadata)

How to Use docker volume plugins

!!! warning "Do not use this strategy to synchronize database volumes."

Instead:
    - optionally configure a distributed database across your nodes (unknown what tools are best).
    - be careful when using `db` services in your recipes. Ignore them in favor of a dedicated database instance, or restrict the services to a dedicated node.
    - in your app configuration, use `DB_HOST` and similar environment variables to point to your network's database host.

!!! warning "Use the S3 protocol wherever a recipe allows it."

S3 is much more efficient at synchronizing large amounts of data. We recommend [garage](https://git.coopcloud.tech/coop-cloud/garage).
On the other hand, avoid using `s3fs` to store docker volumes - this can cause race conditions because S3 does not allow file locking.

**1. Choose a machine to be your dedicated volume data store.

**2. On your data storage machine, choose and install a network-accessible file store. Just about anything can be made to work (with tradeoffs): ssh, nfs, NextCloud, WebDav, ProtonDrive, etc.

**3. Ensure your file store can be securely accessed from every worker node.

**4. On every node in your swarm, install a dedicated docker volume plugin for your file store, or use a generic middleware like Rclone.

**5. In your abra config, create a custom compose file and configure the plugin on all of your recipe's volumes. E.g.:

~/.abra/compose/compose.nextcloud-rclone.yml

---
version: "3.8"

volumes:
  nextcloud:
    driver: rclone
    driver_opts:
        ...
  nextapps:
    driver: rclone
    driver_opts:
        ...
  nextdata: # note: you should use S3 for the majority of your data.
    driver: rclone
    driver_opts:
        ...
  nextconfig:
    driver: rclone
    driver_opts:
        ...
 # here we skip the redis volume, because its cache data
 # doesn't need to be persisted between nodes

**6. Include the compose file in your instance's config. E.g.

abra app config my.app

...
COMPOSE_FILE="compose.yml"
COMPOSE_FILE="$COMPOSE_FILE:../../compose/compose.nextcloud-rclone.yml"
...

**7. Deploy the app

**8. Optionally, verify the volumes are no longer on disk:

docker volume ls
sudo ls /var/lib/docker/volumes/

**9. Optionally, destroy any volumes that were previously created:

docker volume prune -a

Additional Notes

• Manager nodes should be placed on your most reliable machines, since without them workers cannot deploy services. Because of this, you may want to constrain critical apps to run only on managers: node.role == manager.
• If an app needs certain resources to be available (e.g. 4 GiB RAM), you can set a resource constraint, which ensures the app can only be scheduled on nodes with that resource available. See docker docs
• A Docker swarm cluster should have an odd number of manager nodes. 1 manager is sufficient, and 3 managers is the minimum for redundancy. See docker docs
• When using a swarm with worker nodes, some information will be invisible to abra. In particular, abra app logs and abra app ps will (at time of writing) give incomplete information. Here are some workarounds:
  • Instead of abra app logs example.com app, call docker service logs example_com_app on a manager node.
  • In addition to abra app ps example.com, try: docker node ps <node_name> | grep example_com, docker service ps example_com_app, docker service ls -f name=example_com, docker stats
• Abra can only interact with manager nodes; worker nodes lack permissions to run most of abra's commands. When creating an abra server, remember to connect to a manager node.

Additional Resources

• nix-config by papiris (multi-node Co-op Cloud, docker within systemd-nspawn)
• SweHarris blog post about docker swarm placement
• OneUpTime article by @nawazdhandala about docker swarm placement