forked from toolshed/docs.coopcloud.tech
Compare commits
14
Commits
main
...
multinode-howto
| Author | SHA1 | Date | |
|---|---|---|---|
 Amras
|
0c804a382b | ||
 stevensting
|
bbb7c6347b | ||
|
stevensting
|
2399092150 | ||
 oxaliq
|
468188c088 | ||
|
oxaliq
|
27d1941e44 | ||
|
oxaliq
|
c61f8ac550 | ||
|
oxaliq
|
8a7f0f08ca | ||
|
oxaliq
|
b81069a85e | ||
 devydave
|
47ff902c8a | ||
|
devydave
|
49b13faade | ||
 dannygroenewegen
|
ca01c39132
|
||
 maren
|
e8d36bcca5 | ||
|
maren
|
5774f73689 | ||
 3wordchant
|
b5f27f82e1 |
@@ -26,6 +26,32 @@ wget -q -O - https://install.abra.coopcloud.tech | bash
|
||||
curl https://install.abra.coopcloud.tech | bash
|
||||
```
|
||||
|
||||
### NixOS
|
||||
|
||||
The install example is based on the [using nix flakes wiki page](https://nixos.wiki/wiki/flakes#Using_nix_flakes_with_NixOS).
|
||||
|
||||
1. Add the flake to your inputs
|
||||
```nix
|
||||
inputs = {
|
||||
abra = {
|
||||
url = "git+https://git.coopcloud.tech/toolshed/abra.git";
|
||||
};
|
||||
};
|
||||
```
|
||||
2. Add abra package to system packages, replace _x86_64-linux_ with the value according to your system
|
||||
```nix
|
||||
{
|
||||
pkgs,
|
||||
abra,
|
||||
...
|
||||
}:
|
||||
{
|
||||
environment.systemPackages = with pkgs; [
|
||||
abra.packages.x86_64-linux.default
|
||||
];
|
||||
}
|
||||
```
|
||||
|
||||
## Release candidate
|
||||
|
||||
### Wget
|
||||
|
||||
+1
-1
@@ -45,7 +45,7 @@ use it! We could really use your input.
|
||||
|
||||
| Feature | Explanation |
|
||||
| ----------- | ----------- |
|
||||
| Multi-node | It is possible but it doesn't seem like anyone in our community is really doing this? Please report in to `#coop-cloud-tech-future-brain:autonomic.zone` to discuss your usage if you are using multi-node swarm! We believe the majority of Co-op Cloud installs are single node. There is also a lack of [CSI](https://github.com/olljanat/csi-plugins-for-docker-swarm?tab=readme-ov-file) support for coordinating storage across multiple hosts when using Swarm mode. This means we kind of throw out [a bunch](https://docs.docker.com/engine/swarm/#feature-highlights) of the features of Swarm mode. |
|
||||
| Multi-node | The vast majority of Co-op Cloud installs are single node. There is a lack of [CSI](https://github.com/olljanat/csi-plugins-for-docker-swarm?tab=readme-ov-file) support for coordinating storage across multiple hosts when using Swarm mode, which means we kind of throw out [a bunch](https://docs.docker.com/engine/swarm/#feature-highlights) of the features of Swarm mode. However, there is growing interest in multinode setups and a push to improve our understanding. [Here's what we know so far, and some hints on getting started.](/operators/multinode) |
|
||||
|
||||
## Limitations
|
||||
|
||||
|
||||
+20
-37
@@ -7,27 +7,24 @@ title: Finance
|
||||
|
||||
## Agreeing to spend money (Budgets)
|
||||
|
||||
The Federation agrees on Budget items as large decisions that must pass a resolution [reference](https://docs.coopcloud.tech/federation/resolutions/passed/004/). All paid work must be within a Budget. Each budget has an associated "Project" on OpenCollective. Current approved budgets are:
|
||||
|
||||
* [Abra Critical Fixes](https://opencollective.com/coop-cloud/projects/4-abra-critical-fixes) - see also [critical fixes budget resolution](https://docs.coopcloud.tech/federation/resolutions/passed/010/)
|
||||
* [Kite Flying](https://opencollective.com/coop-cloud/projects/13-kite-flying) - see also [kite flying budget resolution](https://docs.coopcloud.tech/federation/resolutions/passed/024/)
|
||||
* [Federation Radmin](https://opencollective.com/coop-cloud/projects/014-federation-radmin) - see also [radmin budget resolution](https://docs.coopcloud.tech/federation/resolutions/passed/029/)
|
||||
* [Website Development](https://opencollective.com/coop-cloud/projects/041-website-development) - see also [website development budget resolution](https://docs.coopcloud.tech/federation/resolutions/passed/041/)
|
||||
|
||||
## Sending and receiving money
|
||||
|
||||
It's slightly complicated, because money is complicated, but here's how it works. There are two moving parts:
|
||||
Money is managed through The Co-op Cloud Open Collective account. Platform 6 is [the fiscal host](https://docs.opencollective.com/help/fiscal-hosts/fiscal-hosts) for the Co-op Cloud Open Collective account.
|
||||
|
||||
* The Co-op Cloud Open Collective
|
||||
* The Autonomic Wise account
|
||||
|
||||
Autonomic is [the fiscal host](https://docs.opencollective.com/help/fiscal-hosts/fiscal-hosts) for the Co-op Cloud Open Collective (OC).
|
||||
|
||||
OC helps us make all expenses and transfers transparent to the Federation. No actual money is handled via the OC interface. All payments are done via the Autonomic [Wise](https://wise.com) account. The total sum of the available funds shows on the OC page is the actual amount that is held in the Autonomic Wise account.
|
||||
|
||||
Autonomic Co-op members commit to support the federation by doing the financial adminstration work for the time being. Autonomic is publicy registered, has a bank account, files taxes etc. All financial comings/goings are kept on the books internally at Autonomic. This could be further mutualised or another collective could pick this up in the future.
|
||||
|
||||
Autonomic does not eat the transfer costs from the Wise account when paying out expense for members. That is charged to the Federation common fund.
|
||||
Transactions are submitted through Open Collective, approved by the Federation and paid out by our fiscal host.
|
||||
|
||||
### How to get paid via Open Collective
|
||||
|
||||
* [Create an account on Open Collective](https://opencollective.com/create-account)
|
||||
* Go to the [Co-op Cloud Open Collective](https://opencollective.com/coop-cloud)
|
||||
* Go to the [Co-op Cloud Open Collective](https://opencollective.com/coop-cloud/projects)
|
||||
* Find the appropriate Project
|
||||
* Click [SUBMIT EXPENSE](https://opencollective.com/coop-cloud/expenses/new)
|
||||
|
||||
**Important** Please include bank details in your expense so that we can make a bank transfer. We do not currently support payments via Paypal and other platforms.
|
||||
@@ -36,36 +33,22 @@ If you urgently need the money, please let us know on the Co-op Cloud Finance ch
|
||||
|
||||
Finally, please let us know what your username/email is for your Open Collective account so we can add you to the team. This helps us build up the view of our community from the perspective of our Open Collective page.
|
||||
|
||||
### How to pay someone via Wise
|
||||
#### Invoice Requirements
|
||||
|
||||
> **Note**: only Autonomic Co-op members can do this
|
||||
Invoices are required to be submitted with your expense. Open Collective can generate an invoice for you if you select this option. Please include the hours worked and any work tickets if applicable in the line items of your invoice.
|
||||
|
||||
* First off, be wary of two things: 1) the currency conversion 2) the transaction fees of Wise. For 1) we have the complicating factor that the OC represents the common fund in GBP but our internal Wise jar is EUR. Then you're getting deeper into trouble if someone wants to get paid in e.g. USD.
|
||||
#### What transfer type do we use for payments?
|
||||
|
||||
* In order to cover the transaction fee, you need to fake do the transfer to see what you'll be charged and then add that to what you withdraw from the jar. This is because Autonomic does not eat the cost of the transfer from Wise, that is charged to the Federation.
|
||||
Platform 6 dispenses other payments as bank transfers via [Wise](https://wise.com), please allow up to a week for pay out.
|
||||
|
||||
* First step is to withdraw cash from the Co-op Cloud jar. It will automatically be transferred to the general EUR jar because the Co-op Cloud jar is also in EUR.
|
||||
### Contributing to Co-op Cloud
|
||||
|
||||
* To transfer to USD, you don't have to use USD, you can use GBP/EUR directly. It's easier to make the direct payment from the jar you transferred it to. This is purely because it is easier to follow it in the accounting bookkeeping later on.
|
||||
|
||||
* When making the payment, do the following:
|
||||
* Select international transfer, choose your requird `$currency`
|
||||
* Put correct amount in "recipient gets exactly" to get Wise to figure out the correct amount
|
||||
* Open the invoice in Open Collective and look for the expense number, e.g. "Expense #132373" and put this in the reference number of the payment
|
||||
* Note how long the transfer will take (Wise should tell you)
|
||||
|
||||
* Mark the expense as paid in Open Collective. Use the "manual" method.
|
||||
|
||||
* Let the member know the payment is on the way and how long it will take (if you have time).
|
||||
|
||||
#### FAQ
|
||||
|
||||
##### What transfer type do we use for USD?
|
||||
|
||||
`ACH`. If you see `Abartn`, that is the `ACH routing number`.
|
||||
|
||||
### Tiers on Open Collective
|
||||
[Contribute to Co-op Cloud](https://opencollective.com/coop-cloud#category-CONTRIBUTE)
|
||||
|
||||
* Infrastructure Sustainability: Folks who are making use of Co-op Cloud digital infrastructure (e.g. [git.coopcloud.tech](https://git.coopcloud.tech)) and want to help out with maintenance costs. All recurring donations are spent directly on running costs and system adminstration labour. Thanks for considering!
|
||||
* Federation Membership: Dues paid by members of the Co-op Cloud Federation. Please see [Resolution 002: Membership/Dues 2023-03-22](https://docs.coopcloud.tech/federation/resolutions/passed/002/) for more information. There may be further decisions made around dues, please refer to the Federation documentation on [docs.coopcloud.tech/federation](https://docs.coopcloud.tech/federation).
|
||||
* One-time Contribution: Just stopping in and want to support? Thanks! [Come say hi!](https://docs.coopcloud.tech/intro/contact/)
|
||||
|
||||
* Federation Membership: Dues paid by members of the Co-op Cloud Federation. Please see "Resolution 002: Membership/Dues 2023-03-22" for more information. There may be further decisions made around dues, please refer to the Federation documentation on [docs.coopcloud.tech/federation](https://docs.coopcloud.tech/federation).
|
||||
When scheduling membership dues, please make them from the OpenCollective account associated with your co-op/collective/project. If you do not have an account for your project and are paying with a personal account, please let us know in the Co-op Cloud Finance channel.
|
||||
|
||||
*Please note that Open Collective charges recurring contributions on the 1st of the month*
|
||||
|
||||
@@ -44,7 +44,7 @@ Beyond our grant funding, and support in terms of time and technical resources f
|
||||
|
||||
### 4. Compare your own project with existing or historical efforts. (eg what is new, more thorough, or otherwise different)
|
||||
|
||||
We maintain an ongoing analysis of Co-op Cloud compared to other options in the [Co-op Cloud documentation](https://docs.coopcloud.tech/faq/#what-about-alternative).
|
||||
We maintain an ongoing analysis of Co-op Cloud compared to other options in the [Co-op Cloud documentation](https://docs.coopcloud.tech/intro/faq/#what-about-alternative).
|
||||
|
||||
Overall, Co-op Cloud has architectural and organisational advantages over existing libre options like Yunohost and [Caprover](https://caprover.com), and our open governance and libre licencing make Co-op Cloud a better long-term, pro-social choice than proprietary platforms like [Cloudron](https://cloudron.io). Versus options like [Ansible](https://ansible.com) or [Kubernetes](https://kubernetes.io), Co-op Cloud aims to be usable by less-technical users, to reduce their reliance on third parties to manage their data and tools.
|
||||
|
||||
|
||||
@@ -6,21 +6,25 @@ title: Membership
|
||||
|
||||
| Name | Dues Paid | Notes | Contact |
|
||||
| --------- | --------- | -------- |-------- |
|
||||
| Agaric | - | - | `@wolcen:matrix.org` |
|
||||
| [Autonomic](https://autonomic.zone) | - | - | `@3wc`, `@cas`, `@knoflook`, `@travvy`, `@aadil` |
|
||||
| [Agaric](https://agaric.coop/) | ⭕ | - | `@wolcen:matrix.org` |
|
||||
| [Autonomic](https://autonomic.zone) | ⭕ | - | `@3wc`, `@cas`, `@knoflook`, `@travvy`, `@aadil` |
|
||||
| [Bonfire](https://bonfirenetworks.org) | ✅ | - | `@mayel:matrix.org` + Ivan (`@cambriale:matrix.org`) |
|
||||
| [Doop.coop](https://doop.coop) | - | - | `@yusf:gottsnack.net` |
|
||||
| [EOTL](https://eotl.supply) | - | - | `@basebuilder:pub.solar` |
|
||||
| [Karrot](https://karrot.world) | - | - | `@nicksellen:matrix.org` |
|
||||
| [Klasse & Methode](https://klasse-methode.it) | - | - | `@p4u1_f4u1:matrix.org` |
|
||||
| [Local IT](https://local-it.org/) | - | - | `@moritz:matrix.local-it.org` + `@simon_sth:matrix.org`|
|
||||
| Mirsal ™ | - | - | `@mirsal:1312.media` |
|
||||
| [UTAW](https://utaw.tech) | - | - | `@javielico:matrix.org` |
|
||||
| `@decentral1se` | Waiver | - | `@decentral1se` |
|
||||
| [ruangrupa](https://ruangrupa.id) | - | - | Henry `@babystepper:matrix.org` |
|
||||
| [RTM](https://resisttechmonopolies.online) | ✅ | - | `@ammaratef45:matrix.org` + `@linnealovespie:matrix.org`|
|
||||
| [Bunk Computer Cooperative](https://bunk.computer) | ✅ | - | `@maren@bunk.computer` + `@sorrel@bunk.computer` |
|
||||
| [CoQuest - IT Coop Stuttgart](https://coquest.coop/en) | ✅ | voting rights waived | `hallo@coquest.coop` |
|
||||
| [Democratic Tech Fund](https://democratictech.fund/) | ✅ | - | `@wtebbens` + `@mikemh` |
|
||||
| [Doop.coop](https://doop.coop) | ⭕ | - | `@yusf:gottsnack.net` |
|
||||
| [eCommons](https://ecommons.nl) | ✅ | - | `@dannygroenewegen:matrix.org` |
|
||||
| [EOTL](https://eotl.supply) | ✅ | - | `@basebuilder:pub.solar` |
|
||||
| [Karrot](https://karrot.world) | ⭕ | - | `@nicksellen:matrix.org` |
|
||||
| [Klasse & Methode](https://klasse-methode.it) | Waiver | - | `@p4u1_f4u1:matrix.org` |
|
||||
| [Local IT](https://local-it.org/) | ✅ | - | `@moritz:matrix.local-it.org` + `@simon_sth:matrix.org`|
|
||||
| [Merri-bek tech](https://www.merri-bek.tech/) | ✅ | - | `coop-cloud-delegate@merri-bek.tech`|
|
||||
| [MIR](https://mirnet.org/) | ✅ | - | `@sixsmith:matrix.org` |
|
||||
| [Red Abya Yala](https://abyayala.sutty.nl/) | - | - | `@fauno:sutty.nl` |
|
||||
| [Merri-bek tech](https://www.merri-bek.tech/) | - | - | `coop-cloud-delegate@merri-bek.tech`|
|
||||
| Amras | - | - | `coop-cloud@joinmeonmy.quest` |
|
||||
| [CoQuest](https://coquest.coop/en) | - | voting rights waived | `hallo@coquest.coop` |
|
||||
| [Red Abya Yala](https://abyayala.sutty.nl/) | ⭕ | - | `@fauno:sutty.nl` |
|
||||
| [RTM](https://resisttechmonopolies.online) | ✅ | - | `@ammaratef45:matrix.org` + `@linnealovespie:matrix.org`|
|
||||
| [ruangrupa](https://ruangrupa.id) | ⭕ | - | Henry `@babystepper:matrix.org` |
|
||||
| [UTAW](https://utaw.tech) | ⭕ | - | `@javielico:matrix.org` |
|
||||
| Amras | Waiver | - | `coop-cloud@joinmeonmy.quest` |
|
||||
| `@decentral1se` | Waiver | - | `@decentral1se` |
|
||||
| Mirsal ™ | ✅ | - | `@mirsal:1312.media` |
|
||||
|
||||
|
||||
@@ -8,4 +8,4 @@ Participation in kite-flying can be compensated at €20/hour; see Co-op Cloud F
|
||||
|
||||
We take notes and doodle on [this collaboratively editable pad](https://pad.autonomic.zone/VtyrLUl9RWaJGgEDrncQUw?view). If you don't have time to attend, feel free to drop your questions and some contact details also, so we can get in touch.
|
||||
|
||||
To work around Hedgedoc length limits, [we keep past content from the kite-flying pad here](./kite-flying-pad-archive).
|
||||
To work around Hedgedoc length limits, [we keep past content from the kite-flying pad here](/federation/kite-flying-pad-archive).
|
||||
|
||||
@@ -15,3 +15,5 @@ title: Managed hosting
|
||||
- [Autonomic Co-op](https://autonomic.zone) (contact: [`helo@autonomic.zone`](mailto:boop@autonomic.zone))
|
||||
- [soulhub-IT](https://soulhub-it.de) (managed hosting, see [price calculator](https://soulhub-it.de/produkte/))
|
||||
- [Local-IT](https://local-it.org/) ([selfhosting](https://wiki.local-it.org/s/kollicloud-wiki/doc/selfhosting-guide-1xZJt8UIha) & cooperative hosting, contact: [`info@local-it.org`](mailto:info@local-it.org))
|
||||
- [eCommons](https://ecommons.nl) (contact: [`info@ecommons.nl`](mailto:info@ecommons.nl))
|
||||
- [CoQuest - IT Coop Stuttgart](https://coquest.coop) (managed hosting, contact: [`hello@coquest.coop`](mailto:hello@coquest.coop))
|
||||
|
||||
@@ -264,6 +264,10 @@ At time of writing (Jan 2022), we think there is a limitation in our design whic
|
||||
|
||||
This may be possible to overcome if someone really needs it, we encourage people to investigate. We've found that often there are limitations in the actual software which don't support this anyway and several of the current operators simply use a new domain per app.
|
||||
|
||||
## Can I add worker nodes to my docker swarm?
|
||||
|
||||
At time of writing (Sep 2026), we officially only support single-node swarms. However, we are working on improving our tools and understanding to support multi-node in the future. If you'd like to get involved, [here is what we know so far](/operators/multinode).
|
||||
|
||||
## How do I bootstrap a server for running Co-op Cloud apps?
|
||||
|
||||
The requirements are:
|
||||
|
||||
@@ -0,0 +1,272 @@
|
||||
---
|
||||
title: Multinode Best Practices
|
||||
---
|
||||
|
||||
## Who This How-To Guide is For
|
||||
|
||||
Configuring coop cloud on multiple nodes poses some difficult problems, and we have not yet streamlined this process. For the sake of security and sanity, you should have a good understanding of the following before you attempt this:
|
||||
|
||||
- VPN configuration, understanding how authentication and encryption are handled.
|
||||
- Firewalls, and `iptables` in particular.
|
||||
- Managing and synchronizing files on a network (e.g. sftp, nfs, rsync)
|
||||
|
||||
What follows is our best knowledge to date. Tread carefully, for here be dragons.
|
||||
|
||||
## Before You Start
|
||||
|
||||
**1. *On every node*, make sure docker is installed and running.
|
||||
|
||||
```bash
|
||||
docker run hello-world
|
||||
```
|
||||
|
||||
**2. *On every node*, ensure kernel modules `ip_vs`[^ip_vs] and `br_netfilter`[^br_netfilter] are loaded.
|
||||
|
||||
```bash
|
||||
lsmod | grep -E "^ip_vs |^br_netfilter "
|
||||
```
|
||||
|
||||
??? Why?
|
||||
|
||||
- `ip_vs` enables multiple Linux kernels to act as *one* virtual server, coordinating request handling and processing with eachother over IP. [kernelconfig.io](https://www.kernelconfig.io/config_ip_vs)
|
||||
- `br_netfilter` enables the kernelspace netfilter capability for bridge network interfaces. Without this kernel module, docker needs to route swarm packets via a slow and insecure userland proxy to communicate with the other docker networks. [Serverfault answer](https://serverfault.com/a/964491), [kernelconfig.io](https://www.kernelconfig.io/CONFIG_BRIDGE_NETFILTER?q=br_netfilter&kernelversion=7.2.4&arch=x86)
|
||||
|
||||
**3. Choose one node to be the manager node.
|
||||
|
||||
Ensure you can connect to this node with `ssh`.
|
||||
|
||||
!!! info "If your server isn't reachable"
|
||||
|
||||
If your manager node is not your main traefik proxy, `abra` may complain about the server not being reachable from the Internet. You can use the `-D` flag to ignore this warning.
|
||||
|
||||
??? warning "Do you need multiple manager nodes?"
|
||||
|
||||
It's easiest to start your setup by choosing only one node in your swarm to be the manager (the others will be worker nodes). If you need multiple manager nodes, e.g. to improve uptime, [read this first](https://docs.docker.com/engine/swarm/how-swarm-mode-works/nodes/#manager-nodes).
|
||||
|
||||
**4. establish a trusted network connection between each worker node and the manager.
|
||||
|
||||
This can be
|
||||
|
||||
- A standard "hub-and-spokes" VPN like strongswan, where worker nodes are connected to the manager node,
|
||||
- Or a mesh VPN like tailscale, where all nodes are connected together.
|
||||
|
||||
??? info "Use a route-based VPN"
|
||||
|
||||
There are two types of VPN implementations: policy-based and route-based. They differ in that route-based VPNs create a distinct network interface for trusted traffic. We'll be using this interface in the next step.
|
||||
|
||||
**5. *On every node*, using `iptables`[^iptables], block ingress on ports `2377/tcp`, `4789/udp`, `7946/udp`, `7946/tcp` except when it comes from a trusted interface. [Docker docs](https://docs.docker.com/engine/swarm/swarm-tutorial/#open-protocols-and-ports-between-the-hosts).
|
||||
|
||||
E.g., if trusted packets arrive on `vpn0`:
|
||||
|
||||
```bash
|
||||
iptables -A INPUT -p tcp --dport 2377 -i !vpn0 -j DROP
|
||||
iptables -A INPUT -p udp --dport 4789 -i !vpn0 -j DROP
|
||||
iptables -A INPUT -p udp --dport 7946 -i !vpn0 -j DROP
|
||||
iptables -A INPUT -p tcp --dport 7946 -i !vpn0 -j DROP
|
||||
```
|
||||
|
||||
??? warning "Use iptables, not nftables"
|
||||
|
||||
All hosts must use `iptables` as the firewall backend, not `nftables`. Nftables support in Docker is experimental, and docker swarm mode hasn't been migrated to support nftables yet. [docker docs](https://docs.docker.com/engine/network/firewall-nftables).
|
||||
|
||||
## Configuring the Swarm and Abra
|
||||
|
||||
**1. In `abra`, create a server using the manager node. (Refer to the [New Operators' Tutorial](https://docs.coopcloud.tech/operators/tutorial/) for a refresher on how to do this.)
|
||||
|
||||
When calling `docker swarm init`, include `--advertise-addr` and point to the trusted interface of your VPN.
|
||||
|
||||
**2. On the *manager node*, call
|
||||
|
||||
```bash
|
||||
docker swarm join-token worker
|
||||
```
|
||||
|
||||
**3. On a *worker node*, call the command provided, e.g.
|
||||
|
||||
```bash
|
||||
docker swarm join --token MYTOKEN M.Y.I.P:2377
|
||||
```
|
||||
|
||||
**4. On the *manager node*, verify the node is in the swarm:
|
||||
|
||||
```bash
|
||||
docker node ls
|
||||
```
|
||||
|
||||
## Volume Duplication
|
||||
|
||||
!!! warning "Docker swarm does not synchronize volume data between nodes."
|
||||
|
||||
- If services on the same recipe deploy on different nodes, they will not have access to any volumes shared between them.
|
||||
- If a service dies or is stopped, and docker swarm deploys it to a different node, it will not have access to its existing volume data.
|
||||
|
||||
You must choose one of these strategies:
|
||||
|
||||
**1. Using placement constraints, restrict apps to always run on specific nodes.
|
||||
|
||||
**2. Choose a networked filesystem (e.g. nfs, rclone) and use its *docker volume plugin* to synchronize volume data between worker nodes.
|
||||
|
||||
Both strategies are described below.
|
||||
|
||||
### How to restrict apps to specific nodes
|
||||
|
||||
**1. In your *abra config*, create a new compose file and set appropriate deploy.placement.constraints for each service. For example:
|
||||
|
||||
`~/.abra/compose/compose.restrict-nextcloud.yml`
|
||||
|
||||
```yml
|
||||
---
|
||||
version: "3.8"
|
||||
|
||||
services:
|
||||
web:
|
||||
deploy:
|
||||
placement:
|
||||
constraints:
|
||||
- node.labels.nextcloud_node == true
|
||||
app:
|
||||
deploy:
|
||||
placement:
|
||||
constraints:
|
||||
- node.labels.nextcloud_node == true
|
||||
cron:
|
||||
deploy:
|
||||
placement:
|
||||
constraints:
|
||||
- node.labels.nextcloud_node == true
|
||||
cache:
|
||||
deploy:
|
||||
placement:
|
||||
constraints:
|
||||
- node.labels.redis_node == true
|
||||
# nb: redis does not share volumes with other services,
|
||||
# so it could be deployed on a separate node.
|
||||
```
|
||||
|
||||
**2. On your *worker node*, assign the appropriate labels:
|
||||
|
||||
```bash
|
||||
docker node update --label-add nextcloud_node=true --label-add redis_node=true <worker_hostname>
|
||||
```
|
||||
|
||||
**3. On the *abra server*, Add your compose file to your app's config. E.g.:
|
||||
|
||||
```bash
|
||||
abra app config my.app
|
||||
```
|
||||
|
||||
```yml
|
||||
...
|
||||
COMPOSE_FILE="compose.yml"
|
||||
COMPOSE_FILE="$COMPOSE_FILE:../../compose/compose.restrict-nextcloud.yml"
|
||||
...
|
||||
```
|
||||
|
||||
**4. Deploy the app.
|
||||
|
||||
**5. Confirm all the services were correctly assigned:
|
||||
|
||||
```bash
|
||||
docker node ps <worker_hostname> | grep "Running"
|
||||
```
|
||||
|
||||
??? info "You can make much more complex setups with placement constraints and node configurations"
|
||||
|
||||
- [Placement constraints (about)](https://docs.docker.com/engine/swarm/services/#control-service-placement)
|
||||
- [Placement constraints (compose file reference)](https://docs.docker.com/reference/compose-file/deploy/#placement)
|
||||
- [Node labels (how to)](https://docs.docker.com/engine/swarm/manage-nodes/#add-or-remove-label-metadata)
|
||||
|
||||
### How to Use docker volume plugins
|
||||
|
||||
!!! warning "Do not use this strategy to synchronize database volumes."
|
||||
|
||||
Instead:
|
||||
- optionally configure a distributed database across your nodes (unknown what tools are best).
|
||||
- be careful when using `db` services in your recipes. Ignore them in favor of a dedicated database instance, or restrict the services to a dedicated node.
|
||||
- in your app configuration, use `DB_HOST` and similar environment variables to point to your network's database host.
|
||||
|
||||
!!! warning "Use the S3 protocol wherever a recipe allows it."
|
||||
|
||||
S3 is much more efficient at synchronizing large amounts of data. We recommend [garage](https://git.coopcloud.tech/coop-cloud/garage).
|
||||
On the other hand, avoid using `s3fs` to store docker volumes - this can cause race conditions because S3 does not allow file locking.
|
||||
|
||||
**1. Choose a machine to be your dedicated volume data store.
|
||||
|
||||
**2. On your *data storage machine*, choose and install a network-accessible file store. Just about anything can be made to work (with tradeoffs): ssh, nfs, NextCloud, WebDav, ProtonDrive, etc.
|
||||
|
||||
**3. Ensure your file store can be securely accessed from every worker node.
|
||||
|
||||
**4. *On every node in your swarm*, install a dedicated docker volume plugin for your file store, or use a generic middleware like [Rclone](https://rclone.org/docker/).
|
||||
|
||||
**5. In your *abra config*, create a custom compose file and configure the plugin on all of your recipe's volumes. E.g.:
|
||||
|
||||
`~/.abra/compose/compose.nextcloud-rclone.yml`
|
||||
|
||||
```yml
|
||||
---
|
||||
version: "3.8"
|
||||
|
||||
volumes:
|
||||
nextcloud:
|
||||
driver: rclone
|
||||
driver_opts:
|
||||
...
|
||||
nextapps:
|
||||
driver: rclone
|
||||
driver_opts:
|
||||
...
|
||||
nextdata: # note: you should use S3 for the majority of your data.
|
||||
driver: rclone
|
||||
driver_opts:
|
||||
...
|
||||
nextconfig:
|
||||
driver: rclone
|
||||
driver_opts:
|
||||
...
|
||||
# here we skip the redis volume, because its cache data
|
||||
# doesn't need to be persisted between nodes
|
||||
```
|
||||
|
||||
**6. Include the compose file in your instance's config. E.g.
|
||||
|
||||
```bash
|
||||
abra app config my.app
|
||||
```
|
||||
|
||||
```yml
|
||||
...
|
||||
COMPOSE_FILE="compose.yml"
|
||||
COMPOSE_FILE="$COMPOSE_FILE:../../compose/compose.nextcloud-rclone.yml"
|
||||
...
|
||||
```
|
||||
|
||||
**7. Deploy the app
|
||||
|
||||
**8. Optionally, verify the volumes are no longer on disk:
|
||||
|
||||
```bash
|
||||
docker volume ls
|
||||
sudo ls /var/lib/docker/volumes/
|
||||
```
|
||||
|
||||
**9. Optionally, destroy any volumes that were previously created:
|
||||
|
||||
```bash
|
||||
docker volume prune -a
|
||||
```
|
||||
|
||||
## Additional Notes
|
||||
|
||||
- Manager nodes should be placed on your most reliable machines, since without them workers cannot deploy services. Because of this, you may want to constrain critical apps to run only on managers: `node.role == manager`.
|
||||
- If an app needs certain resources to be available (e.g. 4 GiB RAM), you can set a resource constraint, which ensures the app can only be scheduled on nodes with that resource available. [See docker docs](https://docs.docker.com/reference/compose-file/deploy/#resources)
|
||||
- A Docker swarm cluster should have an odd number of manager nodes. 1 manager is sufficient, and 3 managers is the minimum for redundancy. [See docker docs](https://docs.docker.com/engine/swarm/how-swarm-mode-works/nodes/#manager-nodes)
|
||||
- When using a swarm with worker nodes, some information will be invisible to abra. In particular, `abra app logs` and `abra app ps` will (at time of writing) give incomplete information. Here are some workarounds:
|
||||
- Instead of `abra app logs example.com app`, call `docker service logs example_com_app` on a manager node.
|
||||
- In addition to `abra app ps example.com`, try: `docker node ps <node_name> | grep example_com`, `docker service ps example_com_app`, `docker service ls -f name=example_com`, `docker stats`
|
||||
- Abra can only interact with manager nodes; worker nodes lack permissions to run most of abra's commands. When creating an `abra server`, remember to connect to a manager node.
|
||||
|
||||
### Additional Resources
|
||||
|
||||
- [nix-config by papiris](https://codeberg.org/papiris/nix-config/src/branch/main/overlays/virtualisation/docker.nix) (multi-node Co-op Cloud, docker within systemd-nspawn)
|
||||
- [SweHarris blog post](https://www.sweharris.org/post/2017-07-30-docker-placement/) about docker swarm placement
|
||||
- [OneUpTime article](https://oneuptime.com/blog/post/2026-03-20-portainer-service-placement-constraints/view) by @nawazdhandala about docker swarm placement
|
||||
@@ -193,6 +193,7 @@ plugins:
|
||||
- redirects:
|
||||
redirect_maps:
|
||||
"get-involved/support/index.md": intro/support.md
|
||||
"faq.md": intro/faq.md
|
||||
|
||||
repo_name: toolshed/docs.coopcloud.tech
|
||||
repo_url: https://git.coopcloud.tech/toolshed/docs.coopcloud.tech/
|
||||
|
||||
Reference in New Issue
Block a user