12 KiB
title |
---|
Operations handbook |
Understanding app and server configuration
Co-op Cloud stores per-app configuration in the $USER/.abra/servers
directory, on whichever machine you're running abra
on (by default, your own workstation).
The format of these configuration files is the same environment variable syntax used by Docker (with the env_file:
statement in a docker-compose.yml
file, or the --env-file
option to docker run
) and direnv
:
abra app example_wordpress config
TYPE=wordpress
DOMAIN=wordpress.example.com
## Domain aliases
EXTRA_DOMAINS=', `www.wordpress.example.com`'
LETS_ENCRYPT_ENV=production
...
abra
doesn't mind if ~/.abra/servers
, or any of its subdirectories, is a symlink, so you can keep your app definitions wherever you like!
mv ~/.abra/servers/ ~/coop-cloud
ln -s ~/coop-cloud ~/.abra/servers
Backing up your app configuration
Just make sure the ~/.abra/servers
is included in the configuration of your favourite backup tool.
You can optionally also backup ~/.abra/apps
, if you'd like to keep an exact copy of the application versions you currently have deployed. Otherwise, they'll be automatically downloaded the first time you run an abra app...
command.
You don't need to worry about ~/.abra/vendor
or ~/.abra/src
directories, which will be likewise recreated automatically as and when you need them.
Version-control your app configs (using git)
Because ~/.abra/servers
is a collection of plain-text files, it's easy to keep your backup configuration in a version control system (we use git
, others would almost certainly work).
This is particularly recommended if you're collaborating with others, so that you can all run abra app...
commands without having to maintain your own separate, probably-conflicting, configuration files.
In the simple case where you only have one server configured with abra
, or everyone in your team is using the same set of servers, you can version-control the whole ~/.abra/servers
directory:
cd ~/.abra/servers
git init
git add .
git commit -m "Initial import"
!!! warning "Test your revision-control self-discipline"
`abra` does not yet help keep your app definitions are up-to-date.
Make sure to run `git add` / `git commit` after making configuration changes, and `cd ~/.abra/servers && git pull` before running `abra app...` commands.
Patches to add some safety checks and auto-updates would be very welcome! 🙏
Collaborating with multiple teams
In a more complex situation, where you're using Co-op Cloud to manage several servers, and you're collaborating with different people on different servers, you can set up a separate repository for each subdirectory in ~/.abra/servers
, or even a mixture of single-server and multi-server repositories:
ls -l ~/.abra/servers
# Example.com's own app configuration:
lrwxrwxrwx. 1 user user 49 Oct 30 22:42 swarm.example.com -> /home/user/Example/coop-cloud-apps/swarm.example.com
# Configuration for one of Example.com's clients – part of the same repository:
lrwxrwxrwx. 1 user user 49 Oct 30 22:42 swarm.client.com -> /home/user/Example/coop-cloud-apps/swarm.client.com
# A completely separate project, part of a different repository:
lrwxrwxrwx. 1 user user 49 Oct 30 22:42 swarm.demonstration.com -> /home/user/Demonstration/coop-cloud-apps
To make setting up these symlinks easier, you might want to include a simple installer script in your configuration repositories.
We don't have a public example of this yet, but something like this should do the trick:
-
Save this as
Makefile
in your repository:# -s symlink, -f force creation, -F don't create symlink in the target dir link: @mkdir -p ~/.abra/servers/ @for SERVER in $$(find -maxdepth 1 -type d -name "[!.]*"); do \ echo ln -sfF "$$(pwd)/$${SERVER#./}" ~/.abra/servers/ ; \ ln -sfF "$$(pwd)/$${SERVER#./}" ~/.abra/servers/ ; \ done
This will set up symlinks from each directory in your repository to a correspondingly-named directory in
~/.abra/servers
– if your repository has aswarm.example.com
directory, it'll be linked as~/.abra/servers/swarm.example.com
. -
Tell your collaborators (e.g. in the repository's
README
), to runmake
in their repository check-out.
!!! warning "You're on your own!"
As with the [simple repository set-up above](#version-control), `abra` doesn't yet help you update your version control system when you make changes, nor check version control to make sure you have the latest configuration.
Make sure to `commit` and `push` after you make any configuration changes, and `pull` before running any `abra app...` commands.
Even more granularity?
The plain-text, file-based configuration format means that you could even keep the configuration for different apps on the same server in different repositories, e.g. having git.example.com
configuration in a separate repository to wordpress.example.com
, using per-file symlinks.
We don't currently recommend this, because it might set inaccurate expectations about the security model – remember that, by default, any user who can deploy apps to a Docker Swarm can manage any app in that swarm.
Running abra server side
If you're on an environment where it's hard to run Docker, or command-line programs in general, you might want to install abra
on a server instead of your local computer.
To install abra
on a different server than you'll be hosting your apps, just follow getting started guide as normal.
If you want to install abra
on the same server, there's one change.
Instead of providing your SSH connection details when you run abra server add ...
, just use default
:
abra server add default
!!! note "Technical details"
This will tell `abra` to look at the Docker system running on the server, instead of a remote one.
Make sure to back up your ~/abra/
directory on the server, or put it in version control, as well as other files you'd like to.
Managing secret data
Co-op Cloud uses [Docker Secrets] to handle sensitive data, like database passwords and API keys, securely:
DOCKER_CONTEXT=swarm.example.com docker secret ls
example_mediawiki_db_password_v1
example_wordpress_db_password_v1
abra
includes several commands to make it easier to manage secrets:
abra app <app> secret generate
-- to auto-generate a single secret, or all secrets defined by the app, and store them in the Docker Swarm store,abra app <app> secret insert
-- to insert a single secret value from the Docker Swarm store,abra app <app> secret delete
-- to remove a single secret, or all secrets defined in the app, from the Docker Swarm store.
Secret versions
You will notice v1
in the example secret names above: like Docker Configs, Docker Secrets are [immutable], which means that their values can't be changed after they're set. To accommodate this, Co-op Cloud uses the established convention of "secret versions". Every time you change (rotate) a secret, you will insert it as a new version.
Because secret versions are managed per-instance by the people deploying their apps, secret versions are stored in the .env
file for each app:
find -L ~/.abra/servers/ -name '*.env' -print0 | xargs -0 grep -h SECRET
OIDC_CLIENT_SECRET_VERSION=v1
RPC_SECRET_VERSION=v1
CLIENT_SECRET_VERSION=v1
...
If you try and add a secret version which already exists, Docker will helpfully complain:
abra app example_wordpress secret insert db_password v1 foobar
Error response from daemon: rpc error: code = AlreadyExists desc = secret example_wordpress_db_password_v1 already exists
By default, new app instances will look for v1
secrets.
Generating secrets automatically
You can generate secrets in one of two ways:
- While running
abra app new <type>
, by passing--secrets
- At any point once an app instance is defined, by running
abra app <app> secret generate ...
(seeabra help secret generate
for full options)
!!! note "How are secrets generated?"
Depending on how the app is configured, you will require the `pwqgen` (from `passwdqc`) and `pwgen` binaries by default, although you can specify your own password-generation app when running `abra <app> secret generate` by providing the `<cmd>` argument.
Inserting secrets manually
For third-party API tokens, like OAuth client secrets, or keys for services like Mailgun, you will be storing values you already have as the appropriately-named Docker secrets. abra
provides a convenient interface to the underlying docker secret create
command:
abra app example_wordpress secret insert db_password v2 "your-secret-here"
Rotating a secret
So, given how secret versions work, here's how you change a secret:
- Find out the current version number of the secret, e.g. by running
abra app example_wordpress config
, and choose a new one. Let's assume it's currentlyv1
, so by convention the new secret will bev2
. - Generate or insert the new secret:
orabra app example_wordpress secret generate db_password v2
abra app example_wordpress secret insert db_password v2 "foobar"
- Edit the app configuration to change which secret version the app will use:
abra app example_wordpress config
- Re-reploy the app with the new secret version:
abra app example_wordpress deploy
Storing secrets in pass
The Co-op Cloud authors use the [UNIX pass
tool][pass] to share sensitive data, including Co-op Cloud secrets, and abra <app> secret...
commands include a --pass
option to automatically manage generated / inserted secrets:
# Store generated secrets in `pass`:
abra app new wordpress --secrets --pass
abra app example_wordpress secret generate --all --pass
# Store inserted secret in `pass`:
abra app example_wordpress secret insert db_password v2 --pass
# Remove secrets from Docker, and `pass`:
abra app example_wordpress secret rm --all --pass
This functionality currently relies on our specific pass
structure; patches to make that configurable are very welcome!
Networking
!!! note "So dark the con of Docker Networking"
Our understanding of Docker networking is probably wrong. We're working on it. Plz send halp :pray:
Traefik networking
Traefik is our core web proxy, all traffic on a Co-op Cloud deployment goes through a running Traefik container. When setting up a new Co-op Cloud delpyment, abra
creates a "global" overlay network which traefik is hooked up to. This is the network that other apps use to speak to traefik and get traffic routed to them. Not every service in every app is also included in this network and hence not internet-facing (by convention, we name this network internal
, see more below).
App networking
By convention, the main app
service is wired up to the "global" traefik overlay network. This container is the one that should be publicy reachable on the internet. The other services in the app such as the database and caches should be not be publicly reachable or visible to other apps on the same instance.
To deal with this, we make an additional "internal" network for each app which is namespaced to that app. So, if you deploy a Wordpress instance called my_wordpress_blog
then there will be a network called my_wordpress_blog_internal
created. This allows all the services in an app to speak to each other but not be reachable on the public internet.
Avoiding service namespace conflicts
When referencing an app
service in a config file, you should prefix with the STACK_NAME
to avoid namespace conflicts (because all these containers sit on the traefik overlay network). You might want to do something like this {{ env "STACK_NAME" }}_app
(using the often obscure dark magic of the Golang templating language). You can find examples of this approach used in the Peertube recipe.