mayel
/
docs.coopcloud.tech
forked from coop-cloud/docs.coopcloud.tech
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

more operators help

This commit is contained in:
decentral1se 2022-02-03 16:02:11 +01:00
parent e6810cfba9
commit 89d13c31ed
Signed by untrusted user: decentral1se
GPG Key ID: 03789458B3D0C410
3 changed files with 90 additions and 116 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

4
docs/maintainers/handbook.md
View File

@ -45,3 +45,7 @@ TODO.
### /bin/bash is missing?
TODO.
## Referening service names in configs?
When referencing an `app` service in a config file, you should prefix with the `STACK_NAME` to avoid namespace conflicts (because all these containers sit on the traefik overlay network). You might want to do something like this `{{ env "STACK_NAME" }}_app` (using the often obscure dark magic of the Golang templating language). You can find examples of this approach used in the [Peertube recipe](https://git.coopcloud.tech/coop-cloud/peertube/src/commit/d1b297c5a6a23a06bf97bb954104ddfd7f736568/nginx.conf.tmpl#L9).

190
docs/operators/handbook.md
View File

@ -2,45 +2,28 @@
title: Operations handbook
---
## Understanding app and server configuration
## Understanding `~/.abra`
Co-op Cloud stores per-app configuration in the `$USER/.abra/servers` directory, on whichever machine you're running `abra` on (by default, your own workstation).
Co-op Cloud stores per-app configuration in the `$USER/.abra/servers` directory, on whichever machine you're running `abra` on (by default, your own work station). In other words, app configurations are grouped under their relevant server directory. This corresponds to the ordering of the output of `abra app ls`.
The format of these configuration files is the same environment variable syntax used by Docker (with the `env_file:` statement in a `docker-compose.yml` file, or the `--env-file` option to `docker run`) and `direnv`:
!!! question "What format do the `.env` files use?"
```
abra app example_wordpress config
TYPE=wordpress
`.env` files use the same format as used by Docker (with the `env_file:` statement in a `docker-compose.yml` file, or the `--env-file` option to `docker run`) and `direnv`. There is no `export ...=...` required since `abra` will take care to thread the values into the recipe configuration at deploy time.
DOMAIN=wordpress.example.com
## Domain aliases
EXTRA_DOMAINS=', `www.wordpress.example.com`'
LETS_ENCRYPT_ENV=production
...
```
`abra` doesn't mind if `~/.abra/servers`, or any of its subdirectories, is a [symlink], so you can keep your app definitions wherever you like!
`abra` doesn't mind if `~/.abra/servers`, or any of its subdirectories, is a [symlink](https://en.wikipedia.org/wiki/Symlink), so you can keep your app definitions wherever you like!
```
mv ~/.abra/servers/ ~/coop-cloud
ln -s ~/coop-cloud ~/.abra/servers
```
### Backing up your app configuration
You don't need to worry about `~/.abra/{vendor,catalogue,recipes,autocompletion}`, `abra` manages those automagically.
Just make sure the `~/.abra/servers` is included in the configuration of your favourite backup tool.
## Backing up `~/.abra`
You can optionally also backup `~/.abra/apps`, if you'd like to keep an exact copy of the application versions you currently have deployed. Otherwise, they'll be automatically downloaded the first time you run an `abra app...` command.
Just make sure the `~/.abra/servers` is included in the configuration of your favourite backup tool. Because `~/.abra/servers` is a collection of plain-text files, it's easy to keep your backup configuration in a version control system (we use `git`, others would almost certainly work).
You don't need to worry about `~/.abra/vendor` or `~/.abra/src` directories, which will be likewise recreated automatically as and when you need them.
<a id="version-control"></a>
### Version-control your app configs (using git)
Because `~/.abra/servers` is a collection of plain-text files, it's easy to keep your backup configuration in a version control system (we use `git`, others would almost certainly work).
This is particularly recommended if you're collaborating with others, so that you can all run `abra app...` commands without having to maintain your own separate, probably-conflicting, configuration files.
This is particularly recommended if you're collaborating with others, so that you can all run `abra app ...` commands without having to maintain your own separate, probably-conflicting, configuration files.
In the simple case where you only have one server configured with `abra`, or everyone in your team is using the same set of servers, you can version-control the whole `~/.abra/servers` directory:
@ -53,35 +36,31 @@ git commit -m "Initial import"
!!! warning "Test your revision-control self-discipline"
`abra` does not yet help keep your app definitions are up-to-date.
`abra` does not yet help keep your `~/.abra/server` configurations up-to-date! Make sure to run `git add` / `git commit` after making configuration changes, and `cd ~/.abra/servers && git pull` before running `abra app ...` commands. Patches to add some safety checks and auto-updates would be very welcome! 🙏
Make sure to run `git add` / `git commit` after making configuration changes, and `cd ~/.abra/servers && git pull` before running `abra app...` commands.
Patches to add some safety checks and auto-updates would be very welcome! 🙏
### Collaborating with multiple teams
## Sharing `~/.abra`
In a more complex situation, where you're using Co-op Cloud to manage several servers, and you're collaborating with different people on different servers, you can set up **a separate repository for each subdirectory in `~/.abra/servers`**, or even a mixture of single-server and multi-server repositories:
```
ls -l ~/.abra/servers
# Example.com's own app configuration:
lrwxrwxrwx. 1 user user 49 Oct 30 22:42 swarm.example.com -> /home/user/Example/coop-cloud-apps/swarm.example.com
swarm.example.com -> /home/user/Example/coop-cloud-apps/swarm.example.com
# Configuration for one of Example.com's clients – part of the same repository:
lrwxrwxrwx. 1 user user 49 Oct 30 22:42 swarm.client.com -> /home/user/Example/coop-cloud-apps/swarm.client.com
swarm.client.com -> /home/user/Example/coop-cloud-apps/swarm.client.com
# A completely separate project, part of a different repository:
lrwxrwxrwx. 1 user user 49 Oct 30 22:42 swarm.demonstration.com -> /home/user/Demonstration/coop-cloud-apps
swarm.demonstration.com -> /home/user/Demonstration/coop-cloud-apps
```
To make setting up these symlinks easier, you might want to include a simple installer script in your configuration repositories.
We don't have a public example of this yet, but something like this should do the trick:
1. Save this as `Makefile` in your repository:
Save this as `Makefile` in your repository:
```
# -s symlink, -f force creation, -F don't create symlink in the target dir
link:
default:
@mkdir -p ~/.abra/servers/
@for SERVER in $$(find -maxdepth 1 -type d -name "[!.]*"); do \
echo ln -sfF "$$(pwd)/$${SERVER#./}" ~/.abra/servers/ ; \
@ -91,65 +70,47 @@ We don't have a public example of this yet, but something like this should do th
This will set up symlinks from each directory in your repository to a correspondingly-named directory in `~/.abra/servers` – if your repository has a `swarm.example.com` directory, it'll be linked as `~/.abra/servers/swarm.example.com`.
2. Tell your collaborators (e.g. in the repository's `README`), to run `make` in their repository check-out.
Then, tell your collaborators (e.g. in the repository's `README.md`), to run `make` in their repository check-out.
!!! warning "You're on your own!"
As with the [simple repository set-up above](#version-control), `abra` doesn't yet help you update your version control system when you make changes, nor check version control to make sure you have the latest configuration.
As with the [simple repository set-up above](#backing-up-your-abra-configuration), `abra` doesn't yet help you update your version control system when you make changes, nor check version control to make sure you have the latest configuration. Make sure to `commit` and `push` after you make any configuration changes, and `pull` before running any `abra app ...` commands.
Make sure to `commit` and `push` after you make any configuration changes, and `pull` before running any `abra app...` commands.
!!! question "Even more granularity?"
### Even more granularity?
The plain-text, file-based configuration format means that you could even keep the configuration for different apps on the same server in different repositories, e.g. having `git.example.com` configuration in a separate repository to `wordpress.example.com`, using per-file symlinks.
The plain-text, file-based configuration format means that you could even keep the configuration for different apps on the same server in different repositories, e.g. having `git.example.com` configuration in a separate repository to `wordpress.example.com`, using per-file symlinks.
We don't currently recommend this, because it might set inaccurate expectations about the security model – remember that, by default, **any user who can deploy apps to a Docker Swarm can manage _any_ app in that swarm**.
[symlink]: https://en.wikipedia.org/wiki/Symlink
We don't currently recommend this, because it might set inaccurate expectations about the security model – remember that, by default, **any user who can deploy apps to a Docker Swarm can manage _any_ app in that swarm**.
## Running abra server side
If you're on an environment where it's hard to run Docker, or command-line programs in general, you might want to install `abra` on a server instead of your local computer.
If you're on an environment where it's hard to run Docker, or command-line programs in general, you might want to install `abra` on a server instead of your local work station.
To install `abra` on a different server than you'll be hosting your apps, just follow [getting started guide](/overview/) as normal.
If you want to install `abra` on the same server, there's one change.
Instead of providing your SSH connection details when you run `abra server add ...`, just use `default`:
To install `abra` on a different server than you'll be hosting your apps, just follow [getting started guide](/operators/tutorial#deploy-your-first-app) as normal except for one difference. Instead of providing your SSH connection details when you run `abra server add ...`, just pass `--local`.
```
abra server add default
abra server add --local
```
!!! note "Technical details"
This will tell `abra` to look at the Docker system running on the server, instead of a remote one.
This will tell `abra` to look at the Docker system running on the server, instead of a remote one (using the Docker internal `default` context). Once this is wired up, `abra` knows that the deployment target is the local server and not a remote one. This will be handle seamlessly for all other deployments on this server.
Make sure to back up your `~/abra/` directory on the server, or put it in version control, as well as other files you'd like to.
Make sure to back up your `~/.abra` directory on the server, or put it in version control, as well as other files you'd like to keep safe.
## Managing secret data
Co-op Cloud uses [Docker Secrets] to handle sensitive data, like database passwords and API keys, securely:
```
DOCKER_CONTEXT=swarm.example.com docker secret ls
example_mediawiki_db_password_v1
example_wordpress_db_password_v1
```
Co-op Cloud uses [Docker Secrets](https://docs.docker.com/engine/swarm/secrets/) to handle sensitive data, like database passwords and API keys, securely.
`abra` includes several commands to make it easier to manage secrets:
- `abra app <app> secret generate` -- to auto-generate a single secret, or all secrets defined by the app, and store them in the Docker Swarm store,
- `abra app <app> secret insert` -- to insert a single secret value from the Docker Swarm store,
- `abra app <app> secret delete` -- to remove a single secret, or all secrets defined in the app, from the Docker Swarm store.
<a id="versions"></a>
- `abra app secret generate <domain>`: to auto-generate a app secrets
- `abra app secret insert <domain>`: to insert a single secret
- `abra app secret rm <domain>`: to remove secrets
### Secret versions
You will notice `v1` in the example secret names above: like Docker Configs, Docker Secrets are [immutable], which means that their values can't be changed after they're set. To accommodate this, Co-op Cloud uses the established convention of "secret versions". Every time you change (rotate) a secret, you will insert it as a new version.
Because secret versions are managed per-instance by the people deploying their apps, secret versions are stored in the `.env` file for each app:
Docker secrets are immutable, which means that their values can't be changed after they're set. To accommodate this, Co-op Cloud uses the established convention of "secret versions". Every time you change (rotate) a secret, you will insert it as a new version. Because secret versions are managed per-instance by the people deploying their apps, secret versions are stored in the `.env` file for each app:
```
find -L ~/.abra/servers/ -name '*.env' -print0 | xargs -0 grep -h SECRET
@ -162,8 +123,8 @@ CLIENT_SECRET_VERSION=v1
If you try and add a secret version which already exists, Docker will helpfully complain:
```
abra app example_wordpress secret insert db_password v1 foobar
Error response from daemon: rpc error: code = AlreadyExists desc = secret example_wordpress_db_password_v1 already exists
abra app secret insert mywordpress.com db_password v1 foobar
Error response from daemon: rpc error: code = AlreadyExists desc = secret mywordpress_com_db_password_v1 already exists
```
By default, new app instances will look for `v1` secrets.
@ -172,58 +133,43 @@ By default, new app instances will look for `v1` secrets.
You can generate secrets in one of two ways:
1. While running `abra app new <type>`, by passing `--secrets`
2. At any point once an app instance is defined, by running `abra app <app> secret generate ...` (see `abra help secret generate` for full options)
!!! note "How are secrets generated?"
Depending on how the app is configured, you will require the `pwqgen` (from `passwdqc`) and `pwgen` binaries by default, although you can specify your own password-generation app when running `abra <app> secret generate` by providing the `<cmd>` argument.
1. While running `abra app new <recipe>`, by passing `-S/--secrets`
2. At any point once an app instance is defined, by running `abra app secret generate <domain> ...` (see `abra app secret generate -h` for more)
### Inserting secrets manually
For third-party API tokens, like OAuth client secrets, or keys for services like Mailgun, you will be storing values you already have as the appropriately-named Docker secrets. `abra` provides a convenient interface to the underlying `docker secret create` command:
```
abra app example_wordpress secret insert db_password v2 "your-secret-here"
abra app secret insert <domain> db_password v2 "your-secret-value-here"
```
### Rotating a secret
So, given how [secret versions](#versions) work, here's how you change a secret:
So, given how [secret versions](/operators/handbook/#secret-versions) work, here's how you change a secret:
1. Find out the current version number of the secret, e.g. by running `abra app example_wordpress config`, and choose a new one. Let's assume it's currently `v1`, so by convention the new secret will be `v2`.
2. Generate or insert the new secret:
```
abra app example_wordpress secret generate db_password v2
```
or
```
abra app example_wordpress secret insert db_password v2 "foobar"
```
3. Edit the app configuration to change which secret version the app will use:
```
abra app example_wordpress config
```
4. Re-reploy the app with the new secret version:
```
abra app example_wordpress deploy
```
1. Find out the current version number of the secret, e.g. by running `abra app config <domain>`, and choose a new one. Let's assume it's currently `v1`, so by convention the new secret will be `v2`
2. Generate or insert the new secret: `abra app secret generate <domain> db_password v2` or `abra app secret insert <domain> db_password v2 "foobar"`
3. Edit the app configuration to change which secret version the app will use: `abra app config <domain>`
4. Re-reploy the app with the new secret version: `abra app deploy <domain>`
### Storing secrets in `pass`
The Co-op Cloud authors use the [UNIX `pass` tool][pass] to share sensitive data, including Co-op Cloud secrets, and `abra <app> secret...` commands include a `--pass` option to automatically manage generated / inserted secrets:
The Co-op Cloud authors use the [UNIX `pass` tool][pass] to share sensitive data, including Co-op Cloud secrets, and `abra app secret ...` commands include a `--pass` option to automatically manage generated / inserted secrets:
```
# Store generated secrets in `pass`:
abra app new wordpress --secrets --pass
abra app example_wordpress secret generate --all --pass
abra app secret generate mywordpress.com --all --pass
# Store inserted secret in `pass`:
abra app example_wordpress secret insert db_password v2 --pass
abra app secret insert mywordpress.com db_password v2 --pass
# Remove secrets from Docker, and `pass`:
abra app example_wordpress secret rm --all --pass
abra app secret rm mywordpress.com --all --pass
```
This functionality currently relies on our specific `pass` structure; patches to make that configurable are very welcome!
This functionality currently relies on our specific `pass` storage conventions; patches to make that configurable are very welcome!
## Networking
@ -241,13 +187,11 @@ By convention, the main `app` service is wired up to the "global" traefik overla
To deal with this, we make an additional "internal" network for each app which is namespaced to that app. So, if you deploy a Wordpress instance called `my_wordpress_blog` then there will be a network called `my_wordpress_blog_internal` created. This allows all the services in an app to speak to each other but not be reachable on the public internet.
### Avoiding service namespace conflicts
## Multiple apps on the same domain?
When referencing an `app` service in a config file, you should prefix with the `STACK_NAME` to avoid namespace conflicts (because all these containers sit on the traefik overlay network). You might want to do something like this `{{ env "STACK_NAME" }}_app` (using the often obscure dark magic of the Golang templating language). You can find examples of this approach used in the [Peertube recipe](https://git.coopcloud.tech/coop-cloud/peertube/src/commit/d1b297c5a6a23a06bf97bb954104ddfd7f736568/nginx.conf.tmpl#L9).
At time of writing (Jan 2022), we think there is a limitation in our design which doesn't support multiple apps sharing the same domain (e.g. `example.com/app1/` & `example.com/app2/`). `abra` treats each domain as unique and as the singler reference for a single app.
## Domain sharing for apps
At time of writing (Jan 2022), we think there is a limitation in our design which doesn't support multiple apps sharing the same domain (e.g. `example.com/app1/` & `example.com/app2/`). This may be possible to overcome if someone really needs it, we encourage people to investigate. We've found that often, there are limitations in the actual software which don't support this anyway and several of the current operators simply use a new domain per app.
This may be possible to overcome if someone really needs it, we encourage people to investigate. We've found that often, there are limitations in the actual software which don't support this anyway and several of the current operators simply use a new domain per app.
## Validating `abra` binary checksums
@ -257,4 +201,30 @@ At time of writing (Jan 2022), we think there is a limitation in our design whic
grep $(sha256sum abra_[version]_[platform]) checksums.txt > /dev/null && echo "checksum OK"
```
If "checksum OK" appears in your terminal - you're good to go! Otherwise, you have downloaded a corrupted file.
If "checksum OK" appears in your terminal - you're good to go!
Otherwise, you have downloaded a corrupted file.
## Creating a new server
`abra server new` can create servers if you have an account with a supported 3rd party integration. We currently support [Servers.coop](https://servers.coop) & [Hetzner](https://hetzner.com). The process of creating a new server usually goes like this:
1. Create an account with a server hosting provider
2. Generate an API client key which you'll give to `abra`
3. Run `abra server new` & fill in the values
`abra` supports creating, listing and removing servers if the 3rd party integration supports it.
If you want to teach `abra` how to support your favourite server hosting provider, we'd glady accept patches.
## Managing DNS entries
`abra record ...` can help you manage your DNS entries if you have an account with a supported 3rd party provider. We currently support [Gandi](https://gandi.net). The process of managing DNS with `abra` usually goes like this:
1. Create an account with a DNS service provider
2. Generate an API client key which you'll give to `abra`
3. Run `abra record ls` to check everything works
`abra` supports creating, listing and removing DNS entries if the 3rd party integration supports it.
If you want to teach `abra` how to support your favourite server hosting provider, we'd glady accept patches.

12
docs/operators/tutorial.md
View File

@ -8,7 +8,7 @@ Co-op Cloud is made up of a few simple, composable pieces. The system does not r
We want to build a resilient and long-term sustainable project and that means allowing for different implementations, open formats and a diverse project organisation.
Here are the main technical concepts listed below, once you [grok](https://en.wikipedia.org/wiki/Grok) this, you grok the moving parts of the project.
Here are the main technical concepts listed below, once you [grok](https://en.wikipedia.org/wiki/Grok) this, you grok the moving parts of the entire project. You can then move on to [deploying your first app](/operators/tutorial/#deploy-your-first-app).
### Libre software apps
@ -158,11 +158,11 @@ abra server add <server-domain> -p
`abra` uses plain 'ol SSH under the hood and aims to make use of your existing SSH configurations in `~/.ssh/config` and interfaces with your running `ssh-agent` for password protected secret key files.
The `server add` command listed above assumes that that you make SSH connections on port 22 using your current username. If that is not he case, pass the new values as position arguments:
The `server add` command listed above assumes that that you make SSH connections on port 22 using your current username. If that is not he case, pass the new values as positional arguments. See `abra server add -h` for more on this.
abra server add <domain> <user> <port> -p
It's best to take a moment to read [this troubleshooting entry](/abra/trouble/#ssh-connection-issues) if you're running into SSH connection issues with `abra`. Running `server add` with `-d/--debug` should help you debug what is going on under the hood.
Running `server add` with `-d/--debug` should help you debug what is going on under the hood. It's best to take a moment to read [this troubleshooting entry](/abra/trouble/#ssh-connection-issues) if you're running into SSH connection issues with `abra`.
The `-p` or `--provision` flag means that `abra` will install Docker and initialise the [new single-host swarm](https://docs.docker.com/engine/swarm/key-concepts/) on your server.
@ -216,13 +216,13 @@ And now we can deploy apps. Let's create a new Nextcloud app.
abra app new nextcloud -S
```
The `-S` or `--secrets` flag is used to generate secrets for the app: database connection password, root password and admin password. `abra` generates these randomly and on a one-time basis and sends them to be stored encrypted on your server.
The `-S` or `--secrets` flag is used to generate secrets for the app: database connection password, root password and admin password.
!!! warning "Beware of password dragons"
Take care, these secrets are only shown once on the terminal so make sure to take note of them! `abra` makes use of the [Docker secrets](/operators/handbook/#managing-secret-data) mechanism to ship these secrets securely to the server and store them as encrypted data.
Take care, these secrets are only shown once on the terminal so make sure to take note of them! `abra` makes use of the [Docker secrets](/operators/handbook/#managing-secret-data) mechanism to ship these secrets securely to the server and store them as encrypted data. Only the apps themselves have access to the values from here on, they're placed in `/run/secrets` on the container file system.
Then we can deploy the Nextcloud.
Then we can deploy Nextcloud:
```bash
abra app deploy <nextcloud-domain>