From 731367a4633c5e77a2774cd1392c5b72bf4d5300 Mon Sep 17 00:00:00 2001 From: cellarspoon Date: Wed, 26 Jan 2022 11:47:44 +0100 Subject: [PATCH] big pass --- docs/cli/index.md | 3 - docs/glossary/index.md | 60 ++--- docs/index.md | 6 +- docs/maintainers/index.md | 4 +- .../tutorial.md} | 51 +++- docs/operators/handbook.md | 246 ++++++++++++++++++ docs/operators/index.md | 4 +- docs/operators/trouble.md | 7 + docs/operators/tutorial.md | 209 +++++++++++++++ docs/organisers/index.md | 22 +- docs/organisers/schedules.md | 25 ++ docs/organisers/tutorial.md | 5 + docs/x-archive/abra.md | 34 --- docs/x-archive/app-config-guide.md | 15 -- docs/x-archive/config.md | 108 -------- docs/x-archive/deploy.md | 145 ----------- docs/x-archive/networking.md | 21 -- docs/x-archive/overview.md | 64 ----- docs/x-archive/recipe-structure.md | 50 ---- docs/x-archive/secrets.md | 109 -------- docs/x-archive/server-side.md | 24 -- docs/x-archive/troubleshooting.md | 35 --- mkdocs.yml | 16 +- 23 files changed, 591 insertions(+), 672 deletions(-) delete mode 100644 docs/cli/index.md rename docs/{x-archive/recipe-maintainer-guide.md => maintainers/tutorial.md} (70%) create mode 100644 docs/operators/handbook.md create mode 100644 docs/operators/trouble.md create mode 100644 docs/operators/tutorial.md create mode 100644 docs/organisers/schedules.md create mode 100644 docs/organisers/tutorial.md delete mode 100644 docs/x-archive/abra.md delete mode 100644 docs/x-archive/app-config-guide.md delete mode 100644 docs/x-archive/config.md delete mode 100644 docs/x-archive/deploy.md delete mode 100644 docs/x-archive/networking.md delete mode 100644 docs/x-archive/overview.md delete mode 100644 docs/x-archive/recipe-structure.md delete mode 100644 docs/x-archive/secrets.md delete mode 100644 docs/x-archive/server-side.md delete mode 100644 docs/x-archive/troubleshooting.md diff --git a/docs/cli/index.md b/docs/cli/index.md deleted file mode 100644 index 0bda76e1..00000000 --- a/docs/cli/index.md +++ /dev/null @@ -1,3 +0,0 @@ ---- -title: TODO ---- diff --git a/docs/glossary/index.md b/docs/glossary/index.md index 5b1a87b0..ce75d649 100644 --- a/docs/glossary/index.md +++ b/docs/glossary/index.md @@ -2,62 +2,58 @@ title: Glossary --- -## Recipe - -A recipe is what we call the configuration files that are used to deploy an [app](/glossary#app). When you run `abra app deploy `, `abra` is reading a recipe configuration, such as [the gitea recipe](https://git.coopcloud.tech/coop-cloud/gitea), in order to know how to deploy a new Gitea instance. When we speak of a "digital configuraiton commons", we're primarily referring to the [growing collection of recipes](https://git.coopcloud.tech/coop-cloud). - -## App - -An app is a website, tool or otherwise libre software that you use. E.g. Wordpress, Gitea, Jitsi, Nextcloud, etc. - ## Abra A command-line tool that has been developed specifically in the context of the Co-op Cloud project for the purpose of making day-to-day operations for [operators](/operators/) and [maintainers](/maintainers/) as convenient as possible. It is libre software, written in [Go](https://go.dev/) and maintained and extended by the community. You can find the source [here](https://git.coopcloud.tech/coop-cloud/abra). +## App + +An app is a libre software that you use, e.g. Wordpress, Gitea, Jitsi, Nextcloud, etc. When you `abra app deploy `, you deploy an app. It is quite an overloaded term in general, often referring to many different things. We struggled with using this word but it seems to be one word people recognise and have a point of reference for. + ## Container -TODO. - -## Stack - -One or more [services](/glossary#service) running together to provide a functionality. An [app](/glossary#app) - -## Docker - -TODO. - -## Service - -A single [Docker](/glossary#docker) [container](/glossary#container) that is a part of a [stack](glossary#stack). +A [Docker](/glossary#docker) term: a running instance of an [image](/glossary#image), running processes that are isolated from the host system. ## Deployment -The act of assembling a [Docker](glossary#docker) [stack](/glossary#stack) described by a [recipe](/glossary#recipe) configuration. +When you run `abra app deploy `, `abra` reads a [recipe](/glossary#recipe) configuration and creates an [app](/glossary#app). -## Image +## Docker -A template for creating [containers](/glossary#container), describing their file structure and installed binaries. - -## Container - -An instance of an [image](/glossary#image), running processes that are isolated from the host system. +[Docker Inc.](https://www.docker.com/), the company who popularised the concept of [the container](https://www.docker.com/resources/what-container). The same company has created the underlying tools & libraries that `abra` uses to get work done. ## Environment variables Variables passed from the shell to processes invoked by it. They are used for configuring [services](/glossary#service). -## The `.env` file +## Environment file A file contained in a [recipe](/glossary#recipe) describing the contents of [environmental variables](/glossary#environment-variables). +## Image + +A [Docker](/glossary#docker) term: a template for creating [containers](/glossary#container), describing their file structure and installed binaries. + ## Proxy network -A virtual network created on the server machine used for communicating between [services](/glossary#service). Any [service](/glossary#service) can be plugged into more than one [network](/glossary#network), allowing for control over data sharing between them. +A [Docker](glossary#docker) related concept: a virtual network created on the server machine used for communicating between [services](/glossary#service). Any [service](/glossary#service) can be plugged into more than one [network](/glossary#network), allowing for control over data sharing between them. + +## Recipe + +A recipe is what we call the configuration files that are used to deploy an [app](/glossary#app). When you run `abra app deploy `, `abra` is reading a recipe configuration, such as [the gitea recipe](https://git.coopcloud.tech/coop-cloud/gitea), in order to know how to deploy a new Gitea instance. When we speak of a "digital configuraiton commons", we're primarily referring to the [growing collection of recipes](https://git.coopcloud.tech/coop-cloud). ## Secret -[Docker](/glossary#docker) uses those for securely storing data such as passwords. They are stored encrypted on disk and mounted inside the [containers](/glossary#container) as files that can be read that contain the secret. +A [Docker](/glossary#docker) related concept: A way to store passwords encrypted on disk and mounted inside the [containers](/glossary#container) as files that can be read that contain the secret. See the [Docker secrets documentation for more](https://docs.docker.com/engine/swarm/secrets/). `abra` makes use of this approach to store secrets for deployed [apps](/glossary#app). + +## Service + +A [Docker](glossary#docker) term: a single [container](/glossary#container) that is a part of a [stack](glossary#stack). + +## Stack + +A [Docker](glossary#docker) term: one or more [services](/glossary#service) running together to provide a functionality. ## Volume -A directory that can be mounted inside a [Docker](/glossary#docker) [container](/glossary#container) to store data. Because [containers](/glossary#container) are meant to be non-changeable and disposable, any data that is supposed to not be lost between updates or restarts is stored in volumes. +A [Docker](/glossary#docker) term: a directory that can be mounted inside a [container](/glossary#container) to store data. Because [containers](/glossary#container) are meant to be non-changeable and disposable, any data that is supposed to not be lost between updates or restarts is stored in volumes. diff --git a/docs/index.md b/docs/index.md index 47271fe7..7c838ef0 100644 --- a/docs/index.md +++ b/docs/index.md @@ -10,7 +10,7 @@ The documentation is aimed at a technical audience: tech co-ops, collectives and A more general public may still find these pages useful but if you're just looking for a quick overview of the project from a less technical perspective, you can take a look at [coopcloud.tech](https://coopcloud.tech). -We'd be happy to hear feedback about our documentation, if it was helpful, what was missing, what was confusing, etc., please [get in touch!](/intro/contact). +We'd be happy to hear feedback about our documentation, if it was helpful, what was missing, what was confusing, etc., please [get in touch](/intro/contact)! ## Quick start @@ -22,12 +22,10 @@ We'd be happy to hear feedback about our documentation, if it was helpful, what - [Maintainers guide](/maintainers/): You maintain recipes and ensure things run smoothly for operators :tools: -- [Organisers guide](/organisers): You're here to do some good 'ol fashioned community organising :fist: +- [Organisers guide](/organisers): You run meetings, write guidelines & shape our democratic process :fist: - [Recipes](/recipes/): You want to know what recipes are packaged so you can deploy them as apps :nerd: - [Get involved](/get-involved): You'd like to help out with the project, we've love to see you stick around :heart: -- [CLI deference](/cli/): You need a command-line reference (`abra`, `docker`, etc.) :notebook: - - [Glossary](/glossary/): You'd like clarification about project terminology :book: diff --git a/docs/maintainers/index.md b/docs/maintainers/index.md index 0bda76e1..e0692445 100644 --- a/docs/maintainers/index.md +++ b/docs/maintainers/index.md @@ -1,3 +1,5 @@ --- -title: TODO +title: Maintainers guide --- + +TODO. diff --git a/docs/x-archive/recipe-maintainer-guide.md b/docs/maintainers/tutorial.md similarity index 70% rename from docs/x-archive/recipe-maintainer-guide.md rename to docs/maintainers/tutorial.md index 0cbb3079..68f42154 100644 --- a/docs/x-archive/recipe-maintainer-guide.md +++ b/docs/maintainers/tutorial.md @@ -1,5 +1,5 @@ --- -title: Recipe maintainer guide +title: New maintainers tutorial --- ## Package your first recipe @@ -104,3 +104,52 @@ The commands uses for dealing with this logic in `abra` are: ## Style guide - Please don't use `&image` YAML repeat anchors on the `image: ...` key because our `recipe release` logic does not handle it (see [#172](https://git.autonomic.zone/coop-cloud/abra/issues/172)) + +## Recipe structure + +A recipe is a git repository that contains instructions for creating stacks that abra can read and interpret. You'll see a couple of files there: + +- [compose.yml](#composeyml) (required!) +- [.env.sample](#envsample) (required!) +- [abra.sh](#abrash) +- [entrypoint.sh](#entrypointsh) +- [other compose files](#other-compose-files) +- [other files](#other-files) + + +## compose.yml + +this is a [compose specification](https://compose-spec.io/) compliant file that contains a list of: + +- services +- secrets +- networks +- volumes +- configs + +that describe what is needed to run a stack. Whenever you deploy an app, abra reads this file to cook the stack. + + +## .env.sample + +this file is a skeleton for environmental variables that should be adjusted by the user. Examples include: domain or php extention list. Whenever you create a new app with `abra app new` this file gets copied to the ~/.abra/servers/server-domain/app-name.env and when you run `abra app config` you're editing this file. + + +## abra.sh + +`abra.sh` provides shell functions for running non-standard deploy, restore, rollback, backup and upgrade. This is only needed for some packages (such as nextcloud or wordpress) + + +## entrypoint.sh + +after docker creates the filesystem and copies files into a new container it runs what's called an entrypoint. This is usually a shell script that exports some variables and runs the application. Sometimes the vendor entrypoint doesn't do everything that we need it to do. In that case you can write your own entrypoint, do whatever you need to do and then run the vendor entrypoint. For a simple example check [entrypoint.sh for croc](https://git.coopcloud.tech/coop-cloud/croc/src/commit/2f06e8aac52a3850d527434a26de0a242bea0c79/entrypoint.sh). In this case, croc needs the password to be exported as an environmental variable called `CROC_PASS`, and that is exactly what the entrypoint does before running vendor entrypoint. If you write your own entrypoint, it needs to be specified in the `config` section of compose.yml. + + +## other compose files + +i.e. compose.smtp.yml. These are used to provide non-essential functionality such as (registration) e-mails or single sign on. + + +## other files + +if you look at compose.yml (or compose.\*.yml) and see a `configs` section, that means this compose file is putting files in the container. This might be used for changing default (vendor) configuration, such as this [fpm-tune.ini file](https://git.coopcloud.tech/coop-cloud/nextcloud/src/commit/28425b6138603067021757de28c639ad464e9cf8/fpm-tune.ini) used to adjust php-fpm. diff --git a/docs/operators/handbook.md b/docs/operators/handbook.md new file mode 100644 index 00000000..548be9d5 --- /dev/null +++ b/docs/operators/handbook.md @@ -0,0 +1,246 @@ +--- +title: Operations handbook +--- + +## Understanding app and server configuration + +Co-op Cloud stores per-app configuration in the `$USER/.abra/servers` directory, on whichever machine you're running `abra` on (by default, your own workstation). + +The format of these configuration files is the same environment variable syntax used by Docker (with the `env_file:` statement in a `docker-compose.yml` file, or the `--env-file` option to `docker run`) and `direnv`: + +``` +abra app example_wordpress config +TYPE=wordpress + +DOMAIN=wordpress.example.com +## Domain aliases +EXTRA_DOMAINS=', `www.wordpress.example.com`' +LETS_ENCRYPT_ENV=production +... +``` + +`abra` doesn't mind if `~/.abra/servers`, or any of its subdirectories, is a [symlink], so you can keep your app definitions wherever you like! + +``` +mv ~/.abra/servers/ ~/coop-cloud +ln -s ~/coop-cloud ~/.abra/servers +``` + +### Backing up your app configuration + +Just make sure the `~/.abra/servers` is included in the configuration of your favourite backup tool. + +You can optionally also backup `~/.abra/apps`, if you'd like to keep an exact copy of the application versions you currently have deployed. Otherwise, they'll be automatically downloaded the first time you run an `abra app...` command. + +You don't need to worry about `~/.abra/vendor` or `~/.abra/src` directories, which will be likewise recreated automatically as and when you need them. + + + +### Version-control your app configs (using git) + +Because `~/.abra/servers` is a collection of plain-text files, it's easy to keep your backup configuration in a version control system (we use `git`, others would almost certainly work). + +This is particularly recommended if you're collaborating with others, so that you can all run `abra app...` commands without having to maintain your own separate, probably-conflicting, configuration files. + +In the simple case where you only have one server configured with `abra`, or everyone in your team is using the same set of servers, you can version-control the whole `~/.abra/servers` directory: + +``` +cd ~/.abra/servers +git init +git add . +git commit -m "Initial import" +``` + +!!! warning "Test your revision-control self-discipline" + + `abra` does not yet help keep your app definitions are up-to-date. + + Make sure to run `git add` / `git commit` after making configuration changes, and `cd ~/.abra/servers && git pull` before running `abra app...` commands. + + Patches to add some safety checks and auto-updates would be very welcome! 🙏 + +### Collaborating with multiple teams + +In a more complex situation, where you're using Co-op Cloud to manage several servers, and you're collaborating with different people on different servers, you can set up **a separate repository for each subdirectory in `~/.abra/servers`**, or even a mixture of single-server and multi-server repositories: + +``` +ls -l ~/.abra/servers +# Example.com's own app configuration: +lrwxrwxrwx. 1 user user 49 Oct 30 22:42 swarm.example.com -> /home/user/Example/coop-cloud-apps/swarm.example.com +# Configuration for one of Example.com's clients – part of the same repository: +lrwxrwxrwx. 1 user user 49 Oct 30 22:42 swarm.client.com -> /home/user/Example/coop-cloud-apps/swarm.client.com +# A completely separate project, part of a different repository: +lrwxrwxrwx. 1 user user 49 Oct 30 22:42 swarm.demonstration.com -> /home/user/Demonstration/coop-cloud-apps +``` + +To make setting up these symlinks easier, you might want to include a simple installer script in your configuration repositories. + +We don't have a public example of this yet, but something like this should do the trick: + +1. Save this as `Makefile` in your repository: + + ``` + # -s symlink, -f force creation, -F don't create symlink in the target dir + link: + @mkdir -p ~/.abra/servers/ + @for SERVER in $$(find -maxdepth 1 -type d -name "[!.]*"); do \ + echo ln -sfF "$$(pwd)/$${SERVER#./}" ~/.abra/servers/ ; \ + ln -sfF "$$(pwd)/$${SERVER#./}" ~/.abra/servers/ ; \ + done + ``` + + This will set up symlinks from each directory in your repository to a correspondingly-named directory in `~/.abra/servers` – if your repository has a `swarm.example.com` directory, it'll be linked as `~/.abra/servers/swarm.example.com`. + +2. Tell your collaborators (e.g. in the repository's `README`), to run `make` in their repository check-out. + +!!! warning "You're on your own!" + + As with the [simple repository set-up above](#version-control), `abra` doesn't yet help you update your version control system when you make changes, nor check version control to make sure you have the latest configuration. + + Make sure to `commit` and `push` after you make any configuration changes, and `pull` before running any `abra app...` commands. + +### Even more granularity? + +The plain-text, file-based configuration format means that you could even keep the configuration for different apps on the same server in different repositories, e.g. having `git.example.com` configuration in a separate repository to `wordpress.example.com`, using per-file symlinks. + +We don't currently recommend this, because it might set inaccurate expectations about the security model – remember that, by default, **any user who can deploy apps to a Docker Swarm can manage _any_ app in that swarm**. + +[symlink]: https://en.wikipedia.org/wiki/Symlink + +## Running abra server side + +If you're on an environment where it's hard to run Docker, or command-line programs in general, you might want to install `abra` on a server instead of your local computer. + +To install `abra` on a different server than you'll be hosting your apps, just follow [getting started guide](/overview/) as normal. + +If you want to install `abra` on the same server, there's one change. + +Instead of providing your SSH connection details when you run `abra server add ...`, just use `default`: + +``` +abra server add default +``` + +!!! note "Technical details" + + This will tell `abra` to look at the Docker system running on the server, instead of a remote one. + +Make sure to back up your `~/abra/` directory on the server, or put it in version control, as well as other files you'd like to. + +## Managing secret data + +Co-op Cloud uses [Docker Secrets] to handle sensitive data, like database passwords and API keys, securely: + +``` +DOCKER_CONTEXT=swarm.example.com docker secret ls +example_mediawiki_db_password_v1 +example_wordpress_db_password_v1 +``` + +`abra` includes several commands to make it easier to manage secrets: + +- `abra app secret generate` -- to auto-generate a single secret, or all secrets defined by the app, and store them in the Docker Swarm store, +- `abra app secret insert` -- to insert a single secret value from the Docker Swarm store, +- `abra app secret delete` -- to remove a single secret, or all secrets defined in the app, from the Docker Swarm store. + + + +### Secret versions + +You will notice `v1` in the example secret names above: like Docker Configs, Docker Secrets are [immutable], which means that their values can't be changed after they're set. To accommodate this, Co-op Cloud uses the established convention of "secret versions". Every time you change (rotate) a secret, you will insert it as a new version. + +Because secret versions are managed per-instance by the people deploying their apps, secret versions are stored in the `.env` file for each app: + +``` +find -L ~/.abra/servers/ -name '*.env' -print0 | xargs -0 grep -h SECRET +OIDC_CLIENT_SECRET_VERSION=v1 +RPC_SECRET_VERSION=v1 +CLIENT_SECRET_VERSION=v1 +... +``` + +If you try and add a secret version which already exists, Docker will helpfully complain: + +``` +abra app example_wordpress secret insert db_password v1 foobar +Error response from daemon: rpc error: code = AlreadyExists desc = secret example_wordpress_db_password_v1 already exists +``` + +By default, new app instances will look for `v1` secrets. + +### Generating secrets automatically + +You can generate secrets in one of two ways: + +1. While running `abra app new `, by passing `--secrets` +2. At any point once an app instance is defined, by running `abra app secret generate ...` (see `abra help secret generate` for full options) + +!!! note "How are secrets generated?" + + Depending on how the app is configured, you will require the `pwqgen` (from `passwdqc`) and `pwgen` binaries by default, although you can specify your own password-generation app when running `abra secret generate` by providing the `` argument. + +### Inserting secrets manually + +For third-party API tokens, like OAuth client secrets, or keys for services like Mailgun, you will be storing values you already have as the appropriately-named Docker secrets. `abra` provides a convenient interface to the underlying `docker secret create` command: + +``` +abra app example_wordpress secret insert db_password v2 "your-secret-here" +``` + +### Rotating a secret + +So, given how [secret versions](#versions) work, here's how you change a secret: + +1. Find out the current version number of the secret, e.g. by running `abra app example_wordpress config`, and choose a new one. Let's assume it's currently `v1`, so by convention the new secret will be `v2`. +2. Generate or insert the new secret: + ``` + abra app example_wordpress secret generate db_password v2 + ``` + or + ``` + abra app example_wordpress secret insert db_password v2 "foobar" + ``` +3. Edit the app configuration to change which secret version the app will use: + ``` + abra app example_wordpress config + ``` +4. Re-reploy the app with the new secret version: + ``` + abra app example_wordpress deploy + ``` + +### Storing secrets in `pass` + +The Co-op Cloud authors use the [UNIX `pass` tool][pass] to share sensitive data, including Co-op Cloud secrets, and `abra secret...` commands include a `--pass` option to automatically manage generated / inserted secrets: + +``` +# Store generated secrets in `pass`: +abra app new wordpress --secrets --pass +abra app example_wordpress secret generate --all --pass +# Store inserted secret in `pass`: +abra app example_wordpress secret insert db_password v2 --pass +# Remove secrets from Docker, and `pass`: +abra app example_wordpress secret rm --all --pass +``` + +This functionality currently relies on our specific `pass` structure; patches to make that configurable are very welcome! + +## Networking + +!!! note "So dark the con of Docker Networking" + + Our understanding of Docker networking is probably wrong. We're working on it. Plz send halp :pray: + +### Traefik networking + +[Traefik](https://doc.traefik.io/traefik/) is our core web proxy, all traffic on a Co-op Cloud deployment goes through a running Traefik container. When setting up a new Co-op Cloud delpyment, `abra` creates a "global" [overlay network](https://docs.docker.com/network/overlay/) which traefik is hooked up to. This is the network that other apps use to speak to traefik and get traffic routed to them. Not every service in every app is also included in this network and hence not internet-facing (by convention, we name this network `internal`, see more below). + +### App networking + +By convention, the main `app` service is wired up to the "global" traefik overlay network. This container is the one that should be publicy reachable on the internet. The other services in the app such as the database and caches should be not be publicly reachable or visible to other apps on the same instance. + +To deal with this, we make an additional "internal" network for each app which is namespaced to that app. So, if you deploy a Wordpress instance called `my_wordpress_blog` then there will be a network called `my_wordpress_blog_internal` created. This allows all the services in an app to speak to each other but not be reachable on the public internet. + +### Avoiding service namespace conflicts + +When referencing an `app` service in a config file, you should prefix with the `STACK_NAME` to avoid namespace conflicts (because all these containers sit on the traefik overlay network). You might want to do something like this `{{ env "STACK_NAME" }}_app` (using the often obscure dark magic of the Golang templating language). You can find examples of this approach used in the [Peertube recipe](https://git.coopcloud.tech/coop-cloud/peertube/src/commit/d1b297c5a6a23a06bf97bb954104ddfd7f736568/nginx.conf.tmpl#L9). diff --git a/docs/operators/index.md b/docs/operators/index.md index 0bda76e1..2493ce8e 100644 --- a/docs/operators/index.md +++ b/docs/operators/index.md @@ -1,3 +1,5 @@ --- -title: TODO +title: Operators Guide --- + +TODO. diff --git a/docs/operators/trouble.md b/docs/operators/trouble.md new file mode 100644 index 00000000..d0eb750f --- /dev/null +++ b/docs/operators/trouble.md @@ -0,0 +1,7 @@ +--- +title: Troubleshooting +--- + +## Configuring SSH + +## CLI flag handling diff --git a/docs/operators/tutorial.md b/docs/operators/tutorial.md new file mode 100644 index 00000000..0f2966d0 --- /dev/null +++ b/docs/operators/tutorial.md @@ -0,0 +1,209 @@ +--- +title: New operators tutorial +--- + +## The moving parts + +Co-op Cloud is made up of a few simple, composable pieces. The system does not rely on any one specific implementation: each part may be replaced and/or extended as needed. + +We want to build a reliable and long-term sustainable project and that means allowing for different implementations, open formats and a diverse project organisation. + +Here are the main technical concepts listed below, once you [grok](https://en.wikipedia.org/wiki/Grok) this, you grok the moving parts of the project. + +### Libre software apps + +Libre software apps are tools, websites & clients that you may already use in your daily life: [Nextcloud], [Jitsi], [Mediawiki], [Rocket.chat] and [many more]! These are tools that are created by volunteer communities who use [free software licenses] in order to build up the public software commons and offer more digital alternatives to [proprietary systems]. + +The communities who develop these softwares also publish them using [containers]. For example, here is the [Nextcloud hub.docker.com account] which allows end-users to quickly deploy a new Nextcloud instance. There is a growing consensus in the free software community that containers are a useful and time saving format for distribution. + +Learn more about why we use containers [in the FAQ section](/faq/#why-containers). + +[nextcloud]: https://nextcloud.com +[jitsi]: https://jitsi.org +[mediawiki]: https://mediawiki.org +[rocket.chat]: https://rocket.chat +[many more]: /recipes/ +[free software licenses]: https://www.gnu.org/philosophy/free-sw.html +[nextcloud hub.docker.com account]: https://hub.docker.com/_/nextcloud +[proprietary systems]: https://en.wikipedia.org/wiki/Proprietary_software +[containers]: https://www.docker.com/resources/what-container + +### The recipe packaging format + +However, just having a container of an app is often not enough. The work required to deploy that app in a "production ready" setup is still too time intensive and often involves a duplication of effort. Each service provider needs to deal with the same problems: stable versioning, backup plan, secret management, upgrade plan, monitoring and the list goes on. + +Individual free software projects can't take on all this responsibility. They provide the containers as is, in a secure and ready-to-go manner but it is up to service providers to worry about how the app is deployed. + +Therefore, Co-op Cloud proposes a packaging format, which we refer to as a recipe, that describes the entire production state of the app in a single place. This format uses the existing [standards based compose specification]. + +This is a file format which is most commonly used by the [Docker compose] tool but Co-op Cloud **does not** require the use of Docker compose itself. Furthermore, as described below, we also don't rely on the actual Docker CLI itself either. We do however use a lot of the underlying libraries. We're happily `docker` & `docker-compose` CLI independent! Learn more about why we use the compose specification [in the FAQ section](/faq/#why-use-the-compose-specification). + +[Each recipe] that Co-op cloud provides is described using the compose specification and makes use of the upstream project published container. This is the core of our approach to working with the ecosystem of free software communities. We want to maximise the chances of sharing work, knowledge and build solidarity through concrete co-operation. + +[standards based compose specification]: https://compose-spec.io +[docker compose]: https://docs.docker.com/compose/ +[each recipe]: /recipes/ + +### Container orchestrator + +Once we have our app packaged as a recipe, we need a deployment environment. Production deployments are typically expected to support a number of features which give hosters and end-users guarantees for uptime, stability and scale. + +The Co-op cloud makes use of [Docker swarm] as a deployment environment. It offers an approriate feature set which allows us to support zero-down time upgrades, seamless app rollbacks, automatic deploy failure handling, scaling, hybrid cloud setups and maintain a decentralised design. + +Learn more about why we use Docker swarm [in the FAQ section](/faq/#why-docker-swarm). + +[docker swarm]: https://docs.docker.com/engine/swarm/ + +### Command-line tool + +Finally, with an app and deployment environment, we need a tool to read the recipe package format and actually deploy it to that environment. For this, we have developed and published the [abra] command-line tool. + +`abra` aims at providing a simple command-line interface for managing your own Co-op Cloud. You can bootstrap machines with the required tools, create new apps, deploy them, back them up, restore them and so on. `abra` is written in [Go](https://go.dev/) and uses a lot of the libraries that the `docker` and `docker-compose` CLIs use but does not rely on them directly. + +`abra` is our flagship command-line client but it does not need to be the only client. `abra` was designed in such a way that is complements a workflow which can still be done completely manually. If Co-op Cloud goes away tomorrow, our configuration commons would still be useful and usable. + +[abra]: https://git.autonomic.zone/coop-cloud/abra + +## Deploy your first app + +In order to deploy an app you need two things: + +1. a server (e.g. [Hetzner VPS](https://www.hetzner.com/cloud)), with 1. SSH access and 2. a public IP address +2. a DNS provider (e.g. [Gandi](https://www.gandi.net/en)) + +### Create a server + +Co-op Cloud has itself near zero system requirements. You only need to worry about the system resource usage of your apps and the overhead of running containers with the docker runtime (often negligible. If you want to know more, see [this FAQ entry](/faq/#isnt-running-everything-in-containers-inefficient)). + +We will deploy a new Nextcloud instance in this guide, so you will only need 1GB of RAM according to [their documentation](https://docs.nextcloud.com/server/latest/admin_manual/installation/system_requirements.html). You may also be interested in this [FAQ entry](/faq/#arent-containers-horrible-from-a-security-perspective) if you are curious about security in the context of containers. + +### Wire up DNS + +Typically, you'll need two A records, one to point to the VPS itself and another to support sub-domains for the apps. You can then support an app hosted on your root domain (e.g. `example.com`) and other apps on sub-domains (e.g. `foo.example.com`, `bar.example.com`). At time of writing, it is not possible to support multiple apps sharing the same domain. + +Your entries in your DNS provider setup might look like the following. + + @ 1800 IN A 116.203.211.204 + *. 1800 IN A 116.203.211.204 + +Where `116.203.211.204` can be replaced with the IP address of your server. + +### Bootstrap `abra` + +You have two options for setting up your command-line client. You can install `abra` on your local development machine or remotely on the actual server. Please see + +Once your DNS and Docker daemon are up, you can install [`abra`](https://git.coopcloud.tech/coop-cloud/abra) locally on your developer machine and hook it up to your server. + +Firstly, install `abra` locally. + +```bash +curl https://install.abra.autonomic.zone | bash +``` +The source for this script [is here](https://git.coopcloud.tech/coop-cloud/abra/src/branch/main/scripts/installer/installer). + +the installer will verify the checksum. If you want to download abra yourself, you can grab it from the [releases page](https://git.coopcloud.tech/coop-cloud/abra/releases) along with the checksums.txt file. If you decide to do this, please run: +```bash +grep $(sha256sum abra_[version]_[platform]) checksums.txt > /dev/null && echo "checksum OK" +``` +if "checksum OK" appears in your terminal - you're good to go! otherwise, you have downloaded a corrupted file. + +You may need to add the `~/.local/bin/` directory with your `$PATH` in order to run the executable. + +```bash +export PATH=$PATH:$HOME/.local/bin +abra -h # check it works +``` + +Now you can connect `abra` with your new server. + +```bash +abra server add example.com +``` + +Where `example.com` is replaced with your server DNS name. + +!!! note "About SSH" + + `abra` uses Docker's built-in SSH support to make a secure connection to a + remote Docker daemon, to deploy and manage apps from your local development + machine. + + If you need to specify a non-standard port, and/or different username, for SSH, + add them as extra arguments: + + ```bash + abra server add -p example.com username 2222 + ``` + +The `-p` or `--provision` flag means that abra will initialise the [new single-host swarm](https://docs.docker.com/engine/swarm/key-concepts/) on your server. + +You will now have a new `~/.abra/` folder on your local file system which stores all the configuration of your Co-op Cloud instance. You can easily share this as a git repository with others. + +### Deploy Traefik + +In order to have your Co-op cloud installation automagically provision SSL certificates, we will first install [Traefik](https://doc.traefik.io/traefik/). This tool is the main entrypoint for all web requests (e.g. like NGINX) and supports automatic SSL certificate configuration and other quality-of-life features which make deploying libre apps more enjoyable. + +```bash +abra app new traefik +``` + +You will want to take a look at your generated configuration and tweak the `LETS_ENCRYPT_EMAIL` value: + +```bash +abra app config traefik +``` + +Every app you deploy will have one of these `.env` files, which contains variables which will be injected into app configurations when deployed. Variables starting with `#` are optional, others are required. + +``` +abra app deploy traefik +``` + +### Deploy Nextcloud + +And now we can deploy apps. + +Let's create a new Nextcloud app. + +```bash +abra app new nextcloud +``` + +And we need to generate secrets for the app: database connection password, root password and admin password. + +```bash +abra app secret generate --all nextcloud +``` + +!!! warning + + Take care, these secrets are only shown once on the terminal so make sure + to take note of them! `abra` makes use of the [Docker secrets](/secrets/) + mechanism to ship these secrets securely to the server and store them as + encrypted data. + +Then we can deploy the Nextcloud. + +```bash +abra app deploy nextcloud +``` + +We can watch to see that things come up correctly. + +```bash +abra app ps nextcloud # status check +abra app logs nextcloud # logs watch +``` + +!!! note + + Since Nextcloud takes some time to come up live, you can run the `ps` + command under `watch` like so. + + ```bash + watch abra app ps nextcloud + ``` + + And you can wait until you see that all containers have the "Running" state. + +Your `traefik` instance will detect that a new app is coming up and generate SSL certificates for it. diff --git a/docs/organisers/index.md b/docs/organisers/index.md index 87d5b120..2ad27189 100644 --- a/docs/organisers/index.md +++ b/docs/organisers/index.md @@ -2,24 +2,4 @@ title: Organisers Guide --- -## Monthly updates - -We have decided we'll try to do monthly progress updates. These will be published on the Co-op Cloud blog. It's a pretty loose format and we're basically just copy/pasta'ing things to a public pad during the month: ["This month in Co-op Cloud"](https://pad.autonomic.zone/YHKn4vHORmS6wjN1t2zi5A?both). Feel free to add your items to the monthly agenda and they will be included! All the previous posts can be seen [here](https://coopcloud.tech/blog/). - -## Kite Flying Hours - -The "Kite Flying Hour" is a weekly public moment where anyone can "drop by" into a Jitsi call and ask/do/propose whatever and meet some people who are currently working on the project. We haven't worked it all out but our process for now is the following. - -Someone from Autonomic will volunteer to be present and talk about the project for an hour weekly from 15:00 CEST - 16:00 CEST. We announce the hour via our socials: A [pinned toot](https://social.coop/web/statuses/106528094828958420) on [`@coopcloud@social.coop`](https://social.coop/@coopcloud) and a post to the `#coopcloud:autonomic.zone` room. - -Here is some invitation boilerplate which you can use: - -> Hey folks, you're all warmly invited to the Co-op Cloud Kite Flying Hour at `$X_TIME` `$Y_TZ` `$Z_DATE` over in [meet.jit.si/CoopCloudKiteFlyingHour](https://meet.jit.si/CoopCloudKiteFlyingHour)! -> -> Inspired by exquisite childhood memories of [flying kites, eating popsicles and looking at clouds](https://norwichhistory.org/norwich-a-z-j-is-for-jigsaw/), it's an open hour to come hang out online and discuss/co-work/lurk/etc. around the [Co-op Cloud](https://coopcloud.tech/) project. -> -> There are no "stupid questions"! It's a space to inquire, be curious and have a good time and get to know each other. -> -> We take notes and doodle on [this collaboratively editable pad](https://pad.autonomic.zone/LqotfSJJRj69RcTtWmr7iw). If you don't have time to attend, feel free to drop your questions and some contact details also, so we can get in touch. This is only the first Kite Flying Hour in a recurring series of Kite Flying Hours. -> -> Hope to see you there! ☁️ 🌞 🖥️ +TODO. diff --git a/docs/organisers/schedules.md b/docs/organisers/schedules.md new file mode 100644 index 00000000..9e4a0227 --- /dev/null +++ b/docs/organisers/schedules.md @@ -0,0 +1,25 @@ +--- +title: Schedules +--- + +## Monthly updates + +We have decided we'll try to do monthly progress updates. These will be published on the Co-op Cloud blog. It's a pretty loose format and we're basically just copy/pasta'ing things to a public pad during the month: ["This month in Co-op Cloud"](https://pad.autonomic.zone/YHKn4vHORmS6wjN1t2zi5A?both). Feel free to add your items to the monthly agenda and they will be included! All the previous posts can be seen [here](https://coopcloud.tech/blog/). + +## Kite Flying Hours + +The "Kite Flying Hour" is a weekly public moment where anyone can "drop by" into a Jitsi call and ask/do/propose whatever and meet some people who are currently working on the project. We haven't worked it all out but our process for now is the following. + +Someone from Autonomic will volunteer to be present and talk about the project for an hour weekly from 15:00 CEST - 16:00 CEST. We announce the hour via our socials: A [pinned toot](https://social.coop/web/statuses/106528094828958420) on [`@coopcloud@social.coop`](https://social.coop/@coopcloud) and a post to the `#coopcloud:autonomic.zone` room. + +Here is some invitation boilerplate which you can use: + +> Hey folks, you're all warmly invited to the Co-op Cloud Kite Flying Hour at `$X_TIME` `$Y_TZ` `$Z_DATE` over in [meet.jit.si/CoopCloudKiteFlyingHour](https://meet.jit.si/CoopCloudKiteFlyingHour)! +> +> Inspired by exquisite childhood memories of [flying kites, eating popsicles and looking at clouds](https://norwichhistory.org/norwich-a-z-j-is-for-jigsaw/), it's an open hour to come hang out online and discuss/co-work/lurk/etc. around the [Co-op Cloud](https://coopcloud.tech/) project. +> +> There are no "stupid questions"! It's a space to inquire, be curious and have a good time and get to know each other. +> +> We take notes and doodle on [this collaboratively editable pad](https://pad.autonomic.zone/LqotfSJJRj69RcTtWmr7iw). If you don't have time to attend, feel free to drop your questions and some contact details also, so we can get in touch. This is only the first Kite Flying Hour in a recurring series of Kite Flying Hours. +> +> Hope to see you there! ☁️ 🌞 🖥️ diff --git a/docs/organisers/tutorial.md b/docs/organisers/tutorial.md new file mode 100644 index 00000000..1c59676d --- /dev/null +++ b/docs/organisers/tutorial.md @@ -0,0 +1,5 @@ +--- +title: New organisers tutorial +--- + +TODO. diff --git a/docs/x-archive/abra.md b/docs/x-archive/abra.md deleted file mode 100644 index 2a13fdd2..00000000 --- a/docs/x-archive/abra.md +++ /dev/null @@ -1,34 +0,0 @@ ---- -title: Abra ---- - -## Enable auto-completion - -### Bash - -Copy `scripts/autocomplete/bash` into `/etc/bash_completion.d/` and rename -it to abra. - -``` -sudo cp scripts/autocomplete/bash /etc/bash_completion.d/abra -source /etc/bash_completion.d/abra -``` - -In development, you can source the script in your git checkout, just make sure -to set `PROG=abra`, otherwise it'll add completion to the wrong command: - -``` -PROG=abra source /path/to/abra/scripts/autocomplete/bash -``` - -### (Fi)Zsh - -(fi)zsh doesn't have an autocompletion folder by default but you can create one, then copy `scripts/autocomplete/zsh` into it and add a couple lines to your `~/.zshrc` or `~/.fizsh/.fizshrc` - -``` -sudo mkdir /etc/zsh/completion.d/ -sudo cp scripts/autocomplete/zsh /etc/zsh/completion.d/abra -echo "PROG=abra\n_CLI_ZSH_AUTOCOMPLETE_HACK=1\nsource /etc/zsh/completion.d/abra" >> ~/.zshrc -``` - -(replace .zshrc with ~/.fizsh/.fizshrc if you use fizsh) diff --git a/docs/x-archive/app-config-guide.md b/docs/x-archive/app-config-guide.md deleted file mode 100644 index a7599c0e..00000000 --- a/docs/x-archive/app-config-guide.md +++ /dev/null @@ -1,15 +0,0 @@ ---- -title: App config guide ---- - -The tips that were previously on this page have moved to the relevant recipe README files, to keep everything in one place while we figure out the best long-term home for per-app documentation. Find the READMEs here: - -- [Keycloak][keycloak] -- [Nextcloud][nextcloud] -- [Drone][drone] -- [Peertube][peertube] - -[keycloak]: https://git.coopcloud.tech/coop-cloud/keycloak -[nextcloud]: https://git.coopcloud.tech/coop-cloud/nextcloud -[drone]: https://git.coopcloud.tech/coop-cloud/drone -[peertube]: https://git.coopcloud.tech/coop-cloud/peertube diff --git a/docs/x-archive/config.md b/docs/x-archive/config.md deleted file mode 100644 index 34a928b0..00000000 --- a/docs/x-archive/config.md +++ /dev/null @@ -1,108 +0,0 @@ ---- -title: Manage your app configuration ---- - -## Understanding app and server configuration - -Co-op Cloud stores per-app configuration in the `$USER/.abra/servers` directory, on whichever machine you're running `abra` on (by default, your own workstation). - -The format of these configuration files is the same environment variable syntax used by Docker (with the `env_file:` statement in a `docker-compose.yml` file, or the `--env-file` option to `docker run`) and `direnv`: - -``` -abra app example_wordpress config -TYPE=wordpress - -DOMAIN=wordpress.example.com -## Domain aliases -EXTRA_DOMAINS=', `www.wordpress.example.com`' -LETS_ENCRYPT_ENV=production -... -``` - -`abra` doesn't mind if `~/.abra/servers`, or any of its subdirectories, is a [symlink], so you can keep your app definitions wherever you like! - -``` -mv ~/.abra/servers/ ~/coop-cloud -ln -s ~/coop-cloud ~/.abra/servers -``` - -## Backing up your app configuration - -Just make sure the `~/.abra/servers` is included in the configuration of your favourite backup tool. - -You can optionally also backup `~/.abra/apps`, if you'd like to keep an exact copy of the application versions you currently have deployed. Otherwise, they'll be automatically downloaded the first time you run an `abra app...` command. - -You don't need to worry about `~/.abra/vendor` or `~/.abra/src` directories, which will be likewise recreated automatically as and when you need them. - - - -## Version-control your app configs (using git) - -Because `~/.abra/servers` is a collection of plain-text files, it's easy to keep your backup configuration in a version control system (we use `git`, others would almost certainly work). - -This is particularly recommended if you're collaborating with others, so that you can all run `abra app...` commands without having to maintain your own separate, probably-conflicting, configuration files. - -In the simple case where you only have one server configured with `abra`, or everyone in your team is using the same set of servers, you can version-control the whole `~/.abra/servers` directory: - -``` -cd ~/.abra/servers -git init -git add . -git commit -m "Initial import" -``` - -!!! warning "Test your revision-control self-discipline" - - `abra` does not yet help keep your app definitions are up-to-date. - - Make sure to run `git add` / `git commit` after making configuration changes, and `cd ~/.abra/servers && git pull` before running `abra app...` commands. - - Patches to add some safety checks and auto-updates would be very welcome! 🙏 - -## Collaborating with multiple teams - -In a more complex situation, where you're using Co-op Cloud to manage several servers, and you're collaborating with different people on different servers, you can set up **a separate repository for each subdirectory in `~/.abra/servers`**, or even a mixture of single-server and multi-server repositories: - -``` -ls -l ~/.abra/servers -# Example.com's own app configuration: -lrwxrwxrwx. 1 user user 49 Oct 30 22:42 swarm.example.com -> /home/user/Example/coop-cloud-apps/swarm.example.com -# Configuration for one of Example.com's clients – part of the same repository: -lrwxrwxrwx. 1 user user 49 Oct 30 22:42 swarm.client.com -> /home/user/Example/coop-cloud-apps/swarm.client.com -# A completely separate project, part of a different repository: -lrwxrwxrwx. 1 user user 49 Oct 30 22:42 swarm.demonstration.com -> /home/user/Demonstration/coop-cloud-apps -``` - -To make setting up these symlinks easier, you might want to include a simple installer script in your configuration repositories. - -We don't have a public example of this yet, but something like this should do the trick: - -1. Save this as `Makefile` in your repository: - - ``` - # -s symlink, -f force creation, -F don't create symlink in the target dir - link: - @mkdir -p ~/.abra/servers/ - @for SERVER in $$(find -maxdepth 1 -type d -name "[!.]*"); do \ - echo ln -sfF "$$(pwd)/$${SERVER#./}" ~/.abra/servers/ ; \ - ln -sfF "$$(pwd)/$${SERVER#./}" ~/.abra/servers/ ; \ - done - ``` - - This will set up symlinks from each directory in your repository to a correspondingly-named directory in `~/.abra/servers` – if your repository has a `swarm.example.com` directory, it'll be linked as `~/.abra/servers/swarm.example.com`. - -2. Tell your collaborators (e.g. in the repository's `README`), to run `make` in their repository check-out. - -!!! warning "You're on your own!" - - As with the [simple repository set-up above](#version-control), `abra` doesn't yet help you update your version control system when you make changes, nor check version control to make sure you have the latest configuration. - - Make sure to `commit` and `push` after you make any configuration changes, and `pull` before running any `abra app...` commands. - -## Even more granularity? - -The plain-text, file-based configuration format means that you could even keep the configuration for different apps on the same server in different repositories, e.g. having `git.example.com` configuration in a separate repository to `wordpress.example.com`, using per-file symlinks. - -We don't currently recommend this, because it might set inaccurate expectations about the security model – remember that, by default, **any user who can deploy apps to a Docker Swarm can manage _any_ app in that swarm**. - -[symlink]: https://en.wikipedia.org/wiki/Symlink diff --git a/docs/x-archive/deploy.md b/docs/x-archive/deploy.md deleted file mode 100644 index b929d99e..00000000 --- a/docs/x-archive/deploy.md +++ /dev/null @@ -1,145 +0,0 @@ ---- -title: Deploy your first app ---- - -In order to deploy an app you need two things: - -1. a server (e.g. [Hetzner VPS](https://www.hetzner.com/cloud)), with - - SSH access - - a public IP address -2. a DNS provider (e.g. [Gandi](https://www.gandi.net/en)) - -## Create your server - -Co-op Cloud has itself near zero system requirements. You only need to worry about the system resource usage of your apps and the overhead of running containers with the docker runtime (often negligible. If you want to know more, see [this FAQ entry](/faq/#isnt-running-everything-in-containers-inefficient)). We will deploy a new Nextcloud instance in this guide, so you will only need 1GB of RAM according to [their documentation](https://docs.nextcloud.com/server/latest/admin_manual/installation/system_requirements.html). You may also be interested in this [FAQ entry](/faq/#arent-containers-horrible-from-a-security-perspective) if you are curious about security in the context of containers. - -## Wire up your DNS - -Typically, you'll need two A records, one to point to the VPS itself and another to support sub-domains for the apps. You can then support an app hosted on your root domain (e.g. `example.com`) and other apps on sub-domains (e.g. `foo.example.com`, `bar.example.com`). Your entries in your DNS provider setup might look like the following. - - @ 1800 IN A 116.203.211.204 - *. 1800 IN A 116.203.211.204 - -Where `116.203.211.204` can be replaced with the IP address of your server. - -## Install server prerequisites - -You'll want to install [Docker](https://www.docker.com/) on your server. This can be done by following the [install documentation](https://docs.docker.com/engine/install/). - -## Bootstrap `abra` - -Once your DNS and docker daemon are up, you can install [`abra`](https://git.coopcloud.tech/coop-cloud/abra) locally on your developer machine and hook it up to your server. - -Firstly, install `abra` locally. - -```bash -curl https://install.abra.autonomic.zone | bash -``` -The source for this script [is here](https://git.coopcloud.tech/coop-cloud/abra/src/branch/main/scripts/installer/installer). - -the installer will verify the checksum. If you want to download abra yourself, you can grab it from the [releases page](https://git.coopcloud.tech/coop-cloud/abra/releases) along with the checksums.txt file. If you decide to do this, please run: -```bash -grep $(sha256sum abra_[version]_[platform]) checksums.txt > /dev/null && echo "checksum OK" -``` -if "checksum OK" appears in your terminal - you're good to go! otherwise, you have downloaded a corrupted file. - -You may need to add the `~/.local/bin/` directory with your `$PATH` in order to run the executable. - -```bash -export PATH=$PATH:$HOME/.local/bin -abra -h # check it works -``` - -Now you can connect `abra` with your new server. - -```bash -abra server add example.com -``` - -Where `example.com` is replaced with your server DNS name. - -!!! note "About SSH" - - `abra` uses Docker's built-in SSH support to make a secure connection to a - remote Docker daemon, to deploy and manage apps from your local development - machine. - - If you need to specify a non-standard port, and/or different username, for SSH, - add them as extra arguments: - - ```bash - abra server add -p example.com username 2222 - ``` - -The `-p` or `--provision` flag means that abra will initialise the [new single-host swarm](https://docs.docker.com/engine/swarm/key-concepts/) on your server. - -You will now have a new `~/.abra/` folder on your local file system which stores all the configuration of your Co-op Cloud instance. You can easily share this as a git repository with others. - -## Deploy Traefik - -In order to have your Co-op cloud installation automagically provision SSL certificates, we will first install [Traefik](https://doc.traefik.io/traefik/). This tool is the main entrypoint for all web requests (e.g. like NGINX) and supports automatic SSL certificate configuration and other quality-of-life features which make deploying libre apps more enjoyable. - -```bash -abra app new traefik -``` - -You will want to take a look at your generated configuration and tweak the `LETS_ENCRYPT_EMAIL` value: - -```bash -abra app config traefik -``` - -Every app you deploy will have one of these `.env` files, which contains variables which will be injected into app configurations when deployed. Variables starting with `#` are optional, others are required. - -``` -abra app deploy traefik -``` - -## Deploy Nextcloud - -And now we can deploy apps. - -Let's create a new Nextcloud app. - -```bash -abra app new nextcloud -``` - -And we need to generate secrets for the app: database connection password, root password and admin password. - -```bash -abra app secret generate --all nextcloud -``` - -!!! warning - - Take care, these secrets are only shown once on the terminal so make sure - to take note of them! `abra` makes use of the [Docker secrets](/secrets/) - mechanism to ship these secrets securely to the server and store them as - encrypted data. - -Then we can deploy the Nextcloud. - -```bash -abra app deploy nextcloud -``` - -We can watch to see that things come up correctly. - -```bash -abra app ps nextcloud # status check -abra app logs nextcloud # logs watch -``` - -!!! note - - Since Nextcloud takes some time to come up live, you can run the `ps` - command under `watch` like so. - - ```bash - watch abra app ps nextcloud - ``` - - And you can wait until you see that all containers have the "Running" state. - -Your `traefik` instance will detect that a new app is coming up and generate SSL certificates for it. diff --git a/docs/x-archive/networking.md b/docs/x-archive/networking.md deleted file mode 100644 index 75f9846b..00000000 --- a/docs/x-archive/networking.md +++ /dev/null @@ -1,21 +0,0 @@ ---- -title: Docker Networking ---- - -!!! warning - - Our understanding of Docker networking is probably wrong. We're working on it. - -# Traefik networking - -When a new Co-op Cloud instance is made, we make a "global" [overlay network](https://docs.docker.com/network/overlay/) which traefik sits on. This is the network that other apps use to speak to traefik and get traffic routed to them. Not every service in every app is also included in this network and hence not internet-facing. - -# App networking - -One service in an app, typically the one called `app`, sits on the "global" traefik network. This container is the one that should be publicy reachable on the internet. The other services in the app such as the database and caches should be not be publicly reachable or visible to other apps on the same instance. - -To deal with this, we make an additional "internal" network for each app which is namespaced to that app. So, if you deploy a Wordpress instance called `my_wordpress_blog` then there will be a network called `my_wordpress_blog_internal` created. This allows all the services in an app to speak to each other but not be reachable on the public internet. - -# Avoiding namespace conflicts - -When referencing an `app` service in a config file, you should prefix with the `STACK_NAME` to avoid namespace conflicts (because all these containers sit on the traefik overlay network). You might want to do something like this `{{ env "STACK_NAME" }}_app` (using Golang templating). diff --git a/docs/x-archive/overview.md b/docs/x-archive/overview.md deleted file mode 100644 index 8082f129..00000000 --- a/docs/x-archive/overview.md +++ /dev/null @@ -1,64 +0,0 @@ ---- -title: Technical overview ---- - -The Co-op Cloud is made up of a few simple, composable pieces. The system does not rely on any one specific implementation: each part may be replaced and extended as needed. - -- [Libre software apps](#libre-software-apps) -- [The packaging format](#the-packaging-format) -- [Container orchestrator](#container-orchestrator) -- [Command-line tool](#command-line-tool) - -## Libre software apps - -Applications that you may already use in your daily life: [Nextcloud], [Jitsi], [Mediawiki], [Rocket.chat] and [many more]! These are tools that are created by volunteer communities who use [free software licenses] in order to build up the public software commons and offer more digital alternatives. - -The communities who develop these softwares also publish them using containers. For example, here is the [Nextcloud hub.docker.com account] which allows end-users to quickly deploy a new Nextcloud instance. - -Learn more about why we use containers [in the FAQ section](/faq/#why-containers). - -[nextcloud]: https://nextcloud.com -[jitsi]: https://jitsi.org -[mediawiki]: https://mediawiki.org -[rocket.chat]: https://rocket.chat -[many more]: /apps/ -[free software licenses]: https://www.gnu.org/philosophy/free-sw.html -[nextcloud hub.docker.com account]: https://hub.docker.com/_/nextcloud - -## The packaging format - -The work required to take a new instance of an application and make it production ready is still too time intensive and often involves a duplication of effort. Each service provider needs to deal with the same problems: stable versioning, backup plan, secret management, upgrade plan, monitoring and the list goes on. - -Therefore, the Co-op Cloud proposes a packaging format which describes the entire production state of the application in a single place. This format uses the existing [standards based compose specification]. This is a file format which is most commonly used by the [Docker compose] tool but Co-op Cloud **does not** require the use of Docker compose itself. - -[Each application] that the Co-op cloud provides is described using the compose specification and makes use of the upstream project published container. - -Learn more about why we use the compose specification [in the FAQ section](/faq/#why-use-the-compose-specification). - -[standards based compose specification]: https://compose-spec.io -[docker compose]: https://docs.docker.com/compose/ -[each application]: /apps/ - -## Container orchestrator - -Once we have our application packaged, we need a deployment environment. Production deployments are typically expected to support a number of features which give hosters and end-users guarantees for uptime, stability and scale. - -The Co-op cloud makes use of [Docker swarm] as a deployment environment. It offers an approriate feature set which allows us to support zero-down time upgrades, seamless application rollbacks, automatic deploy failure handling, scaling, hybrid cloud setups and maintain a decentralised design. - -Learn more about why we use Docker swarm [in the FAQ section](/faq/#why-docker-swarm). - -[docker swarm]: https://docs.docker.com/engine/swarm/ - -## Command-line tool - -Finally, with an application and an application environment, we need a tool to read that package format and actually deploy it to the environment. For this, we have developed and published the [abra] command-line tool. - -Abra aims at providing a simple command-line interface for managing your own co-op cloud. You can bootstrap machines with the required tools, create new applications, deploy them, back them up, restore them and so on. - -[abra]: https://git.autonomic.zone/coop-cloud/abra - -## Next steps - -Now that you've got an overview, it is time to [deploy your first application]. - -[deploy your first application]: /deploy/ diff --git a/docs/x-archive/recipe-structure.md b/docs/x-archive/recipe-structure.md deleted file mode 100644 index 0c96a7b5..00000000 --- a/docs/x-archive/recipe-structure.md +++ /dev/null @@ -1,50 +0,0 @@ ---- -title: Recipe structure ---- - -A recipe is a git repository that contains instructions for creating stacks that abra can read and interpret. You'll see a couple of files there: - -- [compose.yml](#composeyml) (required!) -- [.env.sample](#envsample) (required!) -- [abra.sh](#abrash) -- [entrypoint.sh](#entrypointsh) -- [other compose files](#other-compose-files) -- [other files](#other-files) - - -## compose.yml - -this is a [compose specification](https://compose-spec.io/) compliant file that contains a list of: - -- services -- secrets -- networks -- volumes -- configs - -that describe what is needed to run a stack. Whenever you deploy an app, abra reads this file to cook the stack. - - -## .env.sample - -this file is a skeleton for environmental variables that should be adjusted by the user. Examples include: domain or php extention list. Whenever you create a new app with `abra app new` this file gets copied to the ~/.abra/servers/server-domain/app-name.env and when you run `abra app config` you're editing this file. - - -## abra.sh - -`abra.sh` provides shell functions for running non-standard deploy, restore, rollback, backup and upgrade. This is only needed for some packages (such as nextcloud or wordpress) - - -## entrypoint.sh - -after docker creates the filesystem and copies files into a new container it runs what's called an entrypoint. This is usually a shell script that exports some variables and runs the application. Sometimes the vendor entrypoint doesn't do everything that we need it to do. In that case you can write your own entrypoint, do whatever you need to do and then run the vendor entrypoint. For a simple example check [entrypoint.sh for croc](https://git.coopcloud.tech/coop-cloud/croc/src/commit/2f06e8aac52a3850d527434a26de0a242bea0c79/entrypoint.sh). In this case, croc needs the password to be exported as an environmental variable called `CROC_PASS`, and that is exactly what the entrypoint does before running vendor entrypoint. If you write your own entrypoint, it needs to be specified in the `config` section of compose.yml. - - -## other compose files - -i.e. compose.smtp.yml. These are used to provide non-essential functionality such as (registration) e-mails or single sign on. - - -## other files - -if you look at compose.yml (or compose.\*.yml) and see a `configs` section, that means this compose file is putting files in the container. This might be used for changing default (vendor) configuration, such as this [fpm-tune.ini file](https://git.coopcloud.tech/coop-cloud/nextcloud/src/commit/28425b6138603067021757de28c639ad464e9cf8/fpm-tune.ini) used to adjust php-fpm. diff --git a/docs/x-archive/secrets.md b/docs/x-archive/secrets.md deleted file mode 100644 index 186096ea..00000000 --- a/docs/x-archive/secrets.md +++ /dev/null @@ -1,109 +0,0 @@ ---- -title: Managing secret data ---- - -# Managing secret data - -Co-op Cloud uses [Docker Secrets] to handle sensitive data, like database passwords and API keys, securely: - -``` -DOCKER_CONTEXT=swarm.example.com docker secret ls -example_mediawiki_db_password_v1 -example_wordpress_db_password_v1 -``` - -`abra` includes several commands to make it easier to manage secrets: - -- `abra app secret generate` -- to auto-generate a single secret, or all secrets defined by the app, and store them in the Docker Swarm store, -- `abra app secret insert` -- to insert a single secret value from the Docker Swarm store, -- `abra app secret delete` -- to remove a single secret, or all secrets defined in the app, from the Docker Swarm store. - - - -## Secret versions - -You will notice `v1` in the example secret names above: like Docker Configs, Docker Secrets are [immutable], which means that their values can't be changed after they're set. To accommodate this, Co-op Cloud uses the established convention of "secret versions". Every time you change (rotate) a secret, you will insert it as a new version. - -Because secret versions are managed per-instance by the people deploying their apps, secret versions are stored in the `.env` file for each app: - -``` -find -L ~/.abra/servers/ -name '*.env' -print0 | xargs -0 grep -h SECRET -OIDC_CLIENT_SECRET_VERSION=v1 -RPC_SECRET_VERSION=v1 -CLIENT_SECRET_VERSION=v1 -... -``` - -If you try and add a secret version which already exists, Docker will helpfully complain: - -``` -abra app example_wordpress secret insert db_password v1 foobar -Error response from daemon: rpc error: code = AlreadyExists desc = secret example_wordpress_db_password_v1 already exists -``` - -By default, new app instances will look for `v1` secrets. - -## Generating secrets automatically - -You can generate secrets in one of two ways: - -1. While running `abra app new `, by passing `--secrets` -2. At any point once an app instance is defined, by running `abra app secret generate ...` (see `abra help secret generate` for full options) - -!!! note "How are secrets generated?" - - Depending on how the app is configured, you will require the `pwqgen` (from `passwdqc`) and `pwgen` binaries by default, although you can specify your own password-generation app when running `abra secret generate` by providing the `` argument. - -## Inserting secrets manually - -For third-party API tokens, like OAuth client secrets, or keys for services like Mailgun, you will be storing values you already have as the appropriately-named Docker secrets. `abra` provides a convenient interface to the underlying `docker secret create` command: - -``` -abra app example_wordpress secret insert db_password v2 "your-secret-here" -``` - -## Rotating a secret - -So, given how [secret versions](#versions) work, here's how you change a secret: - -1. Find out the current version number of the secret, e.g. by running `abra app example_wordpress config`, and choose a new one. Let's assume it's currently `v1`, so by convention the new secret will be `v2`. -2. Generate or insert the new secret: - ``` - abra app example_wordpress secret generate db_password v2 - ``` - or - ``` - abra app example_wordpress secret insert db_password v2 "foobar" - ``` -3. Edit the app configuration to change which secret version the app will use: - ``` - abra app example_wordpress config - ``` -4. Re-reploy the app with the new secret version: - ``` - abra app example_wordpress deploy - ``` - -## Storing secrets in `pass` - -The Co-op Cloud authors use the [UNIX `pass` tool][pass] to share sensitive data, including Co-op Cloud secrets, and `abra secret...` commands include a `--pass` option to automatically manage generated / inserted secrets: - -``` -# Store generated secrets in `pass`: -abra app new wordpress --secrets --pass -abra app example_wordpress secret generate --all --pass -# Store inserted secret in `pass`: -abra app example_wordpress secret insert db_password v2 --pass -# Remove secrets from Docker, and `pass`: -abra app example_wordpress secret rm --all --pass -``` - -This functionality currently relies on our specific `pass` structure; patches to make that configurable are very welcome! - -## What makes secrets secure? - -TODO - -[docker secrets]: https://docs.docker.com/engine/swarm/secrets/ -[immutable]: https://en.wikipedia.org/wiki/Immutable_object -[pass]: https://www.passwordstore.org diff --git a/docs/x-archive/server-side.md b/docs/x-archive/server-side.md deleted file mode 100644 index 2ac884cc..00000000 --- a/docs/x-archive/server-side.md +++ /dev/null @@ -1,24 +0,0 @@ - ---- -title: Running abra on the server ---- - -## Why? - -If you're on an environment where it's hard to run Docker, or command-line programs in general, you might want to install `abra` on a server instead of your local computer. - -To install `abra` on a different server than you'll be hosting your apps, just follow [getting started guide](/overview/) as normal. - -If you want to install `abra` on the same server, there's one change. - -Instead of providing your SSH connection details when you run `abra server add ...`, just use `default`: - -``` -abra server add default -``` - -!!! note "Technical details" - - This will tell `abra` to look at the Docker system running on the server, instead of a remote one. - -Make sure to back up your `~/abra/` directory on the server, or put it in version control, as well as other files you'd like to. diff --git a/docs/x-archive/troubleshooting.md b/docs/x-archive/troubleshooting.md deleted file mode 100644 index 226fef37..00000000 --- a/docs/x-archive/troubleshooting.md +++ /dev/null @@ -1,35 +0,0 @@ ---- -title: Troubleshooting ---- - -## SSH / connection problems - -Assuming: -- Hostname: `coopcloud.example.com` -- User: `username` -- Port: `222` - -### Step 1: Can you SSH to the server normally? - -Does `ssh username@coopcloud.example.com -p2222` work? - -If not, run through your standard oh-no-why-doesn't-SSH-work troubleshooting 🍀. - -### Step 2: Can you run remote Docker commands over SSH? - -Does `ssh username@coopcloud.example.com -p2222 docker ps` work? - -If not: -- Is the remote Docker daemon running? -- Is your user in the `docker` group? - -### Step 3: Does your Docker context work? - -``` -[user@hostname ~]$ DOCKER_CONTEXT=coopcloud.example.com docker ps -CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES -``` - -If you get an error message instead: -- Use `abra server ls` / `docker context ls` to double-check the SSH connection details -- Try removing the context with `docker context rm coopcloud.example.com`, then re-add it diff --git a/mkdocs.yml b/mkdocs.yml index 25d71458..3d6b0ce3 100644 --- a/mkdocs.yml +++ b/mkdocs.yml @@ -45,12 +45,20 @@ nav: - "Managed hosting": intro/managed.md - "Get in touch": intro/contact.md - "Credits": intro/credits.md - - "Operators Guide": operators/index.md - - "Maintainers Guide": maintainers/index.md - - "Organisers Guide": organisers/index.md + - "Operators Guide": + - operators/index.md + - "New operators tutorial": operators/tutorial.md + - "Operations handbook": operators/handbook.md + - "Troubleshooting abra": operators/trouble.md + - "Maintainers Guide": + - maintainers/index.md + - "New maintainers tutorial": maintainers/tutorial.md + - "Organisers Guide": + - organisers/index.md + - "New organisers tutorial": organisers/tutorial.md + - "Schedules": organisers/schedules.md - "Recipes": recipes/index.md - "Get Involved": get-involved/index.md - - "CLI Reference": cli/index.md - "Glossary": glossary/index.md plugins: