mayel
/
docs.coopcloud.tech
forked from coop-cloud/docs.coopcloud.tech
0
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

maintainers pass

This commit is contained in:
decentral1se 2022-02-03 17:42:31 +01:00
parent d9909367b8
commit f26fb3b66f
Signed by untrusted user: decentral1se
GPG Key ID: 03789458B3D0C410
2 changed files with 206 additions and 111 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

191
docs/maintainers/handbook.md
View File

@ -4,15 +4,198 @@ title: Packaging handbook
## Create a new recipe
TODO.
You can run `abra recipe new <recipe>` to generate a new `~/.abra/recipes/<recipe>` repository. See the [maintainers tutorial](/maintainers/tutorial) for help on getting started with packaging.
## Make a new release
## How are recipes are versioned?
TODO.
We'll use an example to work through this. Let's use [Gitea](https://hub.docker.com/r/gitea/gitea).
The Gitea project maintains a version, e.g. `1.14.3`. This version uses the [semver](https://semver.org) strategy for communicating what type of changes are included in each version, i.e., if there is a breaking change, Gitea will release a new version as `2.0.0`.
However, there are other types of changes that can happen for a recipe. Perhaps the database image gets a breaking update or the actual recipe configuration changes some environment variable. This can mean that end-users of the recipe need to do some work to make sure their updates will deploy successfully.
Therefore, we maintain an additional version part, in front of the project version. So, the first release of the Gitea recipe in the Co-op Cloud project has the version of `1.0.0+1.14.3`. This `x.y.z+` is the version part that the recipe maintainer manages. If a new available Gitea version comes out as `1.15` then the recipe maintainer will publish `1.1.0+1.15` as this is a backwards compatible update, following semantic versioning.
In all cases, we follow the semver semantics. So, if we upgrade the Gitea recipe from `1.14.3` to `1.15.3`, we still publish `1.1.0+1.15.3` as our recipe version. In this case, we skipped a few patch releases but it was all backwards compatible, so we only increment the minor version part.
## How do I release a new recipe version?
The commands uses for dealing with recipe versioning in `abra` are:
- `abra recipe upgrade`: upgrade the image tags in the compose configs of a recipe
- `abra recipe sync`: upgrade the deploy labels to match the new recipe version
- `abra recipe release`: publish a git tag for the recipe repo
The `abra` recipe publishing commands have been designed to complement a semi-automatic workflow. If `abra` breaks or doesn't understand what is going on, you can always finish the process manually with a few Git commands and a bit of luck. We designed `abra` to support this way due to the chaotic nature of container publishing versioning schemes. This handbook entry will show you everything you need to know to understand.
Let's take a practical example, publishing a new version of [Wordpress](https://git.coopcloud.tech/coop-cloud/wordpress).
If we run `abra recipe upgrade wordpress` (at time of running), we end up with a prompt to upgrade Wordpress to `5.9.0`. We can skip the database upgrade for now. Here is what that looks like:
```
➜ ~ abra recipe upgrade wordpress
? upgrade to which tag? (service: app, image: wordpress, tag: 5.8.3) 5.9.0
? upgrade to which tag? (service: db, image: mariadb, tag: 10.6) skip
WARN[0004] not upgrading mariadb, skipping as requested
```
Now, what happened? `abra` queried the upstream container repositories of all the images listed in the Wordpress recipe configuration and checked if there are new tags available. Once you make some choices on the prompt, `abra` will update the recipe configurations. Let's take a look by running `cd ~/.abra/recipes/wordpress && git diff`:
```diff
diff --git a/compose.yml b/compose.yml
index 1618ef5..6cd754d 100644
--- a/compose.yml
+++ b/compose.yml
@@ -3,7 +3,7 @@ version: "3.8"
services:
app:
- image: "wordpress:5.8.3"
+ image: "wordpress:5.9.0"
volumes:
- "wordpress_content:/var/www/html/wp-content/"
networks:
```
!!! warning "Here be versioning dragons"
`abra` doesn't understand all image tags unfortunately. There are limitations which we're still running into. You can pass `-a` to have `abra` list all available image tags from the upstream repository and then make a choice manually. See [`tagcmp`](https://git.coopcloud.tech/coop-cloud/tagcmp) for more info on how we implement image parsing.
Next, we need to update the label in the recipe, we can do that with `abra recipe sync wordpress`. You'll be prompted by a question asking what kind of upgrade this is. Take a moment to read the output and if it still doesn't make sense, read [this](/maintainers/handbook/#how-are-recipes-are-versioned). Since we're upgrading from `5.8.3` -> `5.9.0`, it is a minor release, so we choose `minor`:
```
➜ wordpress (master) ✗ abra recipe sync wordpress
...
INFO[0088] synced label coop-cloud.${STACK_NAME}.version=1.1.0+5.9.0 to service app
```
Once again, we can run `cd ~/.abra/recipes/wordpress && git diff` to see what `abra` has done for us:
```diff
diff --git a/compose.yml b/compose.yml
index 1618ef5..4a08db6 100644
--- a/compose.yml
+++ b/compose.yml
@@ -3,7 +3,7 @@ version: "3.8"
services:
app:
- image: "wordpress:5.8.3"
+ image: "wordpress:5.9.0"
volumes:
- "wordpress_content:/var/www/html/wp-content/"
networks:
@@ -48,7 +48,7 @@ services:
#- "traefik.http.routers.${STACK_NAME}.rule=HostRegexp(`{subdomain:.+}.${DOMAIN}`, `${DOMAIN}`)"
- "traefik.http.routers.${STACK_NAME}.tls.certresolver=${LETS_ENCRYPT_ENV}"
- "traefik.http.routers.${STACK_NAME}.entrypoints=web-secure"
- - "coop-cloud.${STACK_NAME}.version=1.0.2+5.8.3"
+ - "coop-cloud.${STACK_NAME}.version=1.1.0+5.9.0"
- "backupbot.backup=true"
- "backupbot.backup.path=/var/www/html"
```
You'll notice that `abra` figured out how to upgrade the Co-op Cloud version label according to our choice, `1.0.2` -> `1.1.0` is a minor update.
At this point, we're all set, we can run `abra recipe release --publish wordpress`. This will do the following:
1. run `git commit` the new changes
1. run `git tag` to create a new git tag named `1.1.0+5.9.0`
1. run `git push` to publish changes to the Wordpress repository
Here is the output:
```
WARN[0000] discovered 1.1.0+5.9.0 as currently synced recipe label
WARN[0000] previous git tags detected, assuming this is a new semver release
? current: 1.0.2+5.8.3, new: 1.1.0+5.9.0, correct? Yes
new release published: https://git.coopcloud.tech/coop-cloud/wordpress/src/tag/1.1.0+5.9.0
```
And once more, we can valdate this tag has been created with `cd ~/.abra/recipes/wordpress && git tag -l`.
## How are new recipe versions tested?
This is currently a manual process. Our best estimates are to do a backup and run a test deployment and see how things go.
Following the [entry above](/maintainers/handbook/#how-do-i-release-a-new-recipe-version), before running `abra recipe release --publish <recipe>`, you can deploy the new version of the recipe. You find an app that relies on this recipe and pass `-C/--chaos` to `ugrade` so that it accepts the locally unstaged changes.
It is good practice to take note of all the issues you ran into and share them with other operators. See [this entry](/maintainers/handbook/#how-do-i-write-version-release-notes) for more.
If you don't have time or are not an operator, reach out on our communication channels for an operator willing to do some testing.
## How do I write version release notes?
In the root of your recipe repository, run the following (if the folder doesn't already exist):
```
mkdir -p releases
```
And then simply create a text file which corresponds to the version release, e.g. `1.1.0+5.9.0` and write some notes. `abra` will show these when another operator runs `abra app deploy` / `abra app upgrade`.
## Generate the recipe catalogue
TODO.
To generate an entire new copy of the catalogue:
```
abra catalogue generate
```
You will most likely want to pass `--user/--username` / `--pass/--password` with container regsitry credentials to avoid rate limiting.
If you just want to generate a catalogue entry for a single recipe:
```
abra catalogue generate <recipe>
```
The changes are generated and added to `~/.abra/catalogue`, you can validate what is done by running:
```
cd ~/.abra/catalogue
git diff
```
You can pass `--publish` to have `abra` automatically publish those changes. You'll need to have write access to the [git.coopcloud.tech repository](https://git.coopcloud.tech/coop-cloud/recipes.coopcloud.tech).
## How is a recipe structured?
### `compose.yml`
This is a [compose specification](https://compose-spec.io/) compliant file that contains a list of:
- services
- secrets
- networks
- volumes
- configs
It describe what is needed to run an app. Whenever you deploy an app, `abra` reads this file.
### `.env.sample`
This file is a skeleton for environmental variables that should be adjusted by the user. Examples include: domain or php extention list. Whenever you create a new app with `abra app new` this file gets copied to the `~/.abra/servers/<server-domain>/<app-domain>.env` and when you run `abra app config <app-domain>` you're editing this file.
### `abra.sh`
The `abra.sh` provides versions for configs that are vendored by the recipe maintainer.
### `entrypoint.sh`
After docker creates the filesystem and copies files into a new container it runs what's called an entrypoint. This is usually a shell script that exports some variables and runs the application. Sometimes the vendor entrypoint doesn't do everything that we need it to do. In that case you can write your own entrypoint, do whatever you need to do and then run the vendor entrypoint.
For a simple example check the [entrypoint.sh for `croc`](https://git.coopcloud.tech/coop-cloud/croc/src/commit/2f06e8aac52a3850d527434a26de0a242bea0c79/entrypoint.sh). In this case, `croc` needs the password to be exported as an environmental variable called `CROC_PASS`, and that is exactly what the entrypoint does before running vendor entrypoint.
If you write your own entrypoint, it needs to be specified in the `config` section of compose.yml.
### Optional compose files
I.e. `compose.smtp.yml`. These are used to provide non-essential functionality such as (registration) e-mails or single sign on. These are typically loaded by specifying `COMPOSE_FILE="compose.yml:compose.smtp.yml"` in your app `.env` configuration. Then `abra` learns to include these optional files at deploy time. `abra` uses the usual `docker-compose` configuration merging technique when merging all the `compose.**.yml` files together at deploy time.
### Docker configs
If you look at a `compose.yml` file and see a `configs` section, that means this compose file is putting files in the container. This might be used for changing default (vendor) configuration, such as this [fpm-tune.ini file](https://git.coopcloud.tech/coop-cloud/nextcloud/src/commit/28425b6138603067021757de28c639ad464e9cf8/fpm-tune.ini) used to adjust `php-fpm.`
## Enable healthchecks

126
docs/maintainers/tutorial.md
View File

@ -4,24 +4,22 @@ title: New maintainers tutorial
## Package your first recipe
Let's take as an example, [Matomo web analytics](https://matomo.org/).
Packaging a recipe is basically knowing a bag of about 20 tricks. Once you learn them all, there is nothing more to learn. It can seem daunting at first but it's simple and easy to do once you know the tricks.
We'll be making a Docker "swarm-mode" `compose.yml` file.
Let's take as an example, [Matomo web analytics](https://matomo.org/). We'll be making a Docker "swarm-mode" `compose.yml` file. Here's an overview of the options you'd have to package Matomo:
- Tired: Write your own image and compose file
- Wired: Use someone else's image (& maybe compose file)
- Inspired: Upstream image, someone else's compose file
- On fire: Upstream compose file
- **Tired**: Write your own image and compose file from scratch
- **Wired**: Use someone else's image (& maybe compose file)
- **Inspired**: Upstream image, someone else's compose file
- **On fire**: Upstream image, Upstream compose file
I'm feeling lazy so, luckily for me, Matomo already has an example compose file in their repository.
Like a lot of compose files, it's intended for use with `docker-compose`, instead of "swarm mode", but it should be a good start.
Luckily, Matomo already has an example compose file in their repository. Like a lot of compose files, it's intended for use with `docker-compose`, instead of "swarm mode", but it should be a good start. That typically also means "not for production" but we can still use it as a solid base.
First, let's create a directory with the files we need:
```
abra recipe new matomo
cd ~/.abra/apps/matomo
cd ~/.abra/recipes/matomo
```
Then, let's download and edit the `docker-compose.yml` file:
@ -31,11 +29,11 @@ mkdir matomo && cd matomo
wget https://raw.githubusercontent.com/matomo-org/docker/master/.examples/apache/docker-compose.yml -O compose.yml
```
Open the `compose.yml` in your favourite editor and have a gander &#129442;. There are a few things we're looking for -- full list to come -- but some immediate changes could be:
Open the `compose.yml` in your favourite editor and have a gander &#129442;. There are a few things we're looking for, but some immediate changes could be:
1. Let's bump the version to `3.8`, to make sure we can use all the latest swarm coolness
2. We load environment variables separately via [abra](/overview/#command-line-tool), so we'll strip out `env_file`.
3. The `/var/www/html` volume definition on L21 is a bit overzealous; it means a copy of Matomo will be stored separately per app instance, which is a waste of space in most cases. We'll narrow it down according to the documentation -- here, the developers have been nice enough to suggest `logs` and `config` volumes instead, which is a decent start
2. We load environment variables separately via [`abra`](/abra/), so we'll strip out `env_file`
3. The `/var/www/html` volume definition on L21 is a bit overzealous; it means a copy of Matomo will be stored separately per app instance, which is a waste of space in most cases. We'll narrow it down according to the documentation. The developers have been nice enough to suggest `logs` and `config` volumes instead, which is a decent start
4. The MySQL passwords are sent as variables which is fine for basic use, but if we replace them with Docker secrets we can keep them out of our env files if we want to publish those more widely.
5. The MariaDB service doesn't need to be exposed to the internet, so we can define an `internal` network for it to communicate with Matomo.
6. Lastly, we want to use `deploy.labels` and remove the `ports:` definition, to tell Traefik to forward requests to Matomo based on hostname and generate an SSL certificate.
@ -44,112 +42,26 @@ The resulting `compose.yml` is available [here](https://git.autonomic.zone/coop-
!!! note "Running Co-op Cloud server required!"
The rest of this guide assumes you have a Co-op Cloud server going -- we'll use `swarm.example.com`, but replace it with your own server address.
Head over to [the deployment guide](./deploy.md) if you need help setting one up.
The rest of this guide assumes you have a Co-op Cloud server going -- we'll use `swarm.example.com`, but replace it with your own server address. Head over to [the operators tutorial](/operators/tutorial) if you need help setting one up.
Now, we're ready to create a testing instance of Matomo:
```
abra app new matomo --secrets \
--domain matomo.swarm.example.com \
--server swarm.example.com \
--app-name mygreatapp
--server swarm.example.com
```
Depending on whether you defined any extra environment variables -- we didn't so
far, in this example -- you might want to run `abra app mygreatapp config` to
check the configuration.
Depending on whether you defined any extra environment variables, we didn't so
far, in this example, you might want to run `abra app config swarm.example.com`
to check the configuration.
Otherwise, or once you've done that, go ahead and deploy the app:
```
abra app mygreatapp deploy
abra app deploy swarm.example.com
```
Then, open the `DOMAIN` you configured (you might need to wait a while for Traefik to generate SSL certificates) to finish the set-up. Luckily, this container is (mostly) configurable via environment variables -- if we want to auto-generate the configuration we can use a `config` and / or a custom `entrypoint` (see [`coop-cloud/mediawiki`](https://git.autonomic.zone/coop-cloud/mediawiki) for examples of both).
Then, open the `DOMAIN` you configured (you might need to wait a while for Traefik to generate SSL certificates) to finish the set-up. Luckily, this container is (mostly) configurable via environment variables, if we want to auto-generate the configuration we can use a `config` and / or a custom `entrypoint` (see [`coop-cloud/mediawiki`](https://git.autonomic.zone/coop-cloud/mediawiki) for examples of both).
### How recipes are versioned
> We are still working on stabilising this workflow
We'll use an example to work through this. Let's use [Gitea](https://hub.docker.com/r/gitea/gitea).
The Gitea project maintains a version, e.g. `1.14.3`. This version uses the [semver](https://semver.org) strategy for communicating what type of changes are included in each version, i.e., if there is a breaking change, Gitea will release a new version as `2.0.0`.
However, there are other types of changes that can happen for a recipe. Perhaps the database image gets a breaking update or the actual recipe configuration changes some environment variable. This can mean that end-users of the recipe need to do some work to make sure their updates will deploy successfully.
Therefore, we maintain an additional version part, in front of the project version. So, the first release of the Gitea recipe in the Co-op Cloud project has the version of `1.0.0+1.14.3-rootless`. This `x.y.z+` is the version part that the recipe maintainer manages. If a new available Gitea version comes out as `1.15-rootless` then the recipe maintainer will publish `1.1.0+1.15-rootless` as this is a backwards compatible update, following semantic versioning.
In all cases, we follow the semver semantics. So, if we upgrade the Gitea recipe from `1.14.3-rootless` to `1.15.3-rootless`, we still publish `1.1.0+1.15.3-rootless` as our recipe version. In this case, we skipped a few patch releases but it was all backwards compatible, so we only increment the minor version part.
The commands uses for dealing with this logic in `abra` are:
- `abra recipe upgrade`: upgrade the image tags in the compose configs of a recipe
- `abra recipe tag`: publish a git tag for the recipe repo
- `abra recipe sync`: upgrade the deploy labels to match the new recipe version
### How to publish a new recipe version
!!! warning
Due to [a limitation](https://git.coopcloud.tech/coop-cloud/organising/issues/185) in the Git library we're using in `abra`, when tagging new releases, you **must** use annotated git tags (i.e. `git tag -a`) or `abra` won't be able to parse them correctly. On the up side, it is probably better to leave a message with your new Git tag anyway. So maybe this is just "best practice" ;)
> This process is very much still a work in progress...
- `abra recipe upgrade keycloak`
- `abra recipe sync keycloak "1.0.0+13.0.1"`
- `abra recipe release -c --cm "chore: first release" -t "first release" --push`
### Style guide
- Please don't use `&image` YAML repeat anchors on the `image: ...` key because our `recipe release` logic does not handle it (see [#172](https://git.autonomic.zone/coop-cloud/abra/issues/172))
## Recipe structure
A recipe is a git repository that contains instructions for creating stacks that abra can read and interpret. You'll see a couple of files there:
- [compose.yml](#composeyml) (required!)
- [.env.sample](#envsample) (required!)
- [abra.sh](#abrash)
- [entrypoint.sh](#entrypointsh)
- [other compose files](#other-compose-files)
- [other files](#other-files)
### compose.yml
this is a [compose specification](https://compose-spec.io/) compliant file that contains a list of:
- services
- secrets
- networks
- volumes
- configs
that describe what is needed to run a stack. Whenever you deploy an app, abra reads this file to cook the stack.
### .env.sample
this file is a skeleton for environmental variables that should be adjusted by the user. Examples include: domain or php extention list. Whenever you create a new app with `abra app new` this file gets copied to the ~/.abra/servers/server-domain/app-name.env and when you run `abra app config` you're editing this file.
### abra.sh
`abra.sh` provides shell functions for running non-standard deploy, restore, rollback, backup and upgrade. This is only needed for some packages (such as nextcloud or wordpress)
### entrypoint.sh
after docker creates the filesystem and copies files into a new container it runs what's called an entrypoint. This is usually a shell script that exports some variables and runs the application. Sometimes the vendor entrypoint doesn't do everything that we need it to do. In that case you can write your own entrypoint, do whatever you need to do and then run the vendor entrypoint. For a simple example check [entrypoint.sh for croc](https://git.coopcloud.tech/coop-cloud/croc/src/commit/2f06e8aac52a3850d527434a26de0a242bea0c79/entrypoint.sh). In this case, croc needs the password to be exported as an environmental variable called `CROC_PASS`, and that is exactly what the entrypoint does before running vendor entrypoint. If you write your own entrypoint, it needs to be specified in the `config` section of compose.yml.
### other compose files
i.e. compose.smtp.yml. These are used to provide non-essential functionality such as (registration) e-mails or single sign on.
### other files
if you look at compose.yml (or compose.\*.yml) and see a `configs` section, that means this compose file is putting files in the container. This might be used for changing default (vendor) configuration, such as this [fpm-tune.ini file](https://git.coopcloud.tech/coop-cloud/nextcloud/src/commit/28425b6138603067021757de28c639ad464e9cf8/fpm-tune.ini) used to adjust php-fpm.
Congrats, you've packaged your first recipe! You've probably got more questions, check out the [maintainers handbook](/maintainers/handbook)!