Merge branch 'main' into styling-spring22

Follow up to #50

Dockerfile

@@ -1,4 +1,4 @@
-FROM squidfunk/mkdocs-material:8.2.1
+FROM squidfunk/mkdocs-material:8.2.5
 
 EXPOSE 8000
 

docs/abra/hack.md

@@ -16,7 +16,7 @@ Install [Go >= 1.16](https://golang.org/doc/install) and then:
 - `make install` will install it to `$GOPATH/bin`
 - `go get <package>` and `go mod tidy` to add a new dependency
 
-Our [Drone CI configuration](.drone.yml) runs a number of sanity on each pushed commit. See the [Makefile](./Makefile) for more handy targets.
+Our [Drone CI configuration](https://git.coopcloud.tech/coop-cloud/abra/src/branch/main/.drone.yml) runs a number of sanity on each pushed commit. See the [Makefile](./Makefile) for more handy targets.
 
 Please use the [conventional commit format](https://www.conventionalcommits.org/en/v1.0.0/) for your commits so we can automate our change log.
 

docs/abra/trouble.md

@@ -67,3 +67,35 @@ docker context create <domain> --docker "host=ssh://<user>@<domain>:<port>"
 ## Command-line flag handling is weird?
 
 Unfortunately, there is a limitation in our underlying command-line library implementation for `abra` ([ref](https://github.com/urfave/cli/issues/1113)) (and more fundamentally in the design of flags in the Go programming language itself ([ref](https://utcc.utoronto.ca/~cks/space/blog/programming/GoFlagUIImportance))). We're aiming to work with upstream to resolve the flag handling but this it is not yet clear when this will be resolved.
+
+Currently, the following example of flexible flag usage is supported:
+
+```
+abra app new gitea -S  # generate secrets, after args
+abra app new -S gitea  # generate secrets, before args
+```
+
+But something like the following does not work as expected:
+
+```
+abra app new -S gitea -p
+```
+
+Where the position of flags is mixed before & after args. `-p` is ignored :cry:
+
+We're still waiting for upstream patch which resovles this.
+
+## Why can't `abra` support multiline in `.env` files?
+
+We're sorry, it's an issue with an upstream dependency. See [`#291`](https://git.coopcloud.tech/coop-cloud/organising/issues/291) for more.
+
+## I need some feature from the old depreciated bash abra?
+
+There is an archive of the [old code here](https://git.coopcloud.tech/coop-cloud/abra-bash).
+
+You can install it alongside the [supported version of Abra](https://git.coopcloud.tech/coop-cloud/abra) by using these commands:
+
+```bash
+git clone https://git.coopcloud.tech/coop-cloud/abra-bash ~/.abra/bash-src
+ln -s ~/.abra/bash-src/abra ~/.local/bin/babra
+```

docs/abra/upgrade.md

@@ -22,7 +22,12 @@ abra upgrade --rc
 
 Make sure to back up your `~/.abra/servers` configurations first for safety.
 
-- `mv ~/.abra/apps ~/.abra/recipes`
-- `find ~/.abra/servers/ -type f -exec sed -i "s/^TYPE=/RECIPE=/g" {} \;`
+- Please run `mv ~/.abra/apps ~/.abra/recipes`.
+
 - "app name" as a concept went away, `abra` now uses the domain name of an app as the identifier. However, we don't expect to see breaking behaviour if you have `.env` files like `~/.abra/servers/foo.com/mycoolapp.env` and you still want to run `abra app ps mycoolapp`. `abra` still reads the filename to figure out the identifier. When running `abra app new <recipe>`, `abra` will now take the domain name as the name of the `.env` file.
+
 - `abra` has a new SSH implementation which enforces SSH host key checking. You may run into connection issues as a result of this code churn, please see [this entry](/abra/trouble/#ssh-connection-issues) for help navigating a fix.
+
+- CLI flag/args handling has been made more flexible. We're workign within the constraints of an upstream library issue but have hopefully made it easier to mange passing flags to commands with `abra`. See [this troubleshooting entry](/abra/trouble/#command-line-flag-handling-is-weird) for the full review.
+
+- A number of short style flags have been re-mapped and/or added. This is again related to an issue with the upstream CLI library which sometimes understands short style flags as long style flags. E.g. `--ch` instead of `-ch` for `--chaos`. As a concrete example, `--ch` is now `-C` on `abra app deploy`.

docs/maintainers/handbook.md

@@ -6,6 +6,10 @@ title: Packaging handbook
 
 You can run `abra recipe new <recipe>` to generate a new `~/.abra/recipes/<recipe>` repository. The generated repository is a copy of [`coop-cloud/example`](https://git.coopcloud.tech/coop-cloud/example).
 
+## Hacking on an existing recipe
+
+If you want to make changes to an existing recipe then you can simply edit the files in `~/.abra/recipes/<recipe-name>` and run pass `--chaos` to the `deploy` command when deploying those changes. `abra` will not deploy unstaged changes to avoid instability but you can tell it to do so with `--chaos`. This means ou can simple hack away on the existing recipe files on your local file system and then when something is working, submit a change request to the recipe upstream.
+
 ## How is a recipe structured?
 
 ### `compose.yml`
@@ -443,3 +447,22 @@ If you want to get the highest rating on SSL certs, you can use the following tr
 ```
 
 See [this PR](https://git.coopcloud.tech/coop-cloud/traefik/pulls/8/files) for the technical details
+
+## Tweaking secret generation length
+
+It is possible to tell `abra` which length it should generate secrets with from your recipe config.
+
+You do this by adding a inline comment to the secret definition in the `.env.sample` / `.env` file.
+
+Here are examples from the gitea recipe:
+
+```
+SECRET_INTERNAL_TOKEN_VERSION=v1 # length=105
+SECRET_JWT_SECRET_VERSION=v1 # length=43
+SECRET_SECRET_KEY_VERSION=v1 # length=64
+```
+
+When using this length specifier, `abra` will not use the "easy to remember
+word" style generator but instead a string of characters to match the exact
+length. This can be useful if you have to generate "key" style values instead
+of passwords which admins have to type out in database shells.

docs/maintainers/tutorial.md

@@ -8,7 +8,9 @@ title: New maintainers tutorial
 
 Packaging a recipe is basically knowing a bag of about 20 tricks. Once you learn them, there is nothing more to learn. It can seem daunting at first but it's simple and easy to do once you know the tricks.
 
-The nice thing about packaging is that only one person has to do it and then we all benefit. We've seen that over time, the core of the configuration doesn't really change. New options and versions might come but the config remains quite stable. This is good since it meand that your packaging work stays relevant and useful for other maintainers & operators as time goes on.
+The nice thing about packaging is that only one person has to do it and then we all benefit. We've seen that over time, the core of the configuration doesn't really change. New options and versions might come but the config remains quite stable. This is good since it means that your packaging work stays relevant and useful for other maintainers & operators as time goes on.
+
+Depending on your familiarity with recipes, it might be worth reading [how a recipe is structured](/maintainers/handbook/#how-is-a-recipe-structured) and making clear you understand [what a recipe is](/glossary/#recipe) before continuing.
 
 ### Making a plan
 

docs/operators/handbook.md

@@ -86,7 +86,7 @@ Then, tell your collaborators (e.g. in the repository's `README.md`), to run `ma
 
 If you're on an environment where it's hard to run Docker, or command-line programs in general, you might want to install `abra` on a server instead of your local work station.
 
-To install `abra` on a different server than you'll be hosting your apps, just follow [getting started guide](/operators/tutorial#deploy-your-first-app) as normal except for one difference. Instead of providing your SSH connection details when you run `abra server add ...`, just pass `--local`.
+To install `abra` on the same server where you'll be hosting your apps, just follow [getting started guide](/operators/tutorial#deploy-your-first-app) as normal except for one difference. Instead of providing your SSH connection details when you run `abra server add ...`, just pass `--local`.
 
 ```
 abra server add --local
@@ -104,7 +104,7 @@ Co-op Cloud uses [Docker Secrets](https://docs.docker.com/engine/swarm/secrets/)
 
 `abra` includes several commands to make it easier to manage secrets:
 
-- `abra app secret generate <domain>`: to auto-generate a app secrets
+- `abra app secret generate <domain>`: to auto-generate app secrets
 - `abra app secret insert <domain>`: to insert a single secret
 - `abra app secret rm <domain>`: to remove secrets
 
@@ -217,6 +217,29 @@ Otherwise, you have downloaded a corrupted file.
 
 If you want to teach `abra` how to support your favourite server hosting provider, we'd glady accept patches.
 
+## How do I bootstrap a server for running Co-op Cloud apps?
+
+The requirements are:
+
+1. Docker installed
+1. User in Docker user group
+1. Swarm mode initialised
+1. Proxy network created
+
+```
+# docker install convenience script
+wget -O- https://get.docker.com | bash
+
+# add user to docker group
+usermod -aG docker $YOURUSERNAMEHERE
+
+# setup swarm
+docker swarm init
+docker network create -d overlay proxy
+```
+
+`abra` will do this for you when you run `abra server add --provision`.
+
 ## Managing DNS entries
 
 `abra record ...` can help you manage your DNS entries if you have an account with a supported 3rd party provider. We currently support [Gandi](https://gandi.net). The process of managing DNS with `abra` usually goes like this:
@@ -228,3 +251,47 @@ If you want to teach `abra` how to support your favourite server hosting provide
 `abra` supports creating, listing and removing DNS entries if the 3rd party integration supports it.
 
 If you want to teach `abra` how to support your favourite server hosting provider, we'd glady accept patches.
+
+## How do I persist container logs after they go away?
+
+This is a big topic but in general, if you're looking for something quick & easy, you can use the [journald logging driver](https://docs.docker.com/config/containers/logging/journald/). This will hook the container logs into systemd which can handle persistent log collection & managing log file size.
+
+You need to add the following to your `/etc/docker/daemon.json` file on the server:
+
+```json
+{
+    "log-driver": "journald",
+    "log-opts": {
+      "labels":"com.docker.swarm.service.name"
+    }
+}
+```
+
+And for log size management, edit `/etc/systemd/journal.conf`:
+
+```
+[Journal]
+Storage=persistent
+SystemMaxUse=5G
+MaxFileSec=1month
+```
+
+Tne restart `docker` & `journald`:
+
+```
+systemctl restart docker
+systemctl restart systemd-journald
+```
+
+Now when you use `docker service logs` or `abra app logs`, it will read from the systemd journald logger seamlessly! Some useful `journalctl` commands are as follows, if you're doing some more fine grained logs investigation:
+
+- `journalctl -f`
+- `journalctl CONTAINER_NAME=my_git_com_app.1.jxn9r85el63pdz42ykjnmh792 -f`
+- `journalctl COM_DOCKER_SWARM_SERVICE_NAME=my_git_com_app --since="2020-09-18 13:00:00" --until="2020-09-18 13:01:00"`
+- `journalctl CONTAINER_ID=$(docker ps -qf name=my_git_com_app) -f`
+
+Also, for more system wide analysis stuff:
+
+- `journalctl --disk-usage`
+- `du -sh /var/log/journal/*`
+- `man journalctl` / `man systemd-journald` / `man journald.conf`

docs/operators/tutorial.md

@@ -102,6 +102,20 @@ Most Co-op Cloud deployments have been run on Debian machines so far. Some exper
 
 You need to keep port `:80` and `:443` free on your server for web proxying to your apps. Typically, you don't need to keep any other ports free as the core web proxy ([Traefik](https://traefik.io)) keeps all app ports internal to its network. Sometimes however, you need to expose an app port when you need to use a transport which would perform better or more reliably without proxying.
 
+`abra` has support for both creating servers (`abra server new`) & provisioning them (passing `--provision` to `abra server add`) but those are more advanced automation options which are covered in the [handbook](/operators/handbook). For this tutorial, we'll focus on the basics. Assuming you've managed to create a testing VPS with some `$hosting_provider`, you'll need to install Docker, add your user to the Docker group & setup swarm mode:
+
+```
+# docker install convenience script
+wget -O- https://get.docker.com | bash
+
+# add user to docker group
+usermod -aG docker $YOURUSERNAMEHERE
+
+# setup swarm
+docker swarm init
+docker network create -d overlay proxy
+```
+
 !!! question "Do you support multiple web proxies?"
 
     We do not know if it is feasible and convenient to set things up on an existing server with another web proxy which uses ports `:80` & `:443`. We'd happily receive reports and documentation on how to do this if you manage to set it up!
@@ -154,17 +168,7 @@ Now you can connect `abra` with your server. You need to have a working SSH conf
 abra server add <server-domain> -p
 ```
 
-!!! warning "Beware of SSH dragons"
-
-    `abra` uses plain 'ol SSH under the hood and aims to make use of your existing SSH configurations in `~/.ssh/config` and interfaces with your running `ssh-agent` for password protected secret key files.
-
-    The `server add` command listed above assumes that that you make SSH connections on port 22 using your current username. If that is not he case, pass the new values as positional arguments. See `abra server add -h` for more on this.
-
-        abra server add <domain> <user> <port> -p
-
-    Running `server add` with `-d/--debug` should help you debug what is going on under the hood. It's best to take a moment to read [this troubleshooting entry](/abra/trouble/#ssh-connection-issues) if you're running into SSH connection issues with `abra`.
-
-The `-p` or `--provision` flag means that `abra` will install Docker and initialise the [new single-host swarm](https://docs.docker.com/engine/swarm/key-concepts/) on your server.
+The `-p` or `--provision` flag means that `abra` will install Docker and initialise the [new single-host swarm](https://docs.docker.com/engine/swarm/key-concepts/) on your server. If you've already followed the steps in [the server setup](/operators/tutorial/#server-setup) step, then `abra` should not need to do any work.
 
 It is important to note that `<domain>` here is a publicy accessible domain name which points to your server IP address. `abra` does make sure this is the case and this is done to avoid issues with HTTPS certificate rate limiting.
 
@@ -176,6 +180,16 @@ You will now have a new `~/.abra/` folder on your local file system which stores
 abra server ls
 ```
 
+!!! warning "Beware of SSH dragons"
+
+    `abra` uses plain 'ol SSH under the hood and aims to make use of your existing SSH configurations in `~/.ssh/config` and interfaces with your running `ssh-agent` for password protected secret key files.
+
+    The `server add` command listed above assumes that that you make SSH connections on port 22 using your current username. If that is not he case, pass the new values as positional arguments. See `abra server add -h` for more on this.
+
+        abra server add <domain> <user> <port> -p
+
+    Running `server add` with `-d/--debug` should help you debug what is going on under the hood. It's best to take a moment to read [this troubleshooting entry](/abra/trouble/#ssh-connection-issues) if you're running into SSH connection issues with `abra`.
+
 !!! question "How do I share my configs in `~/.abra`?"
 
     It's possible and quite easy, see [this handbook entry](/operators/handbook/#understanding-app-and-server-configuration) for more.

docs/organisers/handbook.md

@@ -4,7 +4,7 @@ title: Organising handbook
 
 ## Where do I find "the community"?
 
-The main place is the [Matrix community chat channels](/intro/contact/#matrix). You'll find a number of folks who are actively interested in the project and/or using `abra` and some of our recipes. Furthermore, Matrix has an excellent network of folks and in practice we've e-met a lot of groups who are interested in Co-op Cloud.
+The main place is the [Matrix community chat channels](/intro/contact/#matrix). You'll find a number of folks who are actively interested in the project and/or using `abra` and some of our recipes. Furthermore, Matrix has an excellent network of Cool People :tm: and in practice we've e-met a lot of groups who are interested in Co-op Cloud.
 
 ## Onboarding new project members
 
@@ -12,7 +12,7 @@ We still haven't worked this out. We're working on it.
 
 ## Gathering new case studies
 
-We try to gather as many "case studies" as possible, stories & concrete examples of Co-op Cloud being used For God :tm: See [coopcloud.tech](https://coopcloud.tech) for our existing examples. These studies help people identify what the purpose of the project is for.
+We try to gather as many "case studies" as possible, stories & concrete examples of Co-op Cloud being used For Good :tm: See [coopcloud.tech](https://coopcloud.tech) for our existing examples. These studies help people identify what the purpose of the project is for.
 
 ## Monthly updates
 

requirements.txt

@@ -1,4 +1,4 @@
 mkdocs-awesome-pages-plugin==2.7.0
 mkdocs-material-extensions==1.0.3
-mkdocs-material==8.2.1
+mkdocs-material==8.2.5
 mkdocs==1.2.3