docs: new ssh consolidation changes

See coop-cloud/abra#255

Dockerfile

@@ -1,4 +1,4 @@
-FROM squidfunk/mkdocs-material:9.0.11
+FROM squidfunk/mkdocs-material:9.0.12
 
 EXPOSE 8000
 

docs/abra/cheat-sheet.md

@@ -28,8 +28,7 @@ flags: `-f/--force`, `-C/--chaos`
 flags: `-f/--force`, `-V/--volumes`
 
 ### add/remove server
-- `abra server add $SERVER $USERNAME $SSH_PORT`
-flags: `-p/--provision`, `-l/--local`
+- `abra server add $SERVER`
 - `abra server remove $SERVER`
 flags: `-s/--server`
 

docs/abra/install.md

@@ -2,6 +2,10 @@
 title: Install
 ---
 
+!!! warning
+
+    We've seen reports that `abra` under [WSL](https://learn.microsoft.com/en-us/windows/wsl/about) doesn't work due to an underlying bug in Docker context handling. See [`coop-cloud/organising#406`](https://git.coopcloud.tech/coop-cloud/organising/issues/406) and [`docker/for-win#13180`](https://github.com/docker/for-win/issues/13180) for more.
+
 ## Stable release
 
 ```

docs/abra/trouble.md

@@ -8,51 +8,21 @@ You can use [this issue tracker](https://git.coopcloud.tech/coop-cloud/abra/issu
 
 ## SSH connection issues?
 
-`abra` tries its best to learn from your system configuration or command-line input what the correct SSH connection details are for a given server. This doesn't always work out. Here are some things to try to fix it.
+When you run `abra server add <host>`, `abra` will read from your `~/.ssh/config` and try to match a `Host <host>` entry. If you can `ssh <host>` then you should be able to `abra server add <host>`.
 
-First, ensure that you can `ssh <my-server>` and things work. If you can't SSH to your server then neither can `abra`. If you have a password protected SSH key, then you'll need to make sure your `ssh-agent` is running and you've added your SSH key part:
+For example, if you do `abra server add example.com`, you should have a matching entry that looks like this:
 
 ```
-eval $(ssh-agent -k)
-ssh-add ~/.ssh/<my-secret-key-part>
-ssh-add -L # validate loaded keys
+Host example.com
+  Hostname example.com
+  User exampleUser
+  Port 12345
+  IdentityFile ~/.ssh/example@somewhere
 ```
 
-The first thing `abra` will check for is the connection details listed in `abra server ls`. Check those details are correct. If you haven't managed to `abra server add` your server yet, then no details will appear in that list. You may need to take a look at [this entry](/abra/trouble/#abra-server-ls-shows-the-wrong-details) to clean up old values depending on your situation.
-
-`abra` will then try to read your `~/.ssh/config` entries and match the server domain against a `Host` entry. So, if you do `ssh myserver.com` and you have:
-
-```
-Host myserver.com
-  Hostname myserver.com
-  User myuser
-  Port 222
-  IdentityFile ~/.ssh/my@myserver.com
-```
-
-Then `abra` should have all it needs to build a working SSH connection. You can validate this by passing `-d/--debug` to your commands.
-
-However, sometimes, you use an alias in your SSH configuration, say:
-
-```
-Host mys
-...
-```
-
-So that you can simply type `ssh mys`. `abra` won't be able to match against those entries to discover connection details. You can use aliases to remedy this:
-
-```
-Host mys, myserver.com
-...
-```
-
-`abra` will try to read the relevant `IdentityFile` entry from your `~/.ssh/config` but if it can't make a match, it will rely on your key being added to the `ssh-agent`.
-
-Due to a limitation in our implementation, `abra` uses 2 methods of making SSH connections, the main `abra` -> `remote docker` connection using `/usr/bin/ssh` which can seamlessly pick up loaded SSH keys. However, for SSH host key checking, `abra` uses an SSH library & Golang SSH internals. We're working on resolving this to a single implementation but it is tricky work.
-
 ## "abra server ls" shows the wrong details?
 
-You can use `abra server rm` to remove the incorrect details. Make sure to take a backup of your `~/.abra/servers/<domain>` first. You can then try to re-create by using `abra server add ...` again, making sure to take care if you need to use `<user> <port>`, see `abra server add -h` for more help on this.
+You can use `abra server rm` to remove the incorrect details. Make sure to take a backup of your `~/.abra/servers/<domain>` first. You can then try to re-create by using `abra server add ...` again.
 
 However, if you have Docker installed on the same machine you have `abra`, then there might be some confusion. If you run `docker context ls` you'll see that Docker uses context connection strings also. `abra` simply uses this approach. Sometimes, your Docker defined context details & your `abra` context details can get out of sync. You can use `docker context rm` to resolve this.
 
@@ -62,7 +32,7 @@ If you need to create a new context from Docker, you can do:
 docker context create <domain> --docker "host=ssh://<user>@<domain>:<port>"
 ```
 
-(This is what we used to before we wrote `abra` to make it more convenient.)
+This is what we used to before we wrote `abra` to make it more convenient.
 
 ## Command-line flag handling is weird?
 
@@ -99,7 +69,3 @@ You can install it alongside the [supported version of Abra](https://git.coopclo
 git clone https://git.coopcloud.tech/coop-cloud/abra-bash ~/.abra/bash-src
 ln -s ~/.abra/bash-src/abra ~/.local/bin/babra
 ```
-
-## I am seeing very weird `lookup <domain> on <ip>: write udp <ip>: write: operation not permitted` errors
-
-You should turn off your VPN. `abra` has trouble dealing with it right now. We welcome change sets to make it work though!

docs/abra/upgrade.md

@@ -18,6 +18,35 @@ abra upgrade --rc
 
 ## Migration guides
 
+### `0.6.x-beta` -> `0.7.x-beta`
+
+> **ALERTA, ALERTA**: this is currently only available via the release
+> candidate channel, using `abra upgrade --rc`. There has been a lot of churn
+> and we're being cautious about releasing this one. Please help us test! We're
+> currently on `0.7.0-rc2-beta`.
+
+- `kadabra`, the app auto-updater is available for general alpha testing! See [these docs](https://docs.coopcloud.tech/operators/tutorial/#automatic-upgrades) for how to get started. Binaries can be found [here](https://git.coopcloud.tech/coop-cloud/abra/releases/tag/0.7.0-rc2-beta).
+
+- **ALERTA, ALERTA**, security related issue: all `$domain.env` env vars are now exposed to the deployment via the `app` service container. Each `FOO=BAR` is exported within the context of the container. If you have any privately committed secrets in your `.env` files, please migrate them to the `secrets: ...` configuration in the recipe. This change was made to facilitate tooling which can support auto-upgrading of apps in a deployment.
+
+- `abra` can no longer install Docker, initialise swarm mode and the proxy network. It will check if a Docker install exists and is in swarm mode or not and error out accordingly. We leave the provisioning to tools that are designed for that and reduce the command-line surface that we have to maintain going forward.
+
+- `abra server add <host> <args>` 👉 `abra server add <host>`. We have finally removed the custom SSH handling code and now solely rely on invoke `/usr/bin/ssh` directly and reading from the `~/.ssh/config`. The `<host>` argument should correspond to a `Host <host>` entry in your `~/.ssh/config` or in an `Include <file>` statement (hosts are retrieved via `ssh -G <host>`). This means "how does `abra` interact with SSH is 1) do you have an `~/.ssh/config` entry for `<host>` 2) can you `ssh <host>` successfully? 3) there is no 3. It's an easier mental model and also the way `abra-bash` works, hence, less weird obscure errors. `<host>` being public a domain name is still required.
+
+- `abra` no longer tries to do the TOFU host key verification prompt. We follow the praxis of the Docker CLI and just give up when host keys are not validated. We leave it to folks to SSH in and verify themselves.
+
+- On the way to [`kadabra`](https://git.coopcloud.tech/coop-cloud/abra/pulls/268), several changes regarding labelling deployments have been merged in this release. This will allow tooling to understand a deployment without having the context of a `~/.abra/...` configuration. This will pave the way for server-side tooling, like `kadabra` which can help operators with different kinds of maintenance tasks.
+
+- Welcome `abra recipe fetch`, which helps retrieve a recipe repository to your local work-station.
+
+- Also say hello to `abra app services <domain>`, which lists the in-deployment service names and corresponding image, e.g. `foo_example_com`.
+
+- Digests have been removed from the catalogue generation.
+
+- Backup files generated by `abra` have a much more human-friendly format.
+
+- Linting for domains is disabled when no `DOMAIN=...` is discovered in the `$odmain.env` file.
+
 ### `0.5.x-beta` -> `0.6.x-beta`
 
 - Using `{{ .Domain }}` in recipe `.envrc.sample` files went away because it

docs/maintainers/handbook.md

@@ -8,6 +8,15 @@ You can run `abra recipe new <recipe>` to generate a new `~/.abra/recipes/<recip
 
 ## Hacking on an existing recipe
 
+!!! warning
+
+    It is *very advisable* to disable any `healthcheck: ...` configuration
+    while hacking on new recipes. This is because it is very easy to mess up
+    and it will stop Traefik or other web proxies routing the app. You can
+    enable a specific healthcheck later when your recipe is stable. The default
+    "unconfigured" healthcheck behaviour is much less strict and it's faster to
+    get something up and running.
+
 If you want to make changes to an existing recipe then you can simply edit the files in `~/.abra/recipes/<recipe-name>` and run pass `--chaos` to the `deploy` command when deploying those changes. `abra` will not deploy unstaged changes to avoid instability but you can tell it to do so with `--chaos`. This means ou can simple hack away on the existing recipe files on your local file system and then when something is working, submit a change request to the recipe upstream.
 
 ## How is a recipe structured?
@@ -30,7 +39,7 @@ After docker creates the filesystem and copies files into a new container it run
 
 For a simple example check the [entrypoint.sh for `croc`](https://git.coopcloud.tech/coop-cloud/croc/src/commit/2f06e8aac52a3850d527434a26de0a242bea0c79/entrypoint.sh). In this case, `croc` needs the password to be exported as an environmental variable called `CROC_PASS`, and that is exactly what the entrypoint does before running vendor entrypoint.
 
-If you write your own entrypoint, it needs to be specified in the `config` section of compose.yml. See [this handbook entry](http://localhost:8000/maintainers/handbook/#entrypoints) for more.
+If you write your own entrypoint, it needs to be specified in the `config` section of compose.yml. See [this handbook entry](/maintainers/handbook/#how-do-i-set-a-custom-entrypoint) for more.
 
 ### `releases/` directory
 
@@ -539,3 +548,91 @@ You can use [this `docker-compose` trick](https://docs.docker.com/compose/extend
 If you have a recipe that is using a `mysql` service and you'd like to use `postgresql` instead, you can create a `compose.psql.yml`!
 
 An example of this is the [`selfoss`](https://git.coopcloud.tech/coop-cloud/selfoss) recipe. The default is `sqlite` but there is a `postgresql` compose configuration there too.
+
+## How do I set a custom entrypoint?
+
+For more context, see the [`entrypoint.sh`](/maintainers/handbook/#entrypointsh) section. The following configuration example is ripped from the [`coop-cloud/peertube`](https://git.coopcloud.tech/coop-cloud/peertube) recipe but shortened down. Here are more or less the steps you need to take:
+
+Define a config:
+
+```yaml
+  app:
+    ...
+    configs:
+      - source: app_entrypoint
+        target: /docker-entrypoint.sh
+        mode: 0555
+    ...
+
+configs:
+  app_entrypoint:
+    name: ${STACK_NAME}_app_entrypoint_${APP_ENTRYPOINT_VERSION}
+    file: entrypoint.sh.tmpl
+    template_driver: golang
+```
+
+Define a `entrypoint.sh.tmpl`:
+
+```
+#!/bin/bash
+
+set -e
+
+file_env() {
+   local var="$1"
+   local fileVar="${var}_FILE"
+   local def="${2:-}"
+
+   if [ "${!var:-}" ] && [ "${!fileVar:-}" ]; then
+      echo >&2 "error: both $var and $fileVar are set (but are exclusive)"
+      exit 1
+   fi
+
+   local val="$def"
+
+   if [ "${!var:-}" ]; then
+      val="${!var}"
+   elif [ "${!fileVar:-}" ]; then
+      val="$(< "${!fileVar}")"
+   fi
+
+   export "$var"="$val"
+   unset "$fileVar"
+}
+
+file_env "PEERTUBE_DB_PASSWORD"
+
+{{ if eq (env "PEERTUBE_SMTP_ENABLED") "1" }}
+file_env "PEERTUBE_SMTP_PASSWORD"
+{{ end }}
+
+{{ if eq (env "PEERTUBE_LIVE_CHAT_ENABLED") "1" }}
+apt -y update && apt install -y prosody && apt -y clean
+mkdir -p /run/prosody && chown prosody:prosody /run/prosody
+{{ end }}
+
+# Copy the client files over to a named volume
+# so that they may be served by nginx directly
+cp -ar /app/client/dist /srv/client
+
+# upstream entrypoint
+# https://github.com/Chocobozzz/PeerTube/blob/66f77f63437c6774acbd72584a9839a7636ea167/support/docker/production/entrypoint.sh
+/usr/local/bin/entrypoint.sh "$@"
+```
+
+Please note:
+
+1. The `file_env` // `_FILE` hack is to pass secrets into the container runtime without exposing them in plaintext in the configuration. See [this entry](/maintainers/handbook/#exposing-secrets) for more.
+
+1. In order to pass execution back to the original entrypoint, it's a good idea to find the original entrypoint script and run it from your own entrypoint script. If there is none, you may want to reference the `CMD` definition or if that isn't working, try to actually specify `cmd: ...` in the `compose.yml` definition (there are other recipes which do this).
+
+1. If you're feeling reckless, you can also use the Golang templating engine to do things conditionally.
+
+Then, wire up the vendored config version:
+
+```
+# abra.sh
+export APP_ENTRYPOINT_VERSION=v5
+```
+
+You should be able to deploy this overriden configuration now.

docs/operators/handbook.md

@@ -250,8 +250,6 @@ docker swarm init
 docker network create -d overlay proxy
 ```
 
-`abra` will do this for you when you run `abra server add --provision`.
-
 ## Managing DNS entries
 
 `abra record ...` can help you manage your DNS entries if you have an account with a supported 3rd party provider. We currently support [Gandi](https://gandi.net). The process of managing DNS with `abra` usually goes like this:
@@ -370,3 +368,20 @@ If you get errors about database access:
 ```
 abra app run foo.bar.com db bash -c 'mysqldump -u root -p"$(cat /run/secrets/db_oot_password)" <database>' | gzip > ~/.abra/backups/foo.bar.com_db_`date +%F`.sql.gz
 ```
+
+## Can I deploy a recipe without `abra`?
+
+Yes! It's a design goal to keep the recipes not dependent on `abra` or any
+single tool that we develop. This means the configurationc commons can still be
+useful beyond this project. You can deploy a recipe with standard commands like
+so:
+
+```
+set -a
+source example.com.env
+cd ~/.abra/recipes/myrecipe
+docker stack deploy -c compose.yml example_com
+```
+
+`abra` makes all of this more cenvenient but other tooling could follow this
+approach.

docs/operators/tutorial.md

@@ -102,7 +102,7 @@ Most Co-op Cloud deployments have been run on Debian machines so far. Some exper
 
 You need to keep port `:80` and `:443` free on your server for web proxying to your apps. Typically, you don't need to keep any other ports free as the core web proxy ([Traefik](https://traefik.io)) keeps all app ports internal to its network. Sometimes however, you need to expose an app port when you need to use a transport which would perform better or more reliably without proxying.
 
-`abra` has support for both creating servers (`abra server new`) & provisioning them (passing `--provision` to `abra server add`) but those are more advanced automation options which are covered in the [handbook](/operators/handbook). For this tutorial, we'll focus on the basics. Assuming you've managed to create a testing VPS with some `$hosting_provider`, you'll need to install Docker, add your user to the Docker group & setup swarm mode:
+`abra` has support for creating servers (`abra server new`) but that is a more advanced automation feature which is covered in the [handbook](/operators/handbook). For this tutorial, we'll focus on the basics. Assuming you've managed to create a testing VPS with some `$hosting_provider`, you'll need to install Docker, add your user to the Docker group & setup swarm mode:
 
 ```
 # docker install convenience script
@@ -162,14 +162,13 @@ If you run into issues during installation, [please report a ticket](https://git
 
 #### Add your server
 
-Now you can connect `abra` with your server. You need to have a working SSH configuration before you can do this. That means you can run `ssh <server-domain>` on your command-line and everything Works :tm:.
+Now you can connect `abra` with your server. You should have a working SSH configuration before you can do this (e.g. a matching `Host <server-domain>` entry in `~/.ssh/config` with the correct SSH connection details). That means you can run `ssh <server-domain>` on your command-line and everything Works :tm:.
 
 ```bash
-abra server add <server-domain> -p
+ssh <server-domain> # make sure it works
+abra server add <server-domain>
 ```
 
-The `-p` or `--provision` flag means that `abra` will install Docker and initialise the [new single-host swarm](https://docs.docker.com/engine/swarm/key-concepts/) on your server. If you've already followed the steps in [the server setup](/operators/tutorial/#server-setup) step, then `abra` should not need to do any work.
-
 It is important to note that `<domain>` here is a publicy accessible domain name which points to your server IP address. `abra` does make sure this is the case and this is done to avoid issues with HTTPS certificate rate limiting.
 
 You will now have a new `~/.abra/` folder on your local file system which stores all the configuration of your Co-op Cloud instance.
@@ -184,10 +183,6 @@ abra server ls
 
     `abra` uses plain 'ol SSH under the hood and aims to make use of your existing SSH configurations in `~/.ssh/config` and interfaces with your running `ssh-agent` for password protected secret key files.
 
-    The `server add` command listed above assumes that that you make SSH connections on port 22 using your current username. If that is not he case, pass the new values as positional arguments. See `abra server add -h` for more on this.
-
-        abra server add <domain> <user> <port> -p
-
     Running `server add` with `-d/--debug` should help you debug what is going on under the hood. It's best to take a moment to read [this troubleshooting entry](/abra/trouble/#ssh-connection-issues) if you're running into SSH connection issues with `abra`.
 
 !!! question "How do I share my configs in `~/.abra`?"

requirements.txt

@@ -1,4 +1,4 @@
 mkdocs-awesome-pages-plugin==2.8.0
 mkdocs-material-extensions==1.1.1
-mkdocs-material==9.0.11
+mkdocs-material==9.0.12
 mkdocs==1.4.2