Move uncategorised out of docs for now

2021-02-18 10:42:46 +01:00
parent 18b5837c57
commit 62bca57eb5
8 changed files with 0 additions and 0 deletions
--- a/uncategorised/coop-cloud/index.md
+++ b/uncategorised/coop-cloud/index.md
@ -0,0 +1,26 @@
+# About Coop Cloud
+
+Coöp Cloud is our system for deploying and updating free software applications.
+
+- 🔧 [Our own Coöp Cloud app configurations](https://git.autonomic.zone/autonomic-cooperative/coop-cloud-apps)
+- 🍎 [Supported application list](https://codimd.autonomic.zone/49--AK0GQDWoxMq6-9ngTQ)
+- 👁 [Coöp cloud overview](https://codimd.autonomic.zone/_M81xUukTCiBK96pgyC3DQ#) (portal to other CodiMD pads, including marketing copy)
+
+We're currently using Coöp Cloud for some of our internal infrastructure:
+
+- git.autonomic.zone
+- drone.autonomic.zone
+- id.autonomic.zone
+- traefik.autonomic.zone
+
+(all running on [`autonomic-swarm`](/servers/autonomic-swarm.md)), plus as many clients as we can, starting with:
+
+- drone.neuronic-swarm.autonomic.zone (for [Neuronic Games](/clients/neuronic-games), running on [`neuronic-swarm`](/servers/neuronic-swarm.md))
+- tankie.wiki (for Rebellious Data), running on `autonomic-swarm`
+- wiki.jones.iww.org.uk (for IWW wiki), running on `iww-jones`
+
+Relevant HOWTOs:
+
+- [Set up a new Docker swarm box](newswarm.md)
+- [Working with Docker swarm](working.md)
+- [Logging with systemd-journald](logging.md)
--- a/uncategorised/coop-cloud/logging.md
+++ b/uncategorised/coop-cloud/logging.md
@ -0,0 +1,48 @@
+# Logging with systemd-journald
+
+The default Docker swarm logging driver is `json-file`, which is nice but
+whenever a container is kiled or replaced, the logs are lost. This doesn't help
+us when we want to look back and try to diagnose what hapened.
+
+So, we change the default logging driver to use the systemd based journal
+logging system. This is configured in the `/etc/systemd/journal.conf` and in
+the `/etc/docker/daemon.json`. In practice, it means that logs are persistently
+stored after containers go away, they are rotated and can be analysed later on.
+
+`/etc/docker/daemon.json`:
+
+```
+{
+    "log-driver": "journald",
+    "log-opts": {
+      "labels":"com.docker.swarm.service.name"
+    }
+}
+```
+
+`/etc/systemd/journal.conf`:
+
+```
+[Journal]
+Storage=persistent
+SystemMaxUse=5G
+MaxFileSec=1month
+
+```
+
+There is also the [official docker
+documentation](https://docs.docker.com/config/containers/logging/journald/) on
+the journald logging driver.
+
+Some useful commands:
+
+- `journalctl -f`
+- `journalctl CONTAINER_NAME=gitea_gitea.1.jxn9r85el63pdz42ykjnmh792 -f`
+- `journalctl COM_DOCKER_SWARM_SERVICE_NAME=gitea_gitea --since="2020-09-18 13:00:00" --until="2020-09-18 13:01:00"`
+- `journalctl CONTAINER_ID=$(docker ps -qf name=gitea_gitea) -f`
+
+Also, for more system wide analysis stuff:
+
+- `journalctl --disk-usage`
+- `du -sh /var/log/journal/*`
+- `man journalctl` / `man systemd-journald` / `man journald.conf`
--- a/uncategorised/coop-cloud/newswarm.md
+++ b/uncategorised/coop-cloud/newswarm.md
@ -0,0 +1,19 @@
+# Setting up a new Docker swarm box
+
+Create and provision a new VPS with Docker installed.
+
+The easiest way of doing this is using [`actions/newhetzner.yml` in our `infrastructure` repository](https://git.autonomic.zone/autonomic-cooperative/infrastructure/src/branch/master/actions/newhetzner.yml).
+
+Make sure you have `infrastructure` cloned and set up [according to the instructions](https://git.autonomic.zone/autonomic-cooperative/infrastructure/src/branch/master/README.md), then run `ansible-playbook actions/newhetzner.yml` and answer the questions.
+
+1. Add the server to your `~/.ssh/config` file (you'll at least need to specify `Port 222`).
+2. Add the suggested entry to [`inventories/inventory`](https://git.autonomic.zone/autonomic-cooperative/infrastructure/src/branch/master/inventories)
+3. Create a new `servers/<client>/<server>.yml` file, based on `servers/neuronic/swarm.yml` (including the `swarm.single-node` role)
+4. Run `ansible-playbook servers/<client>/<server>.yml` to install Docker
+
+The easiest way forwards from here is to install `abra` and use it to set up remote context and initialise the swarm:
+
+1. `curl -fsSL https://install.abra.autonomic.zone | bash`
+2. `abra context create swarm.client.tld yourusername 222`
+3. `abra context init swarm.client.tld`
+4. `abra context use swarm.client.tld`
--- a/uncategorised/coop-cloud/working.md
+++ b/uncategorised/coop-cloud/working.md
@ -0,0 +1,97 @@
+# Working with Docker Swarm
+
+## Set up remote context
+
+You can use `docker context` to run Docker command-line commands and have them
+point to the Docker API end-point on a remote host.
+
+This means you can run commands locally and control the remote swarm easily
+(e.g. you run `docker ps` and instead of seeing container on your `localhost`
+you see them on `swarm.autonomic.zone`). This allows to do remote deployments
+manually, filter logs, clean-up containers etc.
+
+!!! note "This is optional!"
+If you like, you can SSH to a swarm server, [install `docker-compose`](https://docs.docker.com/compose/install/#install-compose-on-linux-systems), and run normal Docker commands instead.
+
+Here are the 3 steps to set this up.
+
+1. Create the remote docker context locally.
+
+   ```bash
+   # .envrc.sample
+   export PASSWORD_STORE_DIR=$(pwd)/../infrastructure/credentials/password-store
+   ```
+
+   ```bash
+   $ cp .envrc.sample .envrc
+   $ direnv allow  # ensure password store works
+   $ mkdir -vp ~/.docker/swarm.autonomic.zone && \
+     pass show docker/swarm.autonomic.zone/ca.pem > ~/.docker/swarm.autonomic.zone/ca.pem && \
+     pass show docker/swarm.autonomic.zone/cert.pem > ~/.docker/swarm.autonomic.zone/cert.pem && \
+     pass show docker/swarm.autonomic.zone/key.pem > ~/.docker/swarm.autonomic.zone/key.pem
+   $ docker context create swarm.autonomic.zone --docker \
+     "host=tcp://swarm.autonomic.zone:2376,ca=$HOME/.docker/swarm.autonomic.zone/ca.pem,cert=$HOME/.docker/swarm.autonomic.zone/cert.pem,key=$HOME/.docker/swarm.autonomic.zone/key.pem"
+   $ docker context use swarm.autonomic.zone
+   ```
+
+2. Deploy the application to the remote docker context.
+
+(Assuming you're in, say, the [git.autonomic.zone](https://git.autonomic.zone/autonomic-cooperative/git.autonomic.zone) repository)
+
+```bash
+$ docker stack ls
+$ docker stack deploy -c compose.yml gitea
+```
+
+You can track logs via `docker service logs gitea_gitea`.
+
+3. Switch back to your local context.
+
+```
+$ docker context use default
+```
+
+## Useful concepts & commands
+
+Each app is a **stack**, e.g. `drone` (`docker stack ls`), which creates one or
+more **services**, e.g. `drone_drone` (`docker service ls`), each of which has one or more
+**containers** e.g. `drone_drone.1.czq919syweq23x07whj38pb96` (`docker container ls`). All of this is defined in a `docker-compose.yml` file.
+
+Containers are built from **images**, e.g. `nginx:stable`, optionally using a
+`Dockerfile` to add extra commands or resources.
+
+### Secrets
+
+Most apps will need secret values (like API keys), which Docker can store securely using `docker secret`.
+
+As a failsafe, and to help debugging, we also store secrets in `pass`.
+
+You can generate a password, store it to Docker, and save it to `pass` in one
+step using something like this:
+
+```
+pwgen -n 32 1 | tee \
+  >(docker secret create "APP_SECRET_v1" -)
+  >(pass insert -m hosts/HOSTNAME/APP/SECRET)
+```
+
+Use `docker secrets ls` to see the names of all secrets defined in the current
+context, and `docker secrets rm <NAME>` to remove one if you need to reset it.
+
+## Troubleshooting
+
+If a service is trying to start, but you don't see anything in `docker service logs ...`, then try `docker service ps --no-trunc`, which will show you errors
+during container initialisation.
+
+If you still don't see anything there, log into the swarm server and check the
+Docker logs:
+
+```
+sudo journalctl -u docker.service | tail -n 50
+```
+
+## Investigating persistent journald logs
+
+See [systemd-journald
+docs](https://docs.autonomic.zone/coop-cloud/logging-with-systemd-journald/)
+for more information on the systemd journal logging setup.
--- a/uncategorised/coop-cloud/wp.md
+++ b/uncategorised/coop-cloud/wp.md
@ -0,0 +1,12 @@
+# Running wp-cli commands
+
+Here is an example how to drop into a shell and run `wp` commands. Just change the relevant details for your project.
+
+```bash
+export CONTAINER_ID=$(DOCKER_CONTEXT=swarm.autonomic.zone docker container ls -f 'Name=boycott-turkey_net_app' --format '{{ .ID }}'); DOCKER_CONTEXT=swarm.autonomic.zone docker run -it --volumes-from "$CONTAINER_ID" --network "container:$CONTAINER_ID" wordpress:cli
+```
+
+```bash
+export CONTAINER_ID=$(DOCKER_CONTEXT=swarm.autonomic.zone docker container ls -f 'Name=boycott-turkey_net_app' --format '{{ .ID }}'); DOCKER_CONTEXT=swarm.autonomic.zone docker run -it --volumes-from "$CONTAINER_ID" --network "container:$CONTAINER_ID" wordpress:cli user create
+usage: wp user create <user-login> <user-email> [--role=<role>] [--user_pass=<password>] [--user_registered=<yyyy-mm-dd-hh-ii-ss>] [--display_name=<name>] [--user_nicename=<nice_name>] [--user_url=<url>] [--nickname=<nickname>] [--first_name=<first_name>] [--last_name=<last_name>] [--description=<description>] [--rich_editing=<rich_editing>] [--send-email] [--porcelain]
+```
--- a/uncategorised/howto/convert-app.md
+++ b/uncategorised/howto/convert-app.md
@ -0,0 +1,59 @@
+# CoöpCloud-ising an app
+
+Example: Matomo web analytics
+
+- Tired: Write your own image and compose file
+- Wired: Use someone else's image (& maybe compose file)
+- Inspired: Upstream image, someone else's compose file
+- On fire: Upstream compose file
+
+I'm feeling lazy so, luckily for me, Matomo already has an example compose file in their repository! Let's download and edit it:
+
+```
+mkdir matomo && cd matomo
+wget https://raw.githubusercontent.com/matomo-org/docker/master/.examples/apache/docker-compose.yml -O compose.yml
+```
+
+Open `compose.yml` in your favourite editor and have a gander 🦢. There are a few things we're looking for – full list to come – but a few things we can immediately see are:
+
+1. Let's bump the version to `3.8`, to make sure we can use all the latest swarm coolness
+2. We load environment variables separately via `abra`, so we'll strip out `env_file`.
+3. The `/var/www/html` volume definition on L21 is a bit overzealous; it means a copy of Matomo will be stored separately per app instance, which is a waste of space in most cases. We'll narrow it down according to the documentation – here, the developers have been nice enough to suggest `logs` and `config` volumes instead, which is a decent start
+3. The MySQL passwords are sent as variables which is fine for basic use, but if we replace them with Docker secrets we can keep them out of our env files if we want to publish those more widely.
+4. The MariaDB service doesn't need to be exposed to the internet, so we can define an `internal` network for it to communicate with Matomo.
+5. Lastly, we want to use `deploy.labels` and remove the `ports:` definition, to tell Traefik to forward requests to Matomo based on hostname and generate an SSL certificate.
+
+(The default `db` and `app` service names work great for us; if they were called things like `mariadb` and `matomo` we might want to rename them).
+
+The resulting `compose.yml` is here: https://git.autonomic.zone/coop-cloud/matomo/src/branch/main/compose.yml
+
+Now, create an `.envrc` file.
+
+```
+export APP=matomo
+export DOMAIN=matomo.example.com
+export STACK_NAME=matomo
+export LETS_ENCRYPT_ENV=production
+
+export SECRET_DB_PASSWORD_VERSION=v1
+export SECRET_DB_ROOT_PASSWORD_VERSION=v1
+```
+
+(and, if you're using `.envrc`, remember to `source .envrc` or `direnv allow`)
+
+For the moment, `abra` is missing its old features to deploy from a "stack" directory, so we'll go through the usual steps to create a new app:
+
+1. `abra app new matomo`
+2. `abra app YOURDOMAIN config` (edit as appropriate)
+3. `abra app YOURDOMAIN secret auto` (add `–pass` if you'd like to save in
+	 `pass`)
+4. `abra deploy`
+
+Then, open the `DOMAIN` you configured (you might need to wait a while for Traefik to generate SSL certificates) to finish the set-up.
+
+Luckily, this container is (mostly) configurable via environment variables – if we want to auto-generate the configuration we can use a `config` and / or a custom `entrypoint` (see [`coop-cloud/mediawiki`](https://git.autonomic.zone/coop-cloud/mediawiki) for examples of both).
+
+To-do:
+ - Using a custom `entrypoint.sh` to do any initial set-up that relies on docker-compose run (e.g. for matrix-synapse)
+ - How to replace `build:` in stacks
+ - Adapting stacks which use bind-mounted volumes
--- a/uncategorised/howto/html-website.md
+++ b/uncategorised/howto/html-website.md
@ -0,0 +1,20 @@
+# Deploying an HTML website
+
+1. Install abra
+2. `abra app new custom-html`
+3. Configure DNS to point YOUURDOMAIN.TLD to `116.203.211.204` (`swarm.autonomic.zone`)
+3. `abra app YOURDOMAIN.TLD config` if you want to add domain aliases (e.g. `www`)
+4. `abra app YOURDOMAIN.TLD deploy`
+
+Manual deployment:
+
+1. `cd` to the directory with everything in it
+2. `tar cf - * | abra app YOURDOMAIN.TLD cp - app:/usr/share/nginx/html`
+
+Automatic deployment:
+
+1. Create `.drone.yml` file, e.g. here: https://git.autonomic.zone/kawaiipunk/writing/src/branch/main/.drone.yml
+2. Make sure the `autonomic` user has access to the repo on Gitea
+3. Log into Drone as autonomic and click "Sync"
+4. Add the repo in Drone
+5. Commit and push the `.drone.yml` file
--- a/uncategorised/index.md
+++ b/uncategorised/index.md
@ -0,0 +1,57 @@
+---
+title: Welcome
+---
+
+Coöp Cloud (working title; abbreviated CoCl) is a container-based, platform-agnostic, free software registry for small service providers.
+
+ - Production-ready apps in minutes: Wordpress & Nextcloud instances,
+   mail-servers, and more.
+ - Simple Docker-based framework for continuous deployment of your custom apps.
+
+[List of CoCl apps](https://codimd.autonomic.zone/s/HyNtOhwrv){: .md-button }
+
+HOWTOs:
+
+ - [Deploy an HTML site](howto/html-website.md)
+ - [CoCl-ise an app](howto/convert-app.md)
+
+!!! warning "A note about ARM"
+    Not all applications currently support all ARM computers (like PINE64 and
+    Raspberry Pi): e.g. [Traefik will work on ARMv6 & ARM64][traefik], [the
+    official MariaDB app doesn't work on any ARM boards][mariadb]. It might be possible
+    to use third-party ARM images with separate `compose.yml` files, but we
+    haven't exlplored this rabbit-hole much yet.
+
+## Definitions
+
+CoCl is a **philosophy**.
+
+ - To get started, you create a **server**, e.g. a new VPS, including
+   your local configuration to connect to it...
+ - then you install an **app** like wordpress or nextcloud, which is made up of multiple **services**...
+ - .. and configure your app.
+
+## Technical description
+
+Software-wise, CoöpCloud is:
+
+ - [`coop-cloud`](https://git.autonomic.zone/coop-cloud/), a collection of Docker "swarm mode" configurations for popular web apps
+ - [`abra`](https://git.autonomic.zone/autonomic-cooperative/abra), a simple tool for Docker swarm management
+ - a recommended default set of stacks:
+    - Traefik for SSL & routing
+    - `postfix-relay` for outgoing email
+
+## Principles / features:
+
+ - Security by default
+     - Secret storage using `docker secret` (["What makes it secure?"](https://github.com/BretFisher/ama/issues/86))
+     - Automatic SSL using Traefik & LetsEncrypt
+ - Zero-downtime deployments (for apps with healthchecks defined)
+ - Continuous integration testing using Drone and our [`stack-ssh-deploy`](https://git.autonomic.zone/coop-cloud/stack-ssh-deploy) plugin
+ - Low maintenance overhead:
+     - Automatic tracking of upstream Docker images using `renovate-bot`
+     - Avoiding custom Docker images as far as possible
+
+[traefik]: https://hub.docker.com/_/traefik?tab=tags
+[mariadb]: https://hub.docker.com/_/mariadb?tab=tags
+