explain further how to bootstrap

docs/operators/handbook.md

@@ -217,6 +217,27 @@ Otherwise, you have downloaded a corrupted file.
 
 If you want to teach `abra` how to support your favourite server hosting provider, we'd glady accept patches.
 
+## How do I bootstrap a server manually for running Co-op Cloud apps?
+
+The requirements are:
+
+1. Docker installed
+1. User in Docker user group
+1. Swarm mode initialised
+1. Proxy network created
+
+```
+# docker install convenience script
+wget -O- https://get.docker.com | bash
+
+# add user to docker group
+usermod -aG docker $YOURUSERNAMEHERE
+
+# setup swarm
+docker swarm init
+docker network create -d overlay proxy
+```
+
 ## Managing DNS entries
 
 `abra record ...` can help you manage your DNS entries if you have an account with a supported 3rd party provider. We currently support [Gandi](https://gandi.net). The process of managing DNS with `abra` usually goes like this:
@@ -228,3 +249,47 @@ If you want to teach `abra` how to support your favourite server hosting provide
 `abra` supports creating, listing and removing DNS entries if the 3rd party integration supports it.
 
 If you want to teach `abra` how to support your favourite server hosting provider, we'd glady accept patches.
+
+## How do I persist container logs after they go away?
+
+This is a big topic but in general, if you're looking for something quick & easy, you can use the [journald logging driver](https://docs.docker.com/config/containers/logging/journald/). This will hook the container logs into systemd which can handle persistent log collection & managing log file size.
+
+You need to add the following to your `/etc/docker/daemon.json` file on the server:
+
+```json
+{
+    "log-driver": "journald",
+    "log-opts": {
+      "labels":"com.docker.swarm.service.name"
+    }
+}
+```
+
+And for log size management, edit `/etc/systemd/journal.conf`:
+
+```
+[Journal]
+Storage=persistent
+SystemMaxUse=5G
+MaxFileSec=1month
+```
+
+Tne restart `docker` & `journald`:
+
+```
+systemctl restart docker
+systemctl restart systemd-journald
+```
+
+Now when you use `docker service logs` or `abra app logs`, it will read from the systemd journald logger seamlessly! Some useful `journalctl` commands are as follows, if you're doing some more fine grained logs investigation:
+
+- `journalctl -f`
+- `journalctl CONTAINER_NAME=my_git_com_app.1.jxn9r85el63pdz42ykjnmh792 -f`
+- `journalctl COM_DOCKER_SWARM_SERVICE_NAME=my_git_com_app --since="2020-09-18 13:00:00" --until="2020-09-18 13:01:00"`
+- `journalctl CONTAINER_ID=$(docker ps -qf name=my_git_com_app) -f`
+
+Also, for more system wide analysis stuff:
+
+- `journalctl --disk-usage`
+- `du -sh /var/log/journal/*`
+- `man journalctl` / `man systemd-journald` / `man journald.conf`

docs/operators/tutorial.md

@@ -102,6 +102,20 @@ Most Co-op Cloud deployments have been run on Debian machines so far. Some exper
 
 You need to keep port `:80` and `:443` free on your server for web proxying to your apps. Typically, you don't need to keep any other ports free as the core web proxy ([Traefik](https://traefik.io)) keeps all app ports internal to its network. Sometimes however, you need to expose an app port when you need to use a transport which would perform better or more reliably without proxying.
 
+`abra` has support for both creating servers (`abra server new`) & provisioning them (passing `--provision` to `abra server add`) but those are more advanced automation options which are covered in the [handbook](/operators/handbook). For this tutorial, we'll focus on the basics. Assuming you've managed to create a testing VPS with some `$hosting_provider`, you'll need to install Docker, add your user to the Docker group & setup swarm mode:
+
+```
+# docker install convenience script
+wget -O- https://get.docker.com | bash
+
+# add user to docker group
+usermod -aG docker $YOURUSERNAMEHERE
+
+# setup swarm
+docker swarm init
+docker network create -d overlay proxy
+```
+
 !!! question "Do you support multiple web proxies?"
 
     We do not know if it is feasible and convenient to set things up on an existing server with another web proxy which uses ports `:80` & `:443`. We'd happily receive reports and documentation on how to do this if you manage to set it up!
@@ -154,16 +168,6 @@ Now you can connect `abra` with your server. You need to have a working SSH conf
 abra server add <server-domain> -p
 ```
 
-!!! warning "Beware of SSH dragons"
-
-    `abra` uses plain 'ol SSH under the hood and aims to make use of your existing SSH configurations in `~/.ssh/config` and interfaces with your running `ssh-agent` for password protected secret key files.
-
-    The `server add` command listed above assumes that that you make SSH connections on port 22 using your current username. If that is not he case, pass the new values as positional arguments. See `abra server add -h` for more on this.
-
-        abra server add <domain> <user> <port> -p
-
-    Running `server add` with `-d/--debug` should help you debug what is going on under the hood. It's best to take a moment to read [this troubleshooting entry](/abra/trouble/#ssh-connection-issues) if you're running into SSH connection issues with `abra`.
-
 The `-p` or `--provision` flag means that `abra` will install Docker and initialise the [new single-host swarm](https://docs.docker.com/engine/swarm/key-concepts/) on your server.
 
 It is important to note that `<domain>` here is a publicy accessible domain name which points to your server IP address. `abra` does make sure this is the case and this is done to avoid issues with HTTPS certificate rate limiting.
@@ -176,6 +180,16 @@ You will now have a new `~/.abra/` folder on your local file system which stores
 abra server ls
 ```
 
+!!! warning "Beware of SSH dragons"
+
+    `abra` uses plain 'ol SSH under the hood and aims to make use of your existing SSH configurations in `~/.ssh/config` and interfaces with your running `ssh-agent` for password protected secret key files.
+
+    The `server add` command listed above assumes that that you make SSH connections on port 22 using your current username. If that is not he case, pass the new values as positional arguments. See `abra server add -h` for more on this.
+
+        abra server add <domain> <user> <port> -p
+
+    Running `server add` with `-d/--debug` should help you debug what is going on under the hood. It's best to take a moment to read [this troubleshooting entry](/abra/trouble/#ssh-connection-issues) if you're running into SSH connection issues with `abra`.
+
 !!! question "How do I share my configs in `~/.abra`?"
 
     It's possible and quite easy, see [this handbook entry](/operators/handbook/#understanding-app-and-server-configuration) for more.