akshay/docs.coopcloud.tech
0
0
Fork 0
forked from toolshed/docs.coopcloud.tech
Code Issues Pull Requests 1 Projects Releases Wiki Activity

Move uncategorised out of docs for now

Browse Source
This commit is contained in:
Luke Murphy
2021-02-18 10:42:46 +01:00
parent 18b5837c57
commit 62bca57eb5
8 changed files with 0 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files

26
docs/uncategorised/coop-cloud/index.md
View File

@ -1,26 +0,0 @@
# About Coop Cloud
Coöp Cloud is our system for deploying and updating free software applications.
- 🔧 [Our own Coöp Cloud app configurations](https://git.autonomic.zone/autonomic-cooperative/coop-cloud-apps)
- 🍎 [Supported application list](https://codimd.autonomic.zone/49--AK0GQDWoxMq6-9ngTQ)
- 👁 [Coöp cloud overview](https://codimd.autonomic.zone/_M81xUukTCiBK96pgyC3DQ#) (portal to other CodiMD pads, including marketing copy)
We're currently using Coöp Cloud for some of our internal infrastructure:
- git.autonomic.zone
- drone.autonomic.zone
- id.autonomic.zone
- traefik.autonomic.zone
(all running on [`autonomic-swarm`](/servers/autonomic-swarm.md)), plus as many clients as we can, starting with:
- drone.neuronic-swarm.autonomic.zone (for [Neuronic Games](/clients/neuronic-games), running on [`neuronic-swarm`](/servers/neuronic-swarm.md))
- tankie.wiki (for Rebellious Data), running on `autonomic-swarm`
- wiki.jones.iww.org.uk (for IWW wiki), running on `iww-jones`
Relevant HOWTOs:
- [Set up a new Docker swarm box](newswarm.md)
- [Working with Docker swarm](working.md)
- [Logging with systemd-journald](logging.md)

48
docs/uncategorised/coop-cloud/logging.md
View File

@ -1,48 +0,0 @@
# Logging with systemd-journald
The default Docker swarm logging driver is `json-file`, which is nice but
whenever a container is kiled or replaced, the logs are lost. This doesn't help
us when we want to look back and try to diagnose what hapened.
So, we change the default logging driver to use the systemd based journal
logging system. This is configured in the `/etc/systemd/journal.conf` and in
the `/etc/docker/daemon.json`. In practice, it means that logs are persistently
stored after containers go away, they are rotated and can be analysed later on.
`/etc/docker/daemon.json`:
```
{
"log-driver": "journald",
"log-opts": {
"labels":"com.docker.swarm.service.name"
}
}
```
`/etc/systemd/journal.conf`:
```
[Journal]
Storage=persistent
SystemMaxUse=5G
MaxFileSec=1month
```
There is also the [official docker
documentation](https://docs.docker.com/config/containers/logging/journald/) on
the journald logging driver.
Some useful commands:
- `journalctl -f`
- `journalctl CONTAINER_NAME=gitea_gitea.1.jxn9r85el63pdz42ykjnmh792 -f`
- `journalctl COM_DOCKER_SWARM_SERVICE_NAME=gitea_gitea --since="2020-09-18 13:00:00" --until="2020-09-18 13:01:00"`
- `journalctl CONTAINER_ID=$(docker ps -qf name=gitea_gitea) -f`
Also, for more system wide analysis stuff:
- `journalctl --disk-usage`
- `du -sh /var/log/journal/*`
- `man journalctl` / `man systemd-journald` / `man journald.conf`

19
docs/uncategorised/coop-cloud/newswarm.md
View File

@ -1,19 +0,0 @@
# Setting up a new Docker swarm box
Create and provision a new VPS with Docker installed.
The easiest way of doing this is using [`actions/newhetzner.yml` in our `infrastructure` repository](https://git.autonomic.zone/autonomic-cooperative/infrastructure/src/branch/master/actions/newhetzner.yml).
Make sure you have `infrastructure` cloned and set up [according to the instructions](https://git.autonomic.zone/autonomic-cooperative/infrastructure/src/branch/master/README.md), then run `ansible-playbook actions/newhetzner.yml` and answer the questions.
1. Add the server to your `~/.ssh/config` file (you'll at least need to specify `Port 222`).
2. Add the suggested entry to [`inventories/inventory`](https://git.autonomic.zone/autonomic-cooperative/infrastructure/src/branch/master/inventories)
3. Create a new `servers/<client>/<server>.yml` file, based on `servers/neuronic/swarm.yml` (including the `swarm.single-node` role)
4. Run `ansible-playbook servers/<client>/<server>.yml` to install Docker
The easiest way forwards from here is to install `abra` and use it to set up remote context and initialise the swarm:
1. `curl -fsSL https://install.abra.autonomic.zone | bash`
2. `abra context create swarm.client.tld yourusername 222`
3. `abra context init swarm.client.tld`
4. `abra context use swarm.client.tld`

97
docs/uncategorised/coop-cloud/working.md
View File

@ -1,97 +0,0 @@
# Working with Docker Swarm
## Set up remote context
You can use `docker context` to run Docker command-line commands and have them
point to the Docker API end-point on a remote host.
This means you can run commands locally and control the remote swarm easily
(e.g. you run `docker ps` and instead of seeing container on your `localhost`
you see them on `swarm.autonomic.zone`). This allows to do remote deployments
manually, filter logs, clean-up containers etc.
!!! note "This is optional!"
If you like, you can SSH to a swarm server, [install `docker-compose`](https://docs.docker.com/compose/install/#install-compose-on-linux-systems), and run normal Docker commands instead.
Here are the 3 steps to set this up.
1. Create the remote docker context locally.
```bash
# .envrc.sample
export PASSWORD_STORE_DIR=$(pwd)/../infrastructure/credentials/password-store
```
```bash
$ cp .envrc.sample .envrc
$ direnv allow # ensure password store works
$ mkdir -vp ~/.docker/swarm.autonomic.zone && \
pass show docker/swarm.autonomic.zone/ca.pem > ~/.docker/swarm.autonomic.zone/ca.pem && \
pass show docker/swarm.autonomic.zone/cert.pem > ~/.docker/swarm.autonomic.zone/cert.pem && \
pass show docker/swarm.autonomic.zone/key.pem > ~/.docker/swarm.autonomic.zone/key.pem
$ docker context create swarm.autonomic.zone --docker \
"host=tcp://swarm.autonomic.zone:2376,ca=$HOME/.docker/swarm.autonomic.zone/ca.pem,cert=$HOME/.docker/swarm.autonomic.zone/cert.pem,key=$HOME/.docker/swarm.autonomic.zone/key.pem"
$ docker context use swarm.autonomic.zone
```
2. Deploy the application to the remote docker context.
(Assuming you're in, say, the [git.autonomic.zone](https://git.autonomic.zone/autonomic-cooperative/git.autonomic.zone) repository)
```bash
$ docker stack ls
$ docker stack deploy -c compose.yml gitea
```
You can track logs via `docker service logs gitea_gitea`.
3. Switch back to your local context.
```
$ docker context use default
```
## Useful concepts & commands
Each app is a **stack**, e.g. `drone` (`docker stack ls`), which creates one or
more **services**, e.g. `drone_drone` (`docker service ls`), each of which has one or more
**containers** e.g. `drone_drone.1.czq919syweq23x07whj38pb96` (`docker container ls`). All of this is defined in a `docker-compose.yml` file.
Containers are built from **images**, e.g. `nginx:stable`, optionally using a
`Dockerfile` to add extra commands or resources.
### Secrets
Most apps will need secret values (like API keys), which Docker can store securely using `docker secret`.
As a failsafe, and to help debugging, we also store secrets in `pass`.
You can generate a password, store it to Docker, and save it to `pass` in one
step using something like this:
```
pwgen -n 32 1 | tee \
>(docker secret create "APP_SECRET_v1" -)
>(pass insert -m hosts/HOSTNAME/APP/SECRET)
```
Use `docker secrets ls` to see the names of all secrets defined in the current
context, and `docker secrets rm <NAME>` to remove one if you need to reset it.
## Troubleshooting
If a service is trying to start, but you don't see anything in `docker service logs ...`, then try `docker service ps --no-trunc`, which will show you errors
during container initialisation.
If you still don't see anything there, log into the swarm server and check the
Docker logs:
```
sudo journalctl -u docker.service | tail -n 50
```
## Investigating persistent journald logs
See [systemd-journald
docs](https://docs.autonomic.zone/coop-cloud/logging-with-systemd-journald/)
for more information on the systemd journal logging setup.

12
docs/uncategorised/coop-cloud/wp.md
View File

@ -1,12 +0,0 @@
# Running wp-cli commands
Here is an example how to drop into a shell and run `wp` commands. Just change the relevant details for your project.
```bash
export CONTAINER_ID=$(DOCKER_CONTEXT=swarm.autonomic.zone docker container ls -f 'Name=boycott-turkey_net_app' --format '{{ .ID }}'); DOCKER_CONTEXT=swarm.autonomic.zone docker run -it --volumes-from "$CONTAINER_ID" --network "container:$CONTAINER_ID" wordpress:cli
```
```bash
export CONTAINER_ID=$(DOCKER_CONTEXT=swarm.autonomic.zone docker container ls -f 'Name=boycott-turkey_net_app' --format '{{ .ID }}'); DOCKER_CONTEXT=swarm.autonomic.zone docker run -it --volumes-from "$CONTAINER_ID" --network "container:$CONTAINER_ID" wordpress:cli user create
usage: wp user create <user-login> <user-email> [--role=<role>] [--user_pass=<password>] [--user_registered=<yyyy-mm-dd-hh-ii-ss>] [--display_name=<name>] [--user_nicename=<nice_name>] [--user_url=<url>] [--nickname=<nickname>] [--first_name=<first_name>] [--last_name=<last_name>] [--description=<description>] [--rich_editing=<rich_editing>] [--send-email] [--porcelain]
```

59
docs/uncategorised/howto/convert-app.md
View File

@ -1,59 +0,0 @@
# CoöpCloud-ising an app
Example: Matomo web analytics
- Tired: Write your own image and compose file
- Wired: Use someone else's image (& maybe compose file)
- Inspired: Upstream image, someone else's compose file
- On fire: Upstream compose file
I'm feeling lazy so, luckily for me, Matomo already has an example compose file in their repository! Let's download and edit it:
```
mkdir matomo && cd matomo
wget https://raw.githubusercontent.com/matomo-org/docker/master/.examples/apache/docker-compose.yml -O compose.yml
```
Open `compose.yml` in your favourite editor and have a gander 🦢. There are a few things we're looking for – full list to come – but a few things we can immediately see are:
1. Let's bump the version to `3.8`, to make sure we can use all the latest swarm coolness
2. We load environment variables separately via `abra`, so we'll strip out `env_file`.
3. The `/var/www/html` volume definition on L21 is a bit overzealous; it means a copy of Matomo will be stored separately per app instance, which is a waste of space in most cases. We'll narrow it down according to the documentation – here, the developers have been nice enough to suggest `logs` and `config` volumes instead, which is a decent start
3. The MySQL passwords are sent as variables which is fine for basic use, but if we replace them with Docker secrets we can keep them out of our env files if we want to publish those more widely.
4. The MariaDB service doesn't need to be exposed to the internet, so we can define an `internal` network for it to communicate with Matomo.
5. Lastly, we want to use `deploy.labels` and remove the `ports:` definition, to tell Traefik to forward requests to Matomo based on hostname and generate an SSL certificate.
(The default `db` and `app` service names work great for us; if they were called things like `mariadb` and `matomo` we might want to rename them).
The resulting `compose.yml` is here: https://git.autonomic.zone/coop-cloud/matomo/src/branch/main/compose.yml
Now, create an `.envrc` file.
```
export APP=matomo
export DOMAIN=matomo.example.com
export STACK_NAME=matomo
export LETS_ENCRYPT_ENV=production
export SECRET_DB_PASSWORD_VERSION=v1
export SECRET_DB_ROOT_PASSWORD_VERSION=v1
```
(and, if you're using `.envrc`, remember to `source .envrc` or `direnv allow`)
For the moment, `abra` is missing its old features to deploy from a "stack" directory, so we'll go through the usual steps to create a new app:
1. `abra app new matomo`
2. `abra app YOURDOMAIN config` (edit as appropriate)
3. `abra app YOURDOMAIN secret auto` (add `–pass` if you'd like to save in
`pass`)
4. `abra deploy`
Then, open the `DOMAIN` you configured (you might need to wait a while for Traefik to generate SSL certificates) to finish the set-up.
Luckily, this container is (mostly) configurable via environment variables – if we want to auto-generate the configuration we can use a `config` and / or a custom `entrypoint` (see [`coop-cloud/mediawiki`](https://git.autonomic.zone/coop-cloud/mediawiki) for examples of both).
To-do:
- Using a custom `entrypoint.sh` to do any initial set-up that relies on docker-compose run (e.g. for matrix-synapse)
- How to replace `build:` in stacks
- Adapting stacks which use bind-mounted volumes

20
docs/uncategorised/howto/html-website.md
View File

@ -1,20 +0,0 @@
# Deploying an HTML website
1. Install abra
2. `abra app new custom-html`
3. Configure DNS to point YOUURDOMAIN.TLD to `116.203.211.204` (`swarm.autonomic.zone`)
3. `abra app YOURDOMAIN.TLD config` if you want to add domain aliases (e.g. `www`)
4. `abra app YOURDOMAIN.TLD deploy`
Manual deployment:
1. `cd` to the directory with everything in it
2. `tar cf - * | abra app YOURDOMAIN.TLD cp - app:/usr/share/nginx/html`
Automatic deployment:
1. Create `.drone.yml` file, e.g. here: https://git.autonomic.zone/kawaiipunk/writing/src/branch/main/.drone.yml
2. Make sure the `autonomic` user has access to the repo on Gitea
3. Log into Drone as autonomic and click "Sync"
4. Add the repo in Drone
5. Commit and push the `.drone.yml` file

57
docs/uncategorised/index.md
View File

@ -1,57 +0,0 @@
---
title: Welcome
---
Coöp Cloud (working title; abbreviated CoCl) is a container-based, platform-agnostic, free software registry for small service providers.
- Production-ready apps in minutes: Wordpress & Nextcloud instances,
mail-servers, and more.
- Simple Docker-based framework for continuous deployment of your custom apps.
[List of CoCl apps](https://codimd.autonomic.zone/s/HyNtOhwrv){: .md-button }
HOWTOs:
- [Deploy an HTML site](howto/html-website.md)
- [CoCl-ise an app](howto/convert-app.md)
!!! warning "A note about ARM"
Not all applications currently support all ARM computers (like PINE64 and
Raspberry Pi): e.g. [Traefik will work on ARMv6 & ARM64][traefik], [the
official MariaDB app doesn't work on any ARM boards][mariadb]. It might be possible
to use third-party ARM images with separate `compose.yml` files, but we
haven't exlplored this rabbit-hole much yet.
## Definitions
CoCl is a **philosophy**.
- To get started, you create a **server**, e.g. a new VPS, including
your local configuration to connect to it...
- then you install an **app** like wordpress or nextcloud, which is made up of multiple **services**...
- .. and configure your app.
## Technical description
Software-wise, CoöpCloud is:
- [`coop-cloud`](https://git.autonomic.zone/coop-cloud/), a collection of Docker "swarm mode" configurations for popular web apps
- [`abra`](https://git.autonomic.zone/autonomic-cooperative/abra), a simple tool for Docker swarm management
- a recommended default set of stacks:
- Traefik for SSL & routing
- `postfix-relay` for outgoing email
## Principles / features:
- Security by default
- Secret storage using `docker secret` (["What makes it secure?"](https://github.com/BretFisher/ama/issues/86))
- Automatic SSL using Traefik & LetsEncrypt
- Zero-downtime deployments (for apps with healthchecks defined)
- Continuous integration testing using Drone and our [`stack-ssh-deploy`](https://git.autonomic.zone/coop-cloud/stack-ssh-deploy) plugin
- Low maintenance overhead:
- Automatic tracking of upstream Docker images using `renovate-bot`
- Avoiding custom Docker images as far as possible
[traefik]: https://hub.docker.com/_/traefik?tab=tags
[mariadb]: https://hub.docker.com/_/mariadb?tab=tags