add redis password and increase version. does not work yet

.drone.yml

@@ -18,6 +18,10 @@ steps:
       STACK_NAME: loomio
       LETS_ENCRYPT_ENV: production
       LOOMIO_ENTRYPOINT_VERSION: v1
+      SECRET_DEVISE_SECRET_VERSION: v1
+      SECRET_SECRET_COOKIE_TOKEN_VERSION: v1
+      SECRET_DB_PASSWORD_VERSION: v1
+      SECRET_SMTP_PASSWORD_VERSION: v1
 trigger:
   branch:
     - main
@@ -33,7 +37,7 @@ steps:
         from_secret: drone_abra-bot_token
       fork: true
       repositories:
-        - coop-cloud/auto-recipes-catalogue-json
+        - toolshed/auto-recipes-catalogue-json
 
 trigger:
   event: tag

.env.sample

@@ -1,31 +1,29 @@
 TYPE=loomio
+COMPOSE_FILE="compose.yml"
 
 DOMAIN=loomio.example.com
 ## Domain aliases
 #EXTRA_DOMAINS=', `www.loomio.example.com`'
 LETS_ENCRYPT_ENV=production
 
-# the number of dots in your hostname
-TLD_LENGTH=3
-# TODO 3wc: is this needed?
-
+# mail setup
+COMPOSE_FILE="$COMPOSE_FILE:compose.smtp.yml"
 SUPPORT_EMAIL=noreply@example.com
 SMTP_AUTH=plain
 SMTP_DOMAIN=example.com
 SMTP_SERVER=example.com
 SMTP_PORT=587
 SMTP_USERNAME=noreply@example.com
-SMTP_PASSWORD=password
 SMTP_USE_SSL=1
 # to disable SSL comment out line rather than changing to 0
+SECRET_SMTP_PASSWORD_VERSION=v1
 
-# Whyyyy does this need to be set separately
+# From field for notification e-mails
 NOTIFICATIONS_EMAIL_ADDRESS=noreply@example.com
 
-REPLY_HOSTNAME=loomio.example.com
+# reply-to in email notifications
+REPLY_HOSTNAME=$DOMAIN
 
-# helper bot is the account which welcomes people to their groups.
-HELPER_BOT_EMAIL=noreply@loomio.example.com
 RAILS_ENV=production
 
 # Number of webserver processes and threads
@@ -42,6 +40,8 @@ USE_RACK_ATTACK=1
 
 SECRET_DEVISE_SECRET_VERSION=v1 #length=64
 SECRET_SECRET_COOKIE_TOKEN_VERSION=v1 #length=64
+SECRET_DB_PASSWORD_VERSION=v1
+SECRET_REDIS_PASSWORD_VERSION=v1
 
 # Send catch up email (missed yesterday) weekly
 # EMAIL_CATCH_UP_WEEKLY=1

README.md

@@ -4,27 +4,31 @@
 
 <!-- metadata -->
 * **Category**: Apps
-* **Status**: 0, work-in-progress
-* **Image**: [`loomio/*`](https://hub.docker.com/r/loomio)
-* **Healthcheck**: No
-* **Backups**: No
-* **Email**: ?
+* **Status**: 3, work-in-progress
+* **Image**: [`loomio/*`](https://hub.docker.com/r/loomio), 4, upstream
+* **Healthcheck**: Yes
+* **Backups**: Yes
+* **Email**: Outgoing yes, incoming no
 * **Tests**: No
 * **SSO**: No
 <!-- endmetadata -->
 
 ## Basic usage
 
-1. Set up Docker Swarm and [`abra`]
-2. Deploy [`coop-cloud/traefik`]
-3. `abra app new loomio` (optionally with `--pass` if you'd like
-   to save secrets in `pass`)
-4. `abra app config YOURAPPDOMAIN` - be sure to change `$DOMAIN` to something that resolves to
-   your Docker swarm box
-5. `abra app deploy YOURAPPDOMAIN`
-6. This should be automated but you also need to run `abra app run loomio_some_domain app rake db:migrate`
-7. Open the configured domain in your browser to finish set-up
-8. Give yourself admin rights by running `User.last.update(is_admin: true)`
+* `abra app new loomio --secrets   ` (optionally with `--pass` if you'd like to save secrets in `pass`)
+* `abra app config <app-name>`
+* insert your smtp password with `abra app secret insert <app-name> smtp_password v1 "<your-password>"`
+* `abra app deploy <app-name>`
+* Open the configured domain in your browser to create your user account (only works in case mail is configured correctly)
+* Give yourself admin rights by running `abra app cmd <app-name> app make_last_user_admin`
 
-[`abra`]: https://git.autonomic.zone/autonomic-cooperative/abra
-[`coop-cloud/traefik`]: https://git.autonomic.zone/coop-cloud/traefik
+## Manuel migration steps when upgrading from 0.6.0+v2.25.2 and earlier to 1.0.0+v2.25.2 and later
+
+* adapt your env file with the new vars, especially SECRET_DB_PASSWORD_VERSION=v1 and SECRET_SMTP_PASSWORD_VERSION=v1
+* insert your smtp password with `abra app secret insert <app-name> smtp_password v1 "<your-password>"`
+* `abra app secret generate <app-name> db_password v1`
+* `abra app deploy <app-name>`
+* set the new password in DB: `abra app cmd <app-name> db set_new_db_password`
+* redeploy if necessary
+
+For more, see [docs.coopcloud.tech](https://docs.coopcloud.tech).

abra.sh

@@ -1 +1,17 @@
-export LOOMIO_ENTRYPOINT_VERSION=v3
+export LOOMIO_ENTRYPOINT_VERSION=v5
+export REDIS_ENTRYPOINT_VERSION=v1
+
+# cannot be integrated into entrypoint.sh as it requires the operator to create a user first
+function make_last_user_admin()
+{
+    export DATABASE_URL="postgresql://${POSTGRES_USER}:$(cat /run/secrets/db_password)@db/${POSTGRES_DB}"
+    SECRET_KEY_BASE=$(rake secret) rails runner "User.last.update(is_admin: true)"
+}
+
+# only run when upgrading from 0.6.0+v2.25.2 and earlier to 1.0.0+v2.25.2 and later
+function set_new_db_password()
+{
+    echo "setting new password for db user..."
+	psql -U $POSTGRES_USER -c "ALTER USER $POSTGRES_USER PASSWORD '$(cat /run/secrets/db_password)';"
+    echo "done"
+}

compose.smtp.yml

@@ -0,0 +1,28 @@
+version: "3.8"
+
+x-mail-env: &mail-env
+    SMTP_AUTH: ${SMTP_AUTH}
+    SMTP_DOMAIN: ${SMTP_DOMAIN}
+    SMTP_SERVER: ${SMTP_SERVER}
+    SMTP_PORT: ${SMTP_PORT}
+    SMTP_USERNAME: ${SMTP_USERNAME}
+    SMTP_PASSWORD:
+    SMTP_PASSWORD_FILE: /run/secrets/smtp_password
+    SMTP_USE_SSL: ${SMTP_USE_SSL}
+
+services:
+  app:
+    secrets:
+      - smtp_password
+    environment:
+      *mail-env
+  worker:
+    secrets:
+      - smtp_password
+    environment:
+      *mail-env
+
+secrets:
+  smtp_password:
+    name: ${STACK_NAME}_smtp_password_${SECRET_SMTP_PASSWORD_VERSION}
+    external: true

compose.yml

@@ -1,24 +1,24 @@
 ---
 version: "3.8"
 
+x-db-env: &db-env
+    POSTGRES_PASSWORD_FILE: /run/secrets/db_password
+    POSTGRES_DB: loomio_production
+    POSTGRES_USER: postgres
+
+x-redis-env: &redis-env
+  REDIS_PASSWORD_FILE: /run/secrets/redis_password
+  REDIS_URL: redis://:{REDIS_PASSWORD}@redis:6379
+
 x-environment: &default-env
-  DATABASE_URL: postgresql://postgres:password@db/loomio_production
-  REDIS_URL: redis://redis:6379
+  <<: *db-env
+  <<: *redis-env
   CANONICAL_HOST: ${DOMAIN}
   VIRTUAL_HOST: ${DOMAIN}
   CHANNELS_URI: wss://channels.${DOMAIN}
-  TLD_LENGTH: ${TLD_LENGTH}
   SUPPORT_EMAIL: ${SUPPORT_EMAIL}
-  SMTP_AUTH: ${SMTP_AUTH}
-  SMTP_DOMAIN: ${SMTP_DOMAIN}
-  SMTP_SERVER: ${SMTP_SERVER}
-  SMTP_PORT: ${SMTP_PORT}
-  SMTP_USERNAME: ${SMTP_USERNAME}
-  SMTP_PASSWORD: ${SMTP_PASSWORD}
-  SMTP_USE_SSL: ${SMTP_USE_SSL}
   NOTIFICATIONS_EMAIL_ADDRESS: ${NOTIFICATIONS_EMAIL_ADDRESS}
   REPLY_HOSTNAME: ${REPLY_HOSTNAME}
-  HELPER_BOT_EMAIL: ${HELPER_BOT_EMAIL}
   RAILS_ENV: ${RAILS_ENV}
   PUMA_WORKERS: ${PUMA_WORKERS}
   MIN_THREADS: ${MIN_THREADS}
@@ -34,7 +34,7 @@ x-environment: &default-env
 
 services:
   app:
-    image: loomio/loomio:v2.19.0
+    image: loomio/loomio:v2.25.3
     configs:
       - source: entrypoint
         target: /entrypoint.sh
@@ -43,6 +43,8 @@ services:
     secrets:
       - devise_secret
       - secret_cookie_token
+      - db_password
+      - redis_password
     volumes:
       - loomio_uploads:/loomio/public/system
       - loomio_storage:/loomio/storage
@@ -56,6 +58,12 @@ services:
       - db
       - redis
     environment: *default-env
+    healthcheck:
+      test: ["CMD", "curl", "-f", "http://localhost:3000"]
+      interval: 30s
+      timeout: 10s
+      retries: 10
+      start_period: 2m
     deploy:
       restart_policy:
         condition: on-failure
@@ -65,9 +73,10 @@ services:
         - "traefik.http.routers.${STACK_NAME}.rule=Host(`${DOMAIN}`${EXTRA_DOMAINS})"
         - "traefik.http.routers.${STACK_NAME}.entrypoints=web-secure"
         - "traefik.http.routers.${STACK_NAME}.tls.certresolver=${LETS_ENCRYPT_ENV}"
-        - "coop-cloud.${STACK_NAME}.version=0.3.1+v2.19.0"
+        - "coop-cloud.${STACK_NAME}.version=2.0.0+v2.25.3"
+        - "backupbot.backup:=${ENABLE_BACKUPS:-true}"
   worker:
-    image: loomio/loomio:v2.19.0
+    image: loomio/loomio:v2.25.3
     configs:
       - source: entrypoint
         target: /entrypoint.sh
@@ -76,6 +85,8 @@ services:
     secrets:
       - devise_secret
       - secret_cookie_token
+      - db_password
+      - redis_password
     networks:
       - backend
     environment:
@@ -89,20 +100,38 @@ services:
       - loomio_files:/loomio/public/files
       - loomio_plugins:/loomio/plugins/docker
   db:
-    image: postgres:12.14
+    image: pgautoupgrade/pgautoupgrade:17-debian
     networks:
       - backend
     volumes:
       - pgdata:/pgdata
-      - pgdumps:/pgdumps
+    secrets:
+      - db_password
     environment:
-      - POSTGRES_PASSWORD=password
-      - POSTGRES_DB=loomio_production
-      - PGDATA=/pgdata
+      <<: *db-env
+      PGDATA: /pgdata
+    deploy:
+      labels:
+        backupbot.backup: "${ENABLE_BACKUPS:-true}"
+        backupbot.backup.pre-hook: sh -c 'pg_dump -U "$$POSTGRES_USER" -Fc "$$POSTGRES_DB" | gzip > "/postgres.dump.gz"'
+        backupbot.backup.path: "/postgres.dump.gz"
+        backupbot.backup.post-hook: "rm -f /postgres.dump.gz"
+        backupbot.restore: "true"
+        backupbot.restore.post-hook: sh -c 'gzip -d /postgres.dump.gz && pg_restore --clean -U "$$POSTGRES_USER" --dbname="$$POSTGRES_DB" < /postgres.dump && rm -f /postgres.dump'
   redis:
-    image: redis:5.0
+    image: redis:7.4
     networks:
       - backend
+    command: /bin/sh -c "redis-server redis.conf --loglevel debug"
+    healthcheck:
+      test: ["CMD", "redis-cli", "ping"]
+    secrets:
+      - redis_password
+    configs:
+      - source: entrypoint_redis
+        target: /entrypoint.sh
+        mode: 0555
+    entrypoint: /entrypoint.sh
   #mailin:
   #  image: loomio/mailin-docker:latest
   #  networks:
@@ -117,19 +146,23 @@ services:
       - backend
     depends_on:
       - redis
+    secrets:
+      - redis_password
     environment:
-      - REDIS_URL=redis://redis:6379
+      <<: *redis-env
   cron:
-    image: loomio/loomio:v2.19.0
+    image: loomio/loomio:v2.25.3
     configs:
       - source: entrypoint
         target: /entrypoint.sh
         mode: 0555
-    entrypoint: [ "/entrypoint.sh", "rake", "loomio:hourly_tasks" ]
+    entrypoint: [ "/entrypoint.sh", "rake loomio:hourly_tasks" ]
     environment: *default-env
     secrets:
       - devise_secret
       - secret_cookie_token
+      - db_password
+      - redis_password
     volumes:
       - loomio_uploads:/loomio/public/system
       - loomio_storage:/loomio/storage
@@ -160,12 +193,14 @@ volumes:
   loomio_plugins:
   loomio_import:
   pgdata:
-  pgdumps:
 
 configs:
   entrypoint:
     name: ${STACK_NAME}_entrypoint_${LOOMIO_ENTRYPOINT_VERSION}
     file: entrypoint.sh
+  entrypoint_redis:
+    name: ${STACK_NAME}_entrypoint_redis_${REDIS_ENTRYPOINT_VERSION}
+    file: entrypoint.redis.sh
 
 secrets:
   devise_secret:
@@ -174,3 +209,9 @@ secrets:
   secret_cookie_token:
     external: true
     name: ${STACK_NAME}_secret_cookie_token_${SECRET_SECRET_COOKIE_TOKEN_VERSION}
+  db_password:
+    external: true
+    name: ${STACK_NAME}_db_password_${SECRET_DB_PASSWORD_VERSION}
+  redis_password:
+    external: true
+    name: ${STACK_NAME}_redis_password_${SECRET_REDIS_PASSWORD_VERSION}

entrypoint.redis.sh

@@ -0,0 +1,6 @@
+#!/usr/bin/env bash
+set -e
+
+echo "creating redis.conf..."
+echo "requirepass $(cat /run/secrets/redis_password)" > redis.conf
+echo "redis.conf created"

entrypoint.sh

@@ -23,16 +23,24 @@ file_env() {
 
 file_env "DEVISE_SECRET"
 file_env "SECRET_COOKIE_TOKEN"
+file_env "POSTGRES_PASSWORD"
+file_env "SMTP_PASSWORD"
+export DATABASE_URL="postgresql://${POSTGRES_USER}:${POSTGRES_PASSWORD}@db/${POSTGRES_DB}"
 
-if test ! -f /loomio/storage/migrations_ran; then
-    echo "first deploy, running migrations..."
-    rake db:setup
-    touch /loomio/storage/migrations_ran
-fi
 if [ -n "$1" ]; then
 	echo "Running '$1'"
 	$1
 else
+	if [ ! -f /loomio/storage/migrations_ran ] && [ "${TASK:-}" = "worker" ]; then
+		echo "first deploy, running DB setup..."
+		rake db:setup
+		touch /loomio/storage/migrations_ran
+	fi
+
+	echo "running DB migrations..."
+	rake db:migrate
+	echo "DB migrations finished"
+
 	echo "starting loomio!"
 	/loomio/docker_start.sh
 fi

release/0.3.1+v2.19.0

@@ -0,0 +1,9 @@
+Loomio seems to have added a new setting, NOTIFICATIONS_EMAIL_ADDRESS, to
+define what address transactional emails should come from. 
+
+If you don't set it, it will default to notifications@$MAIL_DOMAIN, which is
+unlikely to work in many cases.
+
+If you find that transactional emails aren't working, try setting
+NOTIFICATIONS_EMAIL_ADDRESS to the same value as SUPPORT_EMAIL,
+HELPER_BOT_EMAIL, or SMTP_USERNAME.

release/1.0.0+v2.25.2

@@ -0,0 +1,9 @@
+In this release the passwords for smtp and postgres DB were moved into docker secrets. Therefore a few manual steps need to be performed (also available in recipe documentation)
+
+* adapt your env file with the new vars, especially SECRET_DB_PASSWORD_VERSION=v1 and SECRET_SMTP_PASSWORD_VERSION=v1
+* insert your smtp password with abra app secret insert <app-name> smtp_password v1 "<your-password>"
+* abra app secret generate <app-name> db_password v1
+* abra app deploy <app-name>
+* set the new password in DB: abra app cmd <app-name> db set_new_db_password
+* redeploy
+

release/2.0.0+v2.25.3

@@ -0,0 +1,2 @@
+The major change in this release is the upgrade of postgres db from 10 to 17. As we use a container with automatic migration, this should work seemlessly without operator intervention.
+NEVERTHELESS, please create a BACKUP before you upgrade to this version!