little more security

For more information see: toolshed/abra#534

.drone.yml

@@ -0,0 +1,43 @@
+---
+kind: pipeline
+name: deploy to swarm-test.autonomic.zone
+steps:
+  - name: deployment
+    image: git.coopcloud.tech/coop-cloud/stack-ssh-deploy:latest
+    settings:
+      host: swarm-test.autonomic.zone
+      stack: loomio
+      generate_secrets: true
+      purge: true
+      deploy_key:
+        from_secret: drone_ssh_swarm_test
+      networks:
+        - proxy
+    environment:
+      DOMAIN: loomio.swarm-test.autonomic.zone
+      STACK_NAME: loomio
+      LETS_ENCRYPT_ENV: production
+      LOOMIO_ENTRYPOINT_VERSION: v1
+      SECRET_DEVISE_SECRET_VERSION: v1
+      SECRET_SECRET_COOKIE_TOKEN_VERSION: v1
+      SECRET_DB_PASSWORD_VERSION: v1
+      SECRET_SMTP_PASSWORD_VERSION: v1
+trigger:
+  branch:
+    - main
+---
+kind: pipeline
+name: generate recipe catalogue
+steps:
+  - name: release a new version
+    image: plugins/downstream
+    settings:
+      server: https://build.coopcloud.tech
+      token:
+        from_secret: drone_abra-bot_token
+      fork: true
+      repositories:
+        - toolshed/auto-recipes-catalogue-json
+
+trigger:
+  event: tag

.env.sample

@@ -1,28 +1,29 @@
 TYPE=loomio
+COMPOSE_FILE="compose.yml"
 
 DOMAIN=loomio.example.com
 ## Domain aliases
 #EXTRA_DOMAINS=', `www.loomio.example.com`'
 LETS_ENCRYPT_ENV=production
 
-# the number of dots in your hostname
+# mail setup
-TLD_LENGTH=3
+COMPOSE_FILE="$COMPOSE_FILE:compose.smtp.yml"
-# TODO 3wc: is this needed?
-
 SUPPORT_EMAIL=noreply@example.com
 SMTP_AUTH=plain
 SMTP_DOMAIN=example.com
 SMTP_SERVER=example.com
 SMTP_PORT=587
 SMTP_USERNAME=noreply@example.com
-SMTP_PASSWORD=password
 SMTP_USE_SSL=1
 # to disable SSL comment out line rather than changing to 0
+SECRET_SMTP_PASSWORD_VERSION=v1
 
-REPLY_HOSTNAME=loomio.example.com
+# From field for notification e-mails
+NOTIFICATIONS_EMAIL_ADDRESS=noreply@example.com
+
+# reply-to in email notifications
+REPLY_HOSTNAME=$DOMAIN
 
-# helper bot is the account which welcomes people to their groups.
-HELPER_BOT_EMAIL=noreply@loomio.example.com
 RAILS_ENV=production
 
 # Number of webserver processes and threads
@@ -37,6 +38,10 @@ FORCE_SSL=1
 # Enable rate limiting on group creation, other POST actions
 USE_RACK_ATTACK=1
 
+SECRET_DEVISE_SECRET_VERSION=v1 #length=64
+SECRET_SECRET_COOKIE_TOKEN_VERSION=v1 #length=64
+SECRET_DB_PASSWORD_VERSION=v1
+
 # Send catch up email (missed yesterday) weekly
 # EMAIL_CATCH_UP_WEEKLY=1
 

README.md

@@ -4,26 +4,46 @@
 
 <!-- metadata -->
 * **Category**: Apps
-* **Status**: 0, work-in-progress
+* **Status**: 3, work-in-progress
-* **Image**: [`loomio/*`](https://hub.docker.com/r/loomio)
+* **Image**: [`loomio/*`](https://hub.docker.com/r/loomio), 4, upstream
-* **Healthcheck**: No
+* **Healthcheck**: Yes
-* **Backups**: No
+* **Backups**: Yes
-* **Email**: ?
+* **Email**: Outgoing yes, incoming no
 * **Tests**: No
 * **SSO**: No
 <!-- endmetadata -->
 
 ## Basic usage
 
-1. Set up Docker Swarm and [`abra`]
+* `abra app new loomio --secrets   ` (optionally with `--pass` if you'd like to save secrets in `pass`)
-2. Deploy [`coop-cloud/traefik`]
+* `abra app config <app-name>`
-3. `abra app new loomio` (optionally with `--pass` if you'd like
+* insert your smtp password with `abra app secret insert <app-name> smtp_password v1 "<your-password>"`
-   to save secrets in `pass`)
+* `abra app deploy <app-name>`
-4. `abra app YOURAPPDOMAIN config` - be sure to change `$DOMAIN` to something that resolves to
+* Open the configured domain in your browser to create your user account (only works in case mail is configured correctly)
-   your Docker swarm box
+* Give yourself admin rights by running `abra app cmd <app-name> app make_last_user_admin`
-5. `abra app YOURAPPDOMAIN deploy`
+* Deploy [swarm-cronjob](https://recipes.coopcloud.tech/swarm-cronjob) on your server if it is not running yet. This is needed for loomios cron container to be started to do hourly chores.
-6. This should be automated but you also need to run `abra app run loomio_some_domain app rake db:migrate`
-7. Open the configured domain in your browser to finish set-up
 
-[`abra`]: https://git.autonomic.zone/autonomic-cooperative/abra
+## Migration guide
-[`coop-cloud/traefik`]: https://git.autonomic.zone/coop-cloud/traefik
+
+### Upgrading to 4.0.0+v2.25.3 and later
+
+* Before upgrading to `4.0.0+v2.25.3` and later, one successful deployment of `3.0.0+v2.25.3` shall be performed. Doing both at the same time will lead to complications as 3.0.0 runs rails DB migrations and 4.0.0 upgrades the postgres DB version.
+
+### Broken releases
+
+Please do not use these releases, as they are having issues regarding database migrations:
+
+* 1.0.0+v2.25.2
+* 2.0.0+v2.25.3
+* 2.1.0+v2.25.3
+
+### Manual migration steps when upgrading from 0.6.0+v2.25.2 and earlier to 1.0.0+v2.25.2 and later
+
+* adapt your env file with the new vars, especially SECRET_DB_PASSWORD_VERSION=v1 and SECRET_SMTP_PASSWORD_VERSION=v1
+* insert your smtp password with `abra app secret insert <app-name> smtp_password v1 "<your-password>"`
+* `abra app secret generate <app-name> db_password v1`
+* `abra app deploy <app-name>`
+* set the new password in DB: `abra app cmd <app-name> db set_new_db_password`
+* redeploy if necessary
+
+For more, see [docs.coopcloud.tech](https://docs.coopcloud.tech).

abra.sh

@@ -1 +1,16 @@
-export LOOMIO_ENTRYPOINT_VERSION=v1
+export LOOMIO_ENTRYPOINT_VERSION=v6
+
+# cannot be integrated into entrypoint.sh as it requires the operator to create a user first
+function make_last_user_admin()
+{
+    export DATABASE_URL="postgresql://${POSTGRES_USER}:$(cat /run/secrets/db_password)@db/${POSTGRES_DB}"
+    SECRET_KEY_BASE=$(rake secret) rails runner "User.last.update(is_admin: true)"
+}
+
+# only run when upgrading from 0.6.0+v2.25.2 and earlier to 1.0.0+v2.25.2 and later
+function set_new_db_password()
+{
+    echo "setting new password for db user..."
+	psql -U $POSTGRES_USER -c "ALTER USER $POSTGRES_USER PASSWORD '$(cat /run/secrets/db_password)';"
+    echo "done"
+}

compose.smtp.yml

@@ -0,0 +1,28 @@
+version: "3.8"
+
+x-mail-env: &mail-env
+    SMTP_AUTH: ${SMTP_AUTH}
+    SMTP_DOMAIN: ${SMTP_DOMAIN}
+    SMTP_SERVER: ${SMTP_SERVER}
+    SMTP_PORT: ${SMTP_PORT}
+    SMTP_USERNAME: ${SMTP_USERNAME}
+    SMTP_PASSWORD:
+    SMTP_PASSWORD_FILE: /run/secrets/smtp_password
+    SMTP_USE_SSL: ${SMTP_USE_SSL}
+
+services:
+  app:
+    secrets:
+      - smtp_password
+    environment:
+      *mail-env
+  worker:
+    secrets:
+      - smtp_password
+    environment:
+      *mail-env
+
+secrets:
+  smtp_password:
+    name: ${STACK_NAME}_smtp_password_${SECRET_SMTP_PASSWORD_VERSION}
+    external: true

compose.yml

@@ -1,31 +1,30 @@
 ---
 version: "3.8"
 
-x-environment: &default-env
+x-db-env: &db-env
-  DATABASE_URL: postgresql://postgres:password@db/loomio_production
+    POSTGRES_PASSWORD_FILE: /run/secrets/db_password
+    POSTGRES_DB: loomio_production
+    POSTGRES_USER: postgres
+
+x-redis-env: &redis-env
   REDIS_URL: redis://redis:6379
+
+x-environment: &default-env
+  <<: [*db-env, *redis-env]
   CANONICAL_HOST: ${DOMAIN}
   VIRTUAL_HOST: ${DOMAIN}
   CHANNELS_URI: wss://channels.${DOMAIN}
-  TLD_LENGTH: ${TLD_LENGTH}
   SUPPORT_EMAIL: ${SUPPORT_EMAIL}
-  SMTP_AUTH: ${SMTP_AUTH}
+  NOTIFICATIONS_EMAIL_ADDRESS: ${NOTIFICATIONS_EMAIL_ADDRESS}
-  SMTP_DOMAIN: ${SMTP_DOMAIN}
-  SMTP_SERVER: ${SMTP_SERVER}
-  SMTP_PORT: ${SMTP_PORT}
-  SMTP_USERNAME: ${SMTP_USERNAME}
-  SMTP_PASSWORD: ${SMTP_PASSWORD}
-  SMTP_USE_SSL: ${SMTP_USE_SSL}
   REPLY_HOSTNAME: ${REPLY_HOSTNAME}
-  HELPER_BOT_EMAIL: ${HELPER_BOT_EMAIL}
   RAILS_ENV: ${RAILS_ENV}
   PUMA_WORKERS: ${PUMA_WORKERS}
   MIN_THREADS: ${MIN_THREADS}
   MAX_THREADS: ${MAX_THREADS}
   FORCE_SSL: ${FORCE_SSL}
   USE_RACK_ATTACK: ${USE_RACK_ATTACK}
-  DEVISE_SECRET: uuaYZNGgeKCMqGc5pPd3seNSrvKkVO9+pAu6JvrQI4M6T395m8dLJja5qza8HaHY
+  DEVISE_SECRET_FILE: /run/secrets/devise_secret
-  SECRET_COOKIE_TOKEN: uuaYZNGgeKCMqGc5pPd3seNSrvKkVO9+pAu6JvrQI4M6T395m8dLJja5qza8HaHY
+  SECRET_COOKIE_TOKEN_FILE: /run/secrets/secret_cookie_token
   SAML_APP_KEY:
   SAML_IDP_METADATA_URL:
   SAML_ISSUER:
@@ -33,12 +32,16 @@ x-environment: &default-env
 
 services:
   app:
-    image: loomio/loomio:v2.11.13
+    image: loomio/loomio:v2.25.4
     configs:
       - source: entrypoint
         target: /entrypoint.sh
         mode: 0555
     entrypoint: /entrypoint.sh
+    secrets:
+      - devise_secret
+      - secret_cookie_token
+      - db_password
     volumes:
       - loomio_uploads:/loomio/public/system
       - loomio_storage:/loomio/storage
@@ -52,6 +55,12 @@ services:
       - db
       - redis
     environment: *default-env
+    healthcheck:
+      test: ["CMD", "curl", "-f", "http://localhost:3000"]
+      interval: 30s
+      timeout: 10s
+      retries: 10
+      start_period: 2m
     deploy:
       restart_policy:
         condition: on-failure
@@ -61,9 +70,25 @@ services:
         - "traefik.http.routers.${STACK_NAME}.rule=Host(`${DOMAIN}`${EXTRA_DOMAINS})"
         - "traefik.http.routers.${STACK_NAME}.entrypoints=web-secure"
         - "traefik.http.routers.${STACK_NAME}.tls.certresolver=${LETS_ENCRYPT_ENV}"
-        - "coop-cloud.${STACK_NAME}.version=0.1.0+v2.7.10"
+        - "coop-cloud.${STACK_NAME}.version=4.0.2+v2.25.4"
+        - "backupbot.backup:=${ENABLE_BACKUPS:-true}"
+        - "traefik.http.routers.${STACK_NAME}.middlewares=${STACK_NAME}"
+        - "traefik.http.middlewares.${STACK_NAME}.headers.accesscontrolallowmethods=GET,OPTIONS,PUT"
+        - "traefik.http.middlewares.${STACK_NAME}.headers.accesscontrolallowheaders=*"
+        - "traefik.http.middlewares.${STACK_NAME}.headers.accessControlAllowOriginList=https://*.${DOMAIN}"
+        - "traefik.http.middlewares.${STACK_NAME}.headers.accesscontrolmaxage=100"
+        - "traefik.http.middlewares.${STACK_NAME}.headers.addvaryheader=true"
   worker:
-    image: loomio/loomio:v2.11.13
+    image: loomio/loomio:v2.25.4
+    configs:
+      - source: entrypoint
+        target: /entrypoint.sh
+        mode: 0555
+    entrypoint: /entrypoint.sh
+    secrets:
+      - devise_secret
+      - secret_cookie_token
+      - db_password
     networks:
       - backend
     environment:
@@ -77,20 +102,31 @@ services:
       - loomio_files:/loomio/public/files
       - loomio_plugins:/loomio/plugins/docker
   db:
-    image: postgres:12.7
+    # loomio version upgrades and postgres version upgrade should not be performed at the same time.
+    image: pgautoupgrade/pgautoupgrade:17-debian
     networks:
       - backend
     volumes:
       - pgdata:/pgdata
-      - pgdumps:/pgdumps
+    secrets:
+      - db_password
     environment:
-      - POSTGRES_PASSWORD=password
+      <<: *db-env
-      - POSTGRES_DB=loomio_production
+      PGDATA: /pgdata
-      - PGDATA=/pgdata
+    deploy:
+      labels:
+        backupbot.backup: "${ENABLE_BACKUPS:-true}"
+        backupbot.backup.pre-hook: sh -c 'pg_dump -U "$$POSTGRES_USER" -Fc "$$POSTGRES_DB" | gzip > "/postgres.dump.gz"'
+        backupbot.backup.path: "/postgres.dump.gz"
+        backupbot.backup.post-hook: "rm -f /postgres.dump.gz"
+        backupbot.restore: "true"
+        backupbot.restore.post-hook: sh -c 'gzip -d /postgres.dump.gz && pg_restore --clean -U "$$POSTGRES_USER" --dbname="$$POSTGRES_DB" < /postgres.dump && rm -f /postgres.dump'
   redis:
     image: redis:5.0
     networks:
       - backend
+    healthcheck:
+      test: ["CMD", "redis-cli", "ping"]
   #mailin:
   #  image: loomio/mailin-docker:latest
   #  networks:
@@ -103,15 +139,34 @@ services:
     image: loomio/loomio_channel_server
     networks:
       - backend
+      - proxy
     depends_on:
       - redis
     environment:
-      - REDIS_URL=redis://redis:6379
+      <<: *redis-env
+      VIRTUAL_HOST: channels.${DOMAIN}
+    deploy:
+      labels:
+        - "traefik.enable=true"
+        - "traefik.docker.network=proxy"
+        - "traefik.http.routers.channels${STACK_NAME}.rule=Host(`channels.${DOMAIN}`)"
+        - "traefik.http.routers.channels${STACK_NAME}.tls=true"
+        - "traefik.http.routers.channels${STACK_NAME}.tls.certresolver=${LETS_ENCRYPT_ENV}"
+        - "traefik.http.services.channels${STACK_NAME}.loadbalancer.server.port=5000"
+        - "traefik.http.routers.channels${STACK_NAME}.entrypoints=web-secure"
+
   cron:
-    image: loomio/loomio:v2.7.10
+    image: loomio/loomio:v2.25.4
-    entrypoint: "/usr/local/bundle/bin/bundle"
+    configs:
-    command: ["exec", "rake", "loomio:hourly_tasks"]
+      - source: entrypoint
+        target: /entrypoint.sh
+        mode: 0555
+    entrypoint: [ "/entrypoint.sh", "rake loomio:hourly_tasks" ]
     environment: *default-env
+    secrets:
+      - devise_secret
+      - secret_cookie_token
+      - db_password
     volumes:
       - loomio_uploads:/loomio/public/system
       - loomio_storage:/loomio/storage
@@ -142,9 +197,19 @@ volumes:
   loomio_plugins:
   loomio_import:
   pgdata:
-  pgdumps:
 
 configs:
   entrypoint:
     name: ${STACK_NAME}_entrypoint_${LOOMIO_ENTRYPOINT_VERSION}
     file: entrypoint.sh
+
+secrets:
+  devise_secret:
+    external: true
+    name: ${STACK_NAME}_devise_secret_${SECRET_DEVISE_SECRET_VERSION}
+  secret_cookie_token:
+    external: true
+    name: ${STACK_NAME}_secret_cookie_token_${SECRET_SECRET_COOKIE_TOKEN_VERSION}
+  db_password:
+    external: true
+    name: ${STACK_NAME}_db_password_${SECRET_DB_PASSWORD_VERSION}

entrypoint.sh

@@ -1,10 +1,66 @@
-#! /bin/sh
+#!/usr/bin/env bash
 
-if test ! -f /loomio/storage/migrations_ran; then
+file_env() {
-    echo "first deploy, running migrations..."
+	# 3wc: Load $VAR_FILE into $VAR - useful for secrets. See
-    rake db:setup
+	# 	https://medium.com/@adrian.gheorghe.dev/using-docker-secrets-in-your-environment-variables-7a0609659aab
-    touch /loomio/storage/migrations_ran
+	local var="$1"
+	local fileVar="${var}_FILE"
+	local def="${2:-}"
+
+	if [ "${!var:-}" ] && [ "${!fileVar:-}" ]; then
+		echo >&2 "error: both $var and $fileVar are set (but are exclusive)"
+		exit 1
+	fi
+	local val="$def"
+	if [ "${!var:-}" ]; then
+		val="${!var}"
+	elif [ "${!fileVar:-}" ]; then
+		val="$(< "${!fileVar}")"
+	fi
+	export "$var"="$val"
+	unset "$fileVar"
+}
+
+file_env "DEVISE_SECRET"
+file_env "SECRET_COOKIE_TOKEN"
+file_env "POSTGRES_PASSWORD"
+file_env "SMTP_PASSWORD"
+export DB_HOST="db"
+export DATABASE_URL="postgresql://${POSTGRES_USER}:${POSTGRES_PASSWORD}@db/${POSTGRES_DB}"
+
+if [ -n "$1" ]; then
+	echo "Running '$1'"
+	$1
+else
+
+	echo "starting loomio!"
+	if [ "$TASK" = "worker" ]; then
+  		bundle exec sidekiq
+	else
+		sudo apt update -y && sudo apt install -y postgresql-client
+		bundle install
+
+		# running this code instaed of db:prepare in docker_start.sh in loomio container
+		# as postgres container creates empty db, somehow db:prepare cannot cope.
+		# therefore we run db:setup or db:migrate individually
+		if PGPASSWORD=$(cat /run/secrets/db_password) psql -U "$POSTGRES_USER" -h "$DB_HOST" -lqt | cut -d \| -f 1 | grep -wq "$POSTGRES_DB"; then
+			echo "database '$POSTGRES_DB' exists."
+
+			# check if the database contains tables
+			TABLE_COUNT=$(PGPASSWORD=$(cat /run/secrets/db_password) psql -U "$POSTGRES_USER" -h "$DB_HOST" -d "$POSTGRES_DB" -t -c "SELECT count(*) FROM information_schema.tables WHERE table_schema = 'public';")
+
+			if [ "$TABLE_COUNT" -eq 0 ]; then
+				echo "Database '$POSTGRES_DB' is empty, running db:setup."
+				bundle exec rake db:setup
+			else
+				echo "database '$POSTGRES_DB' not empty, running migrations."
+				bundle exec rake db:migrate
+			fi
+		else
+			echo "database '$POSTGRES_DB' does not exist, running db:setup."
+			bundle exec rake db:setup
+		fi
+
+		bundle exec puma -C config/puma.rb
+	fi
 fi
-
-echo "starting loomio!"
-/loomio/docker_start.sh

release/0.3.1+v2.19.0

@@ -0,0 +1,9 @@
+Loomio seems to have added a new setting, NOTIFICATIONS_EMAIL_ADDRESS, to
+define what address transactional emails should come from. 
+
+If you don't set it, it will default to notifications@$MAIL_DOMAIN, which is
+unlikely to work in many cases.
+
+If you find that transactional emails aren't working, try setting
+NOTIFICATIONS_EMAIL_ADDRESS to the same value as SUPPORT_EMAIL,
+HELPER_BOT_EMAIL, or SMTP_USERNAME.

release/1.0.0+v2.25.2

@@ -0,0 +1,14 @@
+In this release the passwords for smtp and postgres DB were moved into docker secrets. Therefore a few manual steps need to be performed (also available in recipe documentation)
+
+* adapt your env file with the new vars, especially `SECRET_DB_PASSWORD_VERSION=v1` and `SECRET_SMTP_PASSWORD_VERSION=v1` (remember, you can use `abra app check <app-name>` to check for any missing variables)
+* insert your existing smtp password with `abra app secret insert <app-name> smtp_password v1 "<your-password>"`
+
+Then, choose whether to keep the existing insecure database password (easy):
+* `abra app secret insert <app-name> db_password v1 password`
+
+Or, switch to a new secure password (harder, better):
+* `abra app secret generate <app-name> db_password v1`
+* `abra app deploy <app-name>`
+* set the new password in DB: `abra app cmd <app-name> db set_new_db_password` (NOTE: if you get "FATA loomio doesn't have a set_new_db_password function" here, run `cd ~/.abra/recipes/loomio && git checkout main`, then re-run the `abra app cmd` command with `-C` at the end)
+* redeploy
+

release/2.0.0+v2.25.3

@@ -0,0 +1,2 @@
+The major change in this release is the upgrade of postgres db from 10 to 17. As we use a container with automatic migration, this should work seemlessly without operator intervention.
+NEVERTHELESS, please create a BACKUP before you upgrade to this version!

release/2.1.0+v2.25.3

@@ -0,0 +1 @@
+DB initialization issues with wrong migration should be fixed now. (But not really sure why)

release/3.0.0+v2.25.3

@@ -0,0 +1 @@
+ATTENTION: do not skip this relase, otherwise it will lead to database migration complications. This release fixes migration issues of 1.x and 2.x releases.

release/4.0.0+v2.25.3

@@ -0,0 +1 @@
+ATTENTION: Perform a deployment of version 3.0.0+v2.25.3 before upgrading to this version. Otherwise there will be complications with database migrations. This version upgrades the used postgres container to version 17 with auto-upgrade functionality.