little more security

For more information see: toolshed/abra#534

.drone.yml

@@ -18,6 +18,10 @@ steps:
       STACK_NAME: loomio
       LETS_ENCRYPT_ENV: production
       LOOMIO_ENTRYPOINT_VERSION: v1
+      SECRET_DEVISE_SECRET_VERSION: v1
+      SECRET_SECRET_COOKIE_TOKEN_VERSION: v1
+      SECRET_DB_PASSWORD_VERSION: v1
+      SECRET_SMTP_PASSWORD_VERSION: v1
 trigger:
   branch:
     - main
@@ -33,7 +37,7 @@ steps:
         from_secret: drone_abra-bot_token
       fork: true
       repositories:
-        - coop-cloud/auto-recipes-catalogue-json
+        - toolshed/auto-recipes-catalogue-json
 
 trigger:
   event: tag

.env.sample

@@ -1,31 +1,29 @@
 TYPE=loomio
+COMPOSE_FILE="compose.yml"
 
 DOMAIN=loomio.example.com
 ## Domain aliases
 #EXTRA_DOMAINS=', `www.loomio.example.com`'
 LETS_ENCRYPT_ENV=production
 
-# the number of dots in your hostname
-TLD_LENGTH=3
-# TODO 3wc: is this needed?
-
+# mail setup
+COMPOSE_FILE="$COMPOSE_FILE:compose.smtp.yml"
 SUPPORT_EMAIL=noreply@example.com
 SMTP_AUTH=plain
 SMTP_DOMAIN=example.com
 SMTP_SERVER=example.com
 SMTP_PORT=587
 SMTP_USERNAME=noreply@example.com
-SMTP_PASSWORD=password
 SMTP_USE_SSL=1
 # to disable SSL comment out line rather than changing to 0
+SECRET_SMTP_PASSWORD_VERSION=v1
 
-# Whyyyy does this need to be set separately
+# From field for notification e-mails
 NOTIFICATIONS_EMAIL_ADDRESS=noreply@example.com
 
-REPLY_HOSTNAME=loomio.example.com
+# reply-to in email notifications
+REPLY_HOSTNAME=$DOMAIN
 
-# helper bot is the account which welcomes people to their groups.
-HELPER_BOT_EMAIL=noreply@loomio.example.com
 RAILS_ENV=production
 
 # Number of webserver processes and threads
@@ -42,6 +40,7 @@ USE_RACK_ATTACK=1
 
 SECRET_DEVISE_SECRET_VERSION=v1 #length=64
 SECRET_SECRET_COOKIE_TOKEN_VERSION=v1 #length=64
+SECRET_DB_PASSWORD_VERSION=v1
 
 # Send catch up email (missed yesterday) weekly
 # EMAIL_CATCH_UP_WEEKLY=1

README.md

@@ -4,27 +4,46 @@
 
 <!-- metadata -->
 * **Category**: Apps
-* **Status**: 0, work-in-progress
-* **Image**: [`loomio/*`](https://hub.docker.com/r/loomio)
-* **Healthcheck**: No
-* **Backups**: No
-* **Email**: ?
+* **Status**: 3, work-in-progress
+* **Image**: [`loomio/*`](https://hub.docker.com/r/loomio), 4, upstream
+* **Healthcheck**: Yes
+* **Backups**: Yes
+* **Email**: Outgoing yes, incoming no
 * **Tests**: No
 * **SSO**: No
 <!-- endmetadata -->
 
 ## Basic usage
 
-1. Set up Docker Swarm and [`abra`]
-2. Deploy [`coop-cloud/traefik`]
-3. `abra app new loomio` (optionally with `--pass` if you'd like
-   to save secrets in `pass`)
-4. `abra app config YOURAPPDOMAIN` - be sure to change `$DOMAIN` to something that resolves to
-   your Docker swarm box
-5. `abra app deploy YOURAPPDOMAIN`
-6. This should be automated but you also need to run `abra app run loomio_some_domain app rake db:migrate`
-7. Open the configured domain in your browser to finish set-up
-8. Give yourself admin rights by running `User.last.update(is_admin: true)`
+* `abra app new loomio --secrets   ` (optionally with `--pass` if you'd like to save secrets in `pass`)
+* `abra app config <app-name>`
+* insert your smtp password with `abra app secret insert <app-name> smtp_password v1 "<your-password>"`
+* `abra app deploy <app-name>`
+* Open the configured domain in your browser to create your user account (only works in case mail is configured correctly)
+* Give yourself admin rights by running `abra app cmd <app-name> app make_last_user_admin`
+* Deploy [swarm-cronjob](https://recipes.coopcloud.tech/swarm-cronjob) on your server if it is not running yet. This is needed for loomios cron container to be started to do hourly chores.
 
-[`abra`]: https://git.autonomic.zone/autonomic-cooperative/abra
-[`coop-cloud/traefik`]: https://git.autonomic.zone/coop-cloud/traefik
+## Migration guide
+
+### Upgrading to 4.0.0+v2.25.3 and later
+
+* Before upgrading to `4.0.0+v2.25.3` and later, one successful deployment of `3.0.0+v2.25.3` shall be performed. Doing both at the same time will lead to complications as 3.0.0 runs rails DB migrations and 4.0.0 upgrades the postgres DB version.
+
+### Broken releases
+
+Please do not use these releases, as they are having issues regarding database migrations:
+
+* 1.0.0+v2.25.2
+* 2.0.0+v2.25.3
+* 2.1.0+v2.25.3
+
+### Manual migration steps when upgrading from 0.6.0+v2.25.2 and earlier to 1.0.0+v2.25.2 and later
+
+* adapt your env file with the new vars, especially SECRET_DB_PASSWORD_VERSION=v1 and SECRET_SMTP_PASSWORD_VERSION=v1
+* insert your smtp password with `abra app secret insert <app-name> smtp_password v1 "<your-password>"`
+* `abra app secret generate <app-name> db_password v1`
+* `abra app deploy <app-name>`
+* set the new password in DB: `abra app cmd <app-name> db set_new_db_password`
+* redeploy if necessary
+
+For more, see [docs.coopcloud.tech](https://docs.coopcloud.tech).

abra.sh

@@ -1 +1,16 @@
-export LOOMIO_ENTRYPOINT_VERSION=v3
+export LOOMIO_ENTRYPOINT_VERSION=v6
+
+# cannot be integrated into entrypoint.sh as it requires the operator to create a user first
+function make_last_user_admin()
+{
+    export DATABASE_URL="postgresql://${POSTGRES_USER}:$(cat /run/secrets/db_password)@db/${POSTGRES_DB}"
+    SECRET_KEY_BASE=$(rake secret) rails runner "User.last.update(is_admin: true)"
+}
+
+# only run when upgrading from 0.6.0+v2.25.2 and earlier to 1.0.0+v2.25.2 and later
+function set_new_db_password()
+{
+    echo "setting new password for db user..."
+	psql -U $POSTGRES_USER -c "ALTER USER $POSTGRES_USER PASSWORD '$(cat /run/secrets/db_password)';"
+    echo "done"
+}

compose.smtp.yml

@@ -0,0 +1,28 @@
+version: "3.8"
+
+x-mail-env: &mail-env
+    SMTP_AUTH: ${SMTP_AUTH}
+    SMTP_DOMAIN: ${SMTP_DOMAIN}
+    SMTP_SERVER: ${SMTP_SERVER}
+    SMTP_PORT: ${SMTP_PORT}
+    SMTP_USERNAME: ${SMTP_USERNAME}
+    SMTP_PASSWORD:
+    SMTP_PASSWORD_FILE: /run/secrets/smtp_password
+    SMTP_USE_SSL: ${SMTP_USE_SSL}
+
+services:
+  app:
+    secrets:
+      - smtp_password
+    environment:
+      *mail-env
+  worker:
+    secrets:
+      - smtp_password
+    environment:
+      *mail-env
+
+secrets:
+  smtp_password:
+    name: ${STACK_NAME}_smtp_password_${SECRET_SMTP_PASSWORD_VERSION}
+    external: true

compose.yml

@@ -1,24 +1,22 @@
 ---
 version: "3.8"
 
-x-environment: &default-env
-  DATABASE_URL: postgresql://postgres:password@db/loomio_production
+x-db-env: &db-env
+    POSTGRES_PASSWORD_FILE: /run/secrets/db_password
+    POSTGRES_DB: loomio_production
+    POSTGRES_USER: postgres
+
+x-redis-env: &redis-env
   REDIS_URL: redis://redis:6379
+
+x-environment: &default-env
+  <<: [*db-env, *redis-env]
   CANONICAL_HOST: ${DOMAIN}
   VIRTUAL_HOST: ${DOMAIN}
   CHANNELS_URI: wss://channels.${DOMAIN}
-  TLD_LENGTH: ${TLD_LENGTH}
   SUPPORT_EMAIL: ${SUPPORT_EMAIL}
-  SMTP_AUTH: ${SMTP_AUTH}
-  SMTP_DOMAIN: ${SMTP_DOMAIN}
-  SMTP_SERVER: ${SMTP_SERVER}
-  SMTP_PORT: ${SMTP_PORT}
-  SMTP_USERNAME: ${SMTP_USERNAME}
-  SMTP_PASSWORD: ${SMTP_PASSWORD}
-  SMTP_USE_SSL: ${SMTP_USE_SSL}
   NOTIFICATIONS_EMAIL_ADDRESS: ${NOTIFICATIONS_EMAIL_ADDRESS}
   REPLY_HOSTNAME: ${REPLY_HOSTNAME}
-  HELPER_BOT_EMAIL: ${HELPER_BOT_EMAIL}
   RAILS_ENV: ${RAILS_ENV}
   PUMA_WORKERS: ${PUMA_WORKERS}
   MIN_THREADS: ${MIN_THREADS}
@@ -34,7 +32,7 @@ x-environment: &default-env
 
 services:
   app:
-    image: loomio/loomio:v2.19.0
+    image: loomio/loomio:v2.25.4
     configs:
       - source: entrypoint
         target: /entrypoint.sh
@@ -43,6 +41,7 @@ services:
     secrets:
       - devise_secret
       - secret_cookie_token
+      - db_password
     volumes:
       - loomio_uploads:/loomio/public/system
       - loomio_storage:/loomio/storage
@@ -56,6 +55,12 @@ services:
       - db
       - redis
     environment: *default-env
+    healthcheck:
+      test: ["CMD", "curl", "-f", "http://localhost:3000"]
+      interval: 30s
+      timeout: 10s
+      retries: 10
+      start_period: 2m
     deploy:
       restart_policy:
         condition: on-failure
@@ -65,9 +70,16 @@ services:
         - "traefik.http.routers.${STACK_NAME}.rule=Host(`${DOMAIN}`${EXTRA_DOMAINS})"
         - "traefik.http.routers.${STACK_NAME}.entrypoints=web-secure"
         - "traefik.http.routers.${STACK_NAME}.tls.certresolver=${LETS_ENCRYPT_ENV}"
-        - "coop-cloud.${STACK_NAME}.version=0.3.1+v2.19.0"
+        - "coop-cloud.${STACK_NAME}.version=4.0.2+v2.25.4"
+        - "backupbot.backup:=${ENABLE_BACKUPS:-true}"
+        - "traefik.http.routers.${STACK_NAME}.middlewares=${STACK_NAME}"
+        - "traefik.http.middlewares.${STACK_NAME}.headers.accesscontrolallowmethods=GET,OPTIONS,PUT"
+        - "traefik.http.middlewares.${STACK_NAME}.headers.accesscontrolallowheaders=*"
+        - "traefik.http.middlewares.${STACK_NAME}.headers.accessControlAllowOriginList=https://*.${DOMAIN}"
+        - "traefik.http.middlewares.${STACK_NAME}.headers.accesscontrolmaxage=100"
+        - "traefik.http.middlewares.${STACK_NAME}.headers.addvaryheader=true"
   worker:
-    image: loomio/loomio:v2.19.0
+    image: loomio/loomio:v2.25.4
     configs:
       - source: entrypoint
         target: /entrypoint.sh
@@ -76,6 +88,7 @@ services:
     secrets:
       - devise_secret
       - secret_cookie_token
+      - db_password
     networks:
       - backend
     environment:
@@ -89,20 +102,31 @@ services:
       - loomio_files:/loomio/public/files
       - loomio_plugins:/loomio/plugins/docker
   db:
-    image: postgres:12.14
+    # loomio version upgrades and postgres version upgrade should not be performed at the same time.
+    image: pgautoupgrade/pgautoupgrade:17-debian
     networks:
       - backend
     volumes:
       - pgdata:/pgdata
-      - pgdumps:/pgdumps
+    secrets:
+      - db_password
     environment:
-      - POSTGRES_PASSWORD=password
-      - POSTGRES_DB=loomio_production
-      - PGDATA=/pgdata
+      <<: *db-env
+      PGDATA: /pgdata
+    deploy:
+      labels:
+        backupbot.backup: "${ENABLE_BACKUPS:-true}"
+        backupbot.backup.pre-hook: sh -c 'pg_dump -U "$$POSTGRES_USER" -Fc "$$POSTGRES_DB" | gzip > "/postgres.dump.gz"'
+        backupbot.backup.path: "/postgres.dump.gz"
+        backupbot.backup.post-hook: "rm -f /postgres.dump.gz"
+        backupbot.restore: "true"
+        backupbot.restore.post-hook: sh -c 'gzip -d /postgres.dump.gz && pg_restore --clean -U "$$POSTGRES_USER" --dbname="$$POSTGRES_DB" < /postgres.dump && rm -f /postgres.dump'
   redis:
     image: redis:5.0
     networks:
       - backend
+    healthcheck:
+      test: ["CMD", "redis-cli", "ping"]
   #mailin:
   #  image: loomio/mailin-docker:latest
   #  networks:
@@ -115,21 +139,34 @@ services:
     image: loomio/loomio_channel_server
     networks:
       - backend
+      - proxy
     depends_on:
       - redis
     environment:
-      - REDIS_URL=redis://redis:6379
+      <<: *redis-env
+      VIRTUAL_HOST: channels.${DOMAIN}
+    deploy:
+      labels:
+        - "traefik.enable=true"
+        - "traefik.docker.network=proxy"
+        - "traefik.http.routers.channels${STACK_NAME}.rule=Host(`channels.${DOMAIN}`)"
+        - "traefik.http.routers.channels${STACK_NAME}.tls=true"
+        - "traefik.http.routers.channels${STACK_NAME}.tls.certresolver=${LETS_ENCRYPT_ENV}"
+        - "traefik.http.services.channels${STACK_NAME}.loadbalancer.server.port=5000"
+        - "traefik.http.routers.channels${STACK_NAME}.entrypoints=web-secure"
+
   cron:
-    image: loomio/loomio:v2.19.0
+    image: loomio/loomio:v2.25.4
     configs:
       - source: entrypoint
         target: /entrypoint.sh
         mode: 0555
-    entrypoint: [ "/entrypoint.sh", "rake", "loomio:hourly_tasks" ]
+    entrypoint: [ "/entrypoint.sh", "rake loomio:hourly_tasks" ]
     environment: *default-env
     secrets:
       - devise_secret
       - secret_cookie_token
+      - db_password
     volumes:
       - loomio_uploads:/loomio/public/system
       - loomio_storage:/loomio/storage
@@ -160,7 +197,6 @@ volumes:
   loomio_plugins:
   loomio_import:
   pgdata:
-  pgdumps:
 
 configs:
   entrypoint:
@@ -174,3 +210,6 @@ secrets:
   secret_cookie_token:
     external: true
     name: ${STACK_NAME}_secret_cookie_token_${SECRET_SECRET_COOKIE_TOKEN_VERSION}
+  db_password:
+    external: true
+    name: ${STACK_NAME}_db_password_${SECRET_DB_PASSWORD_VERSION}

entrypoint.sh

@@ -23,16 +23,44 @@ file_env() {
 
 file_env "DEVISE_SECRET"
 file_env "SECRET_COOKIE_TOKEN"
+file_env "POSTGRES_PASSWORD"
+file_env "SMTP_PASSWORD"
+export DB_HOST="db"
+export DATABASE_URL="postgresql://${POSTGRES_USER}:${POSTGRES_PASSWORD}@db/${POSTGRES_DB}"
 
-if test ! -f /loomio/storage/migrations_ran; then
-    echo "first deploy, running migrations..."
-    rake db:setup
-    touch /loomio/storage/migrations_ran
-fi
 if [ -n "$1" ]; then
 	echo "Running '$1'"
 	$1
 else
+
 	echo "starting loomio!"
-	/loomio/docker_start.sh
+	if [ "$TASK" = "worker" ]; then
+  		bundle exec sidekiq
+	else
+		sudo apt update -y && sudo apt install -y postgresql-client
+		bundle install
+
+		# running this code instaed of db:prepare in docker_start.sh in loomio container
+		# as postgres container creates empty db, somehow db:prepare cannot cope.
+		# therefore we run db:setup or db:migrate individually
+		if PGPASSWORD=$(cat /run/secrets/db_password) psql -U "$POSTGRES_USER" -h "$DB_HOST" -lqt | cut -d \| -f 1 | grep -wq "$POSTGRES_DB"; then
+			echo "database '$POSTGRES_DB' exists."
+
+			# check if the database contains tables
+			TABLE_COUNT=$(PGPASSWORD=$(cat /run/secrets/db_password) psql -U "$POSTGRES_USER" -h "$DB_HOST" -d "$POSTGRES_DB" -t -c "SELECT count(*) FROM information_schema.tables WHERE table_schema = 'public';")
+
+			if [ "$TABLE_COUNT" -eq 0 ]; then
+				echo "Database '$POSTGRES_DB' is empty, running db:setup."
+				bundle exec rake db:setup
+			else
+				echo "database '$POSTGRES_DB' not empty, running migrations."
+				bundle exec rake db:migrate
+			fi
+		else
+			echo "database '$POSTGRES_DB' does not exist, running db:setup."
+			bundle exec rake db:setup
+		fi
+
+		bundle exec puma -C config/puma.rb
+	fi
 fi

release/0.3.1+v2.19.0

@@ -0,0 +1,9 @@
+Loomio seems to have added a new setting, NOTIFICATIONS_EMAIL_ADDRESS, to
+define what address transactional emails should come from. 
+
+If you don't set it, it will default to notifications@$MAIL_DOMAIN, which is
+unlikely to work in many cases.
+
+If you find that transactional emails aren't working, try setting
+NOTIFICATIONS_EMAIL_ADDRESS to the same value as SUPPORT_EMAIL,
+HELPER_BOT_EMAIL, or SMTP_USERNAME.

release/1.0.0+v2.25.2

@@ -0,0 +1,14 @@
+In this release the passwords for smtp and postgres DB were moved into docker secrets. Therefore a few manual steps need to be performed (also available in recipe documentation)
+
+* adapt your env file with the new vars, especially `SECRET_DB_PASSWORD_VERSION=v1` and `SECRET_SMTP_PASSWORD_VERSION=v1` (remember, you can use `abra app check <app-name>` to check for any missing variables)
+* insert your existing smtp password with `abra app secret insert <app-name> smtp_password v1 "<your-password>"`
+
+Then, choose whether to keep the existing insecure database password (easy):
+* `abra app secret insert <app-name> db_password v1 password`
+
+Or, switch to a new secure password (harder, better):
+* `abra app secret generate <app-name> db_password v1`
+* `abra app deploy <app-name>`
+* set the new password in DB: `abra app cmd <app-name> db set_new_db_password` (NOTE: if you get "FATA loomio doesn't have a set_new_db_password function" here, run `cd ~/.abra/recipes/loomio && git checkout main`, then re-run the `abra app cmd` command with `-C` at the end)
+* redeploy
+

release/2.0.0+v2.25.3

@@ -0,0 +1,2 @@
+The major change in this release is the upgrade of postgres db from 10 to 17. As we use a container with automatic migration, this should work seemlessly without operator intervention.
+NEVERTHELESS, please create a BACKUP before you upgrade to this version!

release/2.1.0+v2.25.3

@@ -0,0 +1 @@
+DB initialization issues with wrong migration should be fixed now. (But not really sure why)

release/3.0.0+v2.25.3

@@ -0,0 +1 @@
+ATTENTION: do not skip this relase, otherwise it will lead to database migration complications. This release fixes migration issues of 1.x and 2.x releases.

release/4.0.0+v2.25.3

@@ -0,0 +1 @@
+ATTENTION: Perform a deployment of version 3.0.0+v2.25.3 before upgrading to this version. Otherwise there will be complications with database migrations. This version upgrades the used postgres container to version 17 with auto-upgrade functionality.