Merge pull request 'make channels container externally available' (#12) from fix-channels-container into main

Reviewed-on: #12
Reviewed-by: 3wordchant <3wordchant@noreply.git.coopcloud.tech>

.env.sample

@@ -41,7 +41,6 @@ USE_RACK_ATTACK=1
 SECRET_DEVISE_SECRET_VERSION=v1 #length=64
 SECRET_SECRET_COOKIE_TOKEN_VERSION=v1 #length=64
 SECRET_DB_PASSWORD_VERSION=v1
-SECRET_REDIS_PASSWORD_VERSION=v1
 
 # Send catch up email (missed yesterday) weekly
 # EMAIL_CATCH_UP_WEEKLY=1

README.md

@@ -21,8 +21,23 @@
 * `abra app deploy <app-name>`
 * Open the configured domain in your browser to create your user account (only works in case mail is configured correctly)
 * Give yourself admin rights by running `abra app cmd <app-name> app make_last_user_admin`
+* Deploy [swarm-cronjob](https://recipes.coopcloud.tech/swarm-cronjob) on your server if it is not running yet. This is needed for loomios cron container to be started to do hourly chores.
 
-## Manuel migration steps when upgrading from 0.6.0+v2.25.2 and earlier to 1.0.0+v2.25.2 and later
+## Migration guide
+
+### Upgrading to 4.0.0+v2.25.3 and later
+
+* Before upgrading to `4.0.0+v2.25.3` and later, one successful deployment of `3.0.0+v2.25.3` shall be performed. Doing both at the same time will lead to complications as 3.0.0 runs rails DB migrations and 4.0.0 upgrades the postgres DB version.
+
+### Broken releases
+
+Please do not use these releases, as they are having issues regarding database migrations:
+
+* 1.0.0+v2.25.2
+* 2.0.0+v2.25.3
+* 2.1.0+v2.25.3
+
+### Manual migration steps when upgrading from 0.6.0+v2.25.2 and earlier to 1.0.0+v2.25.2 and later
 
 * adapt your env file with the new vars, especially SECRET_DB_PASSWORD_VERSION=v1 and SECRET_SMTP_PASSWORD_VERSION=v1
 * insert your smtp password with `abra app secret insert <app-name> smtp_password v1 "<your-password>"`

abra.sh

@@ -1,5 +1,4 @@
-export LOOMIO_ENTRYPOINT_VERSION=v5
-export REDIS_ENTRYPOINT_VERSION=v1
+export LOOMIO_ENTRYPOINT_VERSION=v6
 
 # cannot be integrated into entrypoint.sh as it requires the operator to create a user first
 function make_last_user_admin()

compose.yml

@@ -7,12 +7,10 @@ x-db-env: &db-env
     POSTGRES_USER: postgres
 
 x-redis-env: &redis-env
-  REDIS_PASSWORD_FILE: /run/secrets/redis_password
-  REDIS_URL: redis://:{REDIS_PASSWORD}@redis:6379
+  REDIS_URL: redis://redis:6379
 
 x-environment: &default-env
-  <<: *db-env
-  <<: *redis-env
+  <<: [*db-env, *redis-env]
   CANONICAL_HOST: ${DOMAIN}
   VIRTUAL_HOST: ${DOMAIN}
   CHANNELS_URI: wss://channels.${DOMAIN}
@@ -34,7 +32,7 @@ x-environment: &default-env
 
 services:
   app:
-    image: loomio/loomio:v2.25.3
+    image: loomio/loomio:v2.25.4
     configs:
       - source: entrypoint
         target: /entrypoint.sh
@@ -44,7 +42,6 @@ services:
       - devise_secret
       - secret_cookie_token
       - db_password
-      - redis_password
     volumes:
       - loomio_uploads:/loomio/public/system
       - loomio_storage:/loomio/storage
@@ -73,10 +70,16 @@ services:
         - "traefik.http.routers.${STACK_NAME}.rule=Host(`${DOMAIN}`${EXTRA_DOMAINS})"
         - "traefik.http.routers.${STACK_NAME}.entrypoints=web-secure"
         - "traefik.http.routers.${STACK_NAME}.tls.certresolver=${LETS_ENCRYPT_ENV}"
-        - "coop-cloud.${STACK_NAME}.version=2.0.0+v2.25.3"
+        - "coop-cloud.${STACK_NAME}.version=4.0.2+v2.25.4"
         - "backupbot.backup:=${ENABLE_BACKUPS:-true}"
+        - "traefik.http.routers.${STACK_NAME}.middlewares=${STACK_NAME}"
+        - "traefik.http.middlewares.${STACK_NAME}.headers.accesscontrolallowmethods=GET,OPTIONS,PUT"
+        - "traefik.http.middlewares.${STACK_NAME}.headers.accesscontrolallowheaders=*"
+        - "traefik.http.middlewares.${STACK_NAME}.headers.accessControlAllowOriginList=https://*.${DOMAIN}"
+        - "traefik.http.middlewares.${STACK_NAME}.headers.accesscontrolmaxage=100"
+        - "traefik.http.middlewares.${STACK_NAME}.headers.addvaryheader=true"
   worker:
-    image: loomio/loomio:v2.25.3
+    image: loomio/loomio:v2.25.4
     configs:
       - source: entrypoint
         target: /entrypoint.sh
@@ -86,7 +89,6 @@ services:
       - devise_secret
       - secret_cookie_token
       - db_password
-      - redis_password
     networks:
       - backend
     environment:
@@ -100,6 +102,7 @@ services:
       - loomio_files:/loomio/public/files
       - loomio_plugins:/loomio/plugins/docker
   db:
+    # loomio version upgrades and postgres version upgrade should not be performed at the same time.
     image: pgautoupgrade/pgautoupgrade:17-debian
     networks:
       - backend
@@ -119,19 +122,11 @@ services:
         backupbot.restore: "true"
         backupbot.restore.post-hook: sh -c 'gzip -d /postgres.dump.gz && pg_restore --clean -U "$$POSTGRES_USER" --dbname="$$POSTGRES_DB" < /postgres.dump && rm -f /postgres.dump'
   redis:
-    image: redis:7.4
+    image: redis:5.0
     networks:
       - backend
-    command: /bin/sh -c "redis-server redis.conf --loglevel debug"
     healthcheck:
       test: ["CMD", "redis-cli", "ping"]
-    secrets:
-      - redis_password
-    configs:
-      - source: entrypoint_redis
-        target: /entrypoint.sh
-        mode: 0555
-    entrypoint: /entrypoint.sh
   #mailin:
   #  image: loomio/mailin-docker:latest
   #  networks:
@@ -144,14 +139,24 @@ services:
     image: loomio/loomio_channel_server
     networks:
       - backend
+      - proxy
     depends_on:
       - redis
-    secrets:
-      - redis_password
     environment:
       <<: *redis-env
+      VIRTUAL_HOST: channels.${DOMAIN}
+    deploy:
+      labels:
+        - "traefik.enable=true"
+        - "traefik.docker.network=proxy"
+        - "traefik.http.routers.channels${STACK_NAME}.rule=Host(`channels.${DOMAIN}`)"
+        - "traefik.http.routers.channels${STACK_NAME}.tls=true"
+        - "traefik.http.routers.channels${STACK_NAME}.tls.certresolver=${LETS_ENCRYPT_ENV}"
+        - "traefik.http.services.channels${STACK_NAME}.loadbalancer.server.port=5000"
+        - "traefik.http.routers.channels${STACK_NAME}.entrypoints=web-secure"
+
   cron:
-    image: loomio/loomio:v2.25.3
+    image: loomio/loomio:v2.25.4
     configs:
       - source: entrypoint
         target: /entrypoint.sh
@@ -162,7 +167,6 @@ services:
       - devise_secret
       - secret_cookie_token
       - db_password
-      - redis_password
     volumes:
       - loomio_uploads:/loomio/public/system
       - loomio_storage:/loomio/storage
@@ -198,9 +202,6 @@ configs:
   entrypoint:
     name: ${STACK_NAME}_entrypoint_${LOOMIO_ENTRYPOINT_VERSION}
     file: entrypoint.sh
-  entrypoint_redis:
-    name: ${STACK_NAME}_entrypoint_redis_${REDIS_ENTRYPOINT_VERSION}
-    file: entrypoint.redis.sh
 
 secrets:
   devise_secret:
@@ -212,6 +213,3 @@ secrets:
   db_password:
     external: true
     name: ${STACK_NAME}_db_password_${SECRET_DB_PASSWORD_VERSION}
-  redis_password:
-    external: true
-    name: ${STACK_NAME}_redis_password_${SECRET_REDIS_PASSWORD_VERSION}

entrypoint.redis.sh

@@ -1,6 +0,0 @@
-#!/usr/bin/env bash
-set -e
-
-echo "creating redis.conf..."
-echo "requirepass $(cat /run/secrets/redis_password)" > redis.conf
-echo "redis.conf created"

entrypoint.sh

@@ -25,22 +25,42 @@ file_env "DEVISE_SECRET"
 file_env "SECRET_COOKIE_TOKEN"
 file_env "POSTGRES_PASSWORD"
 file_env "SMTP_PASSWORD"
+export DB_HOST="db"
 export DATABASE_URL="postgresql://${POSTGRES_USER}:${POSTGRES_PASSWORD}@db/${POSTGRES_DB}"
 
 if [ -n "$1" ]; then
 	echo "Running '$1'"
 	$1
 else
-	if [ ! -f /loomio/storage/migrations_ran ] && [ "${TASK:-}" = "worker" ]; then
-		echo "first deploy, running DB setup..."
-		rake db:setup
-		touch /loomio/storage/migrations_ran
-	fi
-
-	echo "running DB migrations..."
-	rake db:migrate
-	echo "DB migrations finished"
 
 	echo "starting loomio!"
-	/loomio/docker_start.sh
+	if [ "$TASK" = "worker" ]; then
+  		bundle exec sidekiq
+	else
+		sudo apt update -y && sudo apt install -y postgresql-client
+		bundle install
+
+		# running this code instaed of db:prepare in docker_start.sh in loomio container
+		# as postgres container creates empty db, somehow db:prepare cannot cope.
+		# therefore we run db:setup or db:migrate individually
+		if PGPASSWORD=$(cat /run/secrets/db_password) psql -U "$POSTGRES_USER" -h "$DB_HOST" -lqt | cut -d \| -f 1 | grep -wq "$POSTGRES_DB"; then
+			echo "database '$POSTGRES_DB' exists."
+
+			# check if the database contains tables
+			TABLE_COUNT=$(PGPASSWORD=$(cat /run/secrets/db_password) psql -U "$POSTGRES_USER" -h "$DB_HOST" -d "$POSTGRES_DB" -t -c "SELECT count(*) FROM information_schema.tables WHERE table_schema = 'public';")
+
+			if [ "$TABLE_COUNT" -eq 0 ]; then
+				echo "Database '$POSTGRES_DB' is empty, running db:setup."
+				bundle exec rake db:setup
+			else
+				echo "database '$POSTGRES_DB' not empty, running migrations."
+				bundle exec rake db:migrate
+			fi
+		else
+			echo "database '$POSTGRES_DB' does not exist, running db:setup."
+			bundle exec rake db:setup
+		fi
+
+		bundle exec puma -C config/puma.rb
+	fi
 fi

release/1.0.0+v2.25.2

@@ -1,9 +1,14 @@
 In this release the passwords for smtp and postgres DB were moved into docker secrets. Therefore a few manual steps need to be performed (also available in recipe documentation)
 
-* adapt your env file with the new vars, especially SECRET_DB_PASSWORD_VERSION=v1 and SECRET_SMTP_PASSWORD_VERSION=v1
-* insert your smtp password with abra app secret insert <app-name> smtp_password v1 "<your-password>"
-* abra app secret generate <app-name> db_password v1
-* abra app deploy <app-name>
-* set the new password in DB: abra app cmd <app-name> db set_new_db_password
+* adapt your env file with the new vars, especially `SECRET_DB_PASSWORD_VERSION=v1` and `SECRET_SMTP_PASSWORD_VERSION=v1` (remember, you can use `abra app check <app-name>` to check for any missing variables)
+* insert your existing smtp password with `abra app secret insert <app-name> smtp_password v1 "<your-password>"`
+
+Then, choose whether to keep the existing insecure database password (easy):
+* `abra app secret insert <app-name> db_password v1 password`
+
+Or, switch to a new secure password (harder, better):
+* `abra app secret generate <app-name> db_password v1`
+* `abra app deploy <app-name>`
+* set the new password in DB: `abra app cmd <app-name> db set_new_db_password` (NOTE: if you get "FATA loomio doesn't have a set_new_db_password function" here, run `cd ~/.abra/recipes/loomio && git checkout main`, then re-run the `abra app cmd` command with `-C` at the end)
 * redeploy
 

release/2.1.0+v2.25.3

@@ -0,0 +1 @@
+DB initialization issues with wrong migration should be fixed now. (But not really sure why)

release/3.0.0+v2.25.3

@@ -0,0 +1 @@
+ATTENTION: do not skip this relase, otherwise it will lead to database migration complications. This release fixes migration issues of 1.x and 2.x releases.

release/4.0.0+v2.25.3

@@ -0,0 +1 @@
+ATTENTION: Perform a deployment of version 3.0.0+v2.25.3 before upgrading to this version. Otherwise there will be complications with database migrations. This version upgrades the used postgres container to version 17 with auto-upgrade functionality.