Compare commits

..

No commits in common. "main" and "main" have entirely different histories.
main ... main

11 changed files with 65 additions and 306 deletions

View File

@ -1,42 +0,0 @@
---
kind: pipeline
name: deploy to swarm-test.autonomic.zone
steps:
- name: deployment
image: git.coopcloud.tech/coop-cloud/stack-ssh-deploy:latest
settings:
host: swarm-test.autonomic.zone
stack: outline
generate_secrets: true
purge: true
deploy_key:
from_secret: drone_ssh_swarm_test
networks:
- proxy
environment:
DOMAIN: outline.swarm-test.autonomic.zone
STACK_NAME: outline
LETS_ENCRYPT_ENV: production
APP_ENTRYPOINT_VERSION: v1
SECRET_DB_PASSWORD_VERSION: v1
SECRET_SECRET_KEY_VERSION: v1 # length=64
SECRET_UTILS_SECRET_VERSION: v1 # length=64
trigger:
branch:
- main
---
kind: pipeline
name: generate recipe catalogue
steps:
- name: release a new version
image: plugins/downstream
settings:
server: https://build.coopcloud.tech
token:
from_secret: drone_abra-bot_token
fork: true
repositories:
- coop-cloud/auto-recipes-catalogue-json
trigger:
event: tag

View File

@ -9,15 +9,23 @@ DOMAIN=outline.example.com
LETS_ENCRYPT_ENV=production
COMPOSE_FILE="compose.yml"
#COMPOSE_YML="compose.yml:compose.oidc.yml"
#COMPOSE_YML="compose.yml:compose.google.yml"
# REQUIRED
SECRET_DB_PASSWORD_VERSION=v1
SECRET_SECRET_KEY_VERSION=v1 # length=64
SECRET_UTILS_SECRET_VERSION=v1 # length=64
SECRET_AWS_SECRET_KEY_VERSION=v1
# Set to s3 to use AWS S3 bucket
FILE_STORAGE=local
AWS_ACCESS_KEY_ID=
AWS_REGION=
AWS_S3_UPLOAD_BUCKET_URL=
AWS_S3_UPLOAD_BUCKET_NAME=
AWS_S3_UPLOAD_MAX_SIZE=26214400
AWS_S3_FORCE_PATH_STYLE=true
AWS_S3_ACL=private
# OPTIONAL
@ -49,22 +57,19 @@ DEBUG=http
# set, all domains are allowed by default when using Google OAuth to signin
ALLOWED_DOMAINS=
# TODO: setup compose.smtp.yml
# To support sending outgoing transactional emails such as "document updated" or
# "you've been invited" you'll need to provide authentication for an SMTP server
# By default, this enables email login. You can disable this in the settings
# for configuration details see https://docs.getoutline.com/s/hosting/doc/smtp-cqCJyZGMIB
#COMPOSE_FILE="$COMPOSE_FILE:compose.smtp.yml"
#SMTP_ENABLED=1
#SMTP_HOST=
#SMTP_PORT=
#SMTP_USERNAME=
#SMTP_PASSWORD=
#SMTP_FROM_EMAIL=
#SMTP_REPLY_EMAIL=
#SMTP_TLS_CIPHERS=
#SMTP_SECURE=true
#SECRET_SMTP_PASSWORD_VERSION=v1
#COMPOSE_FILE="$COMPOSE_FILE:compose.oidc.yml"
#OIDC_ENABLED=1
#OIDC_CLIENT_ID=
#OIDC_AUTH_URI=
@ -75,20 +80,6 @@ ALLOWED_DOMAINS=
#OIDC_SCOPES="openid profile email"
#SECRET_OIDC_CLIENT_SECRET_VERSION=v1
#COMPOSE_FILE="$COMPOSE_FILE:compose.google.yml"
#GOOGLE_ENABLED=1
#GOOGLE_CLIENT_ID=
#SECRET_GOOGLE_CLIENT_SECRET_VERSION=v1
COMPOSE_FILE="$COMPOSE_FILE:compose.local.yml"
FILE_STORAGE_UPLOAD_MAX_SIZE=26214400
#COMPOSE_FILE="$COMPOSE_FILE:compose.aws.yml"
#AWS_ACCESS_KEY_ID=
#AWS_REGION=
#AWS_S3_UPLOAD_BUCKET_URL=
#AWS_S3_UPLOAD_BUCKET_NAME=
#AWS_S3_UPLOAD_MAX_SIZE=26214400
#AWS_S3_FORCE_PATH_STYLE=true
#AWS_S3_ACL=private
#SECRET_AWS_SECRET_KEY_VERSION=v1

View File

@ -5,12 +5,12 @@ Wiki and knowledge base for growing teams
<!-- metadata -->
* **Category**: Apps
* **Status**: 3, beta
* **Image**: [outlinewiki/outline](https://hub.docker.com/r/outlinewiki/outline), 4, upstream
* **Status**: 1, alpha
* **Image**: [outlinewiki/outline](https://hub.docker.com/r/outlinewiki/outline)
* **Healthcheck**: No
* **Backups**: Yes
* **Email**: Yes
* **Tests**: 2
* **Backups**: No
* **Email**: No
* **Tests**: No
* **SSO**: 3 (OAuth)
<!-- endmetadata -->
@ -19,27 +19,26 @@ Wiki and knowledge base for growing teams
1. Set up Docker Swarm and [`abra`]
2. Deploy [`coop-cloud/traefik`]
3. `abra app new ${REPO_NAME}`
- **WARNING**: Choose "n" when `abra` asks if you'd like to generate secrets
3. `abra app new ${REPO_NAME} --secrets` (optionally with `--pass` if you'd like
to save secrets in `pass`)
4. `abra app config YOURAPPNAME` - be sure to change `$DOMAIN` to something that resolves to
your Docker swarm box
5. Insert secrets:
- `abra app secret insert YOURAPPNAME secret_key v1 $(openssl rand -hex 32)` #12
- `abra app secret generate -a YOURAPPNAME`
6. `abra app deploy YOURAPPNAME`
8. Open the configured domain in your browser to finish set-up
5. `abra app deploy YOURAPPNAME`
7. Open the configured domain in your browser to finish set-up
[`abra`]: https://git.coopcloud.tech/coop-cloud/abra
[`coop-cloud/traefik`]: https://git.coopcloud.tech/coop-cloud/traefik
## Tips & Tricks
### Create an initial admin user
### Post-deploy migration
```
abra app cmd YOURAPPNAME app create_email_user test@example.com
abra app cmd YOURAPPNAME app migrate
```
_As of 2022-03-30, this requires `abra` RC version, run `abra upgrade --rc`._
### Setting up your `.env` config
Avoid the use of quotes (`"..."`) as much as possible, the NodeJS scripts flip out for some reason on some vars.
@ -52,31 +51,4 @@ Where `<username-to-delete>` is the username of the user to be removed, and
`<username-to-replace>` is the username of another user, to assign documents and
revisions to (instead of deleting them).
### Migrate from S3 to local storage
- `abra app config <domain>`, add
- `COMPOSE_FILE="$COMPOSE_FILE:compose.local.yml"`
- `FILE_STORAGE_UPLOAD_MAX_SIZE=26214400`
- `abra app deploy <domain> -f`
- compose.aws.yml should still be deployed!
- `abra app undeploy <domain>`
- on the docker host, find mountpoint of newly created volume via `docker volume ls` and `docker volume inspect`
- volume name is smth like `<domain>_storage-data`
- take note which linux user owns `<storage_mountpoint>` (likely `1001`)
- use s3cmd/rclone/... to sync your bucket to `<storage_mountpoint>`
- `chown -R <storage_user>:<storage_user> <storage_mountpoint>`
- `abra app config <domain>`, switch storage backend
- remove `AWS_*` vars, `SECRET_AWS_SECRET_KEY_VERSION` and `COMPOSE_FILE="$COMPOSE_FILE:compose.aws.yml"`
- set `FILE_STORAGE=local`
- `abra app deploy <domain> -f`
- enjoy getting rid of S3 🥳
## Single Sign On with Keycloak
`abra app config YOURAPPNAME`, then uncomment everything in the `OIDC_` section.
Create a new client in Keycloak:
- **Valid Redirect URIs**: `https://YOURAPPDOMAIN/auth/oidc.callback`
`abra app deploy YOURAPPDOMAIN`
_As of 2022-03-30, this requires `abra` RC version, run `abra upgrade --rc`._

91
abra.sh
View File

@ -1,17 +1,4 @@
export APP_ENTRYPOINT_VERSION=v9
export DB_ENTRYPOINT_VERSION=v2
create_email_user() {
if [ -z "$1" ]; then
echo "Usage: ... create_email_user <email_address>"
exit 1
fi
export DATABASE_PASSWORD=$(cat /run/secrets/db_password)
export DATABASE_URL="postgres://outline:${DATABASE_PASSWORD}@${STACK_NAME}_db:5432/outline"
export UTILS_SECRET=$(cat /run/secrets/utils_secret)
export SECRET_KEY=$(cat /run/secrets/secret_key)
node build/server/scripts/seed.js "$1"
}
export APP_ENTRYPOINT_VERSION=v5
migrate() {
export DATABASE_PASSWORD=$(cat /run/secrets/db_password)
@ -19,42 +6,9 @@ migrate() {
yarn db:migrate --env=production-ssl-disabled
}
delete_user_by_id() {
if [ -z "$1" ] || [ -z "$2" ]; then
echo "Usage: ... delete_user_by_id <userid-to-delete> <userid-to-replace>"
exit 1
fi
USERID_REPLACE="$2"
USERID_REMOVE="$1"
psql -U outline outline <<- SQL
UPDATE documents SET "userId" = '$USERID_REPLACE' WHERE "userId" = '$USERID_REMOVE';
UPDATE groups SET "createdById" = '$USERID_REPLACE' WHERE "createdById" = '$USERID_REMOVE';
UPDATE pins SET "createdById" = '$USERID_REPLACE' WHERE "createdById" = '$USERID_REMOVE';
UPDATE group_users SET "createdById" = '$USERID_REPLACE' WHERE "createdById" = '$USERID_REMOVE';
UPDATE collections SET "createdById" = '$USERID_REPLACE' WHERE "createdById" = '$USERID_REMOVE';
UPDATE collection_users SET "createdById" = '$USERID_REPLACE' WHERE "createdById" = '$USERID_REMOVE';
UPDATE collection_groups SET "createdById" = '$USERID_REPLACE' WHERE "createdById" = '$USERID_REMOVE';
UPDATE documents SET "lastModifiedById" = '$USERID_REPLACE' WHERE "lastModifiedById" = '$USERID_REMOVE';
UPDATE documents SET "createdById" = '$USERID_REPLACE' WHERE "createdById" = '$USERID_REMOVE';
UPDATE revisions SET "userId" = '$USERID_REPLACE' WHERE "userId" = '$USERID_REMOVE';
UPDATE attachments SET "userId" = '$USERID_REPLACE' WHERE "userId" = '$USERID_REMOVE';
UPDATE backlinks SET "userId" = '$USERID_REPLACE' WHERE "userId" = '$USERID_REMOVE';
UPDATE file_operations SET "userId" = '$USERID_REPLACE' WHERE "userId" = '$USERID_REMOVE';
UPDATE users SET "suspendedById" = '$USERID_REPLACE' WHERE "suspendedById" = '$USERID_REMOVE';
DELETE FROM search_queries WHERE "userId" = '$USERID_REMOVE';
DELETE FROM shares WHERE "userId" = '$USERID_REMOVE';
DELETE FROM notification_settings WHERE "userId" = '$USERID_REMOVE';
DELETE FROM events WHERE "actorId" = '$USERID_REMOVE';
DELETE FROM events WHERE "userId" = '$USERID_REMOVE';
DELETE FROM users WHERE "id" = '$USERID_REMOVE';
SQL
}
delete_user() {
if [ -z "$1" ] || [ -z "$2" ]; then
echo "Usage: ... delete_user <userid-to-delete> <userid-to-replace>"
echo "Usage: ... delete_user <user-to-delete> <user-to-replace>"
exit 1
fi
@ -70,26 +24,23 @@ delete_user() {
echo "Can't find ID of '$2'"
exit 1
fi
delete_user_by_id "$USERID_REMOVE" "$USERID_REPLACE"
}
delete_duplicate_users() {
if [ -z "$1" ]; then
echo "Usage: ... delete_duplicate_users <username>"
exit 1
fi
USERIDS=$(echo "SELECT id FROM users WHERE username = '$1' ORDER BY users.\"createdAt\" DESC" | psql -t -A -U outline outline)
if [ ! "$(echo "$USERIDS" | wc -l)" -gt 1 ]; then
echo "Only one user exists, bailing"
exit 1
fi
USERID_NEW=$(echo "$USERIDS" | head -n1)
for USERID_OLD in $(echo "$USERIDS" | tail -n+2); do
delete_user_by_id "$USERID_OLD" "$USERID_NEW"
done
psql -U outline outline <<- SQL
UPDATE documents SET "userId" = '$USERID_REPLACE' WHERE "userId" = '$USERID_REMOVE';
UPDATE groups SET "createdById" = '$USERID_REPLACE' WHERE "createdById" = '$USERID_REMOVE';
UPDATE pins SET "createdById" = '$USERID_REPLACE' WHERE "createdById" = '$USERID_REMOVE';
UPDATE group_users SET "createdById" = '$USERID_REPLACE' WHERE "createdById" = '$USERID_REMOVE';
UPDATE collections SET "createdById" = '$USERID_REPLACE' WHERE "createdById" = '$USERID_REMOVE';
UPDATE collection_users SET "createdById" = '$USERID_REPLACE' WHERE "createdById" = '$USERID_REMOVE';
UPDATE documents SET "lastModifiedById" = '$USERID_REPLACE' WHERE "lastModifiedById" = '$USERID_REMOVE';
UPDATE documents SET "createdById" = '$USERID_REPLACE' WHERE "createdById" = '$USERID_REMOVE';
UPDATE revisions SET "userId" = '$USERID_REPLACE' WHERE "userId" = '$USERID_REMOVE';
UPDATE attachments SET "userId" = '$USERID_REPLACE' WHERE "userId" = '$USERID_REMOVE';
DELETE FROM search_queries WHERE "userId" = '$USERID_REMOVE';
DELETE FROM shares WHERE "userId" = '$USERID_REMOVE';
DELETE FROM notification_settings WHERE "userId" = '$USERID_REMOVE';
DELETE FROM events WHERE "actorId" = '$USERID_REMOVE';
DELETE FROM events WHERE "userId" = '$USERID_REMOVE';
DELETE FROM users WHERE username = '$1';
SQL
}

View File

@ -1,22 +0,0 @@
---
version: "3.8"
services:
app:
secrets:
- aws_secret_key
environment:
- AWS_ACCESS_KEY_ID
- AWS_REGION
- AWS_S3_ACL
- AWS_S3_FORCE_PATH_STYLE
- AWS_S3_UPLOAD_BUCKET_NAME
- AWS_S3_UPLOAD_BUCKET_URL
- AWS_S3_UPLOAD_MAX_SIZE
- AWS_SDK_LOAD_CONFIG=0
- AWS_SECRET_KEY_FILE=/run/secrets/aws_secret_key
secrets:
aws_secret_key:
name: ${STACK_NAME}_aws_secret_key_${SECRET_AWS_SECRET_KEY_VERSION}
external: true

View File

@ -1,13 +0,0 @@
---
version: "3.8"
services:
app:
volumes:
- storage-data:/var/lib/outline/data
environment:
- FILE_STORAGE
- FILE_STORAGE_UPLOAD_MAX_SIZE
volumes:
storage-data:

View File

@ -1,18 +0,0 @@
version: "3.8"
services:
app:
secrets:
- smtp_password
environment:
- SMTP_HOST
- SMTP_PORT
- SMTP_USERNAME
- SMTP_FROM_EMAIL
- SMTP_REPLY_EMAIL
- SMTP_TLS_CIPHERS
- SMTP_SECURE
secrets:
smtp_password:
external: true
name: ${STACK_NAME}_smtp_password_${SECRET_SMTP_PASSWORD_VERSION}

View File

@ -6,8 +6,9 @@ services:
networks:
- backend
- proxy
image: outlinewiki/outline:0.75.2
image: outlinewiki/outline:0.64.4
secrets:
- aws_secret_key
- db_password
- secret_key
- utils_secret
@ -16,7 +17,15 @@ services:
target: /docker-entrypoint.sh
mode: 0555
environment:
- FILE_STORAGE
- AWS_ACCESS_KEY_ID
- AWS_REGION
- AWS_S3_ACL
- AWS_S3_FORCE_PATH_STYLE
- AWS_S3_UPLOAD_BUCKET_NAME
- AWS_S3_UPLOAD_BUCKET_URL
- AWS_S3_UPLOAD_MAX_SIZE
- AWS_SDK_LOAD_CONFIG=0
- AWS_SECRET_KEY_FILE=/run/secrets/aws_secret_key
- DATABASE_PASSWORD_FILE=/run/secrets/db_password
- FORCE_HTTPS=true
- PGSSLMODE=disable
@ -34,40 +43,29 @@ services:
- "traefik.http.routers.${STACK_NAME}.rule=Host(`${DOMAIN}`${EXTRA_DOMAINS})"
- "traefik.http.routers.${STACK_NAME}.entrypoints=web-secure"
- "traefik.http.routers.${STACK_NAME}.tls.certresolver=${LETS_ENCRYPT_ENV}"
- "coop-cloud.${STACK_NAME}.version=2.2.0+0.75.2"
- "coop-cloud.${STACK_NAME}.version=0.4.1+0.64.4"
## Redirect from EXTRA_DOMAINS to DOMAIN
#- "traefik.http.routers.${STACK_NAME}.middlewares=${STACK_NAME}-redirect"
#- "traefik.http.middlewares.${STACK_NAME}-redirect.headers.SSLForceHost=true"
#- "traefik.http.middlewares.${STACK_NAME}-redirect.headers.SSLHost=${DOMAIN}"
cache:
image: redis:7.2.4
image: redis:6.2.6
networks:
- backend
db:
image: postgres:16.2
image: postgres:11
networks:
- backend
secrets:
- db_password
configs:
- source: db_entrypoint
target: /docker-entrypoint.sh
mode: 0555
environment:
POSTGRES_DB: outline
POSTGRES_PASSWORD_FILE: /run/secrets/db_password
POSTGRES_USER: outline
volumes:
- "postgres_data:/var/lib/postgresql/data"
entrypoint: /docker-entrypoint.sh
deploy:
labels:
backupbot.backup: "true"
backupbot.backup.path: "/var/lib/postgresql/data/dump.sql.gz"
backupbot.backup.post-hook: "rm -f /var/lib/postgresql/data/dump.sql.gz"
backupbot.backup.pre-hook: "sh -c 'PGPASSWORD=$$(cat $${POSTGRES_PASSWORD_FILE}) pg_dump -U outline outline | gzip > /var/lib/postgresql/data/dump.sql.gz'"
secrets:
secret_key:
@ -76,6 +74,9 @@ secrets:
utils_secret:
name: ${STACK_NAME}_utils_secret_${SECRET_UTILS_SECRET_VERSION}
external: true
aws_secret_key:
name: ${STACK_NAME}_aws_secret_key_${SECRET_AWS_SECRET_KEY_VERSION}
external: true
db_password:
name: ${STACK_NAME}_db_password_${SECRET_DB_PASSWORD_VERSION}
external: true
@ -93,7 +94,3 @@ configs:
name: ${STACK_NAME}_app_entrypoint_${APP_ENTRYPOINT_VERSION}
file: entrypoint.sh.tmpl
template_driver: golang
db_entrypoint:
name: ${STACK_NAME}_db_entrypoint_${DB_ENTRYPOINT_VERSION}
file: entrypoint.postgres.sh.tmpl
template_driver: golang

View File

@ -1,44 +0,0 @@
#!/bin/bash
set -e
MIGRATION_MARKER=$PGDATA/migration_in_progress
OLDDATA=$PGDATA/old_data
NEWDATA=$PGDATA/new_data
if [ -e $MIGRATION_MARKER ]; then
echo "FATAL: migration was started but did not complete in a previous run. manual recovery necessary"
exit 1
fi
if [ -f $PGDATA/PG_VERSION ]; then
DATA_VERSION=$(cat $PGDATA/PG_VERSION)
if [ -n "$DATA_VERSION" -a "$PG_MAJOR" != "$DATA_VERSION" ]; then
echo "postgres data version $DATA_VERSION found, but need $PG_MAJOR. Starting migration"
echo "Installing postgres $DATA_VERSION"
sed -i "s/$/ $DATA_VERSION/" /etc/apt/sources.list.d/pgdg.list
apt-get update && apt-get install -y --no-install-recommends \
postgresql-$DATA_VERSION \
&& rm -rf /var/lib/apt/lists/*
echo "shuffling around"
gosu postgres mkdir $OLDDATA $NEWDATA
chmod 700 $OLDDATA $NEWDATA
mv $PGDATA/* $OLDDATA/ || true
touch $MIGRATION_MARKER
echo "running initdb"
# abuse entrypoint script for initdb by making server error out
gosu postgres bash -c "export PGDATA=$NEWDATA ; /usr/local/bin/docker-entrypoint.sh --invalid-arg || true"
echo "running pg_upgrade"
cd /tmp
gosu postgres pg_upgrade --link -b /usr/lib/postgresql/$DATA_VERSION/bin -d $OLDDATA -D $NEWDATA -U $POSTGRES_USER
cp $OLDDATA/pg_hba.conf $NEWDATA/
mv $NEWDATA/* $PGDATA
rm -rf $OLDDATA
rmdir $NEWDATA
rm $MIGRATION_MARKER
echo "migration complete"
fi
fi
/usr/local/bin/docker-entrypoint.sh postgres

View File

@ -1,12 +1,6 @@
#!/bin/sh
{{ if eq (env "FILE_STORAGE") "s3" }}
export AWS_SECRET_ACCESS_KEY=$(cat /run/secrets/aws_secret_key)
{{ end }}
{{ if eq (env "SMTP_ENABLED") "1" }}
export SMTP_PASSWORD=$(cat /run/secrets/smtp_password)
{{ end }}
{{ if eq (env "OIDC_ENABLED") "1" }}
export OIDC_CLIENT_SECRET=$(cat /run/secrets/oidc_client_secret)
@ -21,7 +15,4 @@ export SECRET_KEY=$(cat /run/secrets/secret_key)
export DATABASE_PASSWORD=$(cat /run/secrets/db_password)
export DATABASE_URL="postgres://outline:${DATABASE_PASSWORD}@${STACK_NAME}_db:5432/outline"
if [ ! "$1" = "-e" ]; then
/usr/local/bin/yarn db:migrate --env=production-ssl-disabled
/usr/local/bin/yarn start "$@"
fi
/usr/local/bin/yarn start "$@"

View File

@ -1,4 +0,0 @@
Due to the introduction of local storage, you need to adapt your config to continue using S3 storage. Just add the following lines to your config:
FILE_STORAGE=s3
COMPOSE_FILE="$COMPOSE_FILE:compose.aws.yml"