WIP: Upgrade to drive v0.25.0, replace MinIO with RustFS #26

Draft
ineiti wants to merge 1 commits from update_v0.25 into main
11 changed files with 353 additions and 107 deletions
+31 -5
View File
@@ -1,7 +1,8 @@
TYPE=lasuite-drive
DOMAIN=lasuite-drive.example.com
MINIO_DOMAIN="minio.${DOMAIN}"
# public domain of the RustFS object storage (S3)
RUSTFS_DOMAIN="rustfs.${DOMAIN}"
COLLABORA_DOMAIN="collabora.${DOMAIN}"
ONLY_OFFICE_DOMAIN="onlyoffice.${DOMAIN}"
@@ -20,10 +21,10 @@ SECRET_DJANGO_SK_VERSION=v1
SECRET_OIDC_RPCS_VERSION=v1
# DJANGO_SUPERUSER_PASSWORD
SECRET_DJANGO_SP_VERSION=v1
# MINIO_ROOT_PASSWORD
SECRET_MINIO_RP_VERSION=v1
# MINIO_ROOT_USER
SECRET_MINIO_RU_VERSION=v1
# RUSTFS_SECRET_KEY
SECRET_RUSTFS_RP_VERSION=v1
# RUSTFS_ACCESS_KEY
SECRET_RUSTFS_RU_VERSION=v1
# POSTGRES_PASSWORD
SECRET_POSTGRES_P_VERSION=v1
# DJANGO_HOST_EMAIL_PASSWORD
@@ -87,6 +88,31 @@ LOGGING_LEVEL_LOGGERS_APP=INFO
# Set to false to disable backup-bot labels (default: true)
#ENABLE_BACKUPS=true
##############################################################################
# MAINTENANCE SCHEDULING
##############################################################################
# Celery Beat crontab (m h dom mon dow) followed by the command arguments
# Clean up uploads that never finished (default: daily at 0:30)
#CLEAN_PENDING_ITEMS_SCHEDULE="30 0 * * * --hours=48"
# Permanently purge items deleted from the trash (default: daily at 0:45)
#PURGE_DELETED_ITEMS_SCHEDULE="45 0 * * *"
##############################################################################
# SDK
##############################################################################
# Enable the SDK picker pages (/sdk) and relay API used by other La Suite apps
#SDK_ENABLED=false
##############################################################################
# MINIO MIGRATION
##############################################################################
# Only needed once when upgrading from recipe versions using MinIO (<= 0.13.x),
# see README "Migrating from MinIO"
#COMPOSE_FILE="compose.yml:compose.minio-migration.yml"
# versions of the old MinIO secrets, keep them from your previous config
#SECRET_MINIO_RP_VERSION=v1
#SECRET_MINIO_RU_VERSION=v1
##############################################################################
# WOPI SCHEDULING
##############################################################################
+31 -3
View File
@@ -19,25 +19,53 @@
### Setting Up Domains
This recipe requires four domains. One domain for drive, and one for minio which also needs its own public endpoint (see [https://github.com/suitenumerique/drive/issues/476](https://github.com/suitenumerique/drive/issues/476)), one domain for collabora, and one domain for onlyoffice. For example (`drive.yourdomain.tld`, `minio.drive.yourdomain.tld`, `collabora.drive.yourdomain.tld`, `onlyoffice.drive.yourdomain.tld`). All of these domains need to have an A record pointing the IP address of your server.
This recipe requires four domains. One domain for drive, one for the S3 object storage ([RustFS](https://rustfs.com)) which also needs its own public endpoint (see [https://github.com/suitenumerique/drive/issues/476](https://github.com/suitenumerique/drive/issues/476)), one domain for collabora, and one domain for onlyoffice. For example (`drive.yourdomain.tld`, `rustfs.drive.yourdomain.tld`, `collabora.drive.yourdomain.tld`, `onlyoffice.drive.yourdomain.tld`). All of these domains need to have an A record pointing the IP address of your server.
### Installation Steps
* `abra app new lasuite-drive --secrets`
* `abra app config <app-name>`
- make sure to set MINIO_DOMAIN, COLLABORA_DOMAIN, ONLY_OFFICE_DOMAIN to the domains you set up for each.
- make sure to set RUSTFS_DOMAIN, COLLABORA_DOMAIN, ONLY_OFFICE_DOMAIN to the domains you set up for each.
* `abra app deploy <app-name>`
You should then be able to visit the landing page of your app, but not yet to login. To login, you need to deploy and integrate single sign on (described below in the "Configure Authentication" section).
* Migrations run automatically on backend startup. To trigger manually: `abra app cmd <app-name> backend migrate`
* Minio buckets are created automatically on first deploy. To manually trigger: `abra app cmd <app-name> minio minio_initialize`
* The S3 bucket is created automatically on backend startup. To manually trigger: `abra app cmd <app-name> backend s3_initialize`
Wopi discovery is supposed to happen automatically, but if collabora/onlyoffice are not connecting, you can try running:
* `abra app cmd <app-name> backend trigger_wopi` # connects only office & collabora (if they stop working, try running this again)
Maintenance tasks (cleaning unfinished uploads, purging the trash) are scheduled by celery beat, see `CLEAN_PENDING_ITEMS_SCHEDULE` and `PURGE_DELETED_ITEMS_SCHEDULE` in `.env.sample`.
## Migrating from MinIO
Since upstream drive v0.24, and in this recipe since the version packaging drive v0.25, the S3 object storage is [RustFS](https://rustfs.com) instead of MinIO. RustFS cannot read the MinIO data directory, so the media files have to be copied once. RustFS uses new secrets and a new domain variable.
1. `abra app config <app-name>` and add:
```
RUSTFS_DOMAIN="${MINIO_DOMAIN}"
SECRET_RUSTFS_RP_VERSION=v1
SECRET_RUSTFS_RU_VERSION=v1
COMPOSE_FILE="compose.yml:compose.minio-migration.yml"
```
Keep `MINIO_DOMAIN`, `SECRET_MINIO_RP_VERSION` and `SECRET_MINIO_RU_VERSION`, they are still needed during the migration. The `COMPOSE_FILE` line temporarily runs the old MinIO service on the old `minio` volume, next to RustFS.
2. Generate the RustFS credentials:
```
abra app secret generate <app-name> rustfs_ru v1
abra app secret generate <app-name> rustfs_rp v1
```
3. `abra app upgrade <app-name>` (or `abra app deploy <app-name> --force`)
4. `abra app cmd <app-name> backend migrate_from_minio`
This copies all objects (latest version only) from MinIO to RustFS. Objects that already exist on RustFS are never overwritten, so if the copy is interrupted, just run it again. Once a run has completed, further runs are refused: they would copy back files that users deleted in the meantime. If you really need to, run `abra app cmd <app-name> backend migrate_from_minio -- --force`.
5. Check that your files can be opened in drive, then remove the `COMPOSE_FILE` line from the config and redeploy: `abra app deploy <app-name> --force`. If you like, replace `RUSTFS_DOMAIN="${MINIO_DOMAIN}"` with the actual domain and remove `MINIO_DOMAIN`, `SECRET_MINIO_RP_VERSION` and `SECRET_MINIO_RU_VERSION`.
6. Once you are sure everything is there, you can remove the old volume and secrets on the server: `docker volume rm <stack-name>_minio`, `abra app secret rm <app-name> minio_ru` and `abra app secret rm <app-name> minio_rp`
Until step 4 is done, files uploaded before the upgrade will not be accessible.
## Configure Authentication
lasuite-drive **requires** an OpenID Connect (OIDC) single sign-on provider; deployment has been tested with [Keycloak](https://git.coopcloud.tech/coop-cloud/keycloak), which we recommend, or you could also try [Authentik](https://git.coopcloud.tech/coop-cloud/authentik), both of which are installable using Co-op Cloud.
+2 -2
View File
@@ -3,8 +3,8 @@ set -e
[ -f /run/secrets/postgres_p ] && export DB_PASSWORD="$(cat /run/secrets/postgres_p)"
[ -f /run/secrets/django_sk ] && export DJANGO_SECRET_KEY="$(cat /run/secrets/django_sk)"
[ -f /run/secrets/minio_rp ] && export AWS_S3_SECRET_ACCESS_KEY="$(cat /run/secrets/minio_rp)"
[ -f /run/secrets/minio_ru ] && export AWS_S3_ACCESS_KEY_ID="$(cat /run/secrets/minio_ru)"
[ -f /run/secrets/rustfs_rp ] && export AWS_S3_SECRET_ACCESS_KEY="$(cat /run/secrets/rustfs_rp)"
[ -f /run/secrets/rustfs_ru ] && export AWS_S3_ACCESS_KEY_ID="$(cat /run/secrets/rustfs_ru)"
[ -f /run/secrets/django_sp ] && export DJANGO_SUPERUSER_PASSWORD="$(cat /run/secrets/django_sp)"
[ -f /run/secrets/oidc_rpcs ] && export OIDC_RP_CLIENT_SECRET="$(cat /run/secrets/oidc_rpcs)"
[ -f /run/secrets/email_pass ] && export DJANGO_EMAIL_HOST_PASSWORD="$(cat /run/secrets/email_pass)"
+11 -5
View File
@@ -1,11 +1,12 @@
# Set any config versions here
# Docs: https://docs.coopcloud.tech/maintainers/handbook/#manage-configs
export ABRA_ENTRYPOINT_VERSION=v11
export NGINX_CONF_VERSION=v6
export ABRA_ENTRYPOINT_VERSION=v12
export NGINX_CONF_VERSION=v7
export ONLYOFFICE_CONF_VERSION=v2
export PG_BACKUP_VERSION=v4
export MIGRATE_VERSION=v1
export MINIO_INITIALIZE_VERSION=v2
export S3_INITIALIZE_VERSION=v1
export MIGRATE_FROM_MINIO_VERSION=v1
environment() {
# this exports all the secrets as environment variables
@@ -16,8 +17,13 @@ migrate() {
/migrate.sh
}
minio_initialize() {
/minio-initialize.sh
s3_initialize() {
/s3-initialize.sh
}
# only available when deployed with compose.minio-migration.yml
migrate_from_minio() {
/migrate-from-minio.sh "$@"
}
trigger_wopi() {
+47
View File
@@ -0,0 +1,47 @@
---
# Temporary overlay to migrate the media files from the old MinIO service
# (recipe versions up to 0.13.x) to RustFS. See README "Migrating from MinIO".
services:
minio:
image: bitnamilegacy/minio:2025.7.23-debian-12-r5
user: root
environment:
MINIO_ROOT_USER_FILE: /run/secrets/minio_ru
MINIO_ROOT_PASSWORD_FILE: /run/secrets/minio_rp
healthcheck:
test: ["CMD", "mc", "ready", "local"]
interval: 5s
timeout: 20s
retries: 60
networks:
- backend
volumes:
- minio:/bitnami/minio/data
secrets:
- minio_rp
- minio_ru
backend:
secrets:
- minio_rp
- minio_ru
configs:
- source: migrate_from_minio
target: /migrate-from-minio.sh
mode: 0555
volumes:
minio:
configs:
migrate_from_minio:
name: ${STACK_NAME}_migrate_from_minio_${MIGRATE_FROM_MINIO_VERSION}
file: migrate-from-minio.sh
secrets:
minio_rp:
external: true
name: ${STACK_NAME}_minio_rp_${SECRET_MINIO_RP_VERSION}
minio_ru:
external: true
name: ${STACK_NAME}_minio_ru_${SECRET_MINIO_RU_VERSION}
+53 -52
View File
@@ -34,10 +34,10 @@ x-common-env: &common-env
DRIVE_BASE_URL: "https://${DOMAIN}"
# Media
STORAGES_STATICFILES_BACKEND: django.contrib.staticfiles.storage.StaticFilesStorage
AWS_S3_ENDPOINT_URL: http://minio:9000
AWS_S3_DOMAIN_REPLACE: https://${MINIO_DOMAIN}
# AWS_S3_ACCESS_KEY_ID supplied via secret (this is same MINIO_ROOT_USER)
# AWS_S3_SECRET_ACCESS_KEY supplied via secret (this is same as MINIO_ROOT_PASSWORD)
AWS_S3_ENDPOINT_URL: http://rustfs:9000
AWS_S3_DOMAIN_REPLACE: https://${RUSTFS_DOMAIN}
# AWS_S3_ACCESS_KEY_ID supplied via secret (this is same as RUSTFS_ACCESS_KEY)
# AWS_S3_SECRET_ACCESS_KEY supplied via secret (this is same as RUSTFS_SECRET_KEY)
MEDIA_BASE_URL: https://${DOMAIN}
AWS_STORAGE_BUCKET_NAME: drive-media-storage
# OIDC - settings from .env, see .env.sample
@@ -63,6 +63,8 @@ x-common-env: &common-env
AI_MODEL: llama
# Collaboration
COLLABORATION_API_URL: https://$DOMAIN/collaboration/api/
# SDK picker pages (/sdk) and SDK relay API, used by other La Suite apps
SDK_ENABLED: ${SDK_ENABLED:-false}
# WOPI
WOPI_CLIENTS: "collabora,onlyoffice"
@@ -87,9 +89,10 @@ x-postgres-env: &postgres-env
DB_PORT: 5432
# DB_PASSWORD supplied via secrets (this is same as POSTGRES_PASSWORD)
x-minio-env: &minio-env
MINIO_ROOT_USER_FILE: /run/secrets/minio_ru
MINIO_ROOT_PASSWORD_FILE: /run/secrets/minio_rp
x-rustfs-env: &rustfs-env
RUSTFS_ACCESS_KEY_FILE: /run/secrets/rustfs_ru
RUSTFS_SECRET_KEY_FILE: /run/secrets/rustfs_rp
RUSTFS_CONSOLE_ENABLE: "false"
services:
@@ -119,7 +122,8 @@ services:
image: lasuite/drive-backend:v0.25.0
command: [ "gunicorn", "-c", "/usr/local/etc/gunicorn/drive.py", "drive.wsgi:application" ]
entrypoint: >
sh -c "if [ \"$$AUTO_MIGRATIONS\" = \"true\" ]; then /migrate.sh; fi && exec /abra-entrypoint.sh /usr/local/bin/entrypoint \"$$@\"" --
sh -c "/s3-initialize.sh || echo '⚠ s3-initialize failed, run: abra app cmd <app> backend s3_initialize' ;
if [ \"$$AUTO_MIGRATIONS\" = \"true\" ]; then /migrate.sh; fi && exec /abra-entrypoint.sh /usr/local/bin/entrypoint \"$$@\"" --
environment:
<<: [ *common-env, *postgres-env ]
AUTO_MIGRATIONS: "${AUTO_MIGRATIONS:-true}"
@@ -134,12 +138,15 @@ services:
- source: migrate
target: /migrate.sh
mode: 0555
- source: s3_initialize
target: /s3-initialize.sh
mode: 0555
secrets:
- django_sk
- django_sp
- oidc_rpcs
- minio_rp
- minio_ru
- rustfs_rp
- rustfs_ru
- postgres_p
- email_pass
@@ -161,7 +168,8 @@ services:
- django_sk
- django_sp
- oidc_rpcs
- minio_rp
- rustfs_rp
- rustfs_ru
- postgres_p
- email_pass
@@ -174,6 +182,9 @@ services:
entrypoint: ["/abra-entrypoint.sh", "/usr/local/bin/entrypoint"]
environment:
<<: [*common-env, *postgres-env]
# Maintenance commands scheduled by celery beat: crontab expression followed by command arguments
CLEAN_PENDING_ITEMS_SCHEDULE: ${CLEAN_PENDING_ITEMS_SCHEDULE:-30 0 * * *}
PURGE_DELETED_ITEMS_SCHEDULE: ${PURGE_DELETED_ITEMS_SCHEDULE:-45 0 * * *}
configs:
- source: abra_entrypoint
target: /abra-entrypoint.sh
@@ -182,7 +193,8 @@ services:
- django_sk
- django_sp
- oidc_rpcs
- minio_rp
- rustfs_rp
- rustfs_ru
- postgres_p
- email_pass
@@ -228,49 +240,38 @@ services:
networks:
- backend
minio:
image: bitnamilegacy/minio:2025.7.23-debian-12-r5
user: root
environment: *minio-env
rustfs:
image: rustfs/rustfs:1.0.3
environment: *rustfs-env
healthcheck:
test: ["CMD", "mc", "ready", "local"]
interval: 1s
test: ["CMD", "curl", "-sf", "http://localhost:9000/health"]
interval: 5s
timeout: 20s
retries: 300
retries: 60
networks:
- backend
- proxy
command: /opt/bitnami/scripts/minio/run.sh
entrypoint: >
sh -c "/minio-initialize.sh & exec /opt/bitnami/scripts/minio/entrypoint.sh \"$$@\"" --
volumes:
- minio:/bitnami/minio/data
configs:
- source: abra_entrypoint
target: /abra-entrypoint.sh
mode: 0555
- source: minio_initialize
target: /minio-initialize.sh
mode: 0555
- rustfs:/data
secrets:
- minio_rp
- minio_ru
- rustfs_rp
- rustfs_ru
deploy:
labels:
- "traefik.enable=true"
- "traefik.docker.network=proxy"
- "traefik.http.routers.${STACK_NAME}_minio.rule=Host(`${MINIO_DOMAIN}`)"
- "traefik.http.routers.${STACK_NAME}_minio.entrypoints=web-secure"
- "traefik.http.routers.${STACK_NAME}_minio.tls=true"
- "traefik.http.routers.${STACK_NAME}_minio.tls.certresolver=${LETS_ENCRYPT_ENV}"
- "traefik.http.services.${STACK_NAME}_minio.loadbalancer.server.port=9000"
- "traefik.http.middlewares.${STACK_NAME}_minio-cors.headers.accessControlAllowOriginList=https://${DOMAIN}"
- "traefik.http.middlewares.${STACK_NAME}_minio-cors.headers.accessControlAllowMethods=GET,POST,PUT,DELETE,OPTIONS"
- "traefik.http.middlewares.${STACK_NAME}_minio-cors.headers.accessControlAllowHeaders=*"
- "traefik.http.middlewares.${STACK_NAME}_minio-cors.headers.accessControlExposeHeaders=ETag,Content-Length"
- "traefik.http.middlewares.${STACK_NAME}_minio-cors.headers.accessControlMaxAge=600"
- "traefik.http.middlewares.${STACK_NAME}_minio-cors.headers.addVaryHeader=true"
- "traefik.http.routers.${STACK_NAME}_minio.middlewares=${STACK_NAME}_minio-cors"
- "traefik.http.routers.${STACK_NAME}_rustfs.rule=Host(`${RUSTFS_DOMAIN}`)"
- "traefik.http.routers.${STACK_NAME}_rustfs.entrypoints=web-secure"
- "traefik.http.routers.${STACK_NAME}_rustfs.tls=true"
- "traefik.http.routers.${STACK_NAME}_rustfs.tls.certresolver=${LETS_ENCRYPT_ENV}"
- "traefik.http.services.${STACK_NAME}_rustfs.loadbalancer.server.port=9000"
- "traefik.http.middlewares.${STACK_NAME}_rustfs-cors.headers.accessControlAllowOriginList=https://${DOMAIN}"
- "traefik.http.middlewares.${STACK_NAME}_rustfs-cors.headers.accessControlAllowMethods=GET,POST,PUT,DELETE,OPTIONS"
- "traefik.http.middlewares.${STACK_NAME}_rustfs-cors.headers.accessControlAllowHeaders=*"
- "traefik.http.middlewares.${STACK_NAME}_rustfs-cors.headers.accessControlExposeHeaders=ETag,Content-Length"
- "traefik.http.middlewares.${STACK_NAME}_rustfs-cors.headers.accessControlMaxAge=600"
- "traefik.http.middlewares.${STACK_NAME}_rustfs-cors.headers.addVaryHeader=true"
- "traefik.http.routers.${STACK_NAME}_rustfs.middlewares=${STACK_NAME}_rustfs-cors"
- "backupbot.backup=${ENABLE_BACKUPS:-true}"
collabora:
@@ -380,7 +381,7 @@ networks:
volumes:
postgres:
minio:
rustfs:
configs:
nginx_conf:
@@ -396,9 +397,9 @@ configs:
migrate:
name: ${STACK_NAME}_migrate_${MIGRATE_VERSION}
file: migrate.sh
minio_initialize:
name: ${STACK_NAME}_minio_initialize_${MINIO_INITIALIZE_VERSION}
file: minio-initialize.sh
s3_initialize:
name: ${STACK_NAME}_s3_initialize_${S3_INITIALIZE_VERSION}
file: s3-initialize.sh
onlyoffice_conf:
name: ${STACK_NAME}_onlyoffice_conf_${ONLYOFFICE_CONF_VERSION}
file: onlyoffice-config.json.tmpl
@@ -417,12 +418,12 @@ secrets:
postgres_p:
external: true
name: ${STACK_NAME}_postgres_p_${SECRET_POSTGRES_P_VERSION}
minio_rp:
rustfs_rp:
external: true
name: ${STACK_NAME}_minio_rp_${SECRET_MINIO_RP_VERSION}
minio_ru:
name: ${STACK_NAME}_rustfs_rp_${SECRET_RUSTFS_RP_VERSION}
rustfs_ru:
external: true
name: ${STACK_NAME}_minio_ru_${SECRET_MINIO_RU_VERSION}
name: ${STACK_NAME}_rustfs_ru_${SECRET_RUSTFS_RU_VERSION}
collabora_p:
external: true
name: ${STACK_NAME}_collabora_p_${SECRET_COLLABORA_P_VERSION}
+96
View File
@@ -0,0 +1,96 @@
#!/bin/sh
set -e
# One-shot copy of all objects from the old MinIO service (deployed with
# compose.minio-migration.yml) into the RustFS object storage.
# Only the latest version of each object is copied, and only objects missing
# on RustFS: objects changed on RustFS since (e.g. edited documents) are never
# overwritten. Once a run completes, a marker object is written and further
# runs are refused, unless called with --force.
# Load secrets into environment
source /abra-entrypoint.sh -e
python - "$@" <<'PYEOF'
import os
import sys
import boto3
from botocore.exceptions import ClientError
bucket = os.environ.get("AWS_STORAGE_BUCKET_NAME", "drive-media-storage")
marker = "_minio-migration-done"
force = "--force" in sys.argv[1:]
def secret(name):
with open(f"/run/secrets/{name}") as f:
return f.read().strip()
def client(endpoint, access_key, secret_key):
return boto3.client(
"s3",
endpoint_url=endpoint,
aws_access_key_id=access_key,
aws_secret_access_key=secret_key,
region_name=os.environ.get("AWS_S3_REGION_NAME", "us-east-1"),
)
def exists(s3, key):
try:
s3.head_object(Bucket=bucket, Key=key)
return True
except ClientError as error:
if error.response["Error"]["Code"] in ("404", "NoSuchKey", "NotFound"):
return False
raise
# the old MinIO uses the minio_* secrets, RustFS the rustfs_* secrets
src = client(
os.environ.get("MINIO_MIGRATION_ENDPOINT_URL", "http://minio:9000"),
secret("minio_ru"),
secret("minio_rp"),
)
dst = client(
os.environ["AWS_S3_ENDPOINT_URL"],
os.environ["AWS_S3_ACCESS_KEY_ID"],
os.environ["AWS_S3_SECRET_ACCESS_KEY"],
)
if exists(dst, marker) and not force:
sys.exit(
"migrate-from-minio: the migration already completed, refusing to run again.\n"
"Objects deleted on RustFS since then would be copied back from MinIO.\n"
"Use 'migrate_from_minio --force' to run anyway (existing objects are never overwritten)."
)
try:
src.head_bucket(Bucket=bucket)
except Exception as error:
sys.exit(f"migrate-from-minio: cannot reach bucket '{bucket}' on minio: {error}")
copied = skipped = 0
for page in src.get_paginator("list_objects_v2").paginate(Bucket=bucket):
for obj in page.get("Contents", []):
key = obj["Key"]
# upload_fileobj is atomic: an object only exists once fully copied
if exists(dst, key):
skipped += 1
continue
source = src.get_object(Bucket=bucket, Key=key)
extra = {"ContentType": source.get("ContentType", "binary/octet-stream")}
if source.get("Metadata"):
extra["Metadata"] = source["Metadata"]
if source.get("ContentDisposition"):
extra["ContentDisposition"] = source["ContentDisposition"]
dst.upload_fileobj(source["Body"], bucket, key, ExtraArgs=extra)
copied += 1
if copied % 100 == 0:
print(f"migrate-from-minio: {copied} objects copied...")
dst.put_object(Bucket=bucket, Key=marker, Body=b"")
print(f"migrate-from-minio: done, {copied} copied, {skipped} already present")
PYEOF
-34
View File
@@ -1,34 +0,0 @@
#!/bin/sh
set -e
# Use an isolated mc config dir: the bundled Bitnami image runs its own
# internal mc client (as the "minio" user) against the default /.mc path,
# which collides with this script running as root against the same path.
export MC_CONFIG_DIR=/tmp/mc-drive-init
# Wait for minio to be ready (up to 60 seconds)
i=0
while ! mc ready local 2>/dev/null; do
i=$((i+1))
if [ "$i" -ge 60 ]; then
echo "minio-initialize: timed out waiting for minio to be ready" >&2
exit 1
fi
sleep 1
done
MINIO_ROOT_USER="$(cat /run/secrets/minio_ru)"
MINIO_ROOT_PASSWORD="$(cat /run/secrets/minio_rp)"
mc alias set drive http://localhost:9000 "${MINIO_ROOT_USER}" "${MINIO_ROOT_PASSWORD}"
# Idempotent: skip if bucket already exists
if mc ls drive/drive-media-storage > /dev/null 2>&1; then
echo "minio-initialize: bucket 'drive-media-storage' already exists, skipping"
exit 0
fi
echo "minio-initialize: creating bucket 'drive-media-storage'..."
mc mb drive/drive-media-storage
mc version enable drive/drive-media-storage
echo "minio-initialize: done"
+6 -6
View File
@@ -56,9 +56,9 @@ server {
proxy_set_header X-Amz-Date $authDate;
proxy_set_header X-Amz-Content-SHA256 $authContentSha256;
# Get resource from Minio
proxy_pass http://minio:9000/drive-media-storage/;
proxy_set_header Host minio:9000;
# Get resource from the RustFS object storage
proxy_pass http://rustfs:9000/drive-media-storage/;
proxy_set_header Host rustfs:9000;
}
# Proxy auth for media-preview
@@ -74,9 +74,9 @@ server {
proxy_set_header X-Amz-Date $authDate;
proxy_set_header X-Amz-Content-SHA256 $authContentSha256;
# Get resource from Minio
proxy_pass http://minio:9000/drive-media-storage/;
proxy_set_header Host minio:9000;
# Get resource from the RustFS object storage
proxy_pass http://rustfs:9000/drive-media-storage/;
proxy_set_header Host rustfs:9000;
}
location /media-auth {
+25
View File
@@ -0,0 +1,25 @@
Upgrade to drive v0.25.0.
**Breaking change:** following upstream, the S3 object storage changed from MinIO (`bitnamilegacy/minio`, unmaintained) to RustFS (`rustfs/rustfs`). RustFS cannot read the MinIO data, so existing media files must be copied once:
```
abra app config <app-domain> # add:
# RUSTFS_DOMAIN="${MINIO_DOMAIN}"
# SECRET_RUSTFS_RP_VERSION=v1
# SECRET_RUSTFS_RU_VERSION=v1
# COMPOSE_FILE="compose.yml:compose.minio-migration.yml"
abra app secret generate <app-domain> rustfs_ru v1
abra app secret generate <app-domain> rustfs_rp v1
abra app upgrade <app-domain>
abra app cmd <app-domain> backend migrate_from_minio
abra app config <app-domain> # remove the COMPOSE_FILE line
abra app deploy <app-domain> --force
```
See README "Migrating from MinIO" for details. RustFS uses the new `rustfs_ru`/`rustfs_rp` secrets and `RUSTFS_DOMAIN`. The old `minio_ru`/`minio_rp` secrets and `MINIO_DOMAIN` are only needed during the migration.
Other changes:
- The S3 bucket is now created by the backend on startup (`abra app cmd <app-domain> backend s3_initialize`), the `minio_initialize` command is gone.
- Maintenance commands `clean_pending_items` (daily 0:30) and `purge_deleted_items` (daily 0:45) are now scheduled by celery beat. Configure with `CLEAN_PENDING_ITEMS_SCHEDULE` / `PURGE_DELETED_ITEMS_SCHEDULE`. Note that this permanently purges items deleted from the trash after the upstream grace periods (30 + 7 days).
- New `SDK_ENABLED` setting (default `false`) for the SDK picker used by other La Suite apps.
- The celery workers now get the S3 access key secret too.
+51
View File
@@ -0,0 +1,51 @@
#!/bin/sh
set -e
# Creates the media bucket on the RustFS object storage, with versioning
# enabled. Runs inside the backend container, which ships boto3.
# Load secrets into environment
source /abra-entrypoint.sh -e
python - <<'PYEOF'
import os
import sys
import time
import boto3
from botocore.exceptions import ClientError, EndpointConnectionError
bucket = os.environ.get("AWS_STORAGE_BUCKET_NAME", "drive-media-storage")
s3 = boto3.client(
"s3",
endpoint_url=os.environ["AWS_S3_ENDPOINT_URL"],
aws_access_key_id=os.environ["AWS_S3_ACCESS_KEY_ID"],
aws_secret_access_key=os.environ["AWS_S3_SECRET_ACCESS_KEY"],
region_name=os.environ.get("AWS_S3_REGION_NAME", "us-east-1"),
)
# Wait for the object storage to be ready (up to 60 seconds)
for i in range(60):
try:
s3.list_buckets()
break
except (ClientError, EndpointConnectionError):
time.sleep(1)
else:
sys.exit("s3-initialize: timed out waiting for the object storage")
# Idempotent: skip if bucket already exists
try:
s3.head_bucket(Bucket=bucket)
print(f"s3-initialize: bucket '{bucket}' already exists, skipping")
sys.exit(0)
except ClientError:
pass
print(f"s3-initialize: creating bucket '{bucket}'...")
s3.create_bucket(Bucket=bucket)
s3.put_bucket_versioning(
Bucket=bucket, VersioningConfiguration={"Status": "Enabled"}
)
print("s3-initialize: done")
PYEOF