fix: strip monitoring. prefix from instance labels for cleaner dashboard views

fix: refresh dashboard variables on time range change, not just page load
fix: mount containerd socket read-only, avoid abra#900
2026-08-20 16:50:25 +02:00 · 2026-08-20 14:07:10 +02:00 · 2026-08-17 16:09:44 +02:00 · 2026-08-17 15:58:04 +02:00 · 2026-08-17 15:08:08 +02:00 · 2026-08-17 14:59:03 +02:00
29 changed files with 1181 additions and 881 deletions
@@ -1,19 +1,55 @@
 TYPE=monitoring-ng
 LETS_ENCRYPT_ENV=production
 COMPOSE_FILE=compose.yml
-DOMAIN=monitoring.example.com
-TIMEOUT=120
+DOMAIN=monitoring-ng.example.com
+#TIMEOUT=120
 ENABLE_BACKUPS=true

-## Enable this secret for Promtail / Prometheus
-# SECRET_BASIC_AUTH_VERSION=v1
-#
-# Promtail (Gathering Logs)
-# COMPOSE_FILE="$COMPOSE_FILE:compose.promtail.yml"
-# LOKI_PUSH_URL=https://loki.monitoring.example.org/loki/api/v1/push
+# Secret Alloy authenticates with when writing metrics/logs to Prometheus/
+# Loki and what Traefik's basicauth middleware expects for that
+SECRET_BASIC_AUTH_VERSION=v1
+# Username sent along with SECRET_BASIC_AUTH_VERSION above (default: admin)
+# WRITE_BASIC_AUTH_USERNAME=admin

-## Expose node and cadvisor ports instead of traefik
-# COMPOSE_FILE="$COMPOSE_FILE:compose.expose-ports.yml"
+# Expose Alloy's web UI publicly (behind basic-auth) at alloy.$DOMAIN.
+# COMPOSE_FILE="$COMPOSE_FILE:compose.alloy-webui.yml"
+# Enable Live Debugging in web ui
+# LIVE_DEBUGGING=false
+
+# Enable this to send metrics to a Prometheus server, adapt DOMAIN if
+# server is remote
+# PROMETHEUS_REMOTE_WRITE_URL=https://prometheus.$DOMAIN/api/v1/write
+
+# Enable authenticated scraping of containers that opt in via
+# prometheus.io/auth=basic or prometheus.io/auth=bearer labels (used as
+# password/bearer token respectively). Insert it with:
+# abra app secret insert <domain> scrape_auth v1 <password>
+# COMPOSE_FILE="$COMPOSE_FILE:compose.scrape-auth.yml"
+# SECRET_SCRAPE_AUTH_VERSION=v1
+#
+# Username sent along with it (default: alloy)
+# SCRAPE_BASIC_AUTH_USERNAME=alloy
+
+# Edit this if your distribution sets this socket to another location
+# CONTAINERD_SOCKET=/var/run/containerd/containerd.sock
+
+# Enable this to send logs to a Loki server, adapt DOMAIN if server is
+# remote
+# LOKI_PUSH_URL=https://loki.$DOMAIN/loki/api/v1/push
+# Enable on systemd hosts to read logs from the journal
+# JOURNALD=1
+#
+# Enable on non-systemd hosts (Alpine, older Debian/Ubuntu) to tail
+# /var/log/*log files (syslog, auth.log, kern.log, etc.) that a local
+# syslogd writes. No syslogd reconfiguration needed.
+# SYSLOG_FILES=1
+#
+# Enable to receive syslog messages over the network on port 514/tcp.
+# Use for remote devices that push syslog to this host, or for a 
+# local syslogd configured to forward over the network.
+# Not needed if you just want to read local log files — use SYSLOG_FILES instead.
+# SYSLOG=1
+# COMPOSE_FILE="$COMPOSE_FILE:compose.syslog.yml"

 # Monitoring Server
 #
@@ -39,19 +75,20 @@ ENABLE_BACKUPS=true
 # LOKI_AWS_REGION=eu-west-1
 # LOKI_ACCESS_KEY_ID=bush-debrief-approval-robust-scraggly-molecule
 # LOKI_BUCKET_NAMES=loki
-# SECRET_LOKI_AWS_SECRET_ACCESS_KEY_VERSION=v1
+# SECRET_LOKI_AWS_KEY_VERSION=v1
 #
 ## Grafana
 #
 # COMPOSE_FILE="$COMPOSE_FILE:compose.grafana.yml"
 # GF_SERVER_ROOT_URL=https://monitoring.example.com
-# SECRET_GRAFANA_ADMIN_PASSWORD_VERSION=v1
+# SECRET_GF_ADMINPASSWD_VERSION=v1
 ## Seperate domain for Grafana
 #GRAFANA_DOMAIN=grafana.example.com
 #
 ## Single-Sign-On with OIDC
+# COMPOSE_FILE="$COMPOSE_FILE:compose.grafana-oidc.yml"
 # OIDC_ENABLED=1
-# SECRET_GRAFANA_OIDC_CLIENT_SECRET_VERSION=v1
+# SECRET_GF_OIDC_SECRET_VERSION=v1
 # OIDC_CLIENT_ID=grafana
 # OIDC_AUTH_URL="https://authentik.example.com/application/o/authorize/"
 # OIDC_API_URL="https://authentik.example.com/application/o/userinfo/"
@@ -62,24 +99,26 @@ ENABLE_BACKUPS=true
 # GF_INSTALL_PLUGINS=grafana-piechart-panel
 #
 ## grafana SMTP configuration (optional)
+# COMPOSE_FILE="$COMPOSE_FILE:compose.grafana-smtp.yml"
 # GF_SMTP_HOST=changeme
 # GF_SMTP_USER=changme
 # GF_SMTP_ENABLED=true
 # GF_SMTP_FROM_ADDRESS=grafana@example.com
 # GF_SMTP_SKIP_VERIFY=false
-# SECRET_GRAFANA_SMTP_PASSWORD_VERSION=v1
+# SECRET_GF_SMTP_PASS_VERSION=v1
 #

 ## Grafana Matrix Contact Point (optional)
 #COMPOSE_FILE="$COMPOSE_FILE:compose.matrix-alertmanager-receiver.yml"
-#SECRET_MATRIX_ACCESS_TOKEN_VERSION=v1
+#SECRET_MATRIX_TOKEN_VERSION=v1
 #GF_MATRIX_USER_ID="<user-id>"
 #GF_MATRIX_ROOM_ID="<room-id>"
 #GF_MATRIX_HOMESERVER_URL="<homeserver-url>"

-# ALerts
-#ALERT_BACKUP_FAILED_ENABLED=true
-#ALERT_BACKUP_MISSING_ENABLED=true
-#ALERT_BACKUP_NOT_SUCCESSFULL_ENABLED=true
-#ALERT_NODE_DISK_SPACE_ENABLED=true
-#ALERT_NODE_MEMORY_USAGE_ENABLED=true
+## ALerts
+
+# Node disk space alert will trigger when free disk space left is below the given number in percent
+#ALERT_NODE_DISK_SPACE_LEFT=10
+
+# Node memory usage alert will trigger when memory usage is above the given number in percent
+#ALERT_NODE_MEMORY_USAGE=85
@@ -1,9 +1,10 @@
 # monitoring-ng

 Yet another monitoring stack ...
-This time its a all-in-one grafana/prometheus/loki/node_exporter/cadvisor/promtail stack.
-It's based heavily on the [monitoring-lite](https://git.coopcloud.tech/coop-cloud/monitoring-lite) stack, but has everything in one recipe included now. So you can deploy monitoring instances to only gather metrics / logs (node_exporter/cadvisor/promtail) and also deploy instances with the full monitoring stack (grafana/prometheus/loki) with the same recipe and just different .env configuration.
+This time its a all-in-one grafana/prometheus/loki/alloy stack.
+It's based heavily on the [monitoring-lite](https://git.coopcloud.tech/coop-cloud/monitoring-lite) stack, but has everything in one recipe included now. So you can deploy monitoring instances to only gather metrics / logs (alloy) and also deploy instances with the full monitoring stack (grafana/prometheus/loki) with the same recipe and just different .env configuration.

+Metrics and logs are collected by [Grafana Alloy](https://grafana.com/docs/alloy/latest/) and pushed to a central Prometheus/Loki (via `remote_write`/`loki push`). Every `monitoring-ng` instance runs its own Alloy, whether or not it also runs the central stack.

 <!-- metadata -->

@@ -18,79 +19,93 @@ It's based heavily on the [monitoring-lite](https://git.coopcloud.tech/coop-clou

 <!-- endmetadata -->

-## Setup a Metrics Gathering
+## Setup: gathering-only host

-Where gathering.org is the node you want to gather metrics from.
+Deploys just Alloy, pushing this host's own node/container metrics and logs to a central instance. Use this on every host you want metrics/logs from.

-1. Configure DNS
-  - monitoring.gathering.org
-  - cadvisor.monitoring.gathering.org
-  - node.monitoring.gathering.org
-1. Configure Traefik to use BasicAuth
-  * `abra app config traefik.gathering.org`
-  uncomment
-  ```
-  # BASIC_AUTH
-  COMPOSE_FILE="$COMPOSE_FILE:compose.basicauth.yml"
-  BASIC_AUTH=1
-  SECRET_USERSFILE_VERSION=v1
-  ```
-  - Generate userslist with httpasswd hashed password
-    `abra app secret insert traefik.gathering.org usersfile v1 'admin:<hashed-secret>'`
-    make sure there is no whitespace in between `admin:<hashed-secret>`, it seems to break stuff...
-  - `abra app deploy -f traefik`
-1. `abra app new monitoring-ng`
-1. `abra app config monitoring.gathering.org`
-   for gathering only the main `compose.yml` is needed, nothing more.
-1. `abra app deploy monitoring.gathering.org`
-1. check that endpoints are up and basic-auth works
-  - cadvisor.monitoring.gathering.org
-  - node.monitoring.gathering.org
+1. `abra app new monitoring-ng --server gathering.org`
+2. `abra app config monitoring.gathering.org`
+3. Point it at your central instance:
+   ```
+   PROMETHEUS_REMOTE_WRITE_URL=https://prometheus.example.org/api/v1/write
+   LOKI_PUSH_URL=https://loki.example.org/loki/api/v1/push
+   JOURNALD=1   # or SYSLOG_FILES=1 / SYSLOG=1, see .env.sample
+   ```
+4. `abra app secret insert monitoring.gathering.org basic_auth v1 <password>`. Same username/password as the `usersfile` credential configured for Traefik's basicauth on the central node (see below). This is what Alloy authenticates with when pushing metrics/logs. Alloy defaults to username `admin` for this. Uf the Traefik `usersfile` uses a different username, set `WRITE_BASIC_AUTH_USERNAME` in this recipe to match.
+5. `abra app deploy monitoring.gathering.org`

-### Expose node and cadvisor via ports instead of traefik
+## Setup: full monitoring stack (metrics/logs browser)

-In case you have no traefik running on the machine, you can expose the ports directly by uncommenting the following line:
-```
-# COMPOSE_FILE="$COMPOSE_FILE:compose.expose-ports.yml"
+This is what a gathering host pushes into. It also runs its own Alloy, so it monitors itself too.
+
+1. Configure DNS: `monitoring.example.org`, plus `prometheus.`/`loki.`/`pushgateway.` subdomains for whichever of those you enable below
+2. Traefik on this node needs basic auth configured (`BASIC_AUTH=1`, see the Traefik recipe's "Configuring basic auth" section) — Prometheus/Loki/Pushgateway route through its `basicauth@file` middleware, so without it those endpoints won't work. Use the same username (default `admin`, see `WRITE_BASIC_AUTH_USERNAME`) and password you'll insert as the `basic_auth` secret below when generating Traefik's `usersfile`.
+3. `abra app config monitoring.example.org` Uncomment `compose.prometheus.yml` (metrics), `compose.loki.yml` (logs) and `compose.grafana.yml` (dashboard)
+4. `abra app secret insert monitoring.example.org basic_auth v1 <password>` — this is the password every gathering host's Alloy (including this instance's own) authenticates with; also what Traefik's basicauth expects on the public Prometheus/Loki/Pushgateway endpoints
+5. `abra app secret insert monitoring.example.org gf_adminpasswd v1 <password>`
+6. `abra app deploy monitoring.example.org`
+
+### Post-setup guide
+
+- configure the SMTP mailer under `Alerting > Contact points`
+  - edit the default contact point, choose "Alertmanager" as type & `http://alertmanager:9093` as URL
+  - use the "Test" button to send a test mail. It should fire a request at the alertmanager & that should send a mail
+- from your dashboard panels, choose `Edit > Alert` to create alerts based on those panels
+
+## Additional features
+
+### Discovering metrics from other apps
+
+Alloy auto-discovers and scrapes other Docker Swarm services running on the same host, on the `proxy` network, that opt in via labels. No manual scrape config needed. On the app's `compose.yml`:
+
+```yaml
+deploy:
+  labels:
+    - "prometheus.io/scrape=true"       # required: opt in
+    # - "prometheus.io/port=8082"       # optional: defaults to 80
+    # - "prometheus.io/path=/metrics"   # optional: defaults to /metrics
+    # - "prometheus.io/auth=basic"      # optional: basic auth, see below
+    # - "prometheus.io/auth=bearer"     # optional: bearer token auth
 ```

-## Setup Metrics Browser
+Each scraped target gets `instance` (`<service_name>.<slot>`, stable across redeploys), `domain` (the service's stack namespace with underscores converted back to dots, e.g. `traefik.example.com`) and `task_slot` labels attached automatically.

+If the target needs authentication, configure the monitoring-ng instance with a scrape-auth secret that will be used for targets having the auth label set:
+```
+COMPOSE_FILE="$COMPOSE_FILE:compose.scrape-auth.yml"
+SECRET_SCRAPE_AUTH_VERSION=v1
+```
+`abra app secret insert <domain> scrape_auth v1 <password-or-token>`, then set the scraped app's `prometheus.io/auth` label to `basic` or `bearer` to match how it checks the secret.

-1. Configure DNS
-   - monitoring.example.org
-   - prometheus.monitoring.example.org
-   - loki.monitoring.example.org
-2. Setup monitoring stack
-   - `abra app new monitoring-ng`
-   - `abra app config monitoring.example.org`
-      Uncomment all the stuff
-   - `abra app secret insert monitoring.example.org basic_auth v1 <secret>`
-      this needs the plaintext traefik basic-auth secret, not the hashed one!
-   - `abra app secret ls monitoring.example.org`
-   - `abra app deploy monitoring.example.org`
-3. Add scrape config to prometheus
-  - `abra app cmd monitoring.example.org prometheus gathering.org`
-  - or manually
-    ```
-    cp scrape-config.example.yml gathering.org.yml
-    # adjust domain
-    # mkdir scrape_configs
-    abra app cp monitoring.dev.local-it.cloud gathering.org.yml prometheus:/prometheus/scrape_configs/
-    ```
+Check discovered targets via `alloy.example.org` (needs `compose.alloy-webui.yml`, see below), or query the central Prometheus for `up{job="<service_name>"}`.

-* check that all configured targets are up:
-  https://prometheus.monitoring.example.org/targets
+### Manual scraping

+For targets where auto-discovery doesn't work (e.g. not a Docker Swarm service on the `proxy` network, or missing labels): add them directly to Prometheus instead. Create a scrape config file:
+```yaml
+- targets:
+    - 'metrics.something-external.example.org'
+    - 'app-without-labels.example.org'
+```
+and copy it into Prometheus' scrape config directory:
+```
+abra app cp monitoring.gathering.org targets.yml prometheus:/prometheus/scrape_configs/
+```
+Prometheus picks up files there automatically.

+### Alloy Web UI / Live Debugging

-| Service       | Authentication     | Domain                            |
-| ------------- | ------------------ | --------------------------------- |
-| Grafana       | Email / SSO        | monitoring.example.org            |
-| Prometheus    | traefik basic-auth | prometheus.monitoring.example.org |
-| loki          | traefik basic-auth | loki.monitoring.example.org       |
-| Cadvisor      | traefik basic-auth | cadvisor.monitoring.example.org   |
-| Node Exporter | traefik basic-auth | node.monitoring.example.org       |
+Alloy's own web UI isn't exposed by default, its HTTP server only listens
+on localhost inside its own container. To reach it from outside (e.g. to
+browse its component graph or use live debugging), add:
+```
+COMPOSE_FILE="$COMPOSE_FILE:compose.alloy-webui.yml"
+```
+This exposes it (behind the same basic-auth) at `alloy.$DOMAIN`.
+
+To actually see what's being collected (live-tailing the metrics/logs
+flowing through each component, not just their config) also set
+`LIVE_DEBUGGING=true`. Only enable this while troubleshooting.

 ### Logging from a docker host to loki server without anything else

@@ -107,36 +122,25 @@ $ echo '{
 $ systemctl restart docker.service
 ```

-## Setup Push Gateway
+### Setup Push Gateway

-1. Enable in the env fiöle by uncommenting the following lines:
+1. Enable in the env file by uncommenting the following lines:
 ```
 ## Prometheus Pushgateway
 # COMPOSE_FILE="$COMPOSE_FILE:compose.pushgateway.yml"
 ```
 2. `abra app deploy monitoring.example.org`

-This will expose the pushgateway at `https://pushgateway.${DOMAIN}`.
-It is secured behind the same basic auth as the other services.
-After that you need to add the `pushgateway.${DOMAIN}` to the scare config.
+This will expose the pushgateway at `https://pushgateway.${DOMAIN}`, secured behind the same basic auth as the other services.
+After that you need to add the `pushgateway.${DOMAIN}` to the scrape config of Prometheus.

-## Post-setup guide
-
- configure prometheus/loki/alertmanager as data sources in grafana under `Configuration > Data sources`
-  - for loki, you need to set a "Custom HTTP Header": `X-Scope-OrgID: fake`
- configure the SMTP mailer under `Alerting > Contact points`
-  - edit the default contact point, choose "Alertmanager" as type & `http://alertmanager:9093` as URL
-  - use the "Test" button to send a test mail. It should fire a request at the alertmanager & that should send a mail
- `abra app cp` your `scrap_configs: ...` into `/prometheus/scrape_configs` & log into your prometheus web UI to ensure they're working
- load your dashboards in manually under `Create > Dashboard`
- from your dashboard panels, choose `Edit > Alert` to create alerts based on those panels


 ---

 THX to the previous work of @decentral1se @knooflok @3wc @cellarspoon @mirsal

-## Adding Matrix as Alert Contact point
+### Adding Matrix as Alert Contact point

 1. Enable the [matrix-alertmanager-receiver](https://github.com/metio/matrix-alertmanager-receiver/):
 ```
@@ -145,7 +149,7 @@ COMPOSE_FILE="$COMPOSE_FILE:compose.matrix-alertmanager-receiver.yml"

 2. Insert the matrix access token secret:
 ```
-abra app secret insert monitoring.marx.klasse-methode.it matrix_access_token v1
+abra app secret insert monitoring.marx.klasse-methode.it matrix_token v1
 ```

 3. Set required configurations:
@@ -156,13 +160,9 @@ GF_MATRIX_HOME_SERVER_URL=
 ```
 4. Configure Alertmanager webhook and set the url to `http://matrix-alertmanager-receiver:12345/alerts/<room-id>`

-## alerts
-
-It is possible to enable the following alerts, by setting the corresponding env variable to `true`:
- backupbot failed: `ALERT_BACKUP_FAILED_ENABLED`
- backupbot missing: `ALERT_BACKUP_MISSING_ENABLED`
- backupbot not successfull: `ALERT_BACKUP_NOT_SUCCESSFULL_ENABLED`
- node disk space: `ALERT_NODE_DISK_SPACE_ENABLED`
- node memory usage: `ALERT_NODE_MEMORY_USAGE_ENABLED`
+### Alerts

+It is possible to enable the following alerts, by uncommenting the corresponding env variable:

+- node disk space: `ALERT_NODE_DISK_SPACE_LEFT`
+- node memory usage: `ALERT_NODE_MEMORY_USAGE`
@@ -1,38 +1,108 @@
-export ENTRYPOINT_VERSION=v1
-export GRAFANA_DATASOURCES_YML_VERSION=v1
-export GRAFANA_DASHBOARDS_YML_VERSION=v2
-export GRAFANA_SWARM_DASHBOARD_JSON_VERSION=v2
-export GRAFANA_STACKS_DASHBOARD_JSON_VERSION=v2
-export GRAFANA_TRAEFIK_DASHBOARD_JSON_VERSION=v2
-export GRAFANA_BACKUP_DASHBOARD_JSON_VERSION=v1
-export GRAFANA_ALERTS_JSON_VERSION=v3
-export GRAFANA_CUSTOM_INI_VERSION=v4
-export PROMTAIL_YML_VERSION=v3
-export LOKI_YML_VERSION=v2
+export GF_DATASOURCES_VERSION=v1
+export GF_DASHBOARDS_VERSION=v2
+export GF_SWARM_DASH_VERSION=v2
+export GF_STACKS_DASH_VERSION=v2
+export GF_TRAEFIK_DASH_VERSION=v3
+export GF_BACKUP_DASH_VERSION=v1
+export GF_CUSTOM_INI_VERSION=v4
+export LOKI_YML_VERSION=v3
 export PROMETHEUS_YML_VERSION=v2
-export MATRIX_ALERTMANAGER_CONFIG_VERSION=e
-export MATRIX_ALERTMANAGER_ENTRYPOINT_VERSION=a
+export MATRIX_ALERTMANAGER_CONFIG_VERSION=v1
+export MATRIX_ALERTMANAGER_ENTRYPOINT_VERSION=v1
+export GF_ALERTS_NODE_VERSION=v2
+export CONFIG_ALLOY_VERSION=v11

-# creates a default prometheus scrape config for a given node
-add_node(){
-  name=$1
-  add_domain "$name" "$name:8082"
-  add_domain "$name" "node.monitoring.$name"
-  add_domain "$name" "cadvisor.monitoring.$name"
-  cat "/prometheus/scrape_configs/$name.yml"
-}
+# migrates secrets from old names to new names by reading values from the
+# running containers on the server and re-inserting them under the new names.
+# preview changes: abra app cmd --local <app> migrate_secret_names
+# execute changes: abra app cmd --local <app> migrate_secret_names execute
+migrate_secret_names() {
+  if ! command -v jq &> /dev/null; then
+    echo "jq is required on your local machine to migrate secret names"
+    echo "It could not be found in your PATH, please install jq to proceed."
+    echo "For example: On a debian/ubuntu system, run `apt install jq`"
+    exit 1
+  fi

-# adds a domain to a scrape config or creates a new one
-add_domain(){
-  name=$1
-  domain=$2
-  if [ ! -d "/prometheus/scrape_configs/" ]; then
-    mkdir -p /prometheus/scrape_configs/
+  # Hardcoded migration mappings: old_secret_name|new_secret_name
+  MIGRATIONS="
+grafana_admin_password|gf_adminpasswd
+grafana_smtp_password|gf_smtp_pass
+grafana_oidc_client_secret|gf_oidc_secret
+matrix_access_token|matrix_token
+loki_aws_secret_access_key|loki_aws_key
+"
+
+  # Determine which server the app is deployed on
+  SERVER=$(abra app ls -m | jq -r --arg domain "$APP_NAME" '[.[].apps[] | select(.domain == $domain) | .server] | first' 2>/dev/null)
+
+  if [ -z "$SERVER" ]; then
+    echo "Error: could not determine server for app '$APP_NAME'"
+    exit 1
  fi
-  cd /prometheus/scrape_configs/ || exit 1
-  if [ ! -f "$name.yml" ]; then
-    echo -e "- targets:\n  - '$domain'" > "$name.yml"
-  else
-    echo "  - '$domain'"  >> "$name.yml"
+
+  # Build a lookup table of all secrets currently mounted in this stack.
+  # Each line: <secretID> <containerID> <secretName>
+  LOOKUP=$(ssh "$SERVER" "
+    docker stack services ${STACK_NAME} --format '{{.Name}}' | while read svc; do
+      CID=\$(docker ps --no-trunc -q --filter \"name=\${svc}\" | head -1)
+      docker service inspect \"\$svc\" --format '{{json .Spec.TaskTemplate.ContainerSpec.Secrets}}' | \
+        jq -r --arg cid \"\$CID\" '.[]? | .SecretID + \" \" + \$cid + \" \" + .SecretName'
+    done | sort -k3 -r
+  " 2>/dev/null)
+
+  echo "Secret migration plan for: $APP_NAME (server: $SERVER)"
+  echo ""
+  printf "  %-24s %-8s %s\n" "OLD NAME" "FOUND" "ACTION"
+  printf "  %-24s %-8s %s\n" "--------" "-----" "------"
+
+  # Check each old name against the lookup table and display the plan
+  ANY_FOUND=false
+  while IFS='|' read -r OLD_NAME NEW_NAME; do
+    [ -z "$OLD_NAME" ] && continue
+    MATCH=$(echo "$LOOKUP" | grep " ${STACK_NAME}_${OLD_NAME}_" | head -1)
+    if [ -n "$MATCH" ]; then
+      printf "  %-24s %-8s %s\n" "$OLD_NAME" "yes" "recreate as '$NEW_NAME' version V1"
+      ANY_FOUND=true
+    else
+      printf "  %-24s %-8s %s\n" "$OLD_NAME" "no" "nothing (not found on server)"
+    fi
+  done <<< "$MIGRATIONS"
+
+  echo ""
+
+  if [ "$ANY_FOUND" = false ]; then
+    echo "No old secrets found on server. Nothing to migrate."
+    return 0
  fi
+
+  if [ "$1" != "execute" ]; then
+    echo "To apply the above changes, run:"
+    echo "  abra app cmd --local $APP_NAME migrate_secret_names execute"
+    return 0
+  fi
+
+  # read each found secret from its container and re-insert with the new name
+  while IFS='|' read -r OLD_NAME NEW_NAME; do
+    [ -z "$OLD_NAME" ] && continue
+
+    MATCH=$(echo "$LOOKUP" | grep " ${STACK_NAME}_${OLD_NAME}_" | head -1)
+    [ -z "$MATCH" ] && continue
+
+    SECRET_ID=$(echo "$MATCH" | awk '{print $1}')
+    CID=$(echo "$MATCH" | awk '{print $2}')
+    SECRET_VALUE=$(ssh "$SERVER" "cat /var/lib/docker/containers/${CID}/mounts/secrets/${SECRET_ID} 2>/dev/null || sudo cat /var/lib/docker/containers/${CID}/mounts/secrets/${SECRET_ID} 2>/dev/null")
+
+    if [ -z "$SECRET_VALUE" ]; then
+      echo "Error: could not read value for '$OLD_NAME', skipping"
+      continue
+    fi
+
+    echo "Migrating: '$OLD_NAME' -> '$NEW_NAME' (v1)"
+    printf '%s' "$SECRET_VALUE" | abra app secret insert -C "$APP_NAME" "$NEW_NAME" v1
+    
+  done <<< "$MIGRATIONS"
+
+  echo ""
+  echo "Done."
 }
@@ -12,7 +12,7 @@ http:
 matrix:
  homeserver-url: "{{ env "GF_MATRIX_HOMESERVER_URL" }}"
  user-id: "{{ env "GF_MATRIX_USER_ID" }}"
-  access-token: "{{ secret "matrix_access_token" }}"
+  access-token: "{{ secret "matrix_token" }}"
  room-mapping:
    matrixroom: "{{ env "GF_MATRIX_ROOM_ID" }}"

@@ -0,0 +1,131 @@
+apiVersion: 1
+
+# List of alert rule UIDs that should be deleted
+deleteRules:
+  {{ if not (env "ALERT_NODE_DISK_SPACE_LEFT") }}
+  - orgId: 1
+    uid: coopcloud_node_disk_space_left
+  {{ end }}
+  {{ if not (env "ALERT_NODE_MEMORY_USAGE") }}
+  - orgId: 1
+    uid: coopcloud_node_memory_usage
+  {{ end }}
+
+groups:
+- orgId: 1
+  name: node
+  folder: node
+  interval: 5m
+  rules:
+    {{ if (env "ALERT_NODE_DISK_SPACE_LEFT") }}
+    - uid: coopcloud_node_disk_space_left
+      title: Node Disk Space
+      condition: C
+      data:
+        - refId: A
+          relativeTimeRange:
+            from: 600
+            to: 0
+          datasourceUid: PBFA97CFB590B2093
+          model:
+            editorMode: code
+            expr: (node_filesystem_free_bytes{fstype=~"ext4|xfs"} / node_filesystem_size_bytes{fstype=~"ext4|xfs"}) * 100
+            instant: true
+            intervalMs: 1000
+            legendFormat: __auto
+            maxDataPoints: 43200
+            range: false
+            refId: A
+        - refId: C
+          relativeTimeRange:
+            from: 600
+            to: 0
+          datasourceUid: __expr__
+          model:
+            conditions:
+                - evaluator:
+                    params:
+                        - {{ env "ALERT_NODE_DISK_SPACE_LEFT" }}
+                    type: lt
+                  operator:
+                    type: and
+                  query:
+                    params:
+                        - C
+                  reducer:
+                    params: []
+                    type: last
+                  type: query
+            datasource:
+                type: __expr__
+                uid: __expr__
+            expression: A
+            intervalMs: 1000
+            maxDataPoints: 43200
+            refId: C
+            type: threshold
+      noDataState: NoData
+      execErrState: Error
+      for: 5m
+      annotations:
+        description: ""
+        runbook_url: ""
+        summary: Less than {{ env "ALERT_NODE_DISK_SPACE_LEFT" }}% disk space left on {{`{{ $labels.instance }}`}} ({{`{{ (index $values "A").Value }}`}}% left)
+      labels:
+        "": ""
+      isPaused: false
+    {{ end }}
+    {{ if (env "ALERT_NODE_MEMORY_USAGE")  }}
+    - uid: coopcloud_node_memory_usage
+      title: Node Memory Usage
+      condition: C
+      data:
+        - refId: A
+          relativeTimeRange:
+            from: 600
+            to: 0
+          datasourceUid: PBFA97CFB590B2093
+          model:
+            editorMode: code
+            expr: (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100
+            instant: true
+            intervalMs: 1000
+            legendFormat: __auto
+            maxDataPoints: 43200
+            range: false
+            refId: A
+        - refId: C
+          relativeTimeRange:
+            from: 600
+            to: 0
+          datasourceUid: __expr__
+          model:
+            conditions:
+                - evaluator:
+                    params:
+                        - {{ env "ALERT_NODE_MEMORY_USAGE"  }}
+                    type: gt
+                  operator:
+                    type: and
+                  query:
+                    params:
+                        - C
+                  reducer:
+                    params: []
+                    type: last
+                  type: query
+            datasource:
+                type: __expr__
+                uid: __expr__
+            expression: A
+            intervalMs: 1000
+            maxDataPoints: 43200
+            refId: C
+            type: threshold
+      noDataState: NoData
+      execErrState: Error
+      for: 5m
+      annotations:
+        summary: Memory usage is above {{ env "ALERT_NODE_MEMORY_USAGE"  }}% on {{`{{ $labels.instance }}`}}  ({{`{{ printf "%.2f" (index $values "A").Value }}`}}%  usage)
+      isPaused: false
+    {{ end }}
@@ -0,0 +1,16 @@
+version: "3.8"
+
+services:
+  app:
+    environment:
+      - ALLOY_HTTP_LISTEN_ADDR=0.0.0.0
+    deploy:
+      labels:
+        - "traefik.enable=true"
+        - "traefik.swarm.network=proxy"
+        - "traefik.http.services.${STACK_NAME}-alloy.loadbalancer.server.port=12345"
+        - "traefik.http.routers.${STACK_NAME}-alloy.rule=Host(`alloy.${DOMAIN}`)"
+        - "traefik.http.routers.${STACK_NAME}-alloy.entrypoints=web-secure"
+        - "traefik.http.routers.${STACK_NAME}-alloy.tls=true"
+        - "traefik.http.routers.${STACK_NAME}-alloy.tls.certresolver=${LETS_ENCRYPT_ENV}"
+        - "traefik.http.routers.${STACK_NAME}-alloy.middlewares=basicauth@file"
@@ -1,13 +0,0 @@
---
-version: "3.8"
-
-services:
-  app:
-    ports:
-      - "9100:9100"
-    deploy:
-
-  cadvisor:
-    ports:
-      - "9101:8080"
-    deploy:
@@ -0,0 +1,17 @@
+version: '3.8'
+
+services:
+  grafana:
+    secrets:
+      - gf_oidc_secret
+    environment:
+      - OIDC_API_URL
+      - OIDC_AUTH_URL
+      - OIDC_CLIENT_ID
+      - OIDC_ENABLED
+      - OIDC_TOKEN_URL
+
+secrets:
+  gf_oidc_secret:
+    external: true
+    name: ${STACK_NAME}_gf_oidc_secret_${SECRET_GF_OIDC_SECRET_VERSION}
@@ -0,0 +1,18 @@
+version: '3.8'
+
+services:
+  grafana:
+    secrets:
+      - gf_smtp_pass
+    environment:
+      - GF_SMTP_HOST
+      - GF_SMTP_USER
+      - GF_SMTP_PASSWORD__FILE=/run/secrets/gf_smtp_pass
+      - GF_SMTP_ENABLED
+      - GF_SMTP_FROM_ADDRESS
+      - GF_SMTP_SKIP_VERIFY
+
+secrets:
+  gf_smtp_pass:
+    external: true
+    name: ${STACK_NAME}_gf_smtp_pass_${SECRET_GF_SMTP_PASS_VERSION}
@@ -2,104 +2,87 @@ version: '3.8'

 services:
  grafana:
-    image: grafana/grafana:10.4.14
+    image: grafana/grafana:13.0.6
    volumes:
      - grafana-data:/var/lib/grafana:rw
    secrets:
-      - grafana_admin_password
-      - grafana_oidc_client_secret
-      - grafana_smtp_password
+      - gf_adminpasswd
    configs:
-      - source: grafana_custom_ini
+      - source: gf_custom_ini
        target: /etc/grafana/grafana.ini
-      - source: grafana_datasources_yml
+      - source: gf_datasources
        target: /etc/grafana/provisioning/datasources/datasources.yml
-      - source: grafana_dashboards_yml
+      - source: gf_dashboards
        target: /etc/grafana/provisioning/dashboards/dashboards.yml
-      - source: grafana_swarm_dashboard_json
+      - source: gf_swarm_dash
        target: /var/lib/grafana/dashboards/docker-swarm-nodes.json
-      - source: grafana_stacks_dashboard_json
+      - source: gf_stacks_dash
        target: /var/lib/grafana/dashboards/docker-swarm-stacks.json
-      - source: grafana_traefik_dashboard_json
+      - source: gf_traefik_dash
        target: /var/lib/grafana/dashboards/traefik.json
-      - source: grafana_backup_dashboard_json
+      - source: gf_backup_dash
        target: /var/lib/grafana/dashboards/backup.json
-      - source: grafana_alerts_json
-        target: /var/lib/grafana/alerts/alerts.json
+      - source: gf_alerts_node
+        target: /etc/grafana/provisioning/alerting/node.yml
    networks:
      - proxy
      - internal
    environment:
      - GF_SERVER_ROOT_URL
-      - GF_SECURITY_ADMIN_PASSWORD__FILE=/run/secrets/grafana_admin_password
-      - GF_SMTP_HOST
-      - GF_SMTP_USER
-      - GF_SMTP_PASSWORD__FILE=/run/secrets/grafana_smtp_password
-      - GF_SMTP_ENABLED
-      - GF_SMTP_FROM_ADDRESS
-      - GF_SMTP_SKIP_VERIFY
+      - GF_SECURITY_ADMIN_PASSWORD__FILE=/run/secrets/gf_adminpasswd
      - GF_SECURITY_ALLOW_EMBEDDING
      - GF_INSTALL_PLUGINS
-      - OIDC_API_URL
-      - OIDC_AUTH_URL
-      - OIDC_CLIENT_ID
-      - OIDC_ENABLED
-      - OIDC_TOKEN_URL
+      - ALERT_NODE_DISK_SPACE_LEFT
+      - ALERT_NODE_MEMORY_USAGE
    deploy:
      labels:
        - "traefik.enable=true"
-        - "traefik.docker.network=proxy"
+        - "traefik.swarm.network=proxy"
        - "traefik.http.services.${STACK_NAME}-grafana.loadbalancer.server.port=3000"
        - "traefik.http.routers.${STACK_NAME}-grafana.rule=Host(`${GRAFANA_DOMAIN:-$DOMAIN}`)"
        - "traefik.http.routers.${STACK_NAME}-grafana.entrypoints=web-secure"
        - "traefik.http.routers.${STACK_NAME}-grafana.tls=true"
        - "traefik.http.routers.${STACK_NAME}-grafana.tls.certresolver=${LETS_ENCRYPT_ENV}"
    healthcheck:
-      test: "wget -q http://localhost:3000/ -O/dev/null"
+      test: "wget -q http://localhost:3000/healthz -O/dev/null"
      interval: 5s
      timeout: 10s
      retries: 3
      start_period: 10s

 configs:
-  grafana_custom_ini:
+  gf_custom_ini:
    template_driver: golang
-    name: ${STACK_NAME}_grafana_custom_ini_${GRAFANA_CUSTOM_INI_VERSION}
+    name: ${STACK_NAME}_gf_custom_ini_${GF_CUSTOM_INI_VERSION}
    file: grafana_custom.ini
-  grafana_datasources_yml:
-    name: ${STACK_NAME}_g_datasources_yml_${GRAFANA_DATASOURCES_YML_VERSION}
+  gf_datasources:
+    name: ${STACK_NAME}_gf_datasources_${GF_DATASOURCES_VERSION}
    file: grafana-datasources.yml
-  grafana_dashboards_yml:
-    name: ${STACK_NAME}_g_dashboards_yml_${GRAFANA_DASHBOARDS_YML_VERSION}
+  gf_dashboards:
+    name: ${STACK_NAME}_gf_dashboards_${GF_DASHBOARDS_VERSION}
    file: grafana-dashboards.yml
-  grafana_swarm_dashboard_json:
-    name: ${STACK_NAME}_g_swarm_dashboard_json_${GRAFANA_SWARM_DASHBOARD_JSON_VERSION}
+  gf_swarm_dash:
+    name: ${STACK_NAME}_gf_swarm_dash_${GF_SWARM_DASH_VERSION}
    file: grafana-swarm-dashboard.json
-  grafana_stacks_dashboard_json:
-    name: ${STACK_NAME}_g_stacks_dashboard_json_${GRAFANA_STACKS_DASHBOARD_JSON_VERSION}
+  gf_stacks_dash:
+    name: ${STACK_NAME}_gf_stacks_dash_${GF_STACKS_DASH_VERSION}
    file: grafana-stacks-dashboard.json
-  grafana_traefik_dashboard_json:
-    name: ${STACK_NAME}_g_traefik_dashboard_json_${GRAFANA_TRAEFIK_DASHBOARD_JSON_VERSION}
+  gf_traefik_dash:
+    name: ${STACK_NAME}_gf_traefik_dash_${GF_TRAEFIK_DASH_VERSION}
    file: grafana-traefik-dashboard.json
-  grafana_backup_dashboard_json:
-    name: ${STACK_NAME}_g_backup_dashboard_json_${GRAFANA_BACKUP_DASHBOARD_JSON_VERSION}
+  gf_backup_dash:
+    name: ${STACK_NAME}_gf_backup_dash_${GF_BACKUP_DASH_VERSION}
    file: grafana-backup-dashboard.json
-  grafana_alerts_json:
+  gf_alerts_node:
    template_driver: golang
-    name: ${STACK_NAME}_g_alerts_json_${GRAFANA_ALERTS_JSON_VERSION}
-    file: grafana-alerts.json.tmpl
+    name: ${STACK_NAME}_gf_alerts_node_${GF_ALERTS_NODE_VERSION}
+    file: alerts/node.yml.tmpl

 volumes:
  grafana-data:


 secrets:
-  grafana_admin_password:
+  gf_adminpasswd:
    external: true
-    name: ${STACK_NAME}_grafana_admin_password_${SECRET_GRAFANA_ADMIN_PASSWORD_VERSION}
-  grafana_oidc_client_secret:
-    external: true
-    name: ${STACK_NAME}_grafana_oidc_client_secret_${SECRET_GRAFANA_OIDC_CLIENT_SECRET_VERSION}
-  grafana_smtp_password:
-    external: true
-    name: ${STACK_NAME}_grafana_smtp_password_${SECRET_GRAFANA_SMTP_PASSWORD_VERSION}
+    name: ${STACK_NAME}_gf_adminpasswd_${SECRET_GF_ADMINPASSWD_VERSION}
@@ -2,7 +2,7 @@ version: '3.8'

 services:
  loki:
-    image: grafana/loki:2.9.11
+    image: grafana/loki:3.7.2
    command: -config.file=/etc/loki/local-config.yaml
    networks:
      - proxy
@@ -12,7 +12,7 @@ services:
    volumes:
      - loki-data:/loki
    # secrets:
-    #   - loki_aws_secret_access_key
+    #   - loki_aws_key
    environment:
      - LOKI_ACCESS_KEY_ID
      - LOKI_AWS_ENDPOINT
@@ -27,7 +27,7 @@ services:
        condition: on-failure
      labels:
        - "traefik.enable=true"
-        - "traefik.docker.network=proxy"
+        - "traefik.swarm.network=proxy"
        - "traefik.http.services.${STACK_NAME}-loki.loadbalancer.server.port=3100"
        - "traefik.http.routers.${STACK_NAME}-loki.rule=Host(`loki.${DOMAIN}`)"
        - "traefik.http.routers.${STACK_NAME}-loki.entrypoints=web-secure"
@@ -47,6 +47,6 @@ volumes:
  loki-data:

 # secrets:
-#   loki_aws_secret_access_key:
+#   loki_aws_key:
 #     external: true
-#     name: ${STACK_NAME}_loki_aws_secret_access_key_${SECRET_LOKI_AWS_SECRET_ACCESS_KEY_VERSION}
+#     name: ${STACK_NAME}_loki_aws_key_${SECRET_LOKI_AWS_KEY_VERSION}
@@ -2,9 +2,9 @@ version: '3.8'

 services:
  matrix-alertmanager-receiver:
-    image: metio/matrix-alertmanager-receiver:2025.2.9
+    image: metio/matrix-alertmanager-receiver:2026.2.25
    secrets:
-      - matrix_access_token
+      - matrix_token
    configs:
      - source: matrix-alertmanager-receiver-config
        target: /etc/matrix-alertmanager-receiver/config.yml
@@ -23,6 +23,6 @@ configs:
    file: alertmanager-matrix-config.yml.tmpl

 secrets:
-  matrix_access_token:
+  matrix_token:
    external: true
-    name: ${STACK_NAME}_matrix_access_token_${SECRET_MATRIX_ACCESS_TOKEN_VERSION}
+    name: ${STACK_NAME}_matrix_token_${SECRET_MATRIX_TOKEN_VERSION}
@@ -2,7 +2,7 @@ version: '3.8'

 services:
  prometheus:
-    image: prom/prometheus:v2.55.1
+    image: prom/prometheus:v3.12.0
    secrets:
      - basic_auth
    volumes:
@@ -16,20 +16,28 @@ services:
      - "--web.console.libraries=/usr/share/prometheus/console_libraries"
      - "--web.console.templates=/usr/share/prometheus/consoles"
      - "--storage.tsdb.retention.time=${PROMETHEUS_RETENTION_TIME}"
+      - "--web.enable-remote-write-receiver"
    networks:
      - proxy
      - internal
+    healthcheck:
+      test: "wget -q --spider http://localhost:9090/-/healthy || exit 1"
+      interval: 5s
+      timeout: 10s
+      retries: 3
+      start_period: 30s
    deploy:
      restart_policy:
        condition: on-failure
      labels:
        - "traefik.enable=true"
-        - "traefik.docker.network=proxy"
+        - "traefik.swarm.network=proxy"
        - "traefik.http.services.${STACK_NAME}-prometheus.loadbalancer.server.port=9090"
        - "traefik.http.routers.${STACK_NAME}-prometheus.rule=Host(`prometheus.${DOMAIN}`)"
        - "traefik.http.routers.${STACK_NAME}-prometheus.entrypoints=web-secure"
        - "traefik.http.routers.${STACK_NAME}-prometheus.tls=true"
        - "traefik.http.routers.${STACK_NAME}-prometheus.tls.certresolver=${LETS_ENCRYPT_ENV}"
+        - "traefik.http.routers.${STACK_NAME}-prometheus.middlewares=basicauth@file"

 configs:
  prometheus_yml:
@@ -1,30 +0,0 @@
-version: "3.8"
-
-services:
-  promtail:
-    image: grafana/promtail:2.9.11
-    volumes:
-      - /var/log:/var/log:ro
-      - /var/run/docker.sock:/var/run/docker.sock
-    command: -config.file=/etc/promtail/config.yml
-    configs:
-      - source: promtail_yml
-        target: /etc/promtail/config.yml
-    networks:
-      - internal
-    secrets:
-      - basic_auth
-    environment:
-      - DOMAIN
-      - LOKI_PUSH_URL
-
-configs:
-  promtail_yml:
-    name: ${STACK_NAME}_promtail_yml_${PROMTAIL_YML_VERSION}
-    file: promtail.yml.tmpl
-    template_driver: golang
-
-secrets:
-  basic_auth:
-    external: true
-    name: ${STACK_NAME}_basic_auth_${SECRET_BASIC_AUTH_VERSION}
@@ -2,7 +2,7 @@ version: '3.8'

 services:
  pushgateway:
-    image: prom/pushgateway:v1.10.0
+    image: prom/pushgateway:v1.11.2
    command:
      - '--web.listen-address=:9191'
      - '--push.disable-consistency-check'
@@ -12,12 +12,18 @@ services:
    networks:
      - internal
      - proxy
+    healthcheck:
+      test: "wget -q --spider http://localhost:9191/-/healthy || exit 1"
+      interval: 5s
+      timeout: 10s
+      retries: 3
+      start_period: 10s
    deploy:
      restart_policy:
        condition: on-failure
      labels:
        - "traefik.enable=true"
-        - "traefik.docker.network=proxy"
+        - "traefik.swarm.network=proxy"
        - "traefik.http.services.${STACK_NAME}-pushgateway.loadbalancer.server.port=9191"
        - "traefik.http.routers.${STACK_NAME}-pushgateway.rule=Host(`pushgateway.${DOMAIN}`)"
        - "traefik.http.routers.${STACK_NAME}-pushgateway.entrypoints=web-secure"
@@ -0,0 +1,13 @@
+---
+version: "3.8"
+
+services:
+  app:
+    secrets:
+      - source: scrape_auth
+        target: scrape_auth
+
+secrets:
+  scrape_auth:
+    external: true
+    name: ${STACK_NAME}_scrape_auth_${SECRET_SCRAPE_AUTH_VERSION}
@@ -0,0 +1,6 @@
+---
+version: "3.8"
+services:
+  app:
+    ports:
+    - "514:514"
@@ -3,89 +3,69 @@ version: "3.8"

 services:
  app:
-    image: prom/node-exporter:v1.8.1
-    user: root
-    environment:
-      - NODE_ID={{.Node.ID}}
-    volumes:
-      - /proc:/host/proc:ro
-      - /sys:/host/sys:ro
-      - /:/rootfs:ro
-      - /etc/hostname:/etc/nodename:ro
-    command:
-      - "--path.sysfs=/host/sys"
-      - "--path.procfs=/host/proc"
-      - "--path.rootfs=/rootfs"
-      - "--collector.textfile.directory=/etc/node-exporter/"
-      - "--collector.filesystem.ignored-mount-points=^/(sys|proc|dev|host|etc)($$|/)"
-      - "--no-collector.ipvs"
+    image: grafana/alloy:v1.18.1
+    hostname: "${DOMAIN}"
    configs:
-      - source: entrypoint
-        target: /entrypoint.sh
+      - source: config_alloy
+        target: /etc/alloy/config.alloy
+    volumes:
+      - /:/rootfs:ro
+      - /var/run/docker.sock:/var/run/docker.sock:ro
+      - /sys:/sys:ro
+      - /var/lib/docker:/var/lib/docker:ro
+      # long-form avoids toolshed/abra#900
+      - type: bind
+        source: "${CONTAINERD_SOCKET:-/run/containerd/containerd.sock}"
+        target: /run/containerd/containerd.sock
+        read_only: true
+      - alloy-data:/var/lib/alloy/data
+    # runs through a shell so ALLOY_HTTP_LISTEN_ADDR (set by
+    # compose.alloy-webui.yml) is resolved from the container's own
+    # environment at startup, not by compose at deploy time.
+    # $$ escapes it from compose's own interpolation
+    entrypoint: ["/bin/sh", "-c"]
+    command:
+      - >-
+        exec alloy run
+        --storage.path=/var/lib/alloy/data
+        --server.http.listen-addr=$${ALLOY_HTTP_LISTEN_ADDR:-127.0.0.1}:12345
+        /etc/alloy/config.alloy
    networks:
-      - internal
      - proxy
-    entrypoint: [ "/bin/sh", "-e", "/entrypoint.sh" ]
+      - internal
+    environment:
+      - SCRAPE_BASIC_AUTH_USERNAME=${SCRAPE_BASIC_AUTH_USERNAME:-alloy}
+      - WRITE_BASIC_AUTH_USERNAME=${WRITE_BASIC_AUTH_USERNAME:-admin}
+      - LIVE_DEBUGGING=${LIVE_DEBUGGING:-false}
+      - NODE_ID={{.Node.ID}}
+    secrets:
+      - basic_auth
+    # no wget/curl in this image; bash's /dev/tcp is used instead. Works
+    # against localhost regardless of ALLOY_HTTP_LISTEN_ADDR
+    healthcheck:
+      test: ["CMD", "bash", "-c", "exec 3<>/dev/tcp/localhost/12345 && printf 'GET /-/ready HTTP/1.0\r\nHost: localhost\r\n\r\n' >&3 && head -1 <&3 | grep -q 200"]
+      interval: 5s
+      timeout: 10s
+      retries: 3
+      start_period: 10s
    deploy:
      restart_policy:
        condition: on-failure
      labels:
        - "backupbot.backup=${ENABLE_BACKUPS:-true}"
-        - "traefik.enable=true"
-        - "traefik.docker.network=proxy"
-        - "traefik.http.services.${STACK_NAME}-node.loadbalancer.server.port=9100"
-        - "traefik.http.routers.${STACK_NAME}-node.rule=Host(`node.${DOMAIN}`)"
-        - "traefik.http.routers.${STACK_NAME}-node.entrypoints=web-secure"
-        - "traefik.http.routers.${STACK_NAME}-node.tls=true"
-        - "traefik.http.routers.${STACK_NAME}-node.tls.certresolver=${LETS_ENCRYPT_ENV}"
-        - "traefik.http.routers.${STACK_NAME}-node.middlewares=basicauth@file"
        - "coop-cloud.${STACK_NAME}.version=1.6.0+v1.8.1"
-        - "coop-cloud.${STACK_NAME}.timeout=${TIMEOUT:-120}"
-
-  cadvisor:
-    image: gcr.io/cadvisor/cadvisor:v0.49.2
-    command:
-      - "-logtostderr"
-      - "--enable_metrics=cpu,cpuLoad,disk,diskIO,process,memory,network"
-      # all possible metrics: advtcp,app,cpu,cpuLoad,cpu_topology,cpuset,disk,diskIO,hugetlb,memory,memory_numa,network,oom_event,percpu,perf_event,process,referenced_memory,resctrl,sched,tcp,udp.
-      - "--housekeeping_interval=120s"
-      - "--docker_only=true"
-    volumes:
-      - /var/lib/docker/:/var/lib/docker:ro
-      - /dev/disk/:/dev/disk:ro
-      - /sys:/sys:ro
-      - /var/run:/var/run:ro
-      - /:/rootfs:ro
-    networks:
-      - internal
-      - proxy
-    deploy:
-      restart_policy:
-        condition: on-failure
-      labels:
-        - "traefik.enable=true"
-        - "traefik.docker.network=proxy"
-        - "traefik.http.services.${STACK_NAME}-cadvisor.loadbalancer.server.port=8080"
-        - "traefik.http.routers.${STACK_NAME}-cadvisor.rule=Host(`cadvisor.${DOMAIN}`)"
-        - "traefik.http.routers.${STACK_NAME}-cadvisor.entrypoints=web-secure"
-        - "traefik.http.routers.${STACK_NAME}-cadvisor.tls=true"
-        - "traefik.http.routers.${STACK_NAME}-cadvisor.tls.certresolver=${LETS_ENCRYPT_ENV}"
-        - "traefik.http.routers.${STACK_NAME}-cadvisor.middlewares=basicauth@file"
-    healthcheck:
-      test: wget --quiet --tries=1 --spider http://localhost:8080/healthz || exit 1
-      interval: 15s
-      timeout: 15s
-      retries: 5
-      start_period: 30s
-
 configs:
-  entrypoint:
-    name: ${STACK_NAME}_entrypoint_${ENTRYPOINT_VERSION}
-    file: node-exporter-entrypoint.sh
-
-
-
+  config_alloy:
+    template_driver: golang
+    name: ${STACK_NAME}_config_alloy_${CONFIG_ALLOY_VERSION}
+    file: config.alloy.tmpl
 networks:
  proxy:
    external: true
  internal:
+volumes:
+  alloy-data:
+secrets:
+  basic_auth:
+    external: true
+    name: ${STACK_NAME}_basic_auth_${SECRET_BASIC_AUTH_VERSION}
@@ -0,0 +1,349 @@
+logging {
+  level = "info"
+  format = "logfmt"
+}
+
+livedebugging {
+  enabled = {{ env "LIVE_DEBUGGING" }}
+}
+
+discovery.docker "linux" {
+  host = "unix:///var/run/docker.sock"
+}
+
+{{ if ne (env "PROMETHEUS_REMOTE_WRITE_URL") "" }}
+prometheus.exporter.cadvisor "docker" {
+  docker_only = true
+  enabled_metrics = ["cpu", "cpuLoad", "disk", "diskIO", "memory", "network", "process"]
+}
+
+prometheus.exporter.unix "default" {
+  include_exporter_metrics = true
+  rootfs_path = "/rootfs"
+  procfs_path = "/rootfs/proc"
+  sysfs_path = "/rootfs/sys"
+
+  disable_collectors = ["ipvs"]
+
+  filesystem {
+    fs_types_exclude     = "^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|tmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$"
+    mount_points_exclude = "^/(sys|proc|dev|host|etc)($|/)"
+    mount_timeout        = "5s"
+  }
+
+  netclass { ignored_devices = "^(veth.*)$" }
+  netdev   { device_exclude  = "^(veth.*)$" }
+}
+
+prometheus.exporter.self "alloy" {}
+
+prometheus.scrape "default" {
+  scrape_interval = "120s"
+
+  targets = array.concat(
+    prometheus.exporter.self.alloy.targets,
+    prometheus.exporter.unix.default.targets,
+    prometheus.exporter.cadvisor.docker.targets,
+  )
+
+  forward_to = [prometheus.remote_write.prometheus.receiver]
+}
+
+prometheus.remote_write "prometheus" {
+  endpoint {
+    url = "{{ env "PROMETHEUS_REMOTE_WRITE_URL" }}"
+
+    basic_auth {
+      username = "{{ env "WRITE_BASIC_AUTH_USERNAME" }}"
+      password = "{{ secret "basic_auth" }}"
+    }
+  }
+}
+
+// Scrape Prometheus metrics from other containers on this host.
+// Containers opt in via Docker labels:
+//   prometheus.io/scrape=true        required: enable scraping
+//   prometheus.io/port=9090          optional: port exposing /metrics (defaults to 80 if not set)
+//   prometheus.io/path=/metrics      optional: path to metrics endpoint (default: /metrics)
+//   prometheus.io/auth=basic         optional: use basic auth with the scrape_auth secret (see compose.scrape-auth.yml)
+//   prometheus.io/auth=bearer        optional: use bearer auth with the scrape_auth secret (see compose.scrape-auth.yml)
+discovery.dockerswarm "swarm" {
+  host = "unix:///var/run/docker.sock"
+  // "tasks" not "services": dnsrr-mode services (e.g. traefik) have no VIP
+  // and are invisible to the "services" role
+  role = "tasks"
+}
+
+discovery.relabel "metrics" {
+  targets = discovery.dockerswarm.swarm.targets
+
+  // skip old task history, only scrape currently-running tasks
+  rule {
+    source_labels = ["__meta_dockerswarm_task_desired_state"]
+    regex         = "running"
+    action        = "keep"
+  }
+
+  // only scrape hosts running on this host within the swam
+  rule {
+    source_labels = ["__meta_dockerswarm_node_id"]
+    regex         = "{{ env "NODE_ID" }}"
+    action        = "keep"
+  }
+
+  rule {
+    source_labels = ["__meta_dockerswarm_network_name"]
+    regex         = "proxy"
+    action        = "keep"
+  }
+
+  rule {
+    source_labels = ["__meta_dockerswarm_service_label_prometheus_io_scrape"]
+    regex         = "true"
+    action        = "keep"
+  }
+
+  // default to port 80 when prometheus.io/port isn't set
+  rule {
+    source_labels = ["__meta_dockerswarm_service_label_prometheus_io_port"]
+    regex         = "^$"
+    target_label  = "__meta_dockerswarm_service_label_prometheus_io_port"
+    replacement   = "80"
+  }
+
+  // a task with multiple published ports produces one target per port;
+  // this unifies all of them to the single port above, so duplicates
+  // collapse at scrape time instead of scraping every port
+  rule {
+    source_labels = ["__address__", "__meta_dockerswarm_service_label_prometheus_io_port"]
+    regex         = `(.+):\d+;(\d+)`
+    target_label  = "__address__"
+    replacement   = "$1:$2"
+  }
+
+  rule {
+    source_labels = ["__meta_dockerswarm_service_label_prometheus_io_path"]
+    regex         = `(.+)`
+    target_label  = "__metrics_path__"
+  }
+
+  rule {
+    source_labels = ["__meta_dockerswarm_service_name"]
+    target_label  = "job"
+  }
+
+  // task IDs (and the default address-derived instance label) change on
+  // every redeploy; service+slot is stable across redeploys of the same
+  // replica, so data stays continuous instead of restarting each deploy
+  rule {
+    source_labels = ["__meta_dockerswarm_service_name", "__meta_dockerswarm_task_slot"]
+    separator     = "."
+    target_label  = "instance"
+  }
+
+  rule {
+    source_labels = ["__meta_dockerswarm_task_slot"]
+    target_label  = "task_slot"
+  }
+
+  // coop-cloud's STACK_NAME is the domain with "." replaced by "_"
+  // Derive a readable dotted domain label from it. RE2 has no global
+  // replace, so this is done by chaining multiple replacements. Each
+  // rule swaps the first remaining _ for a "." until none are left.
+  rule {
+    source_labels = ["__meta_dockerswarm_service_label_com_docker_stack_namespace"]
+    target_label  = "domain"
+  }
+  rule {
+    source_labels = ["domain"]
+    regex         = `([^_]*)_(.*)`
+    target_label  = "domain"
+    replacement   = "$1.$2"
+  }
+  rule {
+    source_labels = ["domain"]
+    regex         = `([^_]*)_(.*)`
+    target_label  = "domain"
+    replacement   = "$1.$2"
+  }
+  rule {
+    source_labels = ["domain"]
+    regex         = `([^_]*)_(.*)`
+    target_label  = "domain"
+    replacement   = "$1.$2"
+  }
+  rule {
+    source_labels = ["domain"]
+    regex         = `([^_]*)_(.*)`
+    target_label  = "domain"
+    replacement   = "$1.$2"
+  }
+  rule {
+    source_labels = ["domain"]
+    regex         = `([^_]*)_(.*)`
+    target_label  = "domain"
+    replacement   = "$1.$2"
+  }
+  rule {
+    source_labels = ["domain"]
+    regex         = `([^_]*)_(.*)`
+    target_label  = "domain"
+    replacement   = "$1.$2"
+  }
+}
+
+discovery.relabel "metrics_noauth" {
+  targets = discovery.relabel.metrics.output
+  rule {
+    source_labels = ["__meta_dockerswarm_service_label_prometheus_io_auth"]
+    regex         = "^$"
+    action        = "keep"
+  }
+}
+
+discovery.relabel "metrics_basicauth" {
+  targets = discovery.relabel.metrics.output
+  rule {
+    source_labels = ["__meta_dockerswarm_service_label_prometheus_io_auth"]
+    regex         = "basic"
+    action        = "keep"
+  }
+}
+
+discovery.relabel "metrics_bearerauth" {
+  targets = discovery.relabel.metrics.output
+  rule {
+    source_labels = ["__meta_dockerswarm_service_label_prometheus_io_auth"]
+    regex         = "bearer"
+    action        = "keep"
+  }
+}
+
+prometheus.scrape "containers" {
+  scrape_interval = "120s"
+  targets         = discovery.relabel.metrics_noauth.output
+  forward_to      = [prometheus.remote_write.prometheus.receiver]
+}
+
+{{ if ne (env "SECRET_SCRAPE_AUTH_VERSION") "" }}
+prometheus.scrape "containers_basicauth" {
+  scrape_interval = "120s"
+  targets         = discovery.relabel.metrics_basicauth.output
+  forward_to      = [prometheus.remote_write.prometheus.receiver]
+  basic_auth {
+    username = "{{ env "SCRAPE_BASIC_AUTH_USERNAME" }}"
+    password = "{{ secret "scrape_auth" }}"
+  }
+}
+
+prometheus.scrape "containers_bearerauth" {
+  scrape_interval = "120s"
+  targets         = discovery.relabel.metrics_bearerauth.output
+  forward_to      = [prometheus.remote_write.prometheus.receiver]
+  bearer_token    = "{{ secret "scrape_auth" }}"
+}
+{{ end }}
+{{ end }}
+
+{{ if ne (env "LOKI_PUSH_URL") "" }}
+discovery.relabel "docker" {
+  targets = discovery.docker.linux.targets
+
+  rule {
+    source_labels = ["__meta_docker_container_name"]
+    target_label  = "container_name"
+  }
+  rule {
+    source_labels = ["__meta_docker_container_id"]
+    target_label  = "container_id"
+  }
+  rule {
+    source_labels = ["__meta_docker_container_label_com_docker_stack_namespace"]
+    target_label  = "stack_namespace"
+  }
+  rule {
+    source_labels = ["__meta_docker_container_label_com_docker_swarm_service_name"]
+    target_label  = "service_name"
+  }
+  rule {
+    source_labels = ["__meta_docker_container_log_stream"]
+    target_label  = "stream"
+  }
+}
+
+loki.source.docker "docker" {
+  host       = "unix:///var/run/docker.sock"
+  targets    = discovery.relabel.docker.output
+  labels     = {"app" = "docker"}
+  forward_to = [loki.write.loki.receiver]
+}
+
+// JOURNALD: reads the systemd journal binary log directly.
+// Use on systemd hosts (most modern Linux distros). Requires no syslogd.
+{{ if eq (env "JOURNALD") "1" }}
+loki.source.journal "journal" {
+  path = "/rootfs/var/log/journal"
+  labels = { job = "{{ env "DOMAIN" }}" }
+  forward_to = [loki.write.loki.receiver]
+}
+{{ end }}
+
+// SYSLOG_FILES: tails all /var/log/*log files (syslog, auth.log, kern.log, etc.).
+// Use on non-systemd hosts where a syslogd writes to /var/log.
+{{ if eq (env "SYSLOG_FILES") "1" }}
+local.file_match "syslog_files" {
+  path_targets = [{ __path__ = "/rootfs/var/log/*log" }]
+}
+
+loki.source.file "syslog_files" {
+  targets    = local.file_match.syslog_files.targets
+  forward_to = [loki.process.syslog_files.receiver]
+}
+
+loki.process "syslog_files" {
+  stage.static_labels {
+    values = { job = "syslog" }
+  }
+  forward_to = [loki.write.loki.receiver]
+}
+{{ end }}
+
+// SYSLOG: opens a network syslog listener on port 514.
+// Use when a remote device or a local syslogd configured to 
+// forward over the network sends logs to this host.
+// Requires compose.syslog.yml to publish port 514 to the host.
+// This is NOT needed for reading local log files — use SYSLOG_FILES instead.
+{{ if eq (env "SYSLOG") "1" }}
+loki.relabel "syslog" {
+  rule {
+    action = "labelmap"
+    regex = "__syslog_(.+)"
+  }
+
+  forward_to = []
+}
+
+loki.source.syslog "syslog" {
+  listener {
+    address = "[::]:514"
+    label_structured_data = true
+    labels = { component = "loki.source.syslog" }
+  }
+
+  relabel_rules = loki.relabel.syslog.rules
+  forward_to = [loki.write.loki.receiver]
+}
+{{ end }}
+
+loki.write "loki" {
+  endpoint {
+    url = "{{ env "LOKI_PUSH_URL" }}"
+
+    basic_auth {
+      username = "{{ env "WRITE_BASIC_AUTH_USERNAME" }}"
+      password = "{{ secret "basic_auth" }}"
+    }
+  }
+  external_labels = { hostname = "{{ env "DOMAIN" }}" }
+}
+{{ end }}
@@ -1,315 +0,0 @@
-{
-  "apiVersion": 1,
-  "groups": [
-    {
-      "orgId": 1,
-      "name": "backupbot",
-      "folder": "node",
-      "interval": "1m",
-      "rules": [
-        {{ if eq (env "ALERT_BACKUP_FAILED_ENABLED") "true" }}
-        {
-          "uid": "de8e5xxup7t34a",
-          "title": "Backup Failed",
-          "condition": "C",
-          "data": [
-            {
-              "refId": "A",
-              "relativeTimeRange": { "from": 600, "to": 0 },
-              "datasourceUid": "PBFA97CFB590B2093",
-              "model": {
-                "disableTextWrap": false,
-                "editorMode": "builder",
-                "expr": "backup",
-                "fullMetaSearch": false,
-                "includeNullMetadata": true,
-                "instant": true,
-                "intervalMs": 1000,
-                "legendFormat": "__auto",
-                "maxDataPoints": 43200,
-                "range": false,
-                "refId": "A",
-                "useBackend": false
-              }
-            },
-            {
-              "refId": "C",
-              "relativeTimeRange": { "from": 600, "to": 0 },
-              "datasourceUid": "__expr__",
-              "model": {
-                "conditions": [
-                  {
-                    "evaluator": { "params": [0], "type": "lt" },
-                    "operator": { "type": "and" },
-                    "query": { "params": ["C"] },
-                    "reducer": { "params": [], "type": "last" },
-                    "type": "query"
-                  }
-                ],
-                "datasource": { "type": "__expr__", "uid": "__expr__" },
-                "expression": "A",
-                "intervalMs": 1000,
-                "maxDataPoints": 43200,
-                "refId": "C",
-                "type": "threshold"
-              }
-            }
-          ],
-          "noDataState": "NoData",
-          "execErrState": "Error",
-          "for": "1m",
-          "isPaused": false
-        },
-        {{ end }}
-        {{ if eq (env "ALERT_BACKUP_MISSING_ENABLED") "true" }}
-        {
-          "uid": "ce8e65uddcwe8d",
-          "title": "Backup Missing",
-          "condition": "B",
-          "data": [
-            {
-              "refId": "A",
-              "relativeTimeRange": { "from": 600, "to": 0 },
-              "datasourceUid": "PBFA97CFB590B2093",
-              "model": {
-                "disableTextWrap": false,
-                "editorMode": "builder",
-                "expr": "rate(backup[24h])",
-                "fullMetaSearch": false,
-                "includeNullMetadata": true,
-                "instant": true,
-                "intervalMs": 1000,
-                "legendFormat": "__auto",
-                "maxDataPoints": 43200,
-                "range": false,
-                "refId": "A",
-                "useBackend": false
-              }
-            },
-            {
-              "refId": "B",
-              "relativeTimeRange": { "from": 600, "to": 0 },
-              "datasourceUid": "__expr__",
-              "model": {
-                "conditions": [
-                  {
-                    "evaluator": { "params": [0, 0], "type": "within_range" },
-                    "operator": { "type": "and" },
-                    "query": { "params": ["C"] },
-                    "reducer": { "params": [], "type": "last" },
-                    "type": "query"
-                  }
-                ],
-                "datasource": { "type": "__expr__", "uid": "__expr__" },
-                "expression": "A",
-                "intervalMs": 1000,
-                "maxDataPoints": 43200,
-                "refId": "B",
-                "type": "threshold"
-              }
-            }
-          ],
-          "noDataState": "NoData",
-          "execErrState": "Error",
-          "for": "5m",
-          "isPaused": false
-        },
-        {{ end }}
-        {{ if eq (env "ALERT_BACKUP_NOT_SUCCESSFULL_ENABLED") "true" }}
-        {
-            "uid": "de8e6bc92a8lcc",
-            "title": "Backup Not Successfull",
-            "condition": "B",
-            "data": [
-                {
-                    "refId": "A",
-                    "relativeTimeRange": {
-                        "from": 60,
-                        "to": 0
-                    },
-                    "datasourceUid": "PBFA97CFB590B2093",
-                    "model": {
-                        "disableTextWrap": false,
-                        "editorMode": "builder",
-                        "expr": "backup",
-                        "fullMetaSearch": false,
-                        "includeNullMetadata": true,
-                        "instant": true,
-                        "intervalMs": 1000,
-                        "legendFormat": "__auto",
-                        "maxDataPoints": 43200,
-                        "range": false,
-                        "refId": "A",
-                        "useBackend": false
-                    }
-                },
-                {
-                    "refId": "B",
-                    "relativeTimeRange": {
-                        "from": 60,
-                        "to": 0
-                    },
-                    "datasourceUid": "__expr__",
-                    "model": {
-                        "conditions": [
-                            {
-                                "evaluator": {
-                                    "params": [
-                                        0
-                                    ],
-                                    "type": "gt"
-                                },
-                                "operator": {
-                                    "type": "and"
-                                },
-                                "query": {
-                                    "params": [
-                                        "C"
-                                    ]
-                                },
-                                "reducer": {
-                                    "params": [],
-                                    "type": "last"
-                                },
-                                "type": "query"
-                            }
-                        ],
-                        "datasource": {
-                            "type": "__expr__",
-                            "uid": "__expr__"
-                        },
-                        "expression": "A",
-                        "intervalMs": 1000,
-                        "maxDataPoints": 43200,
-                        "refId": "B",
-                        "type": "threshold"
-                    }
-                }
-            ],
-            "noDataState": "NoData",
-            "execErrState": "Error",
-            "for": "20m",
-            "annotations": {
-                "summary": "Backup did not finish within 20 minutes"
-            },
-            "labels": {},
-            "isPaused": false
-        }
-        {{ end }}
-      ]
-    },
-    {
-      "orgId": 1,
-      "name": "node",
-      "folder": "node",
-      "interval": "5m",
-      "rules": [
-        {{ if eq (env "ALERT_NODE_DISK_SPACE_ENABLED") "true" }}
-        {
-          "uid": "bds8bhxu97pxca",
-          "title": "Node Disk Space",
-          "condition": "C",
-          "data": [
-            {
-              "refId": "A",
-              "relativeTimeRange": { "from": 600, "to": 0 },
-              "datasourceUid": "PBFA97CFB590B2093",
-              "model": {
-                "editorMode": "code",
-                "expr": "(node_filesystem_free_bytes{fstype=\"ext4\",mountpoint=~\"(/$)|(/media.*)\"} / node_filesystem_size_bytes{fstype=\"ext4\",mountpoint=~\"(/$)|(/media.*)\"}) * 100",
-                "instant": true,
-                "intervalMs": 1000,
-                "legendFormat": "__auto",
-                "maxDataPoints": 43200,
-                "range": false,
-                "refId": "A"
-              }
-            },
-            {
-              "refId": "C",
-              "relativeTimeRange": { "from": 600, "to": 0 },
-              "datasourceUid": "__expr__",
-              "model": {
-                "conditions": [
-                  {
-                    "evaluator": { "params": [10], "type": "lt" },
-                    "operator": { "type": "and" },
-                    "query": { "params": ["C"] },
-                    "reducer": { "params": [], "type": "last" },
-                    "type": "query"
-                  }
-                ],
-                "datasource": { "type": "__expr__", "uid": "__expr__" },
-                "expression": "A",
-                "intervalMs": 1000,
-                "maxDataPoints": 43200,
-                "refId": "C",
-                "type": "threshold"
-              }
-            }
-          ],
-          "noDataState": "NoData",
-          "execErrState": "Error",
-          "for": "5m",
-          "annotations": {},
-          "labels": {},
-          "isPaused": false
-        },
-        {{ end }}
-        {{ if eq (env "ALERT_NODE_MEMORY_USAGE_ENABLED") "true" }}
-        {
-          "uid": "ads8cswmly96oa",
-          "title": "Node Memory Usage",
-          "condition": "C",
-          "data": [
-            {
-              "refId": "A",
-              "relativeTimeRange": { "from": 600, "to": 0 },
-              "datasourceUid": "PBFA97CFB590B2093",
-              "model": {
-                "editorMode": "code",
-                "expr": "(node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100",
-                "instant": true,
-                "intervalMs": 1000,
-                "legendFormat": "__auto",
-                "maxDataPoints": 43200,
-                "range": false,
-                "refId": "A"
-              }
-            },
-            {
-              "refId": "C",
-              "relativeTimeRange": { "from": 600, "to": 0 },
-              "datasourceUid": "__expr__",
-              "model": {
-                "conditions": [
-                  {
-                    "evaluator": { "params": [90], "type": "gt" },
-                    "operator": { "type": "and" },
-                    "query": { "params": ["C"] },
-                    "reducer": { "params": [], "type": "last" },
-                    "type": "query"
-                  }
-                ],
-                "datasource": { "type": "__expr__", "uid": "__expr__" },
-                "expression": "A",
-                "intervalMs": 1000,
-                "maxDataPoints": 43200,
-                "refId": "C",
-                "type": "threshold"
-              }
-            }
-          ],
-          "noDataState": "NoData",
-          "execErrState": "Error",
-          "for": "5m",
-          "annotations": {},
-          "labels": {},
-          "isPaused": false
-        }
-        {{ end }}
-      ]
-    }
-  ]
-}
-
@@ -110,7 +110,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "editorMode": "code",
-          "expr": "(time() - min(container_start_time_seconds{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}))",
+          "expr": "(time() - min(container_start_time_seconds{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "",
@@ -215,7 +215,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "editorMode": "code",
-          "expr": "sort(\n    sum(\n        rate(\n            container_cpu_usage_seconds_total{\n                container_label_com_docker_stack_namespace=~\"$stack\", \n                instance=~\"cadvisor.monitoring.$instance.*\"}[$interval]\n            )\n        ) \n    by (container_label_com_docker_swarm_service_name)\n)",
+          "expr": "sort(\n    sum(\n        rate(\n            container_cpu_usage_seconds_total{\n                container_label_com_docker_stack_namespace=~\"$stack\", \n                instance=~\".*$node_id.*\"}[$interval]\n            )\n        ) \n    by (container_label_com_docker_swarm_service_name)\n)",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "{{ container_label_com_docker_swarm_service_name }}",
@@ -283,7 +283,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "editorMode": "code",
-          "expr": "count(rate(container_last_seen{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}[$interval]))",
+          "expr": "count(rate(container_last_seen{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}[$interval]))",
          "format": "time_series",
          "intervalFactor": 2,
          "range": true,
@@ -386,7 +386,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "editorMode": "code",
-          "expr": "sum(\n    container_memory_rss{\n        container_label_com_docker_stack_namespace=~\"$stack\", \n        instance=~\"cadvisor.monitoring.$instance.*\"}\n    ) \nby (container_label_com_docker_swarm_service_name)",
+          "expr": "sum(\n    container_memory_rss{\n        container_label_com_docker_stack_namespace=~\"$stack\", \n        instance=~\".*$node_id.*\"}\n    ) \nby (container_label_com_docker_swarm_service_name)",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 2,
@@ -493,7 +493,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "editorMode": "code",
-          "expr": "sum(rate(container_network_receive_bytes_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
+          "expr": "sum(rate(container_network_receive_bytes_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "RX: {{ container_label_com_docker_swarm_service_name }}",
@@ -507,7 +507,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "editorMode": "code",
-          "expr": "sum(rate(container_network_transmit_bytes_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}[$interval])) by (container_label_com_docker_swarm_service_name) * -1",
+          "expr": "sum(rate(container_network_transmit_bytes_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}[$interval])) by (container_label_com_docker_swarm_service_name) * -1",
          "hide": false,
          "legendFormat": "TX: {{container_label_com_docker_swarm_service_name}}",
          "range": true,
@@ -634,7 +634,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "editorMode": "code",
-          "expr": "sum(container_fs_usage_bytes{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}) by (container_label_com_docker_swarm_service_name)",
+          "expr": "sum(container_fs_usage_bytes{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}) by (container_label_com_docker_swarm_service_name)",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 2,
@@ -742,7 +742,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "editorMode": "code",
-          "expr": "sum(rate(container_fs_usage_bytes{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
+          "expr": "sum(rate(container_fs_usage_bytes{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 2,
@@ -865,7 +865,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "editorMode": "code",
-          "expr": "sum(rate(container_fs_io_time_seconds_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
+          "expr": "sum(rate(container_fs_io_time_seconds_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 2,
@@ -973,7 +973,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "editorMode": "code",
-          "expr": "container_fs_io_current{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}",
+          "expr": "container_fs_io_current{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 2,
@@ -1081,7 +1081,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "editorMode": "code",
-          "expr": "sum(rate(container_fs_reads_bytes_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
+          "expr": "sum(rate(container_fs_reads_bytes_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "{{ container_label_com_docker_swarm_service_name }}",
@@ -1188,7 +1188,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "editorMode": "code",
-          "expr": "sum(rate(container_fs_read_seconds_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
+          "expr": "sum(rate(container_fs_read_seconds_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "{{ container_label_com_docker_swarm_service_name }}",
@@ -1295,7 +1295,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "editorMode": "code",
-          "expr": "sum(rate(container_fs_writes_bytes_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
+          "expr": "sum(rate(container_fs_writes_bytes_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "{{ container_label_com_docker_swarm_service_name }}",
@@ -1402,7 +1402,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "editorMode": "code",
-          "expr": "sum(rate(container_fs_write_seconds_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
+          "expr": "sum(rate(container_fs_write_seconds_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "{{ container_label_com_docker_swarm_service_name }}",
@@ -1453,7 +1453,7 @@
          "query": "label_values(container_label_com_docker_stack_namespace)",
          "refId": "PrometheusVariableQueryEditor-VariableQuery"
        },
-        "refresh": 1,
+        "refresh": 2,
        "regex": "",
        "skipUrlSync": false,
        "sort": 2,
@@ -1567,19 +1567,19 @@
          "type": "prometheus",
          "uid": "PBFA97CFB590B2093"
        },
-        "definition": "label_values(instance)",
+        "definition": "label_values(node_uname_info, instance)",
        "hide": 0,
        "includeAll": true,
-        "label": "instance",
+        "label": "Swarm Node",
        "multi": true,
-        "name": "instance",
+        "name": "node_id",
        "options": [],
        "query": {
-          "query": "label_values(instance)",
+          "query": "label_values(node_uname_info, instance)",
          "refId": "PrometheusVariableQueryEditor-VariableQuery"
        },
-        "refresh": 1,
-        "regex": "/.*cadvisor.monitoring.(?<instance>.*):80/",
+        "refresh": 2,
+        "regex": "/^(?:(?:node|cadvisor|monitoring).)*(.*)$/",
        "skipUrlSync": false,
        "sort": 0,
        "type": "query"
@@ -1620,4 +1620,4 @@
  "uid": "KdVoGQm7z",
  "version": 36,
  "weekStart": ""
-}
+}
@@ -93,7 +93,6 @@
      },
      "hideTimeOverride": true,
      "id": 2,
-      "links": [],
      "maxDataPoints": 100,
      "options": {
        "colorMode": "value",
@@ -107,17 +106,19 @@
          "fields": "",
          "values": false
        },
+        "showPercentChange": false,
        "text": {},
-        "textMode": "auto"
+        "textMode": "auto",
+        "wideLayout": true
      },
-      "pluginVersion": "10.0.2",
+      "pluginVersion": "10.4.14",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "PBFA97CFB590B2093"
          },
-          "expr": "topk(1, sum((node_time_seconds - node_boot_time_seconds) * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"}) by (node_name))",
+          "expr": "topk(1, sum(label_replace(node_time_seconds{instance=~\".*$node_id.*\"} - node_boot_time_seconds{instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) by (instance))",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "",
@@ -172,7 +173,6 @@
        "y": 0
      },
      "id": 1,
-      "links": [],
      "maxDataPoints": 100,
      "options": {
        "colorMode": "value",
@@ -186,17 +186,19 @@
          "fields": "",
          "values": false
        },
+        "showPercentChange": false,
        "text": {},
-        "textMode": "auto"
+        "textMode": "auto",
+        "wideLayout": true
      },
-      "pluginVersion": "10.0.2",
+      "pluginVersion": "10.4.14",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "PBFA97CFB590B2093"
          },
-          "expr": "count(node_meta * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"})",
+          "expr": "count(label_replace(node_uname_info{instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\"))",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "",
@@ -251,7 +253,6 @@
      },
      "hideTimeOverride": true,
      "id": 4,
-      "links": [],
      "maxDataPoints": 100,
      "options": {
        "colorMode": "value",
@@ -265,17 +266,19 @@
          "fields": "",
          "values": false
        },
+        "showPercentChange": false,
        "text": {},
-        "textMode": "auto"
+        "textMode": "auto",
+        "wideLayout": true
      },
-      "pluginVersion": "10.0.2",
+      "pluginVersion": "10.4.14",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "PBFA97CFB590B2093"
          },
-          "expr": "count(node_cpu_seconds_total{mode=\"idle\"} * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"})",
+          "expr": "count(label_replace(node_cpu_seconds_total{mode=\"idle\", instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\"))",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "",
@@ -335,9 +338,10 @@
        "y": 0
      },
      "id": 8,
-      "links": [],
      "maxDataPoints": 100,
      "options": {
+        "minVizHeight": 75,
+        "minVizWidth": 75,
        "orientation": "horizontal",
        "reduceOptions": {
          "calcs": [
@@ -348,9 +352,10 @@
        },
        "showThresholdLabels": false,
        "showThresholdMarkers": true,
+        "sizing": "auto",
        "text": {}
      },
-      "pluginVersion": "10.0.2",
+      "pluginVersion": "10.4.14",
      "targets": [
        {
          "datasource": {
@@ -358,7 +363,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "editorMode": "code",
-          "expr": "sum((node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"} * 100) / count(node_meta * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"})",
+          "expr": "sum(label_replace((node_memory_MemAvailable_bytes{instance=~\".*$node_id.*\"} / node_memory_MemTotal_bytes{instance=~\".*$node_id.*\"}) * 100, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) / count(label_replace(node_uname_info{instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\"))",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "",
@@ -405,13 +410,12 @@
      },
      "lines": true,
      "linewidth": 1,
-      "links": [],
      "nullPointMode": "null",
      "options": {
        "alertThreshold": true
      },
      "percentage": false,
-      "pluginVersion": "10.0.2",
+      "pluginVersion": "10.4.14",
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
@@ -425,11 +429,11 @@
            "type": "prometheus",
            "uid": "PBFA97CFB590B2093"
          },
-          "expr": "node_load5 * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"}",
+          "expr": "label_replace(node_load5{instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 2,
-          "legendFormat": "load5 {{node_name}}",
+          "legendFormat": "load5 {{instance}}",
          "refId": "A",
          "step": 2
        }
@@ -507,7 +511,6 @@
      },
      "hideTimeOverride": true,
      "id": 3,
-      "links": [],
      "maxDataPoints": 100,
      "options": {
        "colorMode": "value",
@@ -521,17 +524,19 @@
          "fields": "",
          "values": false
        },
+        "showPercentChange": false,
        "text": {},
-        "textMode": "auto"
+        "textMode": "auto",
+        "wideLayout": true
      },
-      "pluginVersion": "10.0.2",
+      "pluginVersion": "10.4.14",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "PBFA97CFB590B2093"
          },
-          "expr": "sum(node_memory_MemTotal_bytes * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"})",
+          "expr": "sum(label_replace(node_memory_MemTotal_bytes{instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\"))",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "",
@@ -585,7 +590,6 @@
      },
      "hideTimeOverride": true,
      "id": 9,
-      "links": [],
      "maxDataPoints": 100,
      "options": {
        "colorMode": "value",
@@ -599,10 +603,12 @@
          "fields": "",
          "values": false
        },
+        "showPercentChange": false,
        "text": {},
-        "textMode": "auto"
+        "textMode": "auto",
+        "wideLayout": true
      },
-      "pluginVersion": "10.0.2",
+      "pluginVersion": "10.4.14",
      "targets": [
        {
          "datasource": {
@@ -610,7 +616,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "exemplar": true,
-          "expr": "sum(node_filesystem_size_bytes{fstype=\"ext4\",mountpoint=~\"(/$)|(/media.*)\"} * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"})",
+          "expr": "sum(label_replace(node_filesystem_size_bytes{fstype=~\"ext4|xfs\",mountpoint=~\"(/$)|(/media.*)\", instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\"))",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 2,
@@ -671,9 +677,10 @@
      },
      "hideTimeOverride": true,
      "id": 11,
-      "links": [],
      "maxDataPoints": 100,
      "options": {
+        "minVizHeight": 75,
+        "minVizWidth": 75,
        "orientation": "horizontal",
        "reduceOptions": {
          "calcs": [
@@ -684,16 +691,17 @@
        },
        "showThresholdLabels": false,
        "showThresholdMarkers": true,
+        "sizing": "auto",
        "text": {}
      },
-      "pluginVersion": "10.0.2",
+      "pluginVersion": "10.4.14",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "PBFA97CFB590B2093"
          },
-          "expr": "sum(irate(node_cpu_seconds_total{mode=\"idle\"}[$interval]) * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"}) * 100 / count(node_cpu_seconds_total{mode=\"user\"} * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"}) ",
+          "expr": "sum(label_replace(irate(node_cpu_seconds_total{mode=\"idle\", instance=~\".*$node_id.*\"}[$interval]), \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) * 100 / count(label_replace(node_cpu_seconds_total{mode=\"user\", instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\"))",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "",
@@ -713,7 +721,39 @@
      "fieldConfig": {
        "defaults": {
          "color": {
-            "mode": "thresholds"
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisBorderShow": false,
+            "axisCenteredZero": false,
+            "axisColorMode": "text",
+            "axisLabel": "",
+            "axisPlacement": "left",
+            "barAlignment": 0,
+            "drawStyle": "line",
+            "fillOpacity": 0,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "insertNulls": false,
+            "lineInterpolation": "linear",
+            "lineWidth": 1,
+            "pointSize": 5,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "auto",
+            "spanNulls": false,
+            "stacking": {
+              "group": "A",
+              "mode": "none"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
          },
          "mappings": [
            {
@@ -747,33 +787,42 @@
          },
          "unit": "percent"
        },
-        "overrides": []
+        "overrides": [
+          {
+            "matcher": {
+              "id": "byType",
+              "options": "time"
+            },
+            "properties": [
+              {
+                "id": "custom.axisPlacement",
+                "value": "hidden"
+              }
+            ]
+          }
+        ]
      },
      "gridPos": {
-        "h": 4,
-        "w": 2.6666666666666665,
+        "h": 6,
+        "w": 6,
        "x": 0,
        "y": 8
      },
      "id": 10,
-      "links": [],
      "maxDataPoints": 100,
-      "maxPerRow": 12,
      "options": {
-        "colorMode": "value",
-        "graphMode": "area",
-        "justifyMode": "auto",
-        "orientation": "horizontal",
-        "reduceOptions": {
-          "calcs": [
-            "last"
-          ],
-          "fields": "",
-          "values": false
+        "legend": {
+          "calcs": [],
+          "displayMode": "list",
+          "placement": "bottom",
+          "showLegend": true
        },
-        "textMode": "auto"
+        "tooltip": {
+          "mode": "single",
+          "sort": "none"
+        }
      },
-      "pluginVersion": "10.0.2",
+      "pluginVersion": "10.4.14",
      "repeat": "node_id",
      "repeatDirection": "h",
      "targets": [
@@ -782,18 +831,20 @@
            "type": "prometheus",
            "uid": "PBFA97CFB590B2093"
          },
+          "editorMode": "code",
          "exemplar": true,
-          "expr": "sum((node_filesystem_free_bytes{fstype=\"ext4\",mountpoint=~\"(/$)|(/media.*)\"} / node_filesystem_size_bytes{fstype=\"ext4\",mountpoint=~\"(/$)|(/media.*)\"}) * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"} * 100) / count(node_meta * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"})",
+          "expr": "label_replace(node_filesystem_free_bytes{fstype=~\"ext4|xfs\", instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\") / label_replace(node_filesystem_size_bytes{fstype=~\"ext4|xfs\", instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\") * 100",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 2,
-          "legendFormat": "",
+          "legendFormat": "{{mountpoint}}",
+          "range": true,
          "refId": "A",
          "step": 20
        }
      ],
      "title": "Available Disk Space $node_id",
-      "type": "stat"
+      "type": "timeseries"
    },
    {
      "aliasColors": {},
@@ -811,7 +862,7 @@
        "h": 7,
        "w": 24,
        "x": 0,
-        "y": 12
+        "y": 14
      },
      "hiddenSeries": false,
      "id": 14,
@@ -830,13 +881,12 @@
      },
      "lines": true,
      "linewidth": 1,
-      "links": [],
      "nullPointMode": "null as zero",
      "options": {
        "alertThreshold": true
      },
      "percentage": false,
-      "pluginVersion": "10.0.2",
+      "pluginVersion": "10.4.14",
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
@@ -850,10 +900,10 @@
            "type": "prometheus",
            "uid": "PBFA97CFB590B2093"
          },
-          "expr": "100 - (avg(irate(node_cpu_seconds_total{mode=\"idle\"}[$interval])  * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"} * 100) by (node_name))",
+          "expr": "100 - (avg(label_replace(irate(node_cpu_seconds_total{mode=\"idle\", instance=~\".*$node_id.*\"}[$interval]), \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\") * 100) by (instance))",
          "format": "time_series",
          "intervalFactor": 2,
-          "legendFormat": "{{node_name}}",
+          "legendFormat": "{{instance}}",
          "refId": "A",
          "step": 2
        }
@@ -900,6 +950,7 @@
            "mode": "palette-classic"
          },
          "custom": {
+            "axisBorderShow": false,
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
@@ -913,6 +964,7 @@
              "tooltip": false,
              "viz": false
            },
+            "insertNulls": false,
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
@@ -961,12 +1013,11 @@
      },
      "gridPos": {
        "h": 7,
-        "w": 2.6666666666666665,
+        "w": 6,
        "x": 0,
-        "y": 19
+        "y": 21
      },
      "id": 15,
-      "links": [],
      "maxPerRow": 12,
      "options": {
        "legend": {
@@ -994,12 +1045,12 @@
            "uid": "PBFA97CFB590B2093"
          },
          "editorMode": "code",
-          "expr": "sum((node_memory_MemFree_bytes) * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"}) by (node_name)",
+          "expr": "sum(label_replace(node_memory_MemFree_bytes{instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) by (instance)",
          "format": "time_series",
          "hide": false,
          "interval": "",
          "intervalFactor": 2,
-          "legendFormat": "Free  {{node_name}}",
+          "legendFormat": "Free  {{instance}}",
          "range": true,
          "refId": "free",
          "step": 2
@@ -1010,12 +1061,12 @@
            "uid": "PBFA97CFB590B2093"
          },
          "editorMode": "code",
-          "expr": "sum((node_memory_Cached_bytes + node_memory_Buffers_bytes + node_memory_Slab_bytes) * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"}) by (node_name)",
+          "expr": "sum(label_replace(node_memory_Cached_bytes{instance=~\".*$node_id.*\"} + node_memory_Buffers_bytes{instance=~\".*$node_id.*\"} + node_memory_Slab_bytes{instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) by (instance)",
          "format": "time_series",
          "hide": false,
          "interval": "",
          "intervalFactor": 2,
-          "legendFormat": "cache,buffer,slab  {{node_name}}",
+          "legendFormat": "cache,buffer,slab  {{instance}}",
          "range": true,
          "refId": "cache,buffer,slab",
          "step": 2
@@ -1026,12 +1077,12 @@
            "uid": "PBFA97CFB590B2093"
          },
          "editorMode": "code",
-          "expr": "sum((node_memory_MemTotal_bytes - node_memory_MemFree_bytes - node_memory_Cached_bytes - node_memory_Buffers_bytes - node_memory_Slab_bytes) * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"}) by (node_name)",
+          "expr": "sum(label_replace(node_memory_MemTotal_bytes{instance=~\".*$node_id.*\"} - node_memory_MemFree_bytes{instance=~\".*$node_id.*\"} - node_memory_Cached_bytes{instance=~\".*$node_id.*\"} - node_memory_Buffers_bytes{instance=~\".*$node_id.*\"} - node_memory_Slab_bytes{instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) by (instance)",
          "format": "time_series",
          "hide": false,
          "interval": "",
          "intervalFactor": 2,
-          "legendFormat": "Used {{node_name}}",
+          "legendFormat": "Used {{instance}}",
          "range": true,
          "refId": "Used",
          "step": 2
@@ -1043,13 +1094,13 @@
          },
          "editorMode": "code",
          "exemplar": false,
-          "expr": "sum((node_memory_MemTotal_bytes) * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"}) by (node_name)",
+          "expr": "sum(label_replace(node_memory_MemTotal_bytes{instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) by (instance)",
          "format": "time_series",
          "hide": false,
          "instant": false,
          "interval": "",
          "intervalFactor": 2,
-          "legendFormat": "Total {{node_name}}",
+          "legendFormat": "Total {{instance}}",
          "range": true,
          "refId": "total",
          "step": 2
@@ -1074,7 +1125,7 @@
        "h": 7,
        "w": 24,
        "x": 0,
-        "y": 26
+        "y": 28
      },
      "hiddenSeries": false,
      "id": 16,
@@ -1091,13 +1142,12 @@
      },
      "lines": true,
      "linewidth": 1,
-      "links": [],
      "nullPointMode": "null as zero",
      "options": {
        "alertThreshold": true
      },
      "percentage": false,
-      "pluginVersion": "10.0.2",
+      "pluginVersion": "10.4.14",
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
@@ -1111,11 +1161,11 @@
            "type": "prometheus",
            "uid": "PBFA97CFB590B2093"
          },
-          "expr": "sum(irate(node_disk_read_bytes_total[$interval]) * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"}) by (node_name)",
+          "expr": "sum(label_replace(irate(node_disk_read_bytes_total{instance=~\".*$node_id.*\"}[$interval]), \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) by (instance)",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 2,
-          "legendFormat": "Read {{node_name}}",
+          "legendFormat": "Read {{instance}}",
          "refId": "A",
          "step": 2
        },
@@ -1124,10 +1174,10 @@
            "type": "prometheus",
            "uid": "PBFA97CFB590B2093"
          },
-          "expr": "sum(irate(node_disk_written_bytes_total[$interval]) * on(instance) group_left(node_name) node_meta{node_id=~\"$node_id\"}) by (node_name)",
+          "expr": "sum(label_replace(irate(node_disk_written_bytes_total{instance=~\".*$node_id.*\"}[$interval]), \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) by (instance)",
          "format": "time_series",
          "intervalFactor": 2,
-          "legendFormat": "Written {{node_name}}",
+          "legendFormat": "Written {{instance}}",
          "refId": "B",
          "step": 2
        }
@@ -1178,7 +1228,7 @@
        "h": 7,
        "w": 12,
        "x": 0,
-        "y": 33
+        "y": 35
      },
      "hiddenSeries": false,
      "id": 18,
@@ -1195,7 +1245,6 @@
      },
      "lines": true,
      "linewidth": 1,
-      "links": [],
      "nullPointMode": "null as zero",
      "options": {
        "alertThreshold": true
@@ -1215,10 +1264,10 @@
            "type": "prometheus",
            "uid": "PBFA97CFB590B2093"
          },
-          "expr": "sum(irate(node_disk_reads_completed_total[$interval]) * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"}) by (node_name)",
+          "expr": "sum(label_replace(irate(node_disk_reads_completed_total{instance=~\".*$node_id.*\"}[$interval]), \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) by (instance)",
          "format": "time_series",
          "intervalFactor": 2,
-          "legendFormat": "Reads {{node_name}}",
+          "legendFormat": "Reads {{instance}}",
          "refId": "A",
          "step": 2
        },
@@ -1227,10 +1276,10 @@
            "type": "prometheus",
            "uid": "PBFA97CFB590B2093"
          },
-          "expr": "sum(irate(node_disk_writes_completed_total[$interval]) * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"}) by (node_name)",
+          "expr": "sum(label_replace(irate(node_disk_writes_completed_total{instance=~\".*$node_id.*\"}[$interval]), \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) by (instance)",
          "format": "time_series",
          "intervalFactor": 2,
-          "legendFormat": "Writes {{node_name}}",
+          "legendFormat": "Writes {{instance}}",
          "refId": "B",
          "step": 2
        }
@@ -1281,7 +1330,7 @@
        "h": 7,
        "w": 12,
        "x": 12,
-        "y": 33
+        "y": 35
      },
      "hiddenSeries": false,
      "id": 19,
@@ -1300,7 +1349,6 @@
      },
      "lines": true,
      "linewidth": 1,
-      "links": [],
      "nullPointMode": "null as zero",
      "options": {
        "alertThreshold": true
@@ -1320,10 +1368,10 @@
            "type": "prometheus",
            "uid": "PBFA97CFB590B2093"
          },
-          "expr": "(avg(irate(node_cpu_seconds_total{mode=\"iowait\"}[$interval])  * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"} * 100) by (node_name))",
+          "expr": "(avg(label_replace(irate(node_cpu_seconds_total{mode=\"iowait\", instance=~\".*$node_id.*\"}[$interval]), \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\") * 100) by (instance))",
          "format": "time_series",
          "intervalFactor": 2,
-          "legendFormat": "{{node_name}}",
+          "legendFormat": "{{instance}}",
          "refId": "A",
          "step": 2
        }
@@ -1376,7 +1424,7 @@
        "h": 7,
        "w": 18,
        "x": 0,
-        "y": 40
+        "y": 42
      },
      "hiddenSeries": false,
      "id": 12,
@@ -1397,7 +1445,6 @@
      },
      "lines": true,
      "linewidth": 1,
-      "links": [],
      "nullPointMode": "null",
      "options": {
        "alertThreshold": true
@@ -1417,7 +1464,7 @@
            "type": "prometheus",
            "uid": "PBFA97CFB590B2093"
          },
-          "expr": "sum(rate(container_last_seen[5m]) * on(container_label_com_docker_swarm_node_id) group_left(node_name) node_meta{node_name=~\"${node_id}\"}) by (container_label_com_docker_swarm_service_name)",
+          "expr": "sum(rate(container_last_seen{instance=~\".*$node_id.*\"}[5m])) by (container_label_com_docker_swarm_service_name)",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 10,
@@ -1499,10 +1546,9 @@
        "h": 7,
        "w": 6,
        "x": 18,
-        "y": 40
+        "y": 42
      },
      "id": 7,
-      "links": [],
      "maxDataPoints": 100,
      "options": {
        "colorMode": "none",
@@ -1527,7 +1573,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "exemplar": true,
-          "expr": "count(rate(container_last_seen[5m]) * on(container_label_com_docker_swarm_node_id) group_left(node_name) node_meta{node_name=~\"$node_id\"})",
+          "expr": "count(rate(container_last_seen{instance=~\".*$node_id.*\"}[5m]))",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 2,
@@ -1600,10 +1646,9 @@
        "h": 7,
        "w": 24,
        "x": 0,
-        "y": 47
+        "y": 49
      },
      "id": 17,
-      "links": [],
      "options": {
        "legend": {
          "calcs": [],
@@ -1623,11 +1668,11 @@
            "type": "prometheus",
            "uid": "PBFA97CFB590B2093"
          },
-          "expr": "sum(rate(container_network_receive_bytes_total[$interval]) * on(container_label_com_docker_swarm_node_id) group_left(node_name) node_meta{node_name=~\"${node_id}\"}) by (node_name)",
+          "expr": "sum(label_replace(rate(container_network_receive_bytes_total{instance=~\".*$node_id.*\"}[$interval]), \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) by (instance)",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 2,
-          "legendFormat": "IN {{node_name}}",
+          "legendFormat": "IN {{instance}}",
          "refId": "A",
          "step": 2
        },
@@ -1636,11 +1681,11 @@
            "type": "prometheus",
            "uid": "PBFA97CFB590B2093"
          },
-          "expr": "- sum(rate(container_network_transmit_bytes_total[$interval]) * on(container_label_com_docker_swarm_node_id) group_left(node_name) node_meta{node_name=~\"${node_id}\"}) by (node_name)",
+          "expr": "- sum(label_replace(rate(container_network_transmit_bytes_total{instance=~\".*$node_id.*\"}[$interval]), \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) by (instance)",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 2,
-          "legendFormat": "OUT {{node_name}}",
+          "legendFormat": "OUT {{instance}}",
          "metric": "",
          "refId": "B",
          "step": 2
@@ -1658,7 +1703,7 @@
        "h": 9,
        "w": 24,
        "x": 0,
-        "y": 54
+        "y": 56
      },
      "id": 30,
      "options": {
@@ -1688,8 +1733,7 @@
    }
  ],
  "refresh": "",
-  "schemaVersion": 38,
-  "style": "dark",
+  "schemaVersion": 39,
  "tags": [
    "swarmprom",
    "prometheus",
@@ -1718,11 +1762,11 @@
        "name": "node_id",
        "options": [],
        "query": {
-          "query": "node_meta",
+          "query": "label_values(node_uname_info, instance)",
          "refId": "Prometheus-node_id-Variable-Query"
        },
-        "refresh": 1,
-        "regex": "/node_name=\"([^\"]+)\"/",
+        "refresh": 2,
+        "regex": "/^(?:(?:node|cadvisor|monitoring).)*(.*)$/",
        "skipUrlSync": false,
        "sort": 0,
        "type": "query"
@@ -1836,6 +1880,6 @@
  "timezone": "",
  "title": "Docker Swarm Nodes",
  "uid": "BPlb-Sgik",
-  "version": 24,
+  "version": 7,
  "weekStart": ""
-}
+}
@@ -116,12 +116,12 @@
            "uid": "PBFA97CFB590B2093"
          },
          "exemplar": false,
-          "expr": "time() - process_start_time_seconds{job=\"$job\"}",
+          "expr": "time() - label_replace(process_start_time_seconds{instance=~\".*$domain.*\"} or process_start_time_seconds{domain=\"$domain\"}, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)",
          "format": "time_series",
          "instant": true,
          "interval": "",
          "intervalFactor": 2,
-          "legendFormat": "{{ instance }}",
+          "legendFormat": "{{ domain }}",
          "refId": "A"
        }
      ],
@@ -200,7 +200,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "exemplar": true,
-          "expr": "sum(increase(traefik_service_requests_total{code=\"499\", instance=\"$instance\"}[$interval]))",
+          "expr": "sum(label_replace(increase(traefik_service_requests_total{code=\"499\", instance=~\".*$domain.*\"}[$interval]) or increase(traefik_service_requests_total{code=\"499\", domain=\"$domain\"}[$interval]), \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`))",
          "interval": "",
          "legendFormat": "",
          "refId": "A"
@@ -280,7 +280,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "exemplar": true,
-          "expr": "sum(increase(traefik_service_requests_total{code=\"404\",protocol=~\"$protocol\", instance=\"$instance\"}[$interval]))",
+          "expr": "sum(label_replace(increase(traefik_service_requests_total{code=\"404\",protocol=~\"$protocol\", instance=~\".*$domain.*\"}[$interval]) or increase(traefik_service_requests_total{code=\"404\",protocol=~\"$protocol\", domain=\"$domain\"}[$interval]), \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`))",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 2,
@@ -365,7 +365,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "exemplar": true,
-          "expr": "sum(increase(traefik_service_requests_total{code=\"200\",protocol=~\"$protocol\", instance=\"$instance\"}[$interval]))",
+          "expr": "sum(label_replace(increase(traefik_service_requests_total{code=\"200\",protocol=~\"$protocol\", instance=~\".*$domain.*\"}[$interval]) or increase(traefik_service_requests_total{code=\"200\",protocol=~\"$protocol\", domain=\"$domain\"}[$interval]), \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`))",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 2,
@@ -417,7 +417,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "exemplar": true,
-          "expr": "topk(5, sum(traefik_service_requests_total{protocol=~\"$protocol\", instance=\"$instance\"}) by (code))",
+          "expr": "topk(5, sum(label_replace(traefik_service_requests_total{protocol=~\"$protocol\", instance=~\".*$domain.*\"} or traefik_service_requests_total{protocol=~\"$protocol\", domain=\"$domain\"}, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) by (code))",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 2,
@@ -489,7 +489,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "exemplar": true,
-          "expr": "sum(increase(traefik_service_requests_total{code=\"499\",method=\"GET\", instance=\"$instance\"}[$interval])) by (service)",
+          "expr": "sum(label_replace(increase(traefik_service_requests_total{code=\"499\",method=\"GET\", instance=~\".*$domain.*\"}[$interval]) or increase(traefik_service_requests_total{code=\"499\",method=\"GET\", domain=\"$domain\"}[$interval]), \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) by (service)",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 2,
@@ -590,7 +590,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "exemplar": true,
-          "expr": "(sum(traefik_entrypoint_request_duration_seconds_sum{protocol=~\"$protocol\", instance=\"$instance\"}) / sum(traefik_entrypoint_requests_total{protocol=~\"$protocol\", instance=\"$instance\"}) * 1000) - (sum(traefik_service_request_duration_seconds_sum{protocol=~\"$protocol\", instance=\"$instance\"}) / sum(traefik_service_requests_total{protocol=~\"$protocol\", instance=\"$instance\"}) * 1000)",
+          "expr": "(sum(label_replace(traefik_entrypoint_request_duration_seconds_sum{protocol=~\"$protocol\", instance=~\".*$domain.*\"} or traefik_entrypoint_request_duration_seconds_sum{protocol=~\"$protocol\", domain=\"$domain\"}, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) / sum(label_replace(traefik_entrypoint_requests_total{protocol=~\"$protocol\", instance=~\".*$domain.*\"} or traefik_entrypoint_requests_total{protocol=~\"$protocol\", domain=\"$domain\"}, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) * 1000) - (sum(label_replace(traefik_service_request_duration_seconds_sum{protocol=~\"$protocol\", instance=~\".*$domain.*\"} or traefik_service_request_duration_seconds_sum{protocol=~\"$protocol\", domain=\"$domain\"}, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) / sum(label_replace(traefik_service_requests_total{protocol=~\"$protocol\", instance=~\".*$domain.*\"} or traefik_service_requests_total{protocol=~\"$protocol\", domain=\"$domain\"}, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) * 1000)",
          "format": "time_series",
          "instant": false,
          "interval": "",
@@ -709,7 +709,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "exemplar": true,
-          "expr": "sum(traefik_entrypoint_request_duration_seconds_sum{instance=\"$instance\"}) / sum(traefik_entrypoint_requests_total{instance=\"$instance\"}) * 1000",
+          "expr": "sum(label_replace(traefik_entrypoint_request_duration_seconds_sum{instance=~\".*$domain.*\"} or traefik_entrypoint_request_duration_seconds_sum{domain=\"$domain\"}, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) / sum(label_replace(traefik_entrypoint_requests_total{instance=~\".*$domain.*\"} or traefik_entrypoint_requests_total{domain=\"$domain\"}, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) * 1000",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 2,
@@ -781,7 +781,7 @@
          },
          "editorMode": "code",
          "exemplar": true,
-          "expr": "sum(delta(traefik_service_requests_total{instance=\"${instance:raw}\"}[$interval]))",
+          "expr": "sum(label_replace(delta(traefik_service_requests_total{instance=~\".*$domain.*\"}[$interval]) or delta(traefik_service_requests_total{domain=\"$domain\"}[$interval]), \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`))",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 2,
@@ -879,7 +879,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "exemplar": true,
-          "expr": "sum(rate(traefik_service_requests_total{instance=\"$instance\"}[$interval]))",
+          "expr": "sum(label_replace(rate(traefik_service_requests_total{instance=~\".*$domain.*\"}[$interval]) or rate(traefik_service_requests_total{domain=\"$domain\"}[$interval]), \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`))",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 2,
@@ -984,7 +984,7 @@
          },
          "editorMode": "code",
          "exemplar": true,
-          "expr": "sum(rate(traefik_service_request_duration_seconds_sum{ instance=\"$instance\" }[5m])) by(service)",
+          "expr": "sum(label_replace(rate(traefik_service_request_duration_seconds_sum{instance=~\".*$domain.*\"}[5m]) or rate(traefik_service_request_duration_seconds_sum{domain=\"$domain\"}[5m]), \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) by(service)",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 2,
@@ -1053,11 +1053,11 @@
            "uid": "PBFA97CFB590B2093"
          },
          "exemplar": true,
-          "expr": "process_open_fds{job=~\"$job\", instance=\"$instance\"}",
+          "expr": "label_replace(process_open_fds{instance=~\".*$domain.*\"} or process_open_fds{domain=\"$domain\"}, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 2,
-          "legendFormat": "{{ instance }}",
+          "legendFormat": "{{ domain }}",
          "refId": "A",
          "step": 240
        }
@@ -1154,7 +1154,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "exemplar": true,
-          "expr": "sum(rate(traefik_service_requests_total{protocol=~\"http|https\", instance=\"$instance\"}[$interval])) by (service)",
+          "expr": "sum(label_replace(rate(traefik_service_requests_total{protocol=~\"http|https\", instance=~\".*$domain.*\"}[$interval]) or rate(traefik_service_requests_total{protocol=~\"http|https\", domain=\"$domain\"}[$interval]), \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) by (service)",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 2,
@@ -1255,7 +1255,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "exemplar": true,
-          "expr": "sum(traefik_entrypoint_open_connections{instance=\"$instance\"}) by (method)",
+          "expr": "sum(label_replace(traefik_entrypoint_open_connections{instance=~\".*$domain.*\"} or traefik_entrypoint_open_connections{domain=\"$domain\"}, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) by (method)",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 1,
@@ -1355,7 +1355,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "exemplar": true,
-          "expr": "sum(traefik_service_open_connections{instance=\"$instance\"}) by (method)",
+          "expr": "sum(label_replace(traefik_service_open_connections{instance=~\".*$domain.*\"} or traefik_service_open_connections{domain=\"$domain\"}, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) by (method)",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 1,
@@ -1459,7 +1459,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "exemplar": true,
-          "expr": "sum(increase(traefik_service_requests_total{protocol=~\"$protocol\", instance=\"$instance\"}[$interval])) by (code)",
+          "expr": "sum(label_replace(increase(traefik_service_requests_total{protocol=~\"$protocol\", instance=~\".*$domain.*\"}[$interval]) or increase(traefik_service_requests_total{protocol=~\"$protocol\", domain=\"$domain\"}[$interval]), \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) by (code)",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 2,
@@ -1514,36 +1514,6 @@
  ],
  "templating": {
    "list": [
-      {
-        "current": {
-          "selected": false,
-          "text": "default",
-          "value": "default"
-        },
-        "datasource": {
-          "type": "prometheus",
-          "uid": "PBFA97CFB590B2093"
-        },
-        "definition": "",
-        "hide": 0,
-        "includeAll": false,
-        "label": "Job:",
-        "multi": false,
-        "name": "job",
-        "options": [],
-        "query": {
-          "query": "label_values(job)",
-          "refId": "Prometheus-job-Variable-Query"
-        },
-        "refresh": 1,
-        "regex": "",
-        "skipUrlSync": false,
-        "sort": 2,
-        "tagValuesQuery": "",
-        "tagsQuery": "",
-        "type": "query",
-        "useTags": false
-      },
      {
        "current": {
          "selected": true,
@@ -1569,7 +1539,7 @@
          "query": "label_values(traefik_service_requests_total, protocol)",
          "refId": "Prometheus-protocol-Variable-Query"
        },
-        "refresh": 1,
+        "refresh": 2,
        "regex": "",
        "skipUrlSync": false,
        "sort": 0,
@@ -1663,19 +1633,19 @@
          "type": "prometheus",
          "uid": "PBFA97CFB590B2093"
        },
-        "definition": "label_values(instance)",
+        "definition": "query_result(label_replace(traefik_config_reloads_total, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`))",
        "hide": 0,
        "includeAll": false,
-        "label": "Instance:",
+        "label": "Domain:",
        "multi": false,
-        "name": "instance",
+        "name": "domain",
        "options": [],
        "query": {
-          "query": "label_values(instance)",
+          "query": "query_result(label_replace(traefik_config_reloads_total, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`))",
          "refId": "PrometheusVariableQueryEditor-VariableQuery"
        },
-        "refresh": 1,
-        "regex": ".*8082",
+        "refresh": 2,
+        "regex": "/domain=\"([^\"]+)\"/",
        "skipUrlSync": false,
        "sort": 1,
        "tagValuesQuery": "",
@@ -21,7 +21,7 @@ tls_skip_verify_insecure = false
 allow_sign_up = true
 auto_login = true
 client_id = {{ env "OIDC_CLIENT_ID" }}
-client_secret = {{ secret "grafana_oidc_client_secret" }}
+client_secret = {{ secret "gf_oidc_secret" }}
 auth_url = {{ env "OIDC_AUTH_URL" }}
 token_url = {{ env "OIDC_TOKEN_URL" }}
 api_url = {{ env "OIDC_API_URL" }}
@@ -34,7 +34,6 @@ ingester:
  max_chunk_age: 1h           # All chunks will be flushed when they hit this age, default is 1h
  chunk_target_size: 1048576  # Loki will attempt to build chunks up to 1.5MB, flushing first if chunk_idle_period or max_chunk_age is reached first
  chunk_retain_period: 30s    # Must be greater than index read cache TTL if using an index cache (Default index read cache TTL is 5m)
-  max_transfer_retries: 0     # Chunk transfers disabled
  wal:
    dir: "/tmp/wal"

@@ -53,7 +52,7 @@ schema_config:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
-      schema: v11
+      schema: v13
      index:
        prefix: index_
        period: 24h
@@ -63,7 +62,6 @@ storage_config:
    active_index_directory: /loki/boltdb-shipper-active
    cache_location: /loki/boltdb-shipper-cache
    cache_ttl: 24h         # Can be increased for faster performance over longer query periods, uses more disk space
-    shared_store: filesystem
  filesystem:
    directory: /loki/chunks
 {{ end }}
@@ -72,7 +70,6 @@ schema_config:
  configs:
  - from: 2020-11-25
    store: boltdb-shipper
-    object_store: aws
    schema: v11
    index:
      prefix: index_
@@ -89,7 +86,7 @@ storage_config:
   endpoint: {{ env "LOKI_AWS_ENDPOINT" }}
   region: {{ env "LOKI_AWS_REGION" }}
   access_key_id: {{ env "LOKI_ACCESS_KEY_ID" }}
-   secret_access_key: {{ secret "loki_aws_secret_access_key" }}
+   secret_access_key: {{ secret "loki_aws_key" }}
   bucketnames: {{ env "LOKI_BUCKET_NAMES" }}
   insecure: false
   sse_encryption: false
@@ -103,19 +100,24 @@ storage_config:

 compactor:
  working_directory: /loki/boltdb-shipper-compactor
-  shared_store: filesystem
  compaction_interval: 10m
  retention_enabled: true
  retention_delete_delay: 2h
  retention_delete_worker_count: 150
+{{ if eq (env "LOKI_STORAGE_FILESYSTEM") "1" }}
+  delete_request_store: filesystem
+{{ end }}
+{{ if eq (env "LOKI_STORAGE_S3") "1" }}
+  delete_request_store: aws
+{{ end }}

 limits_config:
-  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 168h
  retention_period: {{ env "LOKI_RETENTION_PERIOD" }}
  split_queries_by_interval: 24h
  max_query_parallelism: 100
+  allow_structured_metadata: false

 query_scheduler:
  max_outstanding_requests_per_tenant: 4096
@@ -123,9 +125,6 @@ query_scheduler:
 frontend:
  max_outstanding_per_tenant: 4096

-chunk_store_config:
-  max_look_back_period: 0s
-
 table_manager:
  retention_deletes_enabled: false
  retention_period: 0s
@@ -1,11 +0,0 @@
-#!/bin/sh -e
-
-NODE_NAME=$(cat /etc/nodename)
-
-mkdir -p /etc/node-exporter
-
-echo "node_meta{node_id=\"$NODE_ID\", container_label_com_docker_swarm_node_id=\"$NODE_ID\", node_name=\"$NODE_NAME\"} 1" > /etc/node-exporter/node-meta.prom
-
-set -- /bin/node_exporter "$@"
-
-exec "$@"
@@ -1,37 +0,0 @@
-server:
-  http_listen_port: 9080
-  grpc_listen_port: 0
-
-positions:
-  filename: /tmp/positions.yaml
-
-clients:
-  - url: {{ env "LOKI_PUSH_URL" }}
-    basic_auth:
-      username: admin
-      password: {{ secret "basic_auth" }}
-    external_labels:
-      hostname: {{ env "DOMAIN" }}
-
-scrape_configs:
- job_name: system
-  static_configs:
-  - targets:
-      - localhost
-    labels:
-      job: varlogs
-      __path__: /var/log/*log
-
- job_name: "docker"
-  docker_sd_configs:
-    - host: "unix:///var/run/docker.sock"
-      refresh_interval: "10s"
-  relabel_configs:
-    - source_labels: ['__meta_docker_container_name']
-      target_label: "container_name"
-    - source_labels: ['__meta_docker_container_id']
-      target_label: "container_id"
-    - source_labels: ['__meta_docker_container_label_com_docker_stack_namespace']
-      target_label: "stack_namespace"
-    - source_labels: ['__meta_docker_container_label_com_docker_swarm_service_name']
-      target_label: "service_name"
@@ -0,0 +1,61 @@
+BREAKING CHANGE
+Migration plan for upgrading from 1.6.0+v1.8.1.
+
+## 1. Reinsert secrets with shortened names
+
+Secret and config names were shortened to max 14 characters to prevent going over Docker's 64 character
+limit when STACK_NAME and VERSION are added to it.
+
+- `abra app secret list <domain>` to see which secrets are missing under their new name
+- `abra app cmd --local <domain> migrate_secret_names` to reinsert all of them automatically
+      (or manually: `abra app secret insert <domain> <secret_name> v1 <value>` per secret)
+
+## 2. If you use OIDC (moved to seperate compose file)
+
+- Add to your .env: `COMPOSE_FILE="$COMPOSE_FILE:compose.grafana-oidc.yml"`
+
+## 3. If you use SMTP (moved to a seperate compose file)
+
+- Add to your .env: `COMPOSE_FILE="$COMPOSE_FILE:compose.grafana-smtp.yml"`
+
+## 4. node_exporter/cadvisor/promtail replaced by Grafana Alloy
+
+Metrics collection changed from Prometheus scraping endpoints
+to Alloy pushing via `remote_write`/`loki push`.
+
+- Remove `compose.promtail.yml`, `compose.expose-ports.yml` and
+      `compose.basic-auth.yml` from your .env if present. They no
+      longer exist. `compose.yml` now declares the `basic_auth` secret directly, so
+      `SECRET_BASIC_AUTH_VERSION` is always required.
+- Add `PROMETHEUS_REMOTE_WRITE_URL=https://prometheus.$DOMAIN/api/v1/write`
+      (`$DOMAIN` if this host also runs `compose.prometheus.yml`, otherwise a remote
+      Prometheus' URL). Without this, Alloy collects no metrics at all.
+- Add `LOKI_PUSH_URL` (existing var, still used) and pick a log source:
+      `JOURNALD=1` (systemd hosts), `SYSLOG_FILES=1` (non-systemd, tails
+      `/var/log/*log`), or `SYSLOG=1` + `compose.syslog.yml` (network syslog listener).
+
+- If this host had its own `node.$DOMAIN`/`cadvisor.$DOMAIN` scrape target
+      configured on a central Prometheus, remove it. Those endpoints are gone.
+- `scrape-config.example.yml` and the `add_node`/`add_domain` abra.sh commands
+      are gone. Replaced by label-based auto-discovery (see README).
+- `docker stack deploy` doesn't prune removed services, so old `cadvisor`/
+      `promtail` containers keep running after a normal `abra app deploy`. Run
+      `abra app undeploy <domain>` then `abra app deploy <domain>` to clear them out.
+- Diff your `.env` against the current `.env.sample`, to verify any other changes.
+
+### New: label-based metrics auto-discovery
+
+Alloy now auto-discovers and scrapes other Docker Swarm services on the same
+host/`proxy` network that opt in via `prometheus.io/scrape=true` deploy labels.
+See the README's "Auto-discovering metrics from other apps" section.
+
+- If you scrape Traefik metrics: the old `metrics.traefik.$domain` pull-based
+      endpoint still works if you keep the scrape config in Prometheus and
+      existing dashboards keep showing its data, but it's recommended to get
+      Traefik onto the new label-based discovery.
+
+### Dashboards
+
+The Swarm, Stacks and Traefik dashboards were reworked to show old (pull-model)
+and new (Alloy push-model) data as one continuous line, so you don't lose history
+across the migration.
@@ -1,4 +0,0 @@
- targets:
-  - 'example.org:8082'
-  - 'node.monitoring.example.org'
-  - 'cadvisor.monitoring.example.org'
Author	SHA1	Message	Date
dannygroenewegen	382f6353fc	fix: strip monitoring. prefix from instance labels for cleaner dashboard views	2026-08-20 16:50:25 +02:00
dannygroenewegen	c8f74e2a3c	fix: refresh dashboard variables on time range change, not just page load	2026-08-20 14:07:10 +02:00
dannygroenewegen	6dbb6c5f55	fix: mount containerd socket read-only, avoid abra#900	2026-08-17 16:09:44 +02:00
dannygroenewegen	17c4f6237b	docs: rewrite README and release notes for the Alloy migration	2026-08-17 15:58:04 +02:00
dannygroenewegen	50fc916107	feat: add healthchecks to Alloy, Prometheus and Pushgateway	2026-08-17 15:08:08 +02:00
dannygroenewegenandClaude Sonnet 5	7480a8d6ab	fix abra linting secret length Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>	2026-08-17 14:59:03 +02:00
dannygroenewegen	067f013fb7	fix: don't expose Alloy's web UI by default, but optionally with compose.alloy-webui.yml	2026-08-17 14:48:44 +02:00
dannygroenewegen	536fa7f37d	fix Traefik dashboard	2026-08-17 13:37:27 +02:00
dannygroenewegen	fafd1f89f0	fix: discover containers by task instead of service for scraping Swarm service discovery misses dnsrr-mode services like traefik; switch to swarm task discovery, scoped to this host, with a stable instance label across redeploys and a default port when none is labeled.	2026-08-10 21:24:38 +02:00
dannygroenewegen	7ab6cc74be	fix: make swarm/stacks dashboards work with old and new metric data Replace node_meta-based joins with label_replace() of the instance label, so old (pull-model) and new (Alloy push) series for the same host render as one continuous series. Each rewritten query has a TODO noting the simpler form to use once old data ages out of retention.	2026-08-07 16:57:39 +02:00
dannygroenewegen	0f989c4567	fix: drop unfiltered discovery.docker.containers.targets from default scrape prometheus.exporter.cadvisor.docker.targets already reports resource metrics for every container on the host. The unfiltered discovery just added targets with internal overlay IPs, most of which failed to scrape anything useful.	2026-08-07 15:52:31 +02:00
dannygroenewegen	402e022114	feat: separate scrape_auth secret, configurable usernames Added an optional scrape_auth secret for authenticating scrapes of containers that opt in via prometheus.io/auth labels, no longer reusing the basic_auth secret meant for Prometheus/Loki writes. Made usernames for both write endpoints and authenticated scraping configurable in .env.	2026-08-07 15:52:19 +02:00
dannygroenewegen	7f3d98fd9f	- Restore GRAFANA_DOMAIN - Start documenting the migration in release/next - Various cleanups	2026-08-07 14:47:22 +02:00
fauno	627d988902	fix: upgrade to grafana 13.0.6	2026-08-07 09:24:53 -03:00
fauno	e75fa4487c	fix: upgrade alloy to 1.18.1	2026-08-06 12:37:25 -03:00
fauno	5f3f9c957c	fix: upgrade to grafana 13.0.5	2026-08-04 10:48:47 -03:00
fauno	7818c89ad7	fix: provide access to containerd socket toolshed/abra#900	2026-07-31 13:57:43 -03:00
fauno	b47f1b021f	fix: upgrade grafana to v13.0.3	2026-06-27 14:43:43 -03:00
fauno	c6efc69859	fix: upgrade alloy to v1.17.0	2026-06-27 14:42:11 -03:00
fauno	6c529f3527	fix: upgrade to grafana v12.4.5	2026-06-27 14:37:58 -03:00
fauno	01b398ddea	fix: unneeded flag	2026-06-27 14:24:59 -03:00
fauno	30ad8e54f6	fix: upgrade to alloy v1.16.3	2026-06-27 14:20:32 -03:00
fauno	c9910eabf4	Merge branch 'main' of https://git.coopcloud.tech/coop-cloud/monitoring-ng into alloy	2026-06-20 23:59:20 -03:00
fauno	1d9eb10004	improve alloy config and match with main branch (#24 ) Reviewed-on: #24 Reviewed-by: fauno <fauno@sutty.coop.ar>	2026-06-17 12:53:08 +00:00
fauno	23acf56637	fix: filter by proxy network	2026-06-16 22:01:50 -03:00
fauno	03227f1907	fix: send container metrics directly to prometheus	2026-06-16 22:01:05 -03:00
fauno	d085c66d68	fix: needed labels come from docker swarm	2026-06-16 22:00:42 -03:00
fauno	1970061ff8	feat: live debugging alloy	2026-06-16 21:59:50 -03:00
fauno	fa76179987	feat: enable alloy web ui	2026-06-16 21:20:29 -03:00
fauno	64cb07a4a2	feat: bearer auth support	2026-06-16 20:03:17 -03:00
dannygroenewegen	e247677433	feat: scrape metrics from containers via Docker label discovery Containers opt in with prometheus.io/scrape=true and optionally set prometheus.io/port, prometheus.io/path, and prometheus.io/auth=basic.	2026-06-14 21:38:43 +02:00
dannygroenewegen	f2310f2b86	improve alloy config and match with main branch - Restrict Alloy UI to loopback - Narrow volume mounts: drop /dev, reduce /var/run to docker.sock:ro - Replace HTTP scrape of :12345 with prometheus.exporter.self Match with main branch (node-exporter / promtail / cadvisor): - Add docker_only and explicit enabled_metrics to cadvisor exporter - Match node-exporter collector config - Match promtail relabeling (container_name, container_id, stack_namespace, service_name) and external hostname label - Add SYSLOG_FILES option to tail /var/log/*log (matches promtail) - Fix journal path and syslog listener address	2026-06-13 22:07:55 +02:00
p4u1	02b01e5c23	feat: make alerts configurable (#19 ) Alert threshholds can now be configured via env variables I also seperated the basic auth, so that it is possible to deploy only prometheus without promtail Reviewed-on: #19 Reviewed-by: ammaratef45 <ammaratef45@proton.me> Reviewed-by: Danny Groenewegen <dannygroenewegen@noreply.git.coopcloud.tech> Co-authored-by: p4u1 <p4u1_f4u1@riseup.net> Co-committed-by: p4u1 <p4u1_f4u1@riseup.net>	2026-06-12 06:49:17 +00:00
p4u1	fce8ea5889	docs: Update deploy docs and adds prometheus basic auth back (#7 ) Reviewed-on: #7 Reviewed-by: Danny Groenewegen <dannygroenewegen@noreply.git.coopcloud.tech> Co-authored-by: p4u1 <p4u1_f4u1@riseup.net> Co-committed-by: p4u1 <p4u1_f4u1@riseup.net>	2026-06-12 06:48:39 +00:00
fauno	f2711fa16e	fix: upgrades	2026-06-03 00:19:41 -03:00
fauno	2870b9486c	fix: use the actual health check path	2026-06-02 21:06:05 -03:00
fauno	3a1fabe4f9	fix: prevent redirections on health check	2026-06-02 21:04:56 -03:00
fauno	a358837922	wip: relabel syslog according to docs	2026-06-02 21:00:34 -03:00
fauno	dd0a0c1bb0	fixup! feat: read syslog	2026-06-02 20:19:45 -03:00
fauno	31cabc36ae	fix: prevent traefik deprecation warnings	2026-06-02 19:16:49 -03:00
fauno	d25986d5cb	fix: README	2026-06-02 18:51:10 -03:00
fauno	f8f8004445	feat: read syslog	2026-06-02 18:50:41 -03:00
fauno	aa05d022da	feat: optionally push to prometheus and loki	2026-06-02 18:50:20 -03:00
fauno	fb52a76247	BREAKING CHANGE: deprecate node-exporter	2026-06-02 18:49:05 -03:00
fauno	2e2a52eae0	BREAKING CHANGE: deprecate promtail	2026-06-02 18:48:20 -03:00
fauno	48419d5afa	fixup! BREAKING CHANGE: no need to expose exporters	2026-06-02 18:46:02 -03:00
fauno	a0a6e2c509	fix: basic auth secret is always needed	2026-06-02 18:44:32 -03:00
fauno	024f2a8aec	feat: send docker logs to loki	2026-06-02 18:39:24 -03:00
fauno	38095e23fa	BREAKING CHANGE: no need to expose exporters	2026-06-02 18:37:56 -03:00
fauno	641161329e	fix: grafana alternate domain doesn't work the variable is not expanded and the domain name label ends up as a literal "$DOMAIN".	2026-06-02 18:00:00 -03:00
fauno	cdacfd035e	fix: prometheus querying panel is accessible through basic auth	2026-06-02 17:52:25 -03:00
fauno	b2d3901f61	fix: bind mounts recommended by docs	2026-06-02 13:24:28 -03:00
fauno	8becf1c1d6	fixup! feat: node exporter	2026-05-29 16:16:37 -03:00
fauno	777b1355dd	fixup! feat: node exporter	2026-05-29 16:16:08 -03:00
fauno	e83433cebd	feat: node exporter	2026-05-29 16:04:19 -03:00
fauno	a713f98ffb	feat: instance name is domain	2026-05-29 16:03:59 -03:00
fauno	8dc84c591c	fixup! feat: enable prometheus remote write receiver	2026-05-29 15:38:52 -03:00
fauno	d9aa05a4b5	feat: send metrics to prometheus	2026-05-28 21:00:10 -03:00
fauno	349df12204	feat: enable prometheus remote write receiver	2026-05-28 20:44:00 -03:00
fauno	6c33089078	feat: cadvisor	2026-05-28 20:38:50 -03:00
fauno	4bedebfab1	BREAKING CHANGES: replace promtail and cadvisor for alloy	2026-05-28 20:33:36 -03:00
moritz	dd320e9f1c	fix: Shorten all secret and config names to max 14 characters (#13 ) Reviewed-on: #13 Reviewed-by: p4u1 <p4u1@noreply.git.coopcloud.tech> Reviewed-by: moritz <moritz@noreply.git.coopcloud.tech>	2026-05-11 15:38:15 +00:00
ammaratef45	9cb997b25a	delete_request_store based on env variable	2026-04-09 04:36:03 +00:00
ammaratef45	48d137d194	update loki config file	2026-04-09 04:36:03 +00:00
ammaratef45	1acb5ebd6a	chore: update image tags	2026-04-09 04:36:03 +00:00
dannygroenewegen	682f30cef1	Add migrate_secret_names() to abra.sh to reinsert all secrets with shortened names in docker	2026-03-25 16:11:37 +01:00
dannygroenewegen	694c8a9875	Add instructions for shorter secret names to release notes	2026-03-25 16:11:28 +01:00
dannygroenewegen	9dfa9cad2a	Shortened all the secret and config names to max 14 characters to prevent running into Docker's 64 character limit when STACK_NAME is appended to it.	2026-03-25 15:58:28 +01:00
dannygroenewegen	99f8790ec4	fix: Update scape-config example to use HTTPS for Traefik metrics (#17 ) This fixes the insecure Traefik metrics endpoint. See coop-cloud/traefik#94 for details. Reviewed-on: #17 Co-authored-by: Danny Groenewegen <mail@dannygroenewegen.nl> Co-committed-by: Danny Groenewegen <mail@dannygroenewegen.nl>	2026-03-24 09:37:05 +00:00
p4u1	310c28e735	refactor: provision alerts instead of putting them in the /var/lib folder (#16 ) Note that I did not copy the backupbot alert since this one gets a rework soon Reviewed-on: #16 Co-authored-by: p4u1 <p4u1_f4u1@riseup.net> Co-committed-by: p4u1 <p4u1_f4u1@riseup.net>	2026-03-20 14:10:10 +00:00
ammaratef45andp4u1	16bd65f417	fix recipe part in the domain (#8 ) I created a new app using this recipe and the domain wasn't automatically replaced, I'm guessing cause the part before the root domain didn't match the recipe name? Just opening a PR real quick so I can get back to it and test the fix later when I have cycles Co-authored-by: p4u1 <p4u1@noreply.git.coopcloud.tech> Reviewed-on: #8 Reviewed-by: p4u1 <p4u1@noreply.git.coopcloud.tech> Co-authored-by: ammaratef45 <ammaratef45@proton.me> Co-committed-by: ammaratef45 <ammaratef45@proton.me>	2026-03-20 09:23:36 +00:00
Apfelwurm	97ebcf306a	add all mountpoints to free disk space in Docker Swarm dashboard (#4 ) Until now, only / and /media were monitored in the Docker Swarm dashboard. We removed the filters and changed the dashboard to a time series, so multiple mounts can be shown at once. We also updated the alert, so it also triggers on all mount ext4 points. Reviewed-on: #4 Co-authored-by: Apfelwurm <Alexander@volzit.de> Co-committed-by: Apfelwurm <Alexander@volzit.de>	2026-03-20 09:15:52 +00:00
p4u1	f93370b9ca	Moves oidc to a seperate compose config (#6 ) Otherwise the secret has to be provided when oidc is not used Reviewed-on: #6 Co-authored-by: p4u1 <p4u1_f4u1@riseup.net> Co-committed-by: p4u1 <p4u1_f4u1@riseup.net>	2026-03-20 09:10:48 +00:00
iexos	83461e2e76	remove default TIMEOUT (abra #596 )	2025-12-30 13:53:47 +01:00