fix: scrape once per hour

Following the discussion from the matrix channel, I made the disk usage metrics optional, as it currently causes ~35% of cpu usage on our system.

[Old cadvisor-config](02b01e5c23/compose.yml) from this repo had a `- "--housekeeping_interval=120s"` option enabled which seems currently not configurable for alloy - maybe this allows us reenabling disk metrics at a later point again.

dropped from 35% to 5% when removing `disk` from
`  enabled_metrics = ["cpu", "cpuLoad", "diskIO", "memory", "network"]` for cadvisor.

go profiling:

```
File: alloy
Build ID: de7ba8978f6753c25c13f6886ab3b896d3b05d3f
Type: cpu
Time: Sep 7, 2026 at 12:53pm (CEST)
Duration: 120s, Total samples = 37.23s (31.02%)
Showing nodes accounting for 32.20s, 86.49% of 37.23s total
Dropped 790 nodes (cum <= 0.19s)
      flat  flat%   sum%        cum   cum%
    28.72s 77.14% 77.14%     28.72s 77.14%  internal/runtime/syscall/linux.Syscall6
     0.47s  1.26% 78.40%      0.48s  1.29%  internal/filepathlite.(*lazybuf).append (inline)
     0.30s  0.81% 79.21%      0.94s  2.52%  internal/filepathlite.Clean
     0.26s   0.7% 79.91%      0.26s   0.7%  runtime.nextFreeFast (inline)
     0.23s  0.62% 80.53%      0.23s  0.62%  runtime.futex
     0.21s  0.56% 81.09%      0.21s  0.56%  runtime.memclrNoHeapPointers
     0.21s  0.56% 81.65%      0.21s  0.56%  runtime.memmove
     0.19s  0.51% 82.16%      6.19s 16.63%  os.(*File).readdir
     0.17s  0.46% 82.62%     31.24s 83.91%  path/filepath.walk
     0.12s  0.32% 82.94%      0.20s  0.54%  runtime.exitsyscall
     0.08s  0.21% 83.16%      0.24s  0.64%  runtime.scanObject
     0.08s  0.21% 83.37%      0.24s  0.64%  runtime.sweepone
     0.08s  0.21% 83.59%      0.29s  0.78%  slices.pdqsortOrdered[go.shape.string]
     0.07s  0.19% 83.78%      0.27s  0.73%  runtime.makeslicecopy
     0.06s  0.16% 83.94%     28.48s 76.50%  syscall.RawSyscall6
     0.05s  0.13% 84.07%      0.27s  0.73%  github.com/google/cadvisor/fs.GetDirUsage.func1
     0.05s  0.13% 84.21%     19.99s 53.69%  os.lstatNolog
     0.05s  0.13% 84.34%      1.26s  3.38%  runtime.mallocgc
     0.05s  0.13% 84.47%      0.43s  1.15%  runtime.mallocgcSmallNoscan
     0.04s  0.11% 84.58%      0.46s  1.24%  runtime.newobject
     0.04s  0.11% 84.69%      0.19s  0.51%  runtime.selectgo
     0.04s  0.11% 84.80%      1.18s  3.17%  runtime.systemstack
```

Reviewed-on: coop-cloud/monitoring-ng#29
Reviewed-by: Danny Groenewegen <247+dannygroenewegen@noreply.git.coopcloud.tech>
Co-authored-by: Simon <s.thiessen@local-it.org>

.env.sample

@@ -1,19 +1,61 @@
 TYPE=monitoring-ng
 LETS_ENCRYPT_ENV=production
 COMPOSE_FILE=compose.yml
-DOMAIN=monitoring.example.com
+DOMAIN=monitoring-ng.example.com
-TIMEOUT=120
+#TIMEOUT=120
 ENABLE_BACKUPS=true
 
-## Enable this secret for Promtail / Prometheus
+# Secret Alloy authenticates with when writing metrics/logs to Prometheus/
-# SECRET_BASIC_AUTH_VERSION=v1
+# Loki and what Traefik's basicauth middleware expects for that
-#
+SECRET_BASIC_AUTH_VERSION=v1
-# Promtail (Gathering Logs)
+# Username sent along with SECRET_BASIC_AUTH_VERSION above (default: admin)
-# COMPOSE_FILE="$COMPOSE_FILE:compose.promtail.yml"
+# WRITE_BASIC_AUTH_USERNAME=admin
-# LOKI_PUSH_URL=https://loki.monitoring.example.org/loki/api/v1/push
 
-## Expose node and cadvisor ports instead of traefik
+# Expose Alloy's web UI publicly (behind basic-auth) at alloy.$DOMAIN.
-# COMPOSE_FILE="$COMPOSE_FILE:compose.expose-ports.yml"
+# COMPOSE_FILE="$COMPOSE_FILE:compose.alloy-webui.yml"
+# Enable Live Debugging in web ui
+# LIVE_DEBUGGING=false
+
+# Enable this to send metrics to a Prometheus server, adapt DOMAIN if
+# server is remote
+# PROMETHEUS_REMOTE_WRITE_URL=https://prometheus.$DOMAIN/api/v1/write
+
+# Enable container filesystem usage metrics (expensive du-scan, ~30% CPU on large hosts)
+# CADVISOR_DISK_USAGE=1
+
+# Enable authenticated scraping of containers that opt in via
+# prometheus.io/auth=basic or prometheus.io/auth=bearer labels (used as
+# password/bearer token respectively). Insert it with:
+# abra app secret insert <domain> scrape_auth v1 <password>
+# COMPOSE_FILE="$COMPOSE_FILE:compose.scrape-auth.yml"
+# SECRET_SCRAPE_AUTH_VERSION=v1
+#
+# Username sent along with it (default: alloy)
+# SCRAPE_BASIC_AUTH_USERNAME=alloy
+
+# Edit this if your distribution sets this socket to another location
+# CONTAINERD_SOCKET=/var/run/containerd/containerd.sock
+
+# Enable this to send logs to a Loki server, adapt DOMAIN if server is
+# remote
+# LOKI_PUSH_URL=https://loki.$DOMAIN/loki/api/v1/push
+# Enable on systemd hosts to read logs from the journal
+# JOURNALD=1
+#
+# Enable on non-systemd hosts (Alpine, older Debian/Ubuntu) to tail
+# /var/log/*log files (syslog, auth.log, kern.log, etc.) that a local
+# syslogd writes. No syslogd reconfiguration needed.
+# SYSLOG_FILES=1
+#
+# Enable to receive syslog messages over the network on port 514/tcp.
+# Use for remote devices that push syslog to this host, or for a 
+# local syslogd configured to forward over the network.
+# Not needed if you just want to read local log files — use SYSLOG_FILES instead.
+# SYSLOG=1
+# COMPOSE_FILE="$COMPOSE_FILE:compose.syslog.yml"
+
+# Monitor physical disks health
+# COMPOSE_FILE="$COMPOSE_FILE:compose.smartctl.yml"
 
 # Monitoring Server
 #
@@ -39,19 +81,21 @@ ENABLE_BACKUPS=true
 # LOKI_AWS_REGION=eu-west-1
 # LOKI_ACCESS_KEY_ID=bush-debrief-approval-robust-scraggly-molecule
 # LOKI_BUCKET_NAMES=loki
-# SECRET_LOKI_AWS_SECRET_ACCESS_KEY_VERSION=v1
+# SECRET_LOKI_AWS_KEY_VERSION=v1
 #
 ## Grafana
 #
 # COMPOSE_FILE="$COMPOSE_FILE:compose.grafana.yml"
 # GF_SERVER_ROOT_URL=https://monitoring.example.com
-# SECRET_GRAFANA_ADMIN_PASSWORD_VERSION=v1
+# SECRET_GF_ADMINPASSWD_VERSION=v1
-## Seperate domain for Grafana
+## Grafana's own domain. Defaults to $DOMAIN
-#GRAFANA_DOMAIN=grafana.example.com
+## Change the value if you want Grafana on another domain.
+# GRAFANA_DOMAIN=$DOMAIN
 #
 ## Single-Sign-On with OIDC
+# COMPOSE_FILE="$COMPOSE_FILE:compose.grafana-oidc.yml"
 # OIDC_ENABLED=1
-# SECRET_GRAFANA_OIDC_CLIENT_SECRET_VERSION=v1
+# SECRET_GF_OIDC_SECRET_VERSION=v1
 # OIDC_CLIENT_ID=grafana
 # OIDC_AUTH_URL="https://authentik.example.com/application/o/authorize/"
 # OIDC_API_URL="https://authentik.example.com/application/o/userinfo/"
@@ -62,24 +106,26 @@ ENABLE_BACKUPS=true
 # GF_INSTALL_PLUGINS=grafana-piechart-panel
 #
 ## grafana SMTP configuration (optional)
+# COMPOSE_FILE="$COMPOSE_FILE:compose.grafana-smtp.yml"
 # GF_SMTP_HOST=changeme
 # GF_SMTP_USER=changme
 # GF_SMTP_ENABLED=true
 # GF_SMTP_FROM_ADDRESS=grafana@example.com
 # GF_SMTP_SKIP_VERIFY=false
-# SECRET_GRAFANA_SMTP_PASSWORD_VERSION=v1
+# SECRET_GF_SMTP_PASS_VERSION=v1
 #
 
 ## Grafana Matrix Contact Point (optional)
 #COMPOSE_FILE="$COMPOSE_FILE:compose.matrix-alertmanager-receiver.yml"
-#SECRET_MATRIX_ACCESS_TOKEN_VERSION=v1
+#SECRET_MATRIX_TOKEN_VERSION=v1
 #GF_MATRIX_USER_ID="<user-id>"
 #GF_MATRIX_ROOM_ID="<room-id>"
 #GF_MATRIX_HOMESERVER_URL="<homeserver-url>"
 
-# ALerts
+## ALerts
-#ALERT_BACKUP_FAILED_ENABLED=true
+
-#ALERT_BACKUP_MISSING_ENABLED=true
+# Node disk space alert will trigger when free disk space left is below the given number in percent
-#ALERT_BACKUP_NOT_SUCCESSFULL_ENABLED=true
+#ALERT_NODE_DISK_SPACE_LEFT=10
-#ALERT_NODE_DISK_SPACE_ENABLED=true
+
-#ALERT_NODE_MEMORY_USAGE_ENABLED=true
+# Node memory usage alert will trigger when memory usage is above the given number in percent
+#ALERT_NODE_MEMORY_USAGE=85

README.md

@@ -1,9 +1,10 @@
 # monitoring-ng
 
 Yet another monitoring stack ...
-This time its a all-in-one grafana/prometheus/loki/node_exporter/cadvisor/promtail stack.
+This time its a all-in-one grafana/prometheus/loki/alloy stack.
-It's based heavily on the [monitoring-lite](https://git.coopcloud.tech/coop-cloud/monitoring-lite) stack, but has everything in one recipe included now. So you can deploy monitoring instances to only gather metrics / logs (node_exporter/cadvisor/promtail) and also deploy instances with the full monitoring stack (grafana/prometheus/loki) with the same recipe and just different .env configuration.
+It's based heavily on the [monitoring-lite](https://git.coopcloud.tech/coop-cloud/monitoring-lite) stack, but has everything in one recipe included now. So you can deploy monitoring instances to only gather metrics / logs (alloy) and also deploy instances with the full monitoring stack (grafana/prometheus/loki) with the same recipe and just different .env configuration.
 
+Metrics and logs are collected by [Grafana Alloy](https://grafana.com/docs/alloy/latest/) and pushed to a central Prometheus/Loki (via `remote_write`/`loki push`). Every `monitoring-ng` instance runs its own Alloy, whether or not it also runs the central stack.
 
 <!-- metadata -->
 
@@ -18,62 +19,93 @@ It's based heavily on the [monitoring-lite](https://git.coopcloud.tech/coop-clou
 
 <!-- endmetadata -->
 
-## Setup Metrics Gathering
+## Setup: gathering-only host
 
-Where gathering.org is the node you want to gather metrics from.
+Deploys just Alloy, pushing this host's own node/container metrics and logs to a central instance. Use this on every host you want metrics/logs from.
 
-1. Configure DNS
+1. `abra app new monitoring-ng --server gathering.org`
-  - cadvisor.monitoring.gathering.org
+2. `abra app config monitoring.gathering.org`
-  - node.monitoring.gathering.org
+3. Point it at your central instance:
-2. [Configure Traefik to use BasicAuth](https://git.coopcloud.tech/coop-cloud/traefik#configuring-wildcard-ssl-using-dns)
+   ```
-3. `abra app new monitoring-ng`
+   PROMETHEUS_REMOTE_WRITE_URL=https://prometheus.example.org/api/v1/write
-4. `abra app config monitoring.gathering.org` (for gathering only the main `compose.yml` is needed, nothing more.)
+   LOKI_PUSH_URL=https://loki.example.org/loki/api/v1/push
+   JOURNALD=1   # or SYSLOG_FILES=1 / SYSLOG=1, see .env.sample
+   ```
+4. `abra app secret insert monitoring.gathering.org basic_auth v1 <password>`. Same username/password as the `usersfile` credential configured for Traefik's basicauth on the central node (see below). This is what Alloy authenticates with when pushing metrics/logs. Alloy defaults to username `admin` for this. Uf the Traefik `usersfile` uses a different username, set `WRITE_BASIC_AUTH_USERNAME` in this recipe to match.
 5. `abra app deploy monitoring.gathering.org`
-6. check that endpoints are up and basic-auth works
-  - cadvisor.monitoring.gathering.org
-  - node.monitoring.gathering.org
 
-### Expose node and cadvisor via ports instead of traefik
+## Setup: full monitoring stack (metrics/logs browser)
 
-In case you have no traefik running on the machine, you can expose the ports directly by uncommenting the following line:
+This is what a gathering host pushes into. It also runs its own Alloy, so it monitors itself too.
-```
+
-# COMPOSE_FILE="$COMPOSE_FILE:compose.expose-ports.yml"
+1. Configure DNS: `monitoring.example.org`, plus `prometheus.`/`loki.`/`pushgateway.` subdomains for whichever of those you enable below
+2. Traefik on this node needs basic auth configured (`BASIC_AUTH=1`, see the Traefik recipe's "Configuring basic auth" section) — Prometheus/Loki/Pushgateway route through its `basicauth@file` middleware, so without it those endpoints won't work. Use the same username (default `admin`, see `WRITE_BASIC_AUTH_USERNAME`) and password you'll insert as the `basic_auth` secret below when generating Traefik's `usersfile`.
+3. `abra app config monitoring.example.org` Uncomment `compose.prometheus.yml` (metrics), `compose.loki.yml` (logs) and `compose.grafana.yml` (dashboard)
+4. `abra app secret insert monitoring.example.org basic_auth v1 <password>` — this is the password every gathering host's Alloy (including this instance's own) authenticates with; also what Traefik's basicauth expects on the public Prometheus/Loki/Pushgateway endpoints
+5. `abra app secret insert monitoring.example.org gf_adminpasswd v1 <password>`
+6. `abra app deploy monitoring.example.org`
+
+### Post-setup guide
+
+- configure the SMTP mailer under `Alerting > Contact points`
+  - edit the default contact point, choose "Alertmanager" as type & `http://alertmanager:9093` as URL
+  - use the "Test" button to send a test mail. It should fire a request at the alertmanager & that should send a mail
+- from your dashboard panels, choose `Edit > Alert` to create alerts based on those panels
+
+## Additional features
+
+### Discovering metrics from other apps
+
+Alloy auto-discovers and scrapes other Docker Swarm services running on the same host, on the `proxy` network, that opt in via labels. No manual scrape config needed. On the app's `compose.yml`:
+
+```yaml
+deploy:
+  labels:
+    - "prometheus.io/scrape=true"       # required: opt in
+    # - "prometheus.io/port=8082"       # optional: defaults to 80
+    # - "prometheus.io/path=/metrics"   # optional: defaults to /metrics
+    # - "prometheus.io/auth=basic"      # optional: basic auth, see below
+    # - "prometheus.io/auth=bearer"     # optional: bearer token auth
 ```
 
-## Setup Metrics Browser
+Each scraped target gets `instance` (`<service_name>.<slot>`, stable across redeploys), `domain` (the service's stack namespace with underscores converted back to dots, e.g. `traefik.example.com`) and `task_slot` labels attached automatically.
 
-This builds upon [Setup Metrics Gathering](#setup-metrics-grathering) so make sure you did that first.
+If the target needs authentication, configure the monitoring-ng instance with a scrape-auth secret that will be used for targets having the auth label set:
+```
+COMPOSE_FILE="$COMPOSE_FILE:compose.scrape-auth.yml"
+SECRET_SCRAPE_AUTH_VERSION=v1
+```
+`abra app secret insert <domain> scrape_auth v1 <password-or-token>`, then set the scraped app's `prometheus.io/auth` label to `basic` or `bearer` to match how it checks the secret.
 
-1. Configure DNS
+Check discovered targets via `alloy.example.org` (needs `compose.alloy-webui.yml`, see below), or query the central Prometheus for `up{job="<service_name>"}`.
-   - monitoring.example.org
-2. Setup monitoring stack
-   - `abra app config monitoring.example.org` Uncomment prometheus, loki and grafana
-   - `abra app secret insert monitoring.example.org basic_auth v1 <password>`
-      this needs the plaintext traefik basic-auth secret, not the hashed one!
-   - `abra app secret ls monitoring.example.org`
-   - `abra app deploy monitoring.example.org`
-3. Add scrape config to prometheus
-  - `abra app cmd monitoring.example.org prometheus gathering.org`
-  - or manually
-    ```
-    cp scrape-config.example.yml gathering.org.yml
-    # adjust domain
-    # mkdir scrape_configs
-    abra app cp monitoring.dev.local-it.cloud gathering.org.yml prometheus:/prometheus/scrape_configs/
-    ```
 
-* check that all configured targets are up:
+### Manual scraping
-  https://prometheus.monitoring.example.org/targets
 
+For targets where auto-discovery doesn't work (e.g. not a Docker Swarm service on the `proxy` network, or missing labels): add them directly to Prometheus instead. Create a scrape config file:
+```yaml
+- targets:
+    - 'metrics.something-external.example.org'
+    - 'app-without-labels.example.org'
+```
+and copy it into Prometheus' scrape config directory:
+```
+abra app cp monitoring.gathering.org targets.yml prometheus:/prometheus/scrape_configs/
+```
+Prometheus picks up files there automatically.
 
+### Alloy Web UI / Live Debugging
 
-| Service       | Authentication     | Domain                            |
+Alloy's own web UI isn't exposed by default, its HTTP server only listens
-| ------------- | ------------------ | --------------------------------- |
+on localhost inside its own container. To reach it from outside (e.g. to
-| Grafana       | Email / SSO        | monitoring.example.org            |
+browse its component graph or use live debugging), add:
-| Prometheus    | traefik basic-auth | prometheus.monitoring.example.org |
+```
-| loki          | traefik basic-auth | loki.monitoring.example.org       |
+COMPOSE_FILE="$COMPOSE_FILE:compose.alloy-webui.yml"
-| Cadvisor      | traefik basic-auth | cadvisor.monitoring.example.org   |
+```
-| Node Exporter | traefik basic-auth | node.monitoring.example.org       |
+This exposes it (behind the same basic-auth) at `alloy.$DOMAIN`.
+
+To actually see what's being collected (live-tailing the metrics/logs
+flowing through each component, not just their config) also set
+`LIVE_DEBUGGING=true`. Only enable this while troubleshooting.
 
 ### Logging from a docker host to loki server without anything else
 
@@ -90,36 +122,25 @@ $ echo '{
 $ systemctl restart docker.service
 ```
 
-## Setup Push Gateway
+### Setup Push Gateway
 
-1. Enable in the env fiöle by uncommenting the following lines:
+1. Enable in the env file by uncommenting the following lines:
 ```
 ## Prometheus Pushgateway
 # COMPOSE_FILE="$COMPOSE_FILE:compose.pushgateway.yml"
 ```
 2. `abra app deploy monitoring.example.org`
 
-This will expose the pushgateway at `https://pushgateway.${DOMAIN}`.
+This will expose the pushgateway at `https://pushgateway.${DOMAIN}`, secured behind the same basic auth as the other services.
-It is secured behind the same basic auth as the other services.
+After that you need to add the `pushgateway.${DOMAIN}` to the scrape config of Prometheus.
-After that you need to add the `pushgateway.${DOMAIN}` to the scare config.
 
-## Post-setup guide
-
-- configure prometheus/loki/alertmanager as data sources in grafana under `Configuration > Data sources`
-  - for loki, you need to set a "Custom HTTP Header": `X-Scope-OrgID: fake`
-- configure the SMTP mailer under `Alerting > Contact points`
-  - edit the default contact point, choose "Alertmanager" as type & `http://alertmanager:9093` as URL
-  - use the "Test" button to send a test mail. It should fire a request at the alertmanager & that should send a mail
-- `abra app cp` your `scrap_configs: ...` into `/prometheus/scrape_configs` & log into your prometheus web UI to ensure they're working
-- load your dashboards in manually under `Create > Dashboard`
-- from your dashboard panels, choose `Edit > Alert` to create alerts based on those panels
 
 
 ---
 
 THX to the previous work of @decentral1se @knooflok @3wc @cellarspoon @mirsal
 
-## Adding Matrix as Alert Contact point
+### Adding Matrix as Alert Contact point
 
 1. Enable the [matrix-alertmanager-receiver](https://github.com/metio/matrix-alertmanager-receiver/):
 ```
@@ -128,7 +149,7 @@ COMPOSE_FILE="$COMPOSE_FILE:compose.matrix-alertmanager-receiver.yml"
 
 2. Insert the matrix access token secret:
 ```
-abra app secret insert monitoring.marx.klasse-methode.it matrix_access_token v1
+abra app secret insert monitoring.marx.klasse-methode.it matrix_token v1
 ```
 
 3. Set required configurations:
@@ -139,13 +160,17 @@ GF_MATRIX_HOME_SERVER_URL=
 ```
 4. Configure Alertmanager webhook and set the url to `http://matrix-alertmanager-receiver:12345/alerts/<room-id>`
 
-## alerts
+### Alerts
 
-It is possible to enable the following alerts, by setting the corresponding env variable to `true`:
+It is possible to enable the following alerts, by uncommenting the corresponding env variable:
-- backupbot failed: `ALERT_BACKUP_FAILED_ENABLED`
-- backupbot missing: `ALERT_BACKUP_MISSING_ENABLED`
-- backupbot not successfull: `ALERT_BACKUP_NOT_SUCCESSFULL_ENABLED`
-- node disk space: `ALERT_NODE_DISK_SPACE_ENABLED`
-- node memory usage: `ALERT_NODE_MEMORY_USAGE_ENABLED`
 
+- node disk space: `ALERT_NODE_DISK_SPACE_LEFT`
+- node memory usage: `ALERT_NODE_MEMORY_USAGE`
 
+## smart monitoring
+
+To be able monitor hard drive health data, you need to configure
+`smartd` to run on the host system, and also the
+`collect-smartctl-json.sh` script provided here (via cronjob or as
+a `smartd` hook).  This is a limitation on Docker Swarm, which prevents
+the `smartctl_exporter` from running on privileged mode.

abra.sh

@@ -1,38 +1,108 @@
-export ENTRYPOINT_VERSION=v1
+export GF_DATASOURCES_VERSION=v1
-export GRAFANA_DATASOURCES_YML_VERSION=v1
+export GF_DASHBOARDS_VERSION=v3
-export GRAFANA_DASHBOARDS_YML_VERSION=v2
+export GF_SWARM_DASH_VERSION=v3
-export GRAFANA_SWARM_DASHBOARD_JSON_VERSION=v2
+export GF_STACKS_DASH_VERSION=v3
-export GRAFANA_STACKS_DASHBOARD_JSON_VERSION=v2
+export GF_TRAEFIK_DASH_VERSION=v3
-export GRAFANA_TRAEFIK_DASHBOARD_JSON_VERSION=v2
+export GF_BACKUP_DASH_VERSION=v1
-export GRAFANA_BACKUP_DASHBOARD_JSON_VERSION=v1
+export GF_CUSTOM_INI_VERSION=v4
-export GRAFANA_ALERTS_JSON_VERSION=v3
+export LOKI_YML_VERSION=v3
-export GRAFANA_CUSTOM_INI_VERSION=v4
-export PROMTAIL_YML_VERSION=v3
-export LOKI_YML_VERSION=v2
 export PROMETHEUS_YML_VERSION=v2
-export MATRIX_ALERTMANAGER_CONFIG_VERSION=e
+export MATRIX_ALERTMANAGER_CONFIG_VERSION=v1
-export MATRIX_ALERTMANAGER_ENTRYPOINT_VERSION=a
+export MATRIX_ALERTMANAGER_ENTRYPOINT_VERSION=v1
+export GF_ALERTS_NODE_VERSION=v3
+export CONFIG_ALLOY_VERSION=v2
 
-# creates a default prometheus scrape config for a given node
+# migrates secrets from old names to new names by reading values from the
-add_node(){
+# running containers on the server and re-inserting them under the new names.
-  name=$1
+# preview changes: abra app cmd --local <app> migrate_secret_names
-  add_domain "$name" "$name:8082"
+# execute changes: abra app cmd --local <app> migrate_secret_names execute
-  add_domain "$name" "node.monitoring.$name"
+migrate_secret_names() {
-  add_domain "$name" "cadvisor.monitoring.$name"
+  if ! command -v jq &> /dev/null; then
-  cat "/prometheus/scrape_configs/$name.yml"
+    echo "jq is required on your local machine to migrate secret names"
-}
+    echo "It could not be found in your PATH, please install jq to proceed."
+    echo "For example: On a debian/ubuntu system, run `apt install jq`"
+    exit 1
+  fi
 
-# adds a domain to a scrape config or creates a new one
+  # Hardcoded migration mappings: old_secret_name|new_secret_name
-add_domain(){
+  MIGRATIONS="
-  name=$1
+grafana_admin_password|gf_adminpasswd
-  domain=$2
+grafana_smtp_password|gf_smtp_pass
-  if [ ! -d "/prometheus/scrape_configs/" ]; then
+grafana_oidc_client_secret|gf_oidc_secret
-    mkdir -p /prometheus/scrape_configs/
+matrix_access_token|matrix_token
+loki_aws_secret_access_key|loki_aws_key
+"
+
+  # Determine which server the app is deployed on
+  SERVER=$(abra app ls -m | jq -r --arg domain "$APP_NAME" '[.[].apps[] | select(.domain == $domain) | .server] | first' 2>/dev/null)
+
+  if [ -z "$SERVER" ]; then
+    echo "Error: could not determine server for app '$APP_NAME'"
+    exit 1
   fi
-  cd /prometheus/scrape_configs/ || exit 1
+
-  if [ ! -f "$name.yml" ]; then
+  # Build a lookup table of all secrets currently mounted in this stack.
-    echo -e "- targets:\n  - '$domain'" > "$name.yml"
+  # Each line: <secretID> <containerID> <secretName>
-  else
+  LOOKUP=$(ssh "$SERVER" "
-    echo "  - '$domain'"  >> "$name.yml"
+    docker stack services ${STACK_NAME} --format '{{.Name}}' | while read svc; do
+      CID=\$(docker ps --no-trunc -q --filter \"name=\${svc}\" | head -1)
+      docker service inspect \"\$svc\" --format '{{json .Spec.TaskTemplate.ContainerSpec.Secrets}}' | \
+        jq -r --arg cid \"\$CID\" '.[]? | .SecretID + \" \" + \$cid + \" \" + .SecretName'
+    done | sort -k3 -r
+  " 2>/dev/null)
+
+  echo "Secret migration plan for: $APP_NAME (server: $SERVER)"
+  echo ""
+  printf "  %-24s %-8s %s\n" "OLD NAME" "FOUND" "ACTION"
+  printf "  %-24s %-8s %s\n" "--------" "-----" "------"
+
+  # Check each old name against the lookup table and display the plan
+  ANY_FOUND=false
+  while IFS='|' read -r OLD_NAME NEW_NAME; do
+    [ -z "$OLD_NAME" ] && continue
+    MATCH=$(echo "$LOOKUP" | grep " ${STACK_NAME}_${OLD_NAME}_" | head -1)
+    if [ -n "$MATCH" ]; then
+      printf "  %-24s %-8s %s\n" "$OLD_NAME" "yes" "recreate as '$NEW_NAME' version V1"
+      ANY_FOUND=true
+    else
+      printf "  %-24s %-8s %s\n" "$OLD_NAME" "no" "nothing (not found on server)"
+    fi
+  done <<< "$MIGRATIONS"
+
+  echo ""
+
+  if [ "$ANY_FOUND" = false ]; then
+    echo "No old secrets found on server. Nothing to migrate."
+    return 0
   fi
+
+  if [ "$1" != "execute" ]; then
+    echo "To apply the above changes, run:"
+    echo "  abra app cmd --local $APP_NAME migrate_secret_names execute"
+    return 0
+  fi
+
+  # read each found secret from its container and re-insert with the new name
+  while IFS='|' read -r OLD_NAME NEW_NAME; do
+    [ -z "$OLD_NAME" ] && continue
+
+    MATCH=$(echo "$LOOKUP" | grep " ${STACK_NAME}_${OLD_NAME}_" | head -1)
+    [ -z "$MATCH" ] && continue
+
+    SECRET_ID=$(echo "$MATCH" | awk '{print $1}')
+    CID=$(echo "$MATCH" | awk '{print $2}')
+    SECRET_VALUE=$(ssh "$SERVER" "cat /var/lib/docker/containers/${CID}/mounts/secrets/${SECRET_ID} 2>/dev/null || sudo cat /var/lib/docker/containers/${CID}/mounts/secrets/${SECRET_ID} 2>/dev/null")
+
+    if [ -z "$SECRET_VALUE" ]; then
+      echo "Error: could not read value for '$OLD_NAME', skipping"
+      continue
+    fi
+
+    echo "Migrating: '$OLD_NAME' -> '$NEW_NAME' (v1)"
+    printf '%s' "$SECRET_VALUE" | abra app secret insert -C "$APP_NAME" "$NEW_NAME" v1
+    
+  done <<< "$MIGRATIONS"
+
+  echo ""
+  echo "Done."
 }

alertmanager-matrix-config.yml.tmpl

@@ -12,7 +12,7 @@ http:
 matrix:
   homeserver-url: "{{ env "GF_MATRIX_HOMESERVER_URL" }}"
   user-id: "{{ env "GF_MATRIX_USER_ID" }}"
-  access-token: "{{ secret "matrix_access_token" }}"
+  access-token: "{{ secret "matrix_token" }}"
   room-mapping:
     matrixroom: "{{ env "GF_MATRIX_ROOM_ID" }}"
 

alerts/node.yml.tmpl

@@ -0,0 +1,131 @@
+apiVersion: 1
+
+# List of alert rule UIDs that should be deleted
+deleteRules:
+  {{ if not (env "ALERT_NODE_DISK_SPACE_LEFT") }}
+  - orgId: 1
+    uid: coopcloud_node_disk_space_left
+  {{ end }}
+  {{ if not (env "ALERT_NODE_MEMORY_USAGE") }}
+  - orgId: 1
+    uid: coopcloud_node_memory_usage
+  {{ end }}
+
+groups:
+- orgId: 1
+  name: node
+  folder: node
+  interval: 5m
+  rules:
+    {{ if (env "ALERT_NODE_DISK_SPACE_LEFT") }}
+    - uid: coopcloud_node_disk_space_left
+      title: Node Disk Space
+      condition: C
+      data:
+        - refId: A
+          relativeTimeRange:
+            from: 600
+            to: 0
+          datasourceUid: PBFA97CFB590B2093
+          model:
+            editorMode: code
+            expr: (node_filesystem_free_bytes{fstype=~"ext4|xfs"} / node_filesystem_size_bytes{fstype=~"ext4|xfs"}) * 100
+            instant: true
+            intervalMs: 1000
+            legendFormat: __auto
+            maxDataPoints: 43200
+            range: false
+            refId: A
+        - refId: C
+          relativeTimeRange:
+            from: 600
+            to: 0
+          datasourceUid: __expr__
+          model:
+            conditions:
+                - evaluator:
+                    params:
+                        - {{ env "ALERT_NODE_DISK_SPACE_LEFT" }}
+                    type: lt
+                  operator:
+                    type: and
+                  query:
+                    params:
+                        - C
+                  reducer:
+                    params: []
+                    type: last
+                  type: query
+            datasource:
+                type: __expr__
+                uid: __expr__
+            expression: A
+            intervalMs: 1000
+            maxDataPoints: 43200
+            refId: C
+            type: threshold
+      noDataState: NoData
+      execErrState: Error
+      for: 5m
+      annotations:
+        description: ""
+        runbook_url: ""
+        summary: Less than {{ env "ALERT_NODE_DISK_SPACE_LEFT" }}% disk space left on {{`{{ $labels.instance }}`}} ({{`{{ (index $values "A").Value }}`}}% left)
+      labels:
+        "": ""
+      isPaused: false
+    {{ end }}
+    {{ if (env "ALERT_NODE_MEMORY_USAGE")  }}
+    - uid: coopcloud_node_memory_usage
+      title: Node Memory Usage
+      condition: C
+      data:
+        - refId: A
+          relativeTimeRange:
+            from: 600
+            to: 0
+          datasourceUid: PBFA97CFB590B2093
+          model:
+            editorMode: code
+            expr: (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100
+            instant: true
+            intervalMs: 1000
+            legendFormat: __auto
+            maxDataPoints: 43200
+            range: false
+            refId: A
+        - refId: C
+          relativeTimeRange:
+            from: 600
+            to: 0
+          datasourceUid: __expr__
+          model:
+            conditions:
+                - evaluator:
+                    params:
+                        - {{ env "ALERT_NODE_MEMORY_USAGE"  }}
+                    type: gt
+                  operator:
+                    type: and
+                  query:
+                    params:
+                        - C
+                  reducer:
+                    params: []
+                    type: last
+                  type: query
+            datasource:
+                type: __expr__
+                uid: __expr__
+            expression: A
+            intervalMs: 1000
+            maxDataPoints: 43200
+            refId: C
+            type: threshold
+      noDataState: NoData
+      execErrState: Error
+      for: 5m
+      annotations:
+        summary: Memory usage is above {{ env "ALERT_NODE_MEMORY_USAGE"  }}% on {{`{{ $labels.instance }}`}}  ({{`{{ printf "%.2f" (index $values "A").Value }}`}}%  usage)
+      isPaused: false
+    {{ end }}

collect-smartctl-json.service

@@ -0,0 +1,6 @@
+[Unit]
+Description=Collect SMART data
+
+[Service]
+Type=oneshot
+ExecStart=/usr/local/bin/collect-smartctl-json.sh

collect-smartctl-json.sh

@@ -0,0 +1,69 @@
+#! /bin/bash
+# Adapted from https://github.com/prometheus-community/smartctl_exporter/blob/master/collect-smartctl-json.sh
+
+script_dir=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
+
+# Data directory to dump smartctl output
+# This directory will be created if it doesn't exist
+data_dir="/var/lib/smartmontools/json"
+
+# The original script used --xall but that doesn't work
+# This matches the command in readSMARTctl()
+smartctl_args="--json --info --health --attributes --tolerance=verypermissive \
+--nocheck=standby --format=brief --log=error"
+
+# Ignore this devices
+smartctl_ignore_dev_regex="^(/dev/bus)"
+
+# Determine the json query tool to use
+if command -v jq >/dev/null; then
+	json_tool="jq"
+	json_args="--raw-output"
+elif command -v yq >/dev/null; then
+	json_tool="yq"
+	json_args="--unwrapScalar"
+else
+	echo -e "One of 'yq' or 'jq' is required. Please try again after \
+installing one of them"
+	exit 1
+fi
+
+if [[ ! "${UID}" -eq 0 ]] && ! command -v sudo >/dev/null; then
+	# Not root and sudo doesn't exist
+	echo "sudo does not exist. Please run this as root"
+	exit 1
+fi
+
+SUDO="sudo"
+if [[ "${UID}" -eq 0 ]]; then
+	# Don't use sudo if root
+	SUDO=""
+fi
+
+[[ ! -d "${data_dir}" ]] && mkdir --parents "${data_dir}"
+
+if [[ $# -ne 0 ]]; then
+	devices="${1}"
+else
+	devices="$(smartctl --scan --json | "${json_tool}" "${json_args}" \
+".devices[].name | select(test(\"${smartctl_ignore_dev_regex}\") | not)")"
+  mapfile -t devices <<< "${devices[@]}"
+fi
+
+for device in "${devices[@]}"
+  do
+	echo -n "Collecting data for '${device}'..."
+	# shellcheck disable=SC2086
+	data="$($SUDO smartctl ${smartctl_args} ${device})"
+	# Accommodate a smartmontools pre-7.3 bug
+	data=${data#"  Pending defect count:"}
+	type="$(echo "${data}" | "${json_tool}" "${json_args}" '.device.type')"
+	family="$(echo "${data}" | "${json_tool}" "${json_args}" \
+'select(.model_family != null) | .model_family | sub(" |/" ; "_" ; "g")
+ | sub("\"|\\(|\\)" ; "" ; "g")')"
+	model="$(echo "${data}" | "${json_tool}" "${json_args}" \
+'.model_name | sub(" |/" ; "_" ; "g") | sub("\"|\\(|\\)" ; "" ; "g")')"
+	device_name="$(basename "${device}")"
+	echo -e "\tSaving to ${device_name}.json"
+	echo "${data}" > "${data_dir}/${device_name}.json"
+done

collect-smartctl-json.timer

@@ -0,0 +1,9 @@
+[Unit]
+Description=Collect SMART data
+
+[Timer]
+OnCalendar=hourly
+Persistent=true
+
+[Install]
+WantedBy=timers.target

compose.alloy-webui.yml

@@ -0,0 +1,16 @@
+version: "3.8"
+
+services:
+  app:
+    environment:
+      - ALLOY_HTTP_LISTEN_ADDR=0.0.0.0
+    deploy:
+      labels:
+        - "traefik.enable=true"
+        - "traefik.swarm.network=proxy"
+        - "traefik.http.services.${STACK_NAME}-alloy.loadbalancer.server.port=12345"
+        - "traefik.http.routers.${STACK_NAME}-alloy.rule=Host(`alloy.${DOMAIN}`)"
+        - "traefik.http.routers.${STACK_NAME}-alloy.entrypoints=web-secure"
+        - "traefik.http.routers.${STACK_NAME}-alloy.tls=true"
+        - "traefik.http.routers.${STACK_NAME}-alloy.tls.certresolver=${LETS_ENCRYPT_ENV}"
+        - "traefik.http.routers.${STACK_NAME}-alloy.middlewares=basicauth@file"

compose.expose-ports.yml

@@ -1,13 +0,0 @@
----
-version: "3.8"
-
-services:
-  app:
-    ports:
-      - "9100:9100"
-    deploy:
-
-  cadvisor:
-    ports:
-      - "9101:8080"
-    deploy:

compose.grafana-oidc.yml

@@ -0,0 +1,17 @@
+version: '3.8'
+
+services:
+  grafana:
+    secrets:
+      - gf_oidc_secret
+    environment:
+      - OIDC_API_URL
+      - OIDC_AUTH_URL
+      - OIDC_CLIENT_ID
+      - OIDC_ENABLED
+      - OIDC_TOKEN_URL
+
+secrets:
+  gf_oidc_secret:
+    external: true
+    name: ${STACK_NAME}_gf_oidc_secret_${SECRET_GF_OIDC_SECRET_VERSION}

compose.grafana-smtp.yml

@@ -0,0 +1,18 @@
+version: '3.8'
+
+services:
+  grafana:
+    secrets:
+      - gf_smtp_pass
+    environment:
+      - GF_SMTP_HOST
+      - GF_SMTP_USER
+      - GF_SMTP_PASSWORD__FILE=/run/secrets/gf_smtp_pass
+      - GF_SMTP_ENABLED
+      - GF_SMTP_FROM_ADDRESS
+      - GF_SMTP_SKIP_VERIFY
+
+secrets:
+  gf_smtp_pass:
+    external: true
+    name: ${STACK_NAME}_gf_smtp_pass_${SECRET_GF_SMTP_PASS_VERSION}

compose.grafana.yml

@@ -2,104 +2,87 @@ version: '3.8'
 
 services:
   grafana:
-    image: grafana/grafana:10.4.14
+    image: grafana/grafana:13.0.6
     volumes:
       - grafana-data:/var/lib/grafana:rw
     secrets:
-      - grafana_admin_password
+      - gf_adminpasswd
-      - grafana_oidc_client_secret
-      - grafana_smtp_password
     configs:
-      - source: grafana_custom_ini
+      - source: gf_custom_ini
         target: /etc/grafana/grafana.ini
-      - source: grafana_datasources_yml
+      - source: gf_datasources
         target: /etc/grafana/provisioning/datasources/datasources.yml
-      - source: grafana_dashboards_yml
+      - source: gf_dashboards
         target: /etc/grafana/provisioning/dashboards/dashboards.yml
-      - source: grafana_swarm_dashboard_json
+      - source: gf_swarm_dash
         target: /var/lib/grafana/dashboards/docker-swarm-nodes.json
-      - source: grafana_stacks_dashboard_json
+      - source: gf_stacks_dash
         target: /var/lib/grafana/dashboards/docker-swarm-stacks.json
-      - source: grafana_traefik_dashboard_json
+      - source: gf_traefik_dash
         target: /var/lib/grafana/dashboards/traefik.json
-      - source: grafana_backup_dashboard_json
+      - source: gf_backup_dash
         target: /var/lib/grafana/dashboards/backup.json
-      - source: grafana_alerts_json
+      - source: gf_alerts_node
-        target: /var/lib/grafana/alerts/alerts.json
+        target: /etc/grafana/provisioning/alerting/node.yml
     networks:
       - proxy
       - internal
     environment:
       - GF_SERVER_ROOT_URL
-      - GF_SECURITY_ADMIN_PASSWORD__FILE=/run/secrets/grafana_admin_password
+      - GF_SECURITY_ADMIN_PASSWORD__FILE=/run/secrets/gf_adminpasswd
-      - GF_SMTP_HOST
-      - GF_SMTP_USER
-      - GF_SMTP_PASSWORD__FILE=/run/secrets/grafana_smtp_password
-      - GF_SMTP_ENABLED
-      - GF_SMTP_FROM_ADDRESS
-      - GF_SMTP_SKIP_VERIFY
       - GF_SECURITY_ALLOW_EMBEDDING
       - GF_INSTALL_PLUGINS
-      - OIDC_API_URL
+      - ALERT_NODE_DISK_SPACE_LEFT
-      - OIDC_AUTH_URL
+      - ALERT_NODE_MEMORY_USAGE
-      - OIDC_CLIENT_ID
-      - OIDC_ENABLED
-      - OIDC_TOKEN_URL
     deploy:
       labels:
         - "traefik.enable=true"
-        - "traefik.docker.network=proxy"
+        - "traefik.swarm.network=proxy"
         - "traefik.http.services.${STACK_NAME}-grafana.loadbalancer.server.port=3000"
         - "traefik.http.routers.${STACK_NAME}-grafana.rule=Host(`${GRAFANA_DOMAIN:-$DOMAIN}`)"
         - "traefik.http.routers.${STACK_NAME}-grafana.entrypoints=web-secure"
         - "traefik.http.routers.${STACK_NAME}-grafana.tls=true"
         - "traefik.http.routers.${STACK_NAME}-grafana.tls.certresolver=${LETS_ENCRYPT_ENV}"
     healthcheck:
-      test: "wget -q http://localhost:3000/ -O/dev/null"
+      test: "wget -q http://localhost:3000/healthz -O/dev/null"
       interval: 5s
       timeout: 10s
       retries: 3
       start_period: 10s
 
 configs:
-  grafana_custom_ini:
+  gf_custom_ini:
     template_driver: golang
-    name: ${STACK_NAME}_grafana_custom_ini_${GRAFANA_CUSTOM_INI_VERSION}
+    name: ${STACK_NAME}_gf_custom_ini_${GF_CUSTOM_INI_VERSION}
     file: grafana_custom.ini
-  grafana_datasources_yml:
+  gf_datasources:
-    name: ${STACK_NAME}_g_datasources_yml_${GRAFANA_DATASOURCES_YML_VERSION}
+    name: ${STACK_NAME}_gf_datasources_${GF_DATASOURCES_VERSION}
     file: grafana-datasources.yml
-  grafana_dashboards_yml:
+  gf_dashboards:
-    name: ${STACK_NAME}_g_dashboards_yml_${GRAFANA_DASHBOARDS_YML_VERSION}
+    name: ${STACK_NAME}_gf_dashboards_${GF_DASHBOARDS_VERSION}
     file: grafana-dashboards.yml
-  grafana_swarm_dashboard_json:
+  gf_swarm_dash:
-    name: ${STACK_NAME}_g_swarm_dashboard_json_${GRAFANA_SWARM_DASHBOARD_JSON_VERSION}
+    name: ${STACK_NAME}_gf_swarm_dash_${GF_SWARM_DASH_VERSION}
     file: grafana-swarm-dashboard.json
-  grafana_stacks_dashboard_json:
+  gf_stacks_dash:
-    name: ${STACK_NAME}_g_stacks_dashboard_json_${GRAFANA_STACKS_DASHBOARD_JSON_VERSION}
+    name: ${STACK_NAME}_gf_stacks_dash_${GF_STACKS_DASH_VERSION}
     file: grafana-stacks-dashboard.json
-  grafana_traefik_dashboard_json:
+  gf_traefik_dash:
-    name: ${STACK_NAME}_g_traefik_dashboard_json_${GRAFANA_TRAEFIK_DASHBOARD_JSON_VERSION}
+    name: ${STACK_NAME}_gf_traefik_dash_${GF_TRAEFIK_DASH_VERSION}
     file: grafana-traefik-dashboard.json
-  grafana_backup_dashboard_json:
+  gf_backup_dash:
-    name: ${STACK_NAME}_g_backup_dashboard_json_${GRAFANA_BACKUP_DASHBOARD_JSON_VERSION}
+    name: ${STACK_NAME}_gf_backup_dash_${GF_BACKUP_DASH_VERSION}
     file: grafana-backup-dashboard.json
-  grafana_alerts_json:
+  gf_alerts_node:
     template_driver: golang
-    name: ${STACK_NAME}_g_alerts_json_${GRAFANA_ALERTS_JSON_VERSION}
+    name: ${STACK_NAME}_gf_alerts_node_${GF_ALERTS_NODE_VERSION}
-    file: grafana-alerts.json.tmpl
+    file: alerts/node.yml.tmpl
 
 volumes:
   grafana-data:
 
 
 secrets:
-  grafana_admin_password:
+  gf_adminpasswd:
     external: true
-    name: ${STACK_NAME}_grafana_admin_password_${SECRET_GRAFANA_ADMIN_PASSWORD_VERSION}
+    name: ${STACK_NAME}_gf_adminpasswd_${SECRET_GF_ADMINPASSWD_VERSION}
-  grafana_oidc_client_secret:
-    external: true
-    name: ${STACK_NAME}_grafana_oidc_client_secret_${SECRET_GRAFANA_OIDC_CLIENT_SECRET_VERSION}
-  grafana_smtp_password:
-    external: true
-    name: ${STACK_NAME}_grafana_smtp_password_${SECRET_GRAFANA_SMTP_PASSWORD_VERSION}

compose.loki.yml

@@ -2,7 +2,7 @@ version: '3.8'
 
 services:
   loki:
-    image: grafana/loki:2.9.11
+    image: grafana/loki:3.7.2
     command: -config.file=/etc/loki/local-config.yaml
     networks:
       - proxy
@@ -12,7 +12,7 @@ services:
     volumes:
       - loki-data:/loki
     # secrets:
-    #   - loki_aws_secret_access_key
+    #   - loki_aws_key
     environment:
       - LOKI_ACCESS_KEY_ID
       - LOKI_AWS_ENDPOINT
@@ -27,7 +27,7 @@ services:
         condition: on-failure
       labels:
         - "traefik.enable=true"
-        - "traefik.docker.network=proxy"
+        - "traefik.swarm.network=proxy"
         - "traefik.http.services.${STACK_NAME}-loki.loadbalancer.server.port=3100"
         - "traefik.http.routers.${STACK_NAME}-loki.rule=Host(`loki.${DOMAIN}`)"
         - "traefik.http.routers.${STACK_NAME}-loki.entrypoints=web-secure"
@@ -47,6 +47,6 @@ volumes:
   loki-data:
 
 # secrets:
-#   loki_aws_secret_access_key:
+#   loki_aws_key:
 #     external: true
-#     name: ${STACK_NAME}_loki_aws_secret_access_key_${SECRET_LOKI_AWS_SECRET_ACCESS_KEY_VERSION}
+#     name: ${STACK_NAME}_loki_aws_key_${SECRET_LOKI_AWS_KEY_VERSION}

compose.matrix-alertmanager-receiver.yml

@@ -2,9 +2,9 @@ version: '3.8'
 
 services:
   matrix-alertmanager-receiver:
-    image: metio/matrix-alertmanager-receiver:2025.2.9
+    image: metio/matrix-alertmanager-receiver:2026.2.25
     secrets:
-      - matrix_access_token
+      - matrix_token
     configs:
       - source: matrix-alertmanager-receiver-config
         target: /etc/matrix-alertmanager-receiver/config.yml
@@ -23,6 +23,6 @@ configs:
     file: alertmanager-matrix-config.yml.tmpl
 
 secrets:
-  matrix_access_token:
+  matrix_token:
     external: true
-    name: ${STACK_NAME}_matrix_access_token_${SECRET_MATRIX_ACCESS_TOKEN_VERSION}
+    name: ${STACK_NAME}_matrix_token_${SECRET_MATRIX_TOKEN_VERSION}

compose.prometheus.yml

@@ -2,7 +2,7 @@ version: '3.8'
 
 services:
   prometheus:
-    image: prom/prometheus:v2.55.1
+    image: prom/prometheus:v3.12.0
     secrets:
       - basic_auth
     volumes:
@@ -16,15 +16,22 @@ services:
       - "--web.console.libraries=/usr/share/prometheus/console_libraries"
       - "--web.console.templates=/usr/share/prometheus/consoles"
       - "--storage.tsdb.retention.time=${PROMETHEUS_RETENTION_TIME}"
+      - "--web.enable-remote-write-receiver"
     networks:
       - proxy
       - internal
+    healthcheck:
+      test: "wget -q --spider http://localhost:9090/-/healthy || exit 1"
+      interval: 5s
+      timeout: 10s
+      retries: 3
+      start_period: 30s
     deploy:
       restart_policy:
         condition: on-failure
       labels:
         - "traefik.enable=true"
-        - "traefik.docker.network=proxy"
+        - "traefik.swarm.network=proxy"
         - "traefik.http.services.${STACK_NAME}-prometheus.loadbalancer.server.port=9090"
         - "traefik.http.routers.${STACK_NAME}-prometheus.rule=Host(`prometheus.${DOMAIN}`)"
         - "traefik.http.routers.${STACK_NAME}-prometheus.entrypoints=web-secure"

compose.promtail.yml

@@ -1,30 +0,0 @@
-version: "3.8"
-
-services:
-  promtail:
-    image: grafana/promtail:2.9.11
-    volumes:
-      - /var/log:/var/log:ro
-      - /var/run/docker.sock:/var/run/docker.sock
-    command: -config.file=/etc/promtail/config.yml
-    configs:
-      - source: promtail_yml
-        target: /etc/promtail/config.yml
-    networks:
-      - internal
-    secrets:
-      - basic_auth
-    environment:
-      - DOMAIN
-      - LOKI_PUSH_URL
-
-configs:
-  promtail_yml:
-    name: ${STACK_NAME}_promtail_yml_${PROMTAIL_YML_VERSION}
-    file: promtail.yml.tmpl
-    template_driver: golang
-
-secrets:
-  basic_auth:
-    external: true
-    name: ${STACK_NAME}_basic_auth_${SECRET_BASIC_AUTH_VERSION}

compose.pushgateway.yml

@@ -2,7 +2,7 @@ version: '3.8'
 
 services:
   pushgateway:
-    image: prom/pushgateway:v1.10.0
+    image: prom/pushgateway:v1.11.2
     command:
       - '--web.listen-address=:9191'
       - '--push.disable-consistency-check'
@@ -12,12 +12,18 @@ services:
     networks:
       - internal
       - proxy
+    healthcheck:
+      test: "wget -q --spider http://localhost:9191/-/healthy || exit 1"
+      interval: 5s
+      timeout: 10s
+      retries: 3
+      start_period: 10s
     deploy:
       restart_policy:
         condition: on-failure
       labels:
         - "traefik.enable=true"
-        - "traefik.docker.network=proxy"
+        - "traefik.swarm.network=proxy"
         - "traefik.http.services.${STACK_NAME}-pushgateway.loadbalancer.server.port=9191"
         - "traefik.http.routers.${STACK_NAME}-pushgateway.rule=Host(`pushgateway.${DOMAIN}`)"
         - "traefik.http.routers.${STACK_NAME}-pushgateway.entrypoints=web-secure"

compose.scrape-auth.yml

@@ -0,0 +1,13 @@
+---
+version: "3.8"
+
+services:
+  app:
+    secrets:
+      - source: scrape_auth
+        target: scrape_auth
+
+secrets:
+  scrape_auth:
+    external: true
+    name: ${STACK_NAME}_scrape_auth_${SECRET_SCRAPE_AUTH_VERSION}

compose.smartctl.yml

@@ -0,0 +1,18 @@
+---
+version: "3.8"
+services:
+  smartctl:
+    image: "prometheuscommunity/smartctl-exporter:v0.14.0"
+    volumes:
+    - "/dev:/dev"
+    - "/var/lib/smartmontools/json:/debug"
+    command:
+    - "--smartctl.fake-data"
+    - "--smartctl.interval=1h"
+    networks:
+    - "proxy"
+    deploy:
+      labels:
+        - "prometheus.io/scrape=true"
+        - "prometheus.io/port=9633"
+        - "prometheus.io/path=/metrics"

compose.syslog.yml

@@ -0,0 +1,6 @@
+---
+version: "3.8"
+services:
+  app:
+    ports:
+    - "514:514"

compose.yml

@@ -3,89 +3,69 @@ version: "3.8"
 
 services:
   app:
-    image: prom/node-exporter:v1.8.1
+    image: grafana/alloy:v1.18.1
-    user: root
+    hostname: "${DOMAIN}"
-    environment:
-      - NODE_ID={{.Node.ID}}
-    volumes:
-      - /proc:/host/proc:ro
-      - /sys:/host/sys:ro
-      - /:/rootfs:ro
-      - /etc/hostname:/etc/nodename:ro
-    command:
-      - "--path.sysfs=/host/sys"
-      - "--path.procfs=/host/proc"
-      - "--path.rootfs=/rootfs"
-      - "--collector.textfile.directory=/etc/node-exporter/"
-      - "--collector.filesystem.ignored-mount-points=^/(sys|proc|dev|host|etc)($$|/)"
-      - "--no-collector.ipvs"
     configs:
-      - source: entrypoint
+      - source: config_alloy
-        target: /entrypoint.sh
+        target: /etc/alloy/config.alloy
+    volumes:
+      - /:/rootfs:ro
+      - /var/run/docker.sock:/var/run/docker.sock:ro
+      - /sys:/sys:ro
+      - /var/lib/docker:/var/lib/docker:ro
+      # long-form avoids toolshed/abra#900
+      - type: bind
+        source: "${CONTAINERD_SOCKET:-/run/containerd/containerd.sock}"
+        target: /run/containerd/containerd.sock
+        read_only: true
+      - alloy-data:/var/lib/alloy/data
+    # runs through a shell so ALLOY_HTTP_LISTEN_ADDR (set by
+    # compose.alloy-webui.yml) is resolved from the container's own
+    # environment at startup, not by compose at deploy time.
+    # $$ escapes it from compose's own interpolation
+    entrypoint: ["/bin/sh", "-c"]
+    command:
+      - >-
+        exec alloy run
+        --storage.path=/var/lib/alloy/data
+        --server.http.listen-addr=$${ALLOY_HTTP_LISTEN_ADDR:-127.0.0.1}:12345
+        /etc/alloy/config.alloy
     networks:
-      - internal
       - proxy
-    entrypoint: [ "/bin/sh", "-e", "/entrypoint.sh" ]
+      - internal
+    environment:
+      - SCRAPE_BASIC_AUTH_USERNAME=${SCRAPE_BASIC_AUTH_USERNAME:-alloy}
+      - WRITE_BASIC_AUTH_USERNAME=${WRITE_BASIC_AUTH_USERNAME:-admin}
+      - LIVE_DEBUGGING=${LIVE_DEBUGGING:-false}
+      - NODE_ID={{.Node.ID}}
+    secrets:
+      - basic_auth
+    # no wget/curl in this image; bash's /dev/tcp is used instead. Works
+    # against localhost regardless of ALLOY_HTTP_LISTEN_ADDR
+    healthcheck:
+      test: ["CMD", "bash", "-c", "exec 3<>/dev/tcp/localhost/12345 && printf 'GET /-/ready HTTP/1.0\r\nHost: localhost\r\n\r\n' >&3 && head -1 <&3 | grep -q 200"]
+      interval: 5s
+      timeout: 10s
+      retries: 3
+      start_period: 10s
     deploy:
       restart_policy:
         condition: on-failure
       labels:
         - "backupbot.backup=${ENABLE_BACKUPS:-true}"
-        - "traefik.enable=true"
+        - "coop-cloud.${STACK_NAME}.version=2.0.0+v1.18.1"
-        - "traefik.docker.network=proxy"
-        - "traefik.http.services.${STACK_NAME}-node.loadbalancer.server.port=9100"
-        - "traefik.http.routers.${STACK_NAME}-node.rule=Host(`node.${DOMAIN}`)"
-        - "traefik.http.routers.${STACK_NAME}-node.entrypoints=web-secure"
-        - "traefik.http.routers.${STACK_NAME}-node.tls=true"
-        - "traefik.http.routers.${STACK_NAME}-node.tls.certresolver=${LETS_ENCRYPT_ENV}"
-        - "traefik.http.routers.${STACK_NAME}-node.middlewares=basicauth@file"
-        - "coop-cloud.${STACK_NAME}.version=1.6.0+v1.8.1"
-        - "coop-cloud.${STACK_NAME}.timeout=${TIMEOUT:-120}"
-
-  cadvisor:
-    image: gcr.io/cadvisor/cadvisor:v0.49.2
-    command:
-      - "-logtostderr"
-      - "--enable_metrics=cpu,cpuLoad,disk,diskIO,process,memory,network"
-      # all possible metrics: advtcp,app,cpu,cpuLoad,cpu_topology,cpuset,disk,diskIO,hugetlb,memory,memory_numa,network,oom_event,percpu,perf_event,process,referenced_memory,resctrl,sched,tcp,udp.
-      - "--housekeeping_interval=120s"
-      - "--docker_only=true"
-    volumes:
-      - /var/lib/docker/:/var/lib/docker:ro
-      - /dev/disk/:/dev/disk:ro
-      - /sys:/sys:ro
-      - /var/run:/var/run:ro
-      - /:/rootfs:ro
-    networks:
-      - internal
-      - proxy
-    deploy:
-      restart_policy:
-        condition: on-failure
-      labels:
-        - "traefik.enable=true"
-        - "traefik.docker.network=proxy"
-        - "traefik.http.services.${STACK_NAME}-cadvisor.loadbalancer.server.port=8080"
-        - "traefik.http.routers.${STACK_NAME}-cadvisor.rule=Host(`cadvisor.${DOMAIN}`)"
-        - "traefik.http.routers.${STACK_NAME}-cadvisor.entrypoints=web-secure"
-        - "traefik.http.routers.${STACK_NAME}-cadvisor.tls=true"
-        - "traefik.http.routers.${STACK_NAME}-cadvisor.tls.certresolver=${LETS_ENCRYPT_ENV}"
-        - "traefik.http.routers.${STACK_NAME}-cadvisor.middlewares=basicauth@file"
-    healthcheck:
-      test: wget --quiet --tries=1 --spider http://localhost:8080/healthz || exit 1
-      interval: 15s
-      timeout: 15s
-      retries: 5
-      start_period: 30s
-
 configs:
-  entrypoint:
+  config_alloy:
-    name: ${STACK_NAME}_entrypoint_${ENTRYPOINT_VERSION}
+    template_driver: golang
-    file: node-exporter-entrypoint.sh
+    name: ${STACK_NAME}_config_alloy_${CONFIG_ALLOY_VERSION}
-
+    file: config.alloy.tmpl
-
-
 networks:
   proxy:
     external: true
   internal:
+volumes:
+  alloy-data:
+secrets:
+  basic_auth:
+    external: true
+    name: ${STACK_NAME}_basic_auth_${SECRET_BASIC_AUTH_VERSION}

config.alloy.tmpl

@@ -0,0 +1,355 @@
+logging {
+  level = "info"
+  format = "logfmt"
+}
+
+livedebugging {
+  enabled = {{ env "LIVE_DEBUGGING" }}
+}
+
+discovery.docker "linux" {
+  host = "unix:///var/run/docker.sock"
+}
+
+{{ if ne (env "PROMETHEUS_REMOTE_WRITE_URL") "" }}
+prometheus.exporter.cadvisor "docker" {
+  docker_only = true
+  {{ if eq (env "CADVISOR_DISK_USAGE") "1" }}
+  enabled_metrics = ["cpu", "cpuLoad", "disk", "diskIO", "memory", "network"]
+  {{ else }}
+  enabled_metrics = ["cpu", "cpuLoad", "diskIO", "memory", "network"]
+  {{ end }}
+  // host-wide totals already come from prometheus.exporter.unix
+  disable_root_cgroup_stats = true
+}
+
+prometheus.exporter.unix "default" {
+  include_exporter_metrics = true
+  rootfs_path = "/rootfs"
+  procfs_path = "/rootfs/proc"
+  sysfs_path = "/rootfs/sys"
+
+  disable_collectors = ["ipvs"]
+
+  filesystem {
+    fs_types_exclude     = "^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|tmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$"
+    mount_points_exclude = "^/(sys|proc|dev|host|etc)($|/)"
+    mount_timeout        = "5s"
+  }
+
+  netclass { ignored_devices = "^(veth.*)$" }
+  netdev   { device_exclude  = "^(veth.*)$" }
+}
+
+prometheus.exporter.self "alloy" {}
+
+prometheus.scrape "default" {
+  scrape_interval = "120s"
+
+  targets = array.concat(
+    prometheus.exporter.self.alloy.targets,
+    prometheus.exporter.unix.default.targets,
+    prometheus.exporter.cadvisor.docker.targets,
+  )
+
+  forward_to = [prometheus.remote_write.prometheus.receiver]
+}
+
+prometheus.remote_write "prometheus" {
+  endpoint {
+    url = "{{ env "PROMETHEUS_REMOTE_WRITE_URL" }}"
+
+    basic_auth {
+      username = "{{ env "WRITE_BASIC_AUTH_USERNAME" }}"
+      password = "{{ secret "basic_auth" }}"
+    }
+  }
+}
+
+// Scrape Prometheus metrics from other containers on this host.
+// Containers opt in via Docker labels:
+//   prometheus.io/scrape=true        required: enable scraping
+//   prometheus.io/port=9090          optional: port exposing /metrics (defaults to 80 if not set)
+//   prometheus.io/path=/metrics      optional: path to metrics endpoint (default: /metrics)
+//   prometheus.io/auth=basic         optional: use basic auth with the scrape_auth secret (see compose.scrape-auth.yml)
+//   prometheus.io/auth=bearer        optional: use bearer auth with the scrape_auth secret (see compose.scrape-auth.yml)
+discovery.dockerswarm "swarm" {
+  host = "unix:///var/run/docker.sock"
+  // "tasks" not "services": dnsrr-mode services (e.g. traefik) have no VIP
+  // and are invisible to the "services" role
+  role = "tasks"
+}
+
+discovery.relabel "metrics" {
+  targets = discovery.dockerswarm.swarm.targets
+
+  // skip old task history, only scrape currently-running tasks
+  rule {
+    source_labels = ["__meta_dockerswarm_task_desired_state"]
+    regex         = "running"
+    action        = "keep"
+  }
+
+  // only scrape hosts running on this host within the swam
+  rule {
+    source_labels = ["__meta_dockerswarm_node_id"]
+    regex         = "{{ env "NODE_ID" }}"
+    action        = "keep"
+  }
+
+  rule {
+    source_labels = ["__meta_dockerswarm_network_name"]
+    regex         = "proxy"
+    action        = "keep"
+  }
+
+  rule {
+    source_labels = ["__meta_dockerswarm_service_label_prometheus_io_scrape"]
+    regex         = "true"
+    action        = "keep"
+  }
+
+  // default to port 80 when prometheus.io/port isn't set
+  rule {
+    source_labels = ["__meta_dockerswarm_service_label_prometheus_io_port"]
+    regex         = "^$"
+    target_label  = "__meta_dockerswarm_service_label_prometheus_io_port"
+    replacement   = "80"
+  }
+
+  // a task with multiple published ports produces one target per port;
+  // this unifies all of them to the single port above, so duplicates
+  // collapse at scrape time instead of scraping every port
+  rule {
+    source_labels = ["__address__", "__meta_dockerswarm_service_label_prometheus_io_port"]
+    regex         = `(.+):\d+;(\d+)`
+    target_label  = "__address__"
+    replacement   = "$1:$2"
+  }
+
+  rule {
+    source_labels = ["__meta_dockerswarm_service_label_prometheus_io_path"]
+    regex         = `(.+)`
+    target_label  = "__metrics_path__"
+  }
+
+  rule {
+    source_labels = ["__meta_dockerswarm_service_name"]
+    target_label  = "job"
+  }
+
+  // task IDs (and the default address-derived instance label) change on
+  // every redeploy; service+slot is stable across redeploys of the same
+  // replica, so data stays continuous instead of restarting each deploy
+  rule {
+    source_labels = ["__meta_dockerswarm_service_name", "__meta_dockerswarm_task_slot"]
+    separator     = "."
+    target_label  = "instance"
+  }
+
+  rule {
+    source_labels = ["__meta_dockerswarm_task_slot"]
+    target_label  = "task_slot"
+  }
+
+  // coop-cloud's STACK_NAME is the domain with "." replaced by "_"
+  // Derive a readable dotted domain label from it. RE2 has no global
+  // replace, so this is done by chaining multiple replacements. Each
+  // rule swaps the first remaining _ for a "." until none are left.
+  rule {
+    source_labels = ["__meta_dockerswarm_service_label_com_docker_stack_namespace"]
+    target_label  = "domain"
+  }
+  rule {
+    source_labels = ["domain"]
+    regex         = `([^_]*)_(.*)`
+    target_label  = "domain"
+    replacement   = "$1.$2"
+  }
+  rule {
+    source_labels = ["domain"]
+    regex         = `([^_]*)_(.*)`
+    target_label  = "domain"
+    replacement   = "$1.$2"
+  }
+  rule {
+    source_labels = ["domain"]
+    regex         = `([^_]*)_(.*)`
+    target_label  = "domain"
+    replacement   = "$1.$2"
+  }
+  rule {
+    source_labels = ["domain"]
+    regex         = `([^_]*)_(.*)`
+    target_label  = "domain"
+    replacement   = "$1.$2"
+  }
+  rule {
+    source_labels = ["domain"]
+    regex         = `([^_]*)_(.*)`
+    target_label  = "domain"
+    replacement   = "$1.$2"
+  }
+  rule {
+    source_labels = ["domain"]
+    regex         = `([^_]*)_(.*)`
+    target_label  = "domain"
+    replacement   = "$1.$2"
+  }
+}
+
+discovery.relabel "metrics_noauth" {
+  targets = discovery.relabel.metrics.output
+  rule {
+    source_labels = ["__meta_dockerswarm_service_label_prometheus_io_auth"]
+    regex         = "^$"
+    action        = "keep"
+  }
+}
+
+discovery.relabel "metrics_basicauth" {
+  targets = discovery.relabel.metrics.output
+  rule {
+    source_labels = ["__meta_dockerswarm_service_label_prometheus_io_auth"]
+    regex         = "basic"
+    action        = "keep"
+  }
+}
+
+discovery.relabel "metrics_bearerauth" {
+  targets = discovery.relabel.metrics.output
+  rule {
+    source_labels = ["__meta_dockerswarm_service_label_prometheus_io_auth"]
+    regex         = "bearer"
+    action        = "keep"
+  }
+}
+
+prometheus.scrape "containers" {
+  scrape_interval = "120s"
+  targets         = discovery.relabel.metrics_noauth.output
+  forward_to      = [prometheus.remote_write.prometheus.receiver]
+}
+
+{{ if ne (env "SECRET_SCRAPE_AUTH_VERSION") "" }}
+prometheus.scrape "containers_basicauth" {
+  scrape_interval = "120s"
+  targets         = discovery.relabel.metrics_basicauth.output
+  forward_to      = [prometheus.remote_write.prometheus.receiver]
+  basic_auth {
+    username = "{{ env "SCRAPE_BASIC_AUTH_USERNAME" }}"
+    password = "{{ secret "scrape_auth" }}"
+  }
+}
+
+prometheus.scrape "containers_bearerauth" {
+  scrape_interval = "120s"
+  targets         = discovery.relabel.metrics_bearerauth.output
+  forward_to      = [prometheus.remote_write.prometheus.receiver]
+  bearer_token    = "{{ secret "scrape_auth" }}"
+}
+{{ end }}
+{{ end }}
+
+{{ if ne (env "LOKI_PUSH_URL") "" }}
+discovery.relabel "docker" {
+  targets = discovery.docker.linux.targets
+
+  rule {
+    source_labels = ["__meta_docker_container_name"]
+    target_label  = "container_name"
+  }
+  rule {
+    source_labels = ["__meta_docker_container_id"]
+    target_label  = "container_id"
+  }
+  rule {
+    source_labels = ["__meta_docker_container_label_com_docker_stack_namespace"]
+    target_label  = "stack_namespace"
+  }
+  rule {
+    source_labels = ["__meta_docker_container_label_com_docker_swarm_service_name"]
+    target_label  = "service_name"
+  }
+  rule {
+    source_labels = ["__meta_docker_container_log_stream"]
+    target_label  = "stream"
+  }
+}
+
+loki.source.docker "docker" {
+  host       = "unix:///var/run/docker.sock"
+  targets    = discovery.relabel.docker.output
+  labels     = {"app" = "docker"}
+  forward_to = [loki.write.loki.receiver]
+}
+
+// JOURNALD: reads the systemd journal binary log directly.
+// Use on systemd hosts (most modern Linux distros). Requires no syslogd.
+{{ if eq (env "JOURNALD") "1" }}
+loki.source.journal "journal" {
+  path = "/rootfs/var/log/journal"
+  labels = { job = "{{ env "DOMAIN" }}" }
+  forward_to = [loki.write.loki.receiver]
+}
+{{ end }}
+
+// SYSLOG_FILES: tails all /var/log/*log files (syslog, auth.log, kern.log, etc.).
+// Use on non-systemd hosts where a syslogd writes to /var/log.
+{{ if eq (env "SYSLOG_FILES") "1" }}
+local.file_match "syslog_files" {
+  path_targets = [{ __path__ = "/rootfs/var/log/*log" }]
+}
+
+loki.source.file "syslog_files" {
+  targets    = local.file_match.syslog_files.targets
+  forward_to = [loki.process.syslog_files.receiver]
+}
+
+loki.process "syslog_files" {
+  stage.static_labels {
+    values = { job = "syslog" }
+  }
+  forward_to = [loki.write.loki.receiver]
+}
+{{ end }}
+
+// SYSLOG: opens a network syslog listener on port 514.
+// Use when a remote device or a local syslogd configured to 
+// forward over the network sends logs to this host.
+// Requires compose.syslog.yml to publish port 514 to the host.
+// This is NOT needed for reading local log files — use SYSLOG_FILES instead.
+{{ if eq (env "SYSLOG") "1" }}
+loki.relabel "syslog" {
+  rule {
+    action = "labelmap"
+    regex = "__syslog_(.+)"
+  }
+
+  forward_to = []
+}
+
+loki.source.syslog "syslog" {
+  listener {
+    address = "[::]:514"
+    label_structured_data = true
+    labels = { component = "loki.source.syslog" }
+  }
+
+  relabel_rules = loki.relabel.syslog.rules
+  forward_to = [loki.write.loki.receiver]
+}
+{{ end }}
+
+loki.write "loki" {
+  endpoint {
+    url = "{{ env "LOKI_PUSH_URL" }}"
+
+    basic_auth {
+      username = "{{ env "WRITE_BASIC_AUTH_USERNAME" }}"
+      password = "{{ secret "basic_auth" }}"
+    }
+  }
+  external_labels = { hostname = "{{ env "DOMAIN" }}" }
+}
+{{ end }}

grafana-alerts.json.tmpl

@@ -1,315 +0,0 @@
-{
-  "apiVersion": 1,
-  "groups": [
-    {
-      "orgId": 1,
-      "name": "backupbot",
-      "folder": "node",
-      "interval": "1m",
-      "rules": [
-        {{ if eq (env "ALERT_BACKUP_FAILED_ENABLED") "true" }}
-        {
-          "uid": "de8e5xxup7t34a",
-          "title": "Backup Failed",
-          "condition": "C",
-          "data": [
-            {
-              "refId": "A",
-              "relativeTimeRange": { "from": 600, "to": 0 },
-              "datasourceUid": "PBFA97CFB590B2093",
-              "model": {
-                "disableTextWrap": false,
-                "editorMode": "builder",
-                "expr": "backup",
-                "fullMetaSearch": false,
-                "includeNullMetadata": true,
-                "instant": true,
-                "intervalMs": 1000,
-                "legendFormat": "__auto",
-                "maxDataPoints": 43200,
-                "range": false,
-                "refId": "A",
-                "useBackend": false
-              }
-            },
-            {
-              "refId": "C",
-              "relativeTimeRange": { "from": 600, "to": 0 },
-              "datasourceUid": "__expr__",
-              "model": {
-                "conditions": [
-                  {
-                    "evaluator": { "params": [0], "type": "lt" },
-                    "operator": { "type": "and" },
-                    "query": { "params": ["C"] },
-                    "reducer": { "params": [], "type": "last" },
-                    "type": "query"
-                  }
-                ],
-                "datasource": { "type": "__expr__", "uid": "__expr__" },
-                "expression": "A",
-                "intervalMs": 1000,
-                "maxDataPoints": 43200,
-                "refId": "C",
-                "type": "threshold"
-              }
-            }
-          ],
-          "noDataState": "NoData",
-          "execErrState": "Error",
-          "for": "1m",
-          "isPaused": false
-        },
-        {{ end }}
-        {{ if eq (env "ALERT_BACKUP_MISSING_ENABLED") "true" }}
-        {
-          "uid": "ce8e65uddcwe8d",
-          "title": "Backup Missing",
-          "condition": "B",
-          "data": [
-            {
-              "refId": "A",
-              "relativeTimeRange": { "from": 600, "to": 0 },
-              "datasourceUid": "PBFA97CFB590B2093",
-              "model": {
-                "disableTextWrap": false,
-                "editorMode": "builder",
-                "expr": "rate(backup[24h])",
-                "fullMetaSearch": false,
-                "includeNullMetadata": true,
-                "instant": true,
-                "intervalMs": 1000,
-                "legendFormat": "__auto",
-                "maxDataPoints": 43200,
-                "range": false,
-                "refId": "A",
-                "useBackend": false
-              }
-            },
-            {
-              "refId": "B",
-              "relativeTimeRange": { "from": 600, "to": 0 },
-              "datasourceUid": "__expr__",
-              "model": {
-                "conditions": [
-                  {
-                    "evaluator": { "params": [0, 0], "type": "within_range" },
-                    "operator": { "type": "and" },
-                    "query": { "params": ["C"] },
-                    "reducer": { "params": [], "type": "last" },
-                    "type": "query"
-                  }
-                ],
-                "datasource": { "type": "__expr__", "uid": "__expr__" },
-                "expression": "A",
-                "intervalMs": 1000,
-                "maxDataPoints": 43200,
-                "refId": "B",
-                "type": "threshold"
-              }
-            }
-          ],
-          "noDataState": "NoData",
-          "execErrState": "Error",
-          "for": "5m",
-          "isPaused": false
-        },
-        {{ end }}
-        {{ if eq (env "ALERT_BACKUP_NOT_SUCCESSFULL_ENABLED") "true" }}
-        {
-            "uid": "de8e6bc92a8lcc",
-            "title": "Backup Not Successfull",
-            "condition": "B",
-            "data": [
-                {
-                    "refId": "A",
-                    "relativeTimeRange": {
-                        "from": 60,
-                        "to": 0
-                    },
-                    "datasourceUid": "PBFA97CFB590B2093",
-                    "model": {
-                        "disableTextWrap": false,
-                        "editorMode": "builder",
-                        "expr": "backup",
-                        "fullMetaSearch": false,
-                        "includeNullMetadata": true,
-                        "instant": true,
-                        "intervalMs": 1000,
-                        "legendFormat": "__auto",
-                        "maxDataPoints": 43200,
-                        "range": false,
-                        "refId": "A",
-                        "useBackend": false
-                    }
-                },
-                {
-                    "refId": "B",
-                    "relativeTimeRange": {
-                        "from": 60,
-                        "to": 0
-                    },
-                    "datasourceUid": "__expr__",
-                    "model": {
-                        "conditions": [
-                            {
-                                "evaluator": {
-                                    "params": [
-                                        0
-                                    ],
-                                    "type": "gt"
-                                },
-                                "operator": {
-                                    "type": "and"
-                                },
-                                "query": {
-                                    "params": [
-                                        "C"
-                                    ]
-                                },
-                                "reducer": {
-                                    "params": [],
-                                    "type": "last"
-                                },
-                                "type": "query"
-                            }
-                        ],
-                        "datasource": {
-                            "type": "__expr__",
-                            "uid": "__expr__"
-                        },
-                        "expression": "A",
-                        "intervalMs": 1000,
-                        "maxDataPoints": 43200,
-                        "refId": "B",
-                        "type": "threshold"
-                    }
-                }
-            ],
-            "noDataState": "NoData",
-            "execErrState": "Error",
-            "for": "20m",
-            "annotations": {
-                "summary": "Backup did not finish within 20 minutes"
-            },
-            "labels": {},
-            "isPaused": false
-        }
-        {{ end }}
-      ]
-    },
-    {
-      "orgId": 1,
-      "name": "node",
-      "folder": "node",
-      "interval": "5m",
-      "rules": [
-        {{ if eq (env "ALERT_NODE_DISK_SPACE_ENABLED") "true" }}
-        {
-          "uid": "bds8bhxu97pxca",
-          "title": "Node Disk Space",
-          "condition": "C",
-          "data": [
-            {
-              "refId": "A",
-              "relativeTimeRange": { "from": 600, "to": 0 },
-              "datasourceUid": "PBFA97CFB590B2093",
-              "model": {
-                "editorMode": "code",
-                "expr": "(node_filesystem_free_bytes{fstype=\"ext4\",mountpoint=~\"(/$)|(/media.*)\"} / node_filesystem_size_bytes{fstype=\"ext4\",mountpoint=~\"(/$)|(/media.*)\"}) * 100",
-                "instant": true,
-                "intervalMs": 1000,
-                "legendFormat": "__auto",
-                "maxDataPoints": 43200,
-                "range": false,
-                "refId": "A"
-              }
-            },
-            {
-              "refId": "C",
-              "relativeTimeRange": { "from": 600, "to": 0 },
-              "datasourceUid": "__expr__",
-              "model": {
-                "conditions": [
-                  {
-                    "evaluator": { "params": [10], "type": "lt" },
-                    "operator": { "type": "and" },
-                    "query": { "params": ["C"] },
-                    "reducer": { "params": [], "type": "last" },
-                    "type": "query"
-                  }
-                ],
-                "datasource": { "type": "__expr__", "uid": "__expr__" },
-                "expression": "A",
-                "intervalMs": 1000,
-                "maxDataPoints": 43200,
-                "refId": "C",
-                "type": "threshold"
-              }
-            }
-          ],
-          "noDataState": "NoData",
-          "execErrState": "Error",
-          "for": "5m",
-          "annotations": {},
-          "labels": {},
-          "isPaused": false
-        },
-        {{ end }}
-        {{ if eq (env "ALERT_NODE_MEMORY_USAGE_ENABLED") "true" }}
-        {
-          "uid": "ads8cswmly96oa",
-          "title": "Node Memory Usage",
-          "condition": "C",
-          "data": [
-            {
-              "refId": "A",
-              "relativeTimeRange": { "from": 600, "to": 0 },
-              "datasourceUid": "PBFA97CFB590B2093",
-              "model": {
-                "editorMode": "code",
-                "expr": "(node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100",
-                "instant": true,
-                "intervalMs": 1000,
-                "legendFormat": "__auto",
-                "maxDataPoints": 43200,
-                "range": false,
-                "refId": "A"
-              }
-            },
-            {
-              "refId": "C",
-              "relativeTimeRange": { "from": 600, "to": 0 },
-              "datasourceUid": "__expr__",
-              "model": {
-                "conditions": [
-                  {
-                    "evaluator": { "params": [90], "type": "gt" },
-                    "operator": { "type": "and" },
-                    "query": { "params": ["C"] },
-                    "reducer": { "params": [], "type": "last" },
-                    "type": "query"
-                  }
-                ],
-                "datasource": { "type": "__expr__", "uid": "__expr__" },
-                "expression": "A",
-                "intervalMs": 1000,
-                "maxDataPoints": 43200,
-                "refId": "C",
-                "type": "threshold"
-              }
-            }
-          ],
-          "noDataState": "NoData",
-          "execErrState": "Error",
-          "for": "5m",
-          "annotations": {},
-          "labels": {},
-          "isPaused": false
-        }
-        {{ end }}
-      ]
-    }
-  ]
-}
-

grafana-dashboards.yml

@@ -11,13 +11,3 @@ providers:
     options:
       path: /var/lib/grafana/dashboards
       foldersFromFilesStructure: true
-  - name: 'default-alert-provider'
-    orgId: 1
-    folder: 'default-alerts'
-    type: file
-    disableDeletion: false
-    updateIntervalSeconds: 10
-    allowUiUpdates: true
-    options:
-      path: /var/lib/grafana/alerts
-      foldersFromFilesStructure: true

grafana-stacks-dashboard.json

@@ -110,7 +110,7 @@
             "uid": "PBFA97CFB590B2093"
           },
           "editorMode": "code",
-          "expr": "(time() - min(container_start_time_seconds{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}))",
+          "expr": "(time() - min(container_start_time_seconds{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}))",
           "format": "time_series",
           "intervalFactor": 1,
           "legendFormat": "",
@@ -215,7 +215,7 @@
             "uid": "PBFA97CFB590B2093"
           },
           "editorMode": "code",
-          "expr": "sort(\n    sum(\n        rate(\n            container_cpu_usage_seconds_total{\n                container_label_com_docker_stack_namespace=~\"$stack\", \n                instance=~\"cadvisor.monitoring.$instance.*\"}[$interval]\n            )\n        ) \n    by (container_label_com_docker_swarm_service_name)\n)",
+          "expr": "sort(\n    sum(\n        rate(\n            container_cpu_usage_seconds_total{\n                container_label_com_docker_stack_namespace=~\"$stack\", \n                instance=~\".*$node_id.*\"}[$interval]\n            )\n        ) \n    by (container_label_com_docker_swarm_service_name)\n)",
           "format": "time_series",
           "intervalFactor": 2,
           "legendFormat": "{{ container_label_com_docker_swarm_service_name }}",
@@ -283,7 +283,7 @@
             "uid": "PBFA97CFB590B2093"
           },
           "editorMode": "code",
-          "expr": "count(rate(container_last_seen{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}[$interval]))",
+          "expr": "count(rate(container_last_seen{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}[$interval]))",
           "format": "time_series",
           "intervalFactor": 2,
           "range": true,
@@ -386,7 +386,7 @@
             "uid": "PBFA97CFB590B2093"
           },
           "editorMode": "code",
-          "expr": "sum(\n    container_memory_rss{\n        container_label_com_docker_stack_namespace=~\"$stack\", \n        instance=~\"cadvisor.monitoring.$instance.*\"}\n    ) \nby (container_label_com_docker_swarm_service_name)",
+          "expr": "sum(\n    container_memory_rss{\n        container_label_com_docker_stack_namespace=~\"$stack\", \n        instance=~\".*$node_id.*\"}\n    ) \nby (container_label_com_docker_swarm_service_name)",
           "format": "time_series",
           "hide": false,
           "intervalFactor": 2,
@@ -493,7 +493,7 @@
             "uid": "PBFA97CFB590B2093"
           },
           "editorMode": "code",
-          "expr": "sum(rate(container_network_receive_bytes_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
+          "expr": "sum(rate(container_network_receive_bytes_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
           "format": "time_series",
           "intervalFactor": 2,
           "legendFormat": "RX: {{ container_label_com_docker_swarm_service_name }}",
@@ -507,7 +507,7 @@
             "uid": "PBFA97CFB590B2093"
           },
           "editorMode": "code",
-          "expr": "sum(rate(container_network_transmit_bytes_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}[$interval])) by (container_label_com_docker_swarm_service_name) * -1",
+          "expr": "sum(rate(container_network_transmit_bytes_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}[$interval])) by (container_label_com_docker_swarm_service_name) * -1",
           "hide": false,
           "legendFormat": "TX: {{container_label_com_docker_swarm_service_name}}",
           "range": true,
@@ -634,7 +634,7 @@
             "uid": "PBFA97CFB590B2093"
           },
           "editorMode": "code",
-          "expr": "sum(container_fs_usage_bytes{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}) by (container_label_com_docker_swarm_service_name)",
+          "expr": "sum(container_fs_usage_bytes{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}) by (container_label_com_docker_swarm_service_name)",
           "format": "time_series",
           "hide": false,
           "intervalFactor": 2,
@@ -742,7 +742,7 @@
             "uid": "PBFA97CFB590B2093"
           },
           "editorMode": "code",
-          "expr": "sum(rate(container_fs_usage_bytes{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
+          "expr": "sum(rate(container_fs_usage_bytes{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
           "format": "time_series",
           "hide": false,
           "intervalFactor": 2,
@@ -865,7 +865,7 @@
             "uid": "PBFA97CFB590B2093"
           },
           "editorMode": "code",
-          "expr": "sum(rate(container_fs_io_time_seconds_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
+          "expr": "sum(rate(container_fs_io_time_seconds_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
           "format": "time_series",
           "hide": false,
           "intervalFactor": 2,
@@ -973,7 +973,7 @@
             "uid": "PBFA97CFB590B2093"
           },
           "editorMode": "code",
-          "expr": "container_fs_io_current{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}",
+          "expr": "container_fs_io_current{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}",
           "format": "time_series",
           "hide": false,
           "intervalFactor": 2,
@@ -1081,7 +1081,7 @@
             "uid": "PBFA97CFB590B2093"
           },
           "editorMode": "code",
-          "expr": "sum(rate(container_fs_reads_bytes_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
+          "expr": "sum(rate(container_fs_reads_bytes_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
           "format": "time_series",
           "intervalFactor": 2,
           "legendFormat": "{{ container_label_com_docker_swarm_service_name }}",
@@ -1188,7 +1188,7 @@
             "uid": "PBFA97CFB590B2093"
           },
           "editorMode": "code",
-          "expr": "sum(rate(container_fs_read_seconds_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
+          "expr": "sum(rate(container_fs_read_seconds_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
           "format": "time_series",
           "intervalFactor": 2,
           "legendFormat": "{{ container_label_com_docker_swarm_service_name }}",
@@ -1295,7 +1295,7 @@
             "uid": "PBFA97CFB590B2093"
           },
           "editorMode": "code",
-          "expr": "sum(rate(container_fs_writes_bytes_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
+          "expr": "sum(rate(container_fs_writes_bytes_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
           "format": "time_series",
           "intervalFactor": 2,
           "legendFormat": "{{ container_label_com_docker_swarm_service_name }}",
@@ -1402,7 +1402,7 @@
             "uid": "PBFA97CFB590B2093"
           },
           "editorMode": "code",
-          "expr": "sum(rate(container_fs_write_seconds_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
+          "expr": "sum(rate(container_fs_write_seconds_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
           "format": "time_series",
           "intervalFactor": 2,
           "legendFormat": "{{ container_label_com_docker_swarm_service_name }}",
@@ -1453,7 +1453,7 @@
           "query": "label_values(container_label_com_docker_stack_namespace)",
           "refId": "PrometheusVariableQueryEditor-VariableQuery"
         },
-        "refresh": 1,
+        "refresh": 2,
         "regex": "",
         "skipUrlSync": false,
         "sort": 2,
@@ -1567,19 +1567,19 @@
           "type": "prometheus",
           "uid": "PBFA97CFB590B2093"
         },
-        "definition": "label_values(instance)",
+        "definition": "label_values(node_uname_info, instance)",
         "hide": 0,
         "includeAll": true,
-        "label": "instance",
+        "label": "Swarm Node",
         "multi": true,
-        "name": "instance",
+        "name": "node_id",
         "options": [],
         "query": {
-          "query": "label_values(instance)",
+          "query": "label_values(node_uname_info, instance)",
           "refId": "PrometheusVariableQueryEditor-VariableQuery"
         },
-        "refresh": 1,
+        "refresh": 2,
-        "regex": "/.*cadvisor.monitoring.(?<instance>.*):80/",
+        "regex": "/^(?:(?:node|cadvisor|monitoring).)*(.*)$/",
         "skipUrlSync": false,
         "sort": 0,
         "type": "query"
@@ -1620,4 +1620,4 @@
   "uid": "KdVoGQm7z",
   "version": 36,
   "weekStart": ""
-}
+}

grafana-swarm-dashboard.json

@@ -93,7 +93,6 @@
       },
       "hideTimeOverride": true,
       "id": 2,
-      "links": [],
       "maxDataPoints": 100,
       "options": {
         "colorMode": "value",
@@ -107,17 +106,19 @@
           "fields": "",
           "values": false
         },
+        "showPercentChange": false,
         "text": {},
-        "textMode": "auto"
+        "textMode": "auto",
+        "wideLayout": true
       },
-      "pluginVersion": "10.0.2",
+      "pluginVersion": "10.4.14",
       "targets": [
         {
           "datasource": {
             "type": "prometheus",
             "uid": "PBFA97CFB590B2093"
           },
-          "expr": "topk(1, sum((node_time_seconds - node_boot_time_seconds) * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"}) by (node_name))",
+          "expr": "topk(1, sum(label_replace(node_time_seconds{instance=~\".*$node_id.*\"} - node_boot_time_seconds{instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) by (instance))",
           "format": "time_series",
           "intervalFactor": 2,
           "legendFormat": "",
@@ -172,7 +173,6 @@
         "y": 0
       },
       "id": 1,
-      "links": [],
       "maxDataPoints": 100,
       "options": {
         "colorMode": "value",
@@ -186,17 +186,19 @@
           "fields": "",
           "values": false
         },
+        "showPercentChange": false,
         "text": {},
-        "textMode": "auto"
+        "textMode": "auto",
+        "wideLayout": true
       },
-      "pluginVersion": "10.0.2",
+      "pluginVersion": "10.4.14",
       "targets": [
         {
           "datasource": {
             "type": "prometheus",
             "uid": "PBFA97CFB590B2093"
           },
-          "expr": "count(node_meta * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"})",
+          "expr": "count(label_replace(node_uname_info{instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\"))",
           "format": "time_series",
           "intervalFactor": 2,
           "legendFormat": "",
@@ -251,7 +253,6 @@
       },
       "hideTimeOverride": true,
       "id": 4,
-      "links": [],
       "maxDataPoints": 100,
       "options": {
         "colorMode": "value",
@@ -265,17 +266,19 @@
           "fields": "",
           "values": false
         },
+        "showPercentChange": false,
         "text": {},
-        "textMode": "auto"
+        "textMode": "auto",
+        "wideLayout": true
       },
-      "pluginVersion": "10.0.2",
+      "pluginVersion": "10.4.14",
       "targets": [
         {
           "datasource": {
             "type": "prometheus",
             "uid": "PBFA97CFB590B2093"
           },
-          "expr": "count(node_cpu_seconds_total{mode=\"idle\"} * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"})",
+          "expr": "count(label_replace(node_cpu_seconds_total{mode=\"idle\", instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\"))",
           "format": "time_series",
           "intervalFactor": 2,
           "legendFormat": "",
@@ -335,9 +338,10 @@
         "y": 0
       },
       "id": 8,
-      "links": [],
       "maxDataPoints": 100,
       "options": {
+        "minVizHeight": 75,
+        "minVizWidth": 75,
         "orientation": "horizontal",
         "reduceOptions": {
           "calcs": [
@@ -348,9 +352,10 @@
         },
         "showThresholdLabels": false,
         "showThresholdMarkers": true,
+        "sizing": "auto",
         "text": {}
       },
-      "pluginVersion": "10.0.2",
+      "pluginVersion": "10.4.14",
       "targets": [
         {
           "datasource": {
@@ -358,7 +363,7 @@
             "uid": "PBFA97CFB590B2093"
           },
           "editorMode": "code",
-          "expr": "sum((node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"} * 100) / count(node_meta * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"})",
+          "expr": "sum(label_replace((node_memory_MemAvailable_bytes{instance=~\".*$node_id.*\"} / node_memory_MemTotal_bytes{instance=~\".*$node_id.*\"}) * 100, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) / count(label_replace(node_uname_info{instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\"))",
           "format": "time_series",
           "intervalFactor": 2,
           "legendFormat": "",
@@ -405,13 +410,12 @@
       },
       "lines": true,
       "linewidth": 1,
-      "links": [],
       "nullPointMode": "null",
       "options": {
         "alertThreshold": true
       },
       "percentage": false,
-      "pluginVersion": "10.0.2",
+      "pluginVersion": "10.4.14",
       "pointradius": 5,
       "points": false,
       "renderer": "flot",
@@ -425,11 +429,11 @@
             "type": "prometheus",
             "uid": "PBFA97CFB590B2093"
           },
-          "expr": "node_load5 * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"}",
+          "expr": "label_replace(node_load5{instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")",
           "format": "time_series",
           "interval": "",
           "intervalFactor": 2,
-          "legendFormat": "load5 {{node_name}}",
+          "legendFormat": "load5 {{instance}}",
           "refId": "A",
           "step": 2
         }
@@ -507,7 +511,6 @@
       },
       "hideTimeOverride": true,
       "id": 3,
-      "links": [],
       "maxDataPoints": 100,
       "options": {
         "colorMode": "value",
@@ -521,17 +524,19 @@
           "fields": "",
           "values": false
         },
+        "showPercentChange": false,
         "text": {},
-        "textMode": "auto"
+        "textMode": "auto",
+        "wideLayout": true
       },
-      "pluginVersion": "10.0.2",
+      "pluginVersion": "10.4.14",
       "targets": [
         {
           "datasource": {
             "type": "prometheus",
             "uid": "PBFA97CFB590B2093"
           },
-          "expr": "sum(node_memory_MemTotal_bytes * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"})",
+          "expr": "sum(label_replace(node_memory_MemTotal_bytes{instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\"))",
           "format": "time_series",
           "intervalFactor": 2,
           "legendFormat": "",
@@ -585,7 +590,6 @@
       },
       "hideTimeOverride": true,
       "id": 9,
-      "links": [],
       "maxDataPoints": 100,
       "options": {
         "colorMode": "value",
@@ -599,10 +603,12 @@
           "fields": "",
           "values": false
         },
+        "showPercentChange": false,
         "text": {},
-        "textMode": "auto"
+        "textMode": "auto",
+        "wideLayout": true
       },
-      "pluginVersion": "10.0.2",
+      "pluginVersion": "10.4.14",
       "targets": [
         {
           "datasource": {
@@ -610,7 +616,7 @@
             "uid": "PBFA97CFB590B2093"
           },
           "exemplar": true,
-          "expr": "sum(node_filesystem_size_bytes{fstype=\"ext4\",mountpoint=~\"(/$)|(/media.*)\"} * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"})",
+          "expr": "sum(label_replace(node_filesystem_size_bytes{fstype=~\"ext4|xfs\",mountpoint=~\"(/$)|(/media.*)\", instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\"))",
           "format": "time_series",
           "interval": "",
           "intervalFactor": 2,
@@ -671,9 +677,10 @@
       },
       "hideTimeOverride": true,
       "id": 11,
-      "links": [],
       "maxDataPoints": 100,
       "options": {
+        "minVizHeight": 75,
+        "minVizWidth": 75,
         "orientation": "horizontal",
         "reduceOptions": {
           "calcs": [
@@ -684,16 +691,17 @@
         },
         "showThresholdLabels": false,
         "showThresholdMarkers": true,
+        "sizing": "auto",
         "text": {}
       },
-      "pluginVersion": "10.0.2",
+      "pluginVersion": "10.4.14",
       "targets": [
         {
           "datasource": {
             "type": "prometheus",
             "uid": "PBFA97CFB590B2093"
           },
-          "expr": "sum(irate(node_cpu_seconds_total{mode=\"idle\"}[$interval]) * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"}) * 100 / count(node_cpu_seconds_total{mode=\"user\"} * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"}) ",
+          "expr": "sum(label_replace(irate(node_cpu_seconds_total{mode=\"idle\", instance=~\".*$node_id.*\"}[$interval]), \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) * 100 / count(label_replace(node_cpu_seconds_total{mode=\"user\", instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\"))",
           "format": "time_series",
           "intervalFactor": 2,
           "legendFormat": "",
@@ -713,7 +721,39 @@
       "fieldConfig": {
         "defaults": {
           "color": {
-            "mode": "thresholds"
+            "mode": "palette-classic"
+          },
+          "custom": {
+            "axisBorderShow": false,
+            "axisCenteredZero": false,
+            "axisColorMode": "text",
+            "axisLabel": "",
+            "axisPlacement": "left",
+            "barAlignment": 0,
+            "drawStyle": "line",
+            "fillOpacity": 0,
+            "gradientMode": "none",
+            "hideFrom": {
+              "legend": false,
+              "tooltip": false,
+              "viz": false
+            },
+            "insertNulls": false,
+            "lineInterpolation": "linear",
+            "lineWidth": 1,
+            "pointSize": 5,
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "showPoints": "auto",
+            "spanNulls": false,
+            "stacking": {
+              "group": "A",
+              "mode": "none"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
           },
           "mappings": [
             {
@@ -747,33 +787,42 @@
           },
           "unit": "percent"
         },
-        "overrides": []
+        "overrides": [
+          {
+            "matcher": {
+              "id": "byType",
+              "options": "time"
+            },
+            "properties": [
+              {
+                "id": "custom.axisPlacement",
+                "value": "hidden"
+              }
+            ]
+          }
+        ]
       },
       "gridPos": {
-        "h": 4,
+        "h": 6,
-        "w": 2.6666666666666665,
+        "w": 6,
         "x": 0,
         "y": 8
       },
       "id": 10,
-      "links": [],
       "maxDataPoints": 100,
-      "maxPerRow": 12,
       "options": {
-        "colorMode": "value",
+        "legend": {
-        "graphMode": "area",
+          "calcs": [],
-        "justifyMode": "auto",
+          "displayMode": "list",
-        "orientation": "horizontal",
+          "placement": "bottom",
-        "reduceOptions": {
+          "showLegend": true
-          "calcs": [
-            "last"
-          ],
-          "fields": "",
-          "values": false
         },
-        "textMode": "auto"
+        "tooltip": {
+          "mode": "single",
+          "sort": "none"
+        }
       },
-      "pluginVersion": "10.0.2",
+      "pluginVersion": "10.4.14",
       "repeat": "node_id",
       "repeatDirection": "h",
       "targets": [
@@ -782,18 +831,20 @@
             "type": "prometheus",
             "uid": "PBFA97CFB590B2093"
           },
+          "editorMode": "code",
           "exemplar": true,
-          "expr": "sum((node_filesystem_free_bytes{fstype=\"ext4\",mountpoint=~\"(/$)|(/media.*)\"} / node_filesystem_size_bytes{fstype=\"ext4\",mountpoint=~\"(/$)|(/media.*)\"}) * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"} * 100) / count(node_meta * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"})",
+          "expr": "label_replace(node_filesystem_free_bytes{fstype=~\"ext4|xfs\", instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\") / label_replace(node_filesystem_size_bytes{fstype=~\"ext4|xfs\", instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\") * 100",
           "format": "time_series",
           "interval": "",
           "intervalFactor": 2,
-          "legendFormat": "",
+          "legendFormat": "{{mountpoint}}",
+          "range": true,
           "refId": "A",
           "step": 20
         }
       ],
       "title": "Available Disk Space $node_id",
-      "type": "stat"
+      "type": "timeseries"
     },
     {
       "aliasColors": {},
@@ -811,7 +862,7 @@
         "h": 7,
         "w": 24,
         "x": 0,
-        "y": 12
+        "y": 14
       },
       "hiddenSeries": false,
       "id": 14,
@@ -830,13 +881,12 @@
       },
       "lines": true,
       "linewidth": 1,
-      "links": [],
       "nullPointMode": "null as zero",
       "options": {
         "alertThreshold": true
       },
       "percentage": false,
-      "pluginVersion": "10.0.2",
+      "pluginVersion": "10.4.14",
       "pointradius": 5,
       "points": false,
       "renderer": "flot",
@@ -850,10 +900,10 @@
             "type": "prometheus",
             "uid": "PBFA97CFB590B2093"
           },
-          "expr": "100 - (avg(irate(node_cpu_seconds_total{mode=\"idle\"}[$interval])  * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"} * 100) by (node_name))",
+          "expr": "100 - (avg(label_replace(irate(node_cpu_seconds_total{mode=\"idle\", instance=~\".*$node_id.*\"}[$interval]), \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\") * 100) by (instance))",
           "format": "time_series",
           "intervalFactor": 2,
-          "legendFormat": "{{node_name}}",
+          "legendFormat": "{{instance}}",
           "refId": "A",
           "step": 2
         }
@@ -900,6 +950,7 @@
             "mode": "palette-classic"
           },
           "custom": {
+            "axisBorderShow": false,
             "axisCenteredZero": false,
             "axisColorMode": "text",
             "axisLabel": "",
@@ -913,6 +964,7 @@
               "tooltip": false,
               "viz": false
             },
+            "insertNulls": false,
             "lineInterpolation": "linear",
             "lineWidth": 1,
             "pointSize": 5,
@@ -961,12 +1013,11 @@
       },
       "gridPos": {
         "h": 7,
-        "w": 2.6666666666666665,
+        "w": 6,
         "x": 0,
-        "y": 19
+        "y": 21
       },
       "id": 15,
-      "links": [],
       "maxPerRow": 12,
       "options": {
         "legend": {
@@ -994,12 +1045,12 @@
             "uid": "PBFA97CFB590B2093"
           },
           "editorMode": "code",
-          "expr": "sum((node_memory_MemFree_bytes) * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"}) by (node_name)",
+          "expr": "sum(label_replace(node_memory_MemFree_bytes{instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) by (instance)",
           "format": "time_series",
           "hide": false,
           "interval": "",
           "intervalFactor": 2,
-          "legendFormat": "Free  {{node_name}}",
+          "legendFormat": "Free  {{instance}}",
           "range": true,
           "refId": "free",
           "step": 2
@@ -1010,12 +1061,12 @@
             "uid": "PBFA97CFB590B2093"
           },
           "editorMode": "code",
-          "expr": "sum((node_memory_Cached_bytes + node_memory_Buffers_bytes + node_memory_Slab_bytes) * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"}) by (node_name)",
+          "expr": "sum(label_replace(node_memory_Cached_bytes{instance=~\".*$node_id.*\"} + node_memory_Buffers_bytes{instance=~\".*$node_id.*\"} + node_memory_Slab_bytes{instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) by (instance)",
           "format": "time_series",
           "hide": false,
           "interval": "",
           "intervalFactor": 2,
-          "legendFormat": "cache,buffer,slab  {{node_name}}",
+          "legendFormat": "cache,buffer,slab  {{instance}}",
           "range": true,
           "refId": "cache,buffer,slab",
           "step": 2
@@ -1026,12 +1077,12 @@
             "uid": "PBFA97CFB590B2093"
           },
           "editorMode": "code",
-          "expr": "sum((node_memory_MemTotal_bytes - node_memory_MemFree_bytes - node_memory_Cached_bytes - node_memory_Buffers_bytes - node_memory_Slab_bytes) * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"}) by (node_name)",
+          "expr": "sum(label_replace(node_memory_MemTotal_bytes{instance=~\".*$node_id.*\"} - node_memory_MemFree_bytes{instance=~\".*$node_id.*\"} - node_memory_Cached_bytes{instance=~\".*$node_id.*\"} - node_memory_Buffers_bytes{instance=~\".*$node_id.*\"} - node_memory_Slab_bytes{instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) by (instance)",
           "format": "time_series",
           "hide": false,
           "interval": "",
           "intervalFactor": 2,
-          "legendFormat": "Used {{node_name}}",
+          "legendFormat": "Used {{instance}}",
           "range": true,
           "refId": "Used",
           "step": 2
@@ -1043,13 +1094,13 @@
           },
           "editorMode": "code",
           "exemplar": false,
-          "expr": "sum((node_memory_MemTotal_bytes) * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"}) by (node_name)",
+          "expr": "sum(label_replace(node_memory_MemTotal_bytes{instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) by (instance)",
           "format": "time_series",
           "hide": false,
           "instant": false,
           "interval": "",
           "intervalFactor": 2,
-          "legendFormat": "Total {{node_name}}",
+          "legendFormat": "Total {{instance}}",
           "range": true,
           "refId": "total",
           "step": 2
@@ -1074,7 +1125,7 @@
         "h": 7,
         "w": 24,
         "x": 0,
-        "y": 26
+        "y": 28
       },
       "hiddenSeries": false,
       "id": 16,
@@ -1091,13 +1142,12 @@
       },
       "lines": true,
       "linewidth": 1,
-      "links": [],
       "nullPointMode": "null as zero",
       "options": {
         "alertThreshold": true
       },
       "percentage": false,
-      "pluginVersion": "10.0.2",
+      "pluginVersion": "10.4.14",
       "pointradius": 5,
       "points": false,
       "renderer": "flot",
@@ -1111,11 +1161,11 @@
             "type": "prometheus",
             "uid": "PBFA97CFB590B2093"
           },
-          "expr": "sum(irate(node_disk_read_bytes_total[$interval]) * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"}) by (node_name)",
+          "expr": "sum(label_replace(irate(node_disk_read_bytes_total{instance=~\".*$node_id.*\"}[$interval]), \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) by (instance)",
           "format": "time_series",
           "interval": "",
           "intervalFactor": 2,
-          "legendFormat": "Read {{node_name}}",
+          "legendFormat": "Read {{instance}}",
           "refId": "A",
           "step": 2
         },
@@ -1124,10 +1174,10 @@
             "type": "prometheus",
             "uid": "PBFA97CFB590B2093"
           },
-          "expr": "sum(irate(node_disk_written_bytes_total[$interval]) * on(instance) group_left(node_name) node_meta{node_id=~\"$node_id\"}) by (node_name)",
+          "expr": "sum(label_replace(irate(node_disk_written_bytes_total{instance=~\".*$node_id.*\"}[$interval]), \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) by (instance)",
           "format": "time_series",
           "intervalFactor": 2,
-          "legendFormat": "Written {{node_name}}",
+          "legendFormat": "Written {{instance}}",
           "refId": "B",
           "step": 2
         }
@@ -1178,7 +1228,7 @@
         "h": 7,
         "w": 12,
         "x": 0,
-        "y": 33
+        "y": 35
       },
       "hiddenSeries": false,
       "id": 18,
@@ -1195,7 +1245,6 @@
       },
       "lines": true,
       "linewidth": 1,
-      "links": [],
       "nullPointMode": "null as zero",
       "options": {
         "alertThreshold": true
@@ -1215,10 +1264,10 @@
             "type": "prometheus",
             "uid": "PBFA97CFB590B2093"
           },
-          "expr": "sum(irate(node_disk_reads_completed_total[$interval]) * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"}) by (node_name)",
+          "expr": "sum(label_replace(irate(node_disk_reads_completed_total{instance=~\".*$node_id.*\"}[$interval]), \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) by (instance)",
           "format": "time_series",
           "intervalFactor": 2,
-          "legendFormat": "Reads {{node_name}}",
+          "legendFormat": "Reads {{instance}}",
           "refId": "A",
           "step": 2
         },
@@ -1227,10 +1276,10 @@
             "type": "prometheus",
             "uid": "PBFA97CFB590B2093"
           },
-          "expr": "sum(irate(node_disk_writes_completed_total[$interval]) * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"}) by (node_name)",
+          "expr": "sum(label_replace(irate(node_disk_writes_completed_total{instance=~\".*$node_id.*\"}[$interval]), \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) by (instance)",
           "format": "time_series",
           "intervalFactor": 2,
-          "legendFormat": "Writes {{node_name}}",
+          "legendFormat": "Writes {{instance}}",
           "refId": "B",
           "step": 2
         }
@@ -1281,7 +1330,7 @@
         "h": 7,
         "w": 12,
         "x": 12,
-        "y": 33
+        "y": 35
       },
       "hiddenSeries": false,
       "id": 19,
@@ -1300,7 +1349,6 @@
       },
       "lines": true,
       "linewidth": 1,
-      "links": [],
       "nullPointMode": "null as zero",
       "options": {
         "alertThreshold": true
@@ -1320,10 +1368,10 @@
             "type": "prometheus",
             "uid": "PBFA97CFB590B2093"
           },
-          "expr": "(avg(irate(node_cpu_seconds_total{mode=\"iowait\"}[$interval])  * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"} * 100) by (node_name))",
+          "expr": "(avg(label_replace(irate(node_cpu_seconds_total{mode=\"iowait\", instance=~\".*$node_id.*\"}[$interval]), \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\") * 100) by (instance))",
           "format": "time_series",
           "intervalFactor": 2,
-          "legendFormat": "{{node_name}}",
+          "legendFormat": "{{instance}}",
           "refId": "A",
           "step": 2
         }
@@ -1376,7 +1424,7 @@
         "h": 7,
         "w": 18,
         "x": 0,
-        "y": 40
+        "y": 42
       },
       "hiddenSeries": false,
       "id": 12,
@@ -1397,7 +1445,6 @@
       },
       "lines": true,
       "linewidth": 1,
-      "links": [],
       "nullPointMode": "null",
       "options": {
         "alertThreshold": true
@@ -1417,7 +1464,7 @@
             "type": "prometheus",
             "uid": "PBFA97CFB590B2093"
           },
-          "expr": "sum(rate(container_last_seen[5m]) * on(container_label_com_docker_swarm_node_id) group_left(node_name) node_meta{node_name=~\"${node_id}\"}) by (container_label_com_docker_swarm_service_name)",
+          "expr": "sum(rate(container_last_seen{instance=~\".*$node_id.*\"}[5m])) by (container_label_com_docker_swarm_service_name)",
           "format": "time_series",
           "hide": false,
           "intervalFactor": 10,
@@ -1499,10 +1546,9 @@
         "h": 7,
         "w": 6,
         "x": 18,
-        "y": 40
+        "y": 42
       },
       "id": 7,
-      "links": [],
       "maxDataPoints": 100,
       "options": {
         "colorMode": "none",
@@ -1527,7 +1573,7 @@
             "uid": "PBFA97CFB590B2093"
           },
           "exemplar": true,
-          "expr": "count(rate(container_last_seen[5m]) * on(container_label_com_docker_swarm_node_id) group_left(node_name) node_meta{node_name=~\"$node_id\"})",
+          "expr": "count(rate(container_last_seen{instance=~\".*$node_id.*\"}[5m]))",
           "format": "time_series",
           "interval": "",
           "intervalFactor": 2,
@@ -1600,10 +1646,9 @@
         "h": 7,
         "w": 24,
         "x": 0,
-        "y": 47
+        "y": 49
       },
       "id": 17,
-      "links": [],
       "options": {
         "legend": {
           "calcs": [],
@@ -1623,11 +1668,11 @@
             "type": "prometheus",
             "uid": "PBFA97CFB590B2093"
           },
-          "expr": "sum(rate(container_network_receive_bytes_total[$interval]) * on(container_label_com_docker_swarm_node_id) group_left(node_name) node_meta{node_name=~\"${node_id}\"}) by (node_name)",
+          "expr": "sum(label_replace(rate(container_network_receive_bytes_total{instance=~\".*$node_id.*\"}[$interval]), \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) by (instance)",
           "format": "time_series",
           "interval": "",
           "intervalFactor": 2,
-          "legendFormat": "IN {{node_name}}",
+          "legendFormat": "IN {{instance}}",
           "refId": "A",
           "step": 2
         },
@@ -1636,11 +1681,11 @@
             "type": "prometheus",
             "uid": "PBFA97CFB590B2093"
           },
-          "expr": "- sum(rate(container_network_transmit_bytes_total[$interval]) * on(container_label_com_docker_swarm_node_id) group_left(node_name) node_meta{node_name=~\"${node_id}\"}) by (node_name)",
+          "expr": "- sum(label_replace(rate(container_network_transmit_bytes_total{instance=~\".*$node_id.*\"}[$interval]), \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) by (instance)",
           "format": "time_series",
           "hide": false,
           "intervalFactor": 2,
-          "legendFormat": "OUT {{node_name}}",
+          "legendFormat": "OUT {{instance}}",
           "metric": "",
           "refId": "B",
           "step": 2
@@ -1658,7 +1703,7 @@
         "h": 9,
         "w": 24,
         "x": 0,
-        "y": 54
+        "y": 56
       },
       "id": 30,
       "options": {
@@ -1688,8 +1733,7 @@
     }
   ],
   "refresh": "",
-  "schemaVersion": 38,
+  "schemaVersion": 39,
-  "style": "dark",
   "tags": [
     "swarmprom",
     "prometheus",
@@ -1718,11 +1762,11 @@
         "name": "node_id",
         "options": [],
         "query": {
-          "query": "node_meta",
+          "query": "label_values(node_uname_info, instance)",
           "refId": "Prometheus-node_id-Variable-Query"
         },
-        "refresh": 1,
+        "refresh": 2,
-        "regex": "/node_name=\"([^\"]+)\"/",
+        "regex": "/^(?:(?:node|cadvisor|monitoring).)*(.*)$/",
         "skipUrlSync": false,
         "sort": 0,
         "type": "query"
@@ -1836,6 +1880,6 @@
   "timezone": "",
   "title": "Docker Swarm Nodes",
   "uid": "BPlb-Sgik",
-  "version": 24,
+  "version": 7,
   "weekStart": ""
-}
+}

grafana-traefik-dashboard.json

@@ -116,12 +116,12 @@
             "uid": "PBFA97CFB590B2093"
           },
           "exemplar": false,
-          "expr": "time() - process_start_time_seconds{job=\"$job\"}",
+          "expr": "time() - label_replace(process_start_time_seconds{instance=~\".*$domain.*\"} or process_start_time_seconds{domain=\"$domain\"}, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)",
           "format": "time_series",
           "instant": true,
           "interval": "",
           "intervalFactor": 2,
-          "legendFormat": "{{ instance }}",
+          "legendFormat": "{{ domain }}",
           "refId": "A"
         }
       ],
@@ -200,7 +200,7 @@
             "uid": "PBFA97CFB590B2093"
           },
           "exemplar": true,
-          "expr": "sum(increase(traefik_service_requests_total{code=\"499\", instance=\"$instance\"}[$interval]))",
+          "expr": "sum(label_replace(increase(traefik_service_requests_total{code=\"499\", instance=~\".*$domain.*\"}[$interval]) or increase(traefik_service_requests_total{code=\"499\", domain=\"$domain\"}[$interval]), \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`))",
           "interval": "",
           "legendFormat": "",
           "refId": "A"
@@ -280,7 +280,7 @@
             "uid": "PBFA97CFB590B2093"
           },
           "exemplar": true,
-          "expr": "sum(increase(traefik_service_requests_total{code=\"404\",protocol=~\"$protocol\", instance=\"$instance\"}[$interval]))",
+          "expr": "sum(label_replace(increase(traefik_service_requests_total{code=\"404\",protocol=~\"$protocol\", instance=~\".*$domain.*\"}[$interval]) or increase(traefik_service_requests_total{code=\"404\",protocol=~\"$protocol\", domain=\"$domain\"}[$interval]), \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`))",
           "format": "time_series",
           "interval": "",
           "intervalFactor": 2,
@@ -365,7 +365,7 @@
             "uid": "PBFA97CFB590B2093"
           },
           "exemplar": true,
-          "expr": "sum(increase(traefik_service_requests_total{code=\"200\",protocol=~\"$protocol\", instance=\"$instance\"}[$interval]))",
+          "expr": "sum(label_replace(increase(traefik_service_requests_total{code=\"200\",protocol=~\"$protocol\", instance=~\".*$domain.*\"}[$interval]) or increase(traefik_service_requests_total{code=\"200\",protocol=~\"$protocol\", domain=\"$domain\"}[$interval]), \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`))",
           "format": "time_series",
           "interval": "",
           "intervalFactor": 2,
@@ -417,7 +417,7 @@
             "uid": "PBFA97CFB590B2093"
           },
           "exemplar": true,
-          "expr": "topk(5, sum(traefik_service_requests_total{protocol=~\"$protocol\", instance=\"$instance\"}) by (code))",
+          "expr": "topk(5, sum(label_replace(traefik_service_requests_total{protocol=~\"$protocol\", instance=~\".*$domain.*\"} or traefik_service_requests_total{protocol=~\"$protocol\", domain=\"$domain\"}, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) by (code))",
           "format": "time_series",
           "interval": "",
           "intervalFactor": 2,
@@ -489,7 +489,7 @@
             "uid": "PBFA97CFB590B2093"
           },
           "exemplar": true,
-          "expr": "sum(increase(traefik_service_requests_total{code=\"499\",method=\"GET\", instance=\"$instance\"}[$interval])) by (service)",
+          "expr": "sum(label_replace(increase(traefik_service_requests_total{code=\"499\",method=\"GET\", instance=~\".*$domain.*\"}[$interval]) or increase(traefik_service_requests_total{code=\"499\",method=\"GET\", domain=\"$domain\"}[$interval]), \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) by (service)",
           "format": "time_series",
           "interval": "",
           "intervalFactor": 2,
@@ -590,7 +590,7 @@
             "uid": "PBFA97CFB590B2093"
           },
           "exemplar": true,
-          "expr": "(sum(traefik_entrypoint_request_duration_seconds_sum{protocol=~\"$protocol\", instance=\"$instance\"}) / sum(traefik_entrypoint_requests_total{protocol=~\"$protocol\", instance=\"$instance\"}) * 1000) - (sum(traefik_service_request_duration_seconds_sum{protocol=~\"$protocol\", instance=\"$instance\"}) / sum(traefik_service_requests_total{protocol=~\"$protocol\", instance=\"$instance\"}) * 1000)",
+          "expr": "(sum(label_replace(traefik_entrypoint_request_duration_seconds_sum{protocol=~\"$protocol\", instance=~\".*$domain.*\"} or traefik_entrypoint_request_duration_seconds_sum{protocol=~\"$protocol\", domain=\"$domain\"}, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) / sum(label_replace(traefik_entrypoint_requests_total{protocol=~\"$protocol\", instance=~\".*$domain.*\"} or traefik_entrypoint_requests_total{protocol=~\"$protocol\", domain=\"$domain\"}, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) * 1000) - (sum(label_replace(traefik_service_request_duration_seconds_sum{protocol=~\"$protocol\", instance=~\".*$domain.*\"} or traefik_service_request_duration_seconds_sum{protocol=~\"$protocol\", domain=\"$domain\"}, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) / sum(label_replace(traefik_service_requests_total{protocol=~\"$protocol\", instance=~\".*$domain.*\"} or traefik_service_requests_total{protocol=~\"$protocol\", domain=\"$domain\"}, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) * 1000)",
           "format": "time_series",
           "instant": false,
           "interval": "",
@@ -709,7 +709,7 @@
             "uid": "PBFA97CFB590B2093"
           },
           "exemplar": true,
-          "expr": "sum(traefik_entrypoint_request_duration_seconds_sum{instance=\"$instance\"}) / sum(traefik_entrypoint_requests_total{instance=\"$instance\"}) * 1000",
+          "expr": "sum(label_replace(traefik_entrypoint_request_duration_seconds_sum{instance=~\".*$domain.*\"} or traefik_entrypoint_request_duration_seconds_sum{domain=\"$domain\"}, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) / sum(label_replace(traefik_entrypoint_requests_total{instance=~\".*$domain.*\"} or traefik_entrypoint_requests_total{domain=\"$domain\"}, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) * 1000",
           "format": "time_series",
           "interval": "",
           "intervalFactor": 2,
@@ -781,7 +781,7 @@
           },
           "editorMode": "code",
           "exemplar": true,
-          "expr": "sum(delta(traefik_service_requests_total{instance=\"${instance:raw}\"}[$interval]))",
+          "expr": "sum(label_replace(delta(traefik_service_requests_total{instance=~\".*$domain.*\"}[$interval]) or delta(traefik_service_requests_total{domain=\"$domain\"}[$interval]), \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`))",
           "format": "time_series",
           "interval": "",
           "intervalFactor": 2,
@@ -879,7 +879,7 @@
             "uid": "PBFA97CFB590B2093"
           },
           "exemplar": true,
-          "expr": "sum(rate(traefik_service_requests_total{instance=\"$instance\"}[$interval]))",
+          "expr": "sum(label_replace(rate(traefik_service_requests_total{instance=~\".*$domain.*\"}[$interval]) or rate(traefik_service_requests_total{domain=\"$domain\"}[$interval]), \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`))",
           "format": "time_series",
           "interval": "",
           "intervalFactor": 2,
@@ -984,7 +984,7 @@
           },
           "editorMode": "code",
           "exemplar": true,
-          "expr": "sum(rate(traefik_service_request_duration_seconds_sum{ instance=\"$instance\" }[5m])) by(service)",
+          "expr": "sum(label_replace(rate(traefik_service_request_duration_seconds_sum{instance=~\".*$domain.*\"}[5m]) or rate(traefik_service_request_duration_seconds_sum{domain=\"$domain\"}[5m]), \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) by(service)",
           "format": "time_series",
           "interval": "",
           "intervalFactor": 2,
@@ -1053,11 +1053,11 @@
             "uid": "PBFA97CFB590B2093"
           },
           "exemplar": true,
-          "expr": "process_open_fds{job=~\"$job\", instance=\"$instance\"}",
+          "expr": "label_replace(process_open_fds{instance=~\".*$domain.*\"} or process_open_fds{domain=\"$domain\"}, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)",
           "format": "time_series",
           "interval": "",
           "intervalFactor": 2,
-          "legendFormat": "{{ instance }}",
+          "legendFormat": "{{ domain }}",
           "refId": "A",
           "step": 240
         }
@@ -1154,7 +1154,7 @@
             "uid": "PBFA97CFB590B2093"
           },
           "exemplar": true,
-          "expr": "sum(rate(traefik_service_requests_total{protocol=~\"http|https\", instance=\"$instance\"}[$interval])) by (service)",
+          "expr": "sum(label_replace(rate(traefik_service_requests_total{protocol=~\"http|https\", instance=~\".*$domain.*\"}[$interval]) or rate(traefik_service_requests_total{protocol=~\"http|https\", domain=\"$domain\"}[$interval]), \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) by (service)",
           "format": "time_series",
           "interval": "",
           "intervalFactor": 2,
@@ -1255,7 +1255,7 @@
             "uid": "PBFA97CFB590B2093"
           },
           "exemplar": true,
-          "expr": "sum(traefik_entrypoint_open_connections{instance=\"$instance\"}) by (method)",
+          "expr": "sum(label_replace(traefik_entrypoint_open_connections{instance=~\".*$domain.*\"} or traefik_entrypoint_open_connections{domain=\"$domain\"}, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) by (method)",
           "format": "time_series",
           "interval": "",
           "intervalFactor": 1,
@@ -1355,7 +1355,7 @@
             "uid": "PBFA97CFB590B2093"
           },
           "exemplar": true,
-          "expr": "sum(traefik_service_open_connections{instance=\"$instance\"}) by (method)",
+          "expr": "sum(label_replace(traefik_service_open_connections{instance=~\".*$domain.*\"} or traefik_service_open_connections{domain=\"$domain\"}, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) by (method)",
           "format": "time_series",
           "interval": "",
           "intervalFactor": 1,
@@ -1459,7 +1459,7 @@
             "uid": "PBFA97CFB590B2093"
           },
           "exemplar": true,
-          "expr": "sum(increase(traefik_service_requests_total{protocol=~\"$protocol\", instance=\"$instance\"}[$interval])) by (code)",
+          "expr": "sum(label_replace(increase(traefik_service_requests_total{protocol=~\"$protocol\", instance=~\".*$domain.*\"}[$interval]) or increase(traefik_service_requests_total{protocol=~\"$protocol\", domain=\"$domain\"}[$interval]), \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) by (code)",
           "format": "time_series",
           "interval": "",
           "intervalFactor": 2,
@@ -1514,36 +1514,6 @@
   ],
   "templating": {
     "list": [
-      {
-        "current": {
-          "selected": false,
-          "text": "default",
-          "value": "default"
-        },
-        "datasource": {
-          "type": "prometheus",
-          "uid": "PBFA97CFB590B2093"
-        },
-        "definition": "",
-        "hide": 0,
-        "includeAll": false,
-        "label": "Job:",
-        "multi": false,
-        "name": "job",
-        "options": [],
-        "query": {
-          "query": "label_values(job)",
-          "refId": "Prometheus-job-Variable-Query"
-        },
-        "refresh": 1,
-        "regex": "",
-        "skipUrlSync": false,
-        "sort": 2,
-        "tagValuesQuery": "",
-        "tagsQuery": "",
-        "type": "query",
-        "useTags": false
-      },
       {
         "current": {
           "selected": true,
@@ -1569,7 +1539,7 @@
           "query": "label_values(traefik_service_requests_total, protocol)",
           "refId": "Prometheus-protocol-Variable-Query"
         },
-        "refresh": 1,
+        "refresh": 2,
         "regex": "",
         "skipUrlSync": false,
         "sort": 0,
@@ -1663,19 +1633,19 @@
           "type": "prometheus",
           "uid": "PBFA97CFB590B2093"
         },
-        "definition": "label_values(instance)",
+        "definition": "query_result(label_replace(traefik_config_reloads_total, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`))",
         "hide": 0,
         "includeAll": false,
-        "label": "Instance:",
+        "label": "Domain:",
         "multi": false,
-        "name": "instance",
+        "name": "domain",
         "options": [],
         "query": {
-          "query": "label_values(instance)",
+          "query": "query_result(label_replace(traefik_config_reloads_total, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`))",
           "refId": "PrometheusVariableQueryEditor-VariableQuery"
         },
-        "refresh": 1,
+        "refresh": 2,
-        "regex": ".*8082",
+        "regex": "/domain=\"([^\"]+)\"/",
         "skipUrlSync": false,
         "sort": 1,
         "tagValuesQuery": "",

grafana_custom.ini

@@ -21,7 +21,7 @@ tls_skip_verify_insecure = false
 allow_sign_up = true
 auto_login = true
 client_id = {{ env "OIDC_CLIENT_ID" }}
-client_secret = {{ secret "grafana_oidc_client_secret" }}
+client_secret = {{ secret "gf_oidc_secret" }}
 auth_url = {{ env "OIDC_AUTH_URL" }}
 token_url = {{ env "OIDC_TOKEN_URL" }}
 api_url = {{ env "OIDC_API_URL" }}

loki.yml.tmpl

@@ -34,7 +34,6 @@ ingester:
   max_chunk_age: 1h           # All chunks will be flushed when they hit this age, default is 1h
   chunk_target_size: 1048576  # Loki will attempt to build chunks up to 1.5MB, flushing first if chunk_idle_period or max_chunk_age is reached first
   chunk_retain_period: 30s    # Must be greater than index read cache TTL if using an index cache (Default index read cache TTL is 5m)
-  max_transfer_retries: 0     # Chunk transfers disabled
   wal:
     dir: "/tmp/wal"
 
@@ -53,7 +52,7 @@ schema_config:
     - from: 2020-10-24
       store: boltdb-shipper
       object_store: filesystem
-      schema: v11
+      schema: v13
       index:
         prefix: index_
         period: 24h
@@ -63,7 +62,6 @@ storage_config:
     active_index_directory: /loki/boltdb-shipper-active
     cache_location: /loki/boltdb-shipper-cache
     cache_ttl: 24h         # Can be increased for faster performance over longer query periods, uses more disk space
-    shared_store: filesystem
   filesystem:
     directory: /loki/chunks
 {{ end }}
@@ -72,7 +70,6 @@ schema_config:
   configs:
   - from: 2020-11-25
     store: boltdb-shipper
-    object_store: aws
     schema: v11
     index:
       prefix: index_
@@ -89,7 +86,7 @@ storage_config:
    endpoint: {{ env "LOKI_AWS_ENDPOINT" }}
    region: {{ env "LOKI_AWS_REGION" }}
    access_key_id: {{ env "LOKI_ACCESS_KEY_ID" }}
-   secret_access_key: {{ secret "loki_aws_secret_access_key" }}
+   secret_access_key: {{ secret "loki_aws_key" }}
    bucketnames: {{ env "LOKI_BUCKET_NAMES" }}
    insecure: false
    sse_encryption: false
@@ -103,19 +100,24 @@ storage_config:
 
 compactor:
   working_directory: /loki/boltdb-shipper-compactor
-  shared_store: filesystem
   compaction_interval: 10m
   retention_enabled: true
   retention_delete_delay: 2h
   retention_delete_worker_count: 150
+{{ if eq (env "LOKI_STORAGE_FILESYSTEM") "1" }}
+  delete_request_store: filesystem
+{{ end }}
+{{ if eq (env "LOKI_STORAGE_S3") "1" }}
+  delete_request_store: aws
+{{ end }}
 
 limits_config:
-  enforce_metric_name: false
   reject_old_samples: true
   reject_old_samples_max_age: 168h
   retention_period: {{ env "LOKI_RETENTION_PERIOD" }}
   split_queries_by_interval: 24h
   max_query_parallelism: 100
+  allow_structured_metadata: false
 
 query_scheduler:
   max_outstanding_requests_per_tenant: 4096
@@ -123,9 +125,6 @@ query_scheduler:
 frontend:
   max_outstanding_per_tenant: 4096
 
-chunk_store_config:
-  max_look_back_period: 0s
-
 table_manager:
   retention_deletes_enabled: false
   retention_period: 0s

node-exporter-entrypoint.sh

@@ -1,11 +0,0 @@
-#!/bin/sh -e
-
-NODE_NAME=$(cat /etc/nodename)
-
-mkdir -p /etc/node-exporter
-
-echo "node_meta{node_id=\"$NODE_ID\", container_label_com_docker_swarm_node_id=\"$NODE_ID\", node_name=\"$NODE_NAME\"} 1" > /etc/node-exporter/node-meta.prom
-
-set -- /bin/node_exporter "$@"
-
-exec "$@"

promtail.yml.tmpl

@@ -1,37 +0,0 @@
-server:
-  http_listen_port: 9080
-  grpc_listen_port: 0
-
-positions:
-  filename: /tmp/positions.yaml
-
-clients:
-  - url: {{ env "LOKI_PUSH_URL" }}
-    basic_auth:
-      username: admin
-      password: {{ secret "basic_auth" }}
-    external_labels:
-      hostname: {{ env "DOMAIN" }}
-
-scrape_configs:
-- job_name: system
-  static_configs:
-  - targets:
-      - localhost
-    labels:
-      job: varlogs
-      __path__: /var/log/*log
-
-- job_name: "docker"
-  docker_sd_configs:
-    - host: "unix:///var/run/docker.sock"
-      refresh_interval: "10s"
-  relabel_configs:
-    - source_labels: ['__meta_docker_container_name']
-      target_label: "container_name"
-    - source_labels: ['__meta_docker_container_id']
-      target_label: "container_id"
-    - source_labels: ['__meta_docker_container_label_com_docker_stack_namespace']
-      target_label: "stack_namespace"
-    - source_labels: ['__meta_docker_container_label_com_docker_swarm_service_name']
-      target_label: "service_name"

release/2.0.0+v1.18.1

@@ -0,0 +1,64 @@
+BREAKING CHANGE
+Migration plan for upgrading from 1.6.0+v1.8.1.
+
+## 1. Reinsert secrets with shortened names
+
+Secret and config names were shortened to max 14 characters to prevent going over Docker's 64 character
+limit when STACK_NAME and VERSION are added to it.
+
+- `abra app secret list <domain>` to see which secrets are missing under their new name
+- `abra app cmd --local <domain> migrate_secret_names` to reinsert all of them automatically
+      (or manually: `abra app secret insert <domain> <secret_name> v1 <value>` per secret)
+
+## 2. If you use OIDC (moved to seperate compose file)
+
+- Add to your .env: `COMPOSE_FILE="$COMPOSE_FILE:compose.grafana-oidc.yml"`
+
+## 3. If you use SMTP (moved to a seperate compose file)
+
+- Add to your .env: `COMPOSE_FILE="$COMPOSE_FILE:compose.grafana-smtp.yml"`
+
+## 4. node_exporter/cadvisor/promtail replaced by Grafana Alloy
+
+Metrics collection changed from Prometheus scraping endpoints
+to Alloy pushing via `remote_write`/`loki push`.
+
+- Remove `compose.promtail.yml`, `compose.expose-ports.yml` and
+      `compose.basic-auth.yml` from your .env if present. They no
+      longer exist. `compose.yml` now declares the `basic_auth` secret directly, so
+      `SECRET_BASIC_AUTH_VERSION` is always required.
+- Add `PROMETHEUS_REMOTE_WRITE_URL=https://prometheus.$DOMAIN/api/v1/write`
+      (`$DOMAIN` if this host also runs `compose.prometheus.yml`, otherwise a remote
+      Prometheus' URL). Without this, Alloy collects no metrics at all.
+- Add `LOKI_PUSH_URL` (existing var, still used) and pick a log source:
+      `JOURNALD=1` (systemd hosts), `SYSLOG_FILES=1` (non-systemd, tails
+      `/var/log/*log`), or `SYSLOG=1` + `compose.syslog.yml` (network syslog listener).
+
+- If this host had its own `node.$DOMAIN`/`cadvisor.$DOMAIN` scrape target
+      configured on a central Prometheus, remove it. Those endpoints are gone.
+- `scrape-config.example.yml` and the `add_node`/`add_domain` abra.sh commands
+      are gone. Replaced by label-based auto-discovery (see README).
+- `docker stack deploy` doesn't prune removed services, so old `cadvisor`/
+      `promtail` containers keep running after a normal `abra app deploy`. Run
+      `abra app undeploy <domain>` then `abra app deploy <domain>` to clear them out.
+- Diff your `.env` against the current `.env.sample`, to verify any other changes.
+
+### New: label-based metrics auto-discovery
+
+Alloy now auto-discovers and scrapes other Docker Swarm services on the same
+host/`proxy` network that opt in via `prometheus.io/scrape=true` deploy labels.
+See the README's "Auto-discovering metrics from other apps" section.
+
+- If you scrape Traefik metrics: the old `metrics.traefik.$domain` pull-based
+      endpoint still works if you keep the scrape config in Prometheus and
+      existing dashboards keep showing its data, but it's recommended to get
+      Traefik onto the new label-based discovery.
+
+### Dashboards
+
+The Swarm, Stacks and Traefik dashboards were reworked to show old (pull-model)
+and new (Alloy push-model) data as one continuous line, so you don't lose history
+across the migration.
+
+The disk usage per container was disabled per default because of the CPU-expensive 
+filesystem scan (30% cpu increase). Enable by uncommenting env CADVISOR_DISK_USAGE=1

scrape-config.example.yml

@@ -1,4 +0,0 @@
-- targets:
-  - 'example.org:8082'
-  - 'node.monitoring.example.org'
-  - 'cadvisor.monitoring.example.org'