make disk usage per container metric optional

closes #20

---------

Co-authored-by: Danny Groenewegen <danny@ecommons.space>
Reviewed-on: #21
Reviewed-by: ammaratef45 <ammaratef45@proton.me>
Reviewed-by: Danny Groenewegen <247+dannygroenewegen@noreply.git.coopcloud.tech>
Co-authored-by: f <f@sutty.nl>

.env.sample

@@ -5,15 +5,54 @@ DOMAIN=monitoring-ng.example.com
 #TIMEOUT=120
 ENABLE_BACKUPS=true
 
-## Enable this secret for Promtail / Prometheus
+# Secret Alloy authenticates with when writing metrics/logs to Prometheus/
-# SECRET_BASIC_AUTH_VERSION=v1
+# Loki and what Traefik's basicauth middleware expects for that
-#
+SECRET_BASIC_AUTH_VERSION=v1
-# Promtail (Gathering Logs)
+# Username sent along with SECRET_BASIC_AUTH_VERSION above (default: admin)
-# COMPOSE_FILE="$COMPOSE_FILE:compose.promtail.yml"
+# WRITE_BASIC_AUTH_USERNAME=admin
-# LOKI_PUSH_URL=https://loki.monitoring.example.org/loki/api/v1/push
 
-## Expose node and cadvisor ports instead of traefik
+# Expose Alloy's web UI publicly (behind basic-auth) at alloy.$DOMAIN.
-# COMPOSE_FILE="$COMPOSE_FILE:compose.expose-ports.yml"
+# COMPOSE_FILE="$COMPOSE_FILE:compose.alloy-webui.yml"
+# Enable Live Debugging in web ui
+# LIVE_DEBUGGING=false
+
+# Enable this to send metrics to a Prometheus server, adapt DOMAIN if
+# server is remote
+# PROMETHEUS_REMOTE_WRITE_URL=https://prometheus.$DOMAIN/api/v1/write
+
+# Enable container filesystem usage metrics (expensive du-scan, ~30% CPU on large hosts)
+# CADVISOR_DISK_USAGE=1
+
+# Enable authenticated scraping of containers that opt in via
+# prometheus.io/auth=basic or prometheus.io/auth=bearer labels (used as
+# password/bearer token respectively). Insert it with:
+# abra app secret insert <domain> scrape_auth v1 <password>
+# COMPOSE_FILE="$COMPOSE_FILE:compose.scrape-auth.yml"
+# SECRET_SCRAPE_AUTH_VERSION=v1
+#
+# Username sent along with it (default: alloy)
+# SCRAPE_BASIC_AUTH_USERNAME=alloy
+
+# Edit this if your distribution sets this socket to another location
+# CONTAINERD_SOCKET=/var/run/containerd/containerd.sock
+
+# Enable this to send logs to a Loki server, adapt DOMAIN if server is
+# remote
+# LOKI_PUSH_URL=https://loki.$DOMAIN/loki/api/v1/push
+# Enable on systemd hosts to read logs from the journal
+# JOURNALD=1
+#
+# Enable on non-systemd hosts (Alpine, older Debian/Ubuntu) to tail
+# /var/log/*log files (syslog, auth.log, kern.log, etc.) that a local
+# syslogd writes. No syslogd reconfiguration needed.
+# SYSLOG_FILES=1
+#
+# Enable to receive syslog messages over the network on port 514/tcp.
+# Use for remote devices that push syslog to this host, or for a 
+# local syslogd configured to forward over the network.
+# Not needed if you just want to read local log files — use SYSLOG_FILES instead.
+# SYSLOG=1
+# COMPOSE_FILE="$COMPOSE_FILE:compose.syslog.yml"
 
 # Monitoring Server
 #
@@ -46,8 +85,9 @@ ENABLE_BACKUPS=true
 # COMPOSE_FILE="$COMPOSE_FILE:compose.grafana.yml"
 # GF_SERVER_ROOT_URL=https://monitoring.example.com
 # SECRET_GF_ADMINPASSWD_VERSION=v1
-## Seperate domain for Grafana
+## Grafana's own domain. Defaults to $DOMAIN
-#GRAFANA_DOMAIN=grafana.example.com
+## Change the value if you want Grafana on another domain.
+# GRAFANA_DOMAIN=$DOMAIN
 #
 ## Single-Sign-On with OIDC
 # COMPOSE_FILE="$COMPOSE_FILE:compose.grafana-oidc.yml"
@@ -69,7 +109,7 @@ ENABLE_BACKUPS=true
 # GF_SMTP_ENABLED=true
 # GF_SMTP_FROM_ADDRESS=grafana@example.com
 # GF_SMTP_SKIP_VERIFY=false
-# SECRET_GF_SMTP_PASSWD_VERSION=v1
+# SECRET_GF_SMTP_PASS_VERSION=v1
 #
 
 ## Grafana Matrix Contact Point (optional)
@@ -79,9 +119,10 @@ ENABLE_BACKUPS=true
 #GF_MATRIX_ROOM_ID="<room-id>"
 #GF_MATRIX_HOMESERVER_URL="<homeserver-url>"
 
-# ALerts
+## ALerts
-#ALERT_BACKUP_FAILED_ENABLED=true
+
-#ALERT_BACKUP_MISSING_ENABLED=true
+# Node disk space alert will trigger when free disk space left is below the given number in percent
-#ALERT_BACKUP_NOT_SUCCESSFULL_ENABLED=true
+#ALERT_NODE_DISK_SPACE_LEFT=10
-#ALERT_NODE_DISK_SPACE_ENABLED=true
+
-#ALERT_NODE_MEMORY_USAGE_ENABLED=true
+# Node memory usage alert will trigger when memory usage is above the given number in percent
+#ALERT_NODE_MEMORY_USAGE=85

README.md

@@ -1,9 +1,10 @@
 # monitoring-ng
 
 Yet another monitoring stack ...
-This time its a all-in-one grafana/prometheus/loki/node_exporter/cadvisor/promtail stack.
+This time its a all-in-one grafana/prometheus/loki/alloy stack.
-It's based heavily on the [monitoring-lite](https://git.coopcloud.tech/coop-cloud/monitoring-lite) stack, but has everything in one recipe included now. So you can deploy monitoring instances to only gather metrics / logs (node_exporter/cadvisor/promtail) and also deploy instances with the full monitoring stack (grafana/prometheus/loki) with the same recipe and just different .env configuration.
+It's based heavily on the [monitoring-lite](https://git.coopcloud.tech/coop-cloud/monitoring-lite) stack, but has everything in one recipe included now. So you can deploy monitoring instances to only gather metrics / logs (alloy) and also deploy instances with the full monitoring stack (grafana/prometheus/loki) with the same recipe and just different .env configuration.
 
+Metrics and logs are collected by [Grafana Alloy](https://grafana.com/docs/alloy/latest/) and pushed to a central Prometheus/Loki (via `remote_write`/`loki push`). Every `monitoring-ng` instance runs its own Alloy, whether or not it also runs the central stack.
 
 <!-- metadata -->
 
@@ -18,79 +19,93 @@ It's based heavily on the [monitoring-lite](https://git.coopcloud.tech/coop-clou
 
 <!-- endmetadata -->
 
-## Setup a Metrics Gathering
+## Setup: gathering-only host
 
-Where gathering.org is the node you want to gather metrics from.
+Deploys just Alloy, pushing this host's own node/container metrics and logs to a central instance. Use this on every host you want metrics/logs from.
 
-1. Configure DNS
+1. `abra app new monitoring-ng --server gathering.org`
-  - monitoring.gathering.org
+2. `abra app config monitoring.gathering.org`
-  - cadvisor.monitoring.gathering.org
+3. Point it at your central instance:
-  - node.monitoring.gathering.org
+   ```
-1. Configure Traefik to use BasicAuth
+   PROMETHEUS_REMOTE_WRITE_URL=https://prometheus.example.org/api/v1/write
-  * `abra app config traefik.gathering.org`
+   LOKI_PUSH_URL=https://loki.example.org/loki/api/v1/push
-  uncomment
+   JOURNALD=1   # or SYSLOG_FILES=1 / SYSLOG=1, see .env.sample
-  ```
+   ```
-  # BASIC_AUTH
+4. `abra app secret insert monitoring.gathering.org basic_auth v1 <password>`. Same username/password as the `usersfile` credential configured for Traefik's basicauth on the central node (see below). This is what Alloy authenticates with when pushing metrics/logs. Alloy defaults to username `admin` for this. Uf the Traefik `usersfile` uses a different username, set `WRITE_BASIC_AUTH_USERNAME` in this recipe to match.
-  COMPOSE_FILE="$COMPOSE_FILE:compose.basicauth.yml"
+5. `abra app deploy monitoring.gathering.org`
-  BASIC_AUTH=1
-  SECRET_USERSFILE_VERSION=v1
-  ```
-  - Generate userslist with httpasswd hashed password
-    `abra app secret insert traefik.gathering.org usersfile v1 'admin:<hashed-secret>'`
-    make sure there is no whitespace in between `admin:<hashed-secret>`, it seems to break stuff...
-  - `abra app deploy -f traefik`
-1. `abra app new monitoring-ng`
-1. `abra app config monitoring.gathering.org`
-   for gathering only the main `compose.yml` is needed, nothing more.
-1. `abra app deploy monitoring.gathering.org`
-1. check that endpoints are up and basic-auth works
-  - cadvisor.monitoring.gathering.org
-  - node.monitoring.gathering.org
 
-### Expose node and cadvisor via ports instead of traefik
+## Setup: full monitoring stack (metrics/logs browser)
 
-In case you have no traefik running on the machine, you can expose the ports directly by uncommenting the following line:
+This is what a gathering host pushes into. It also runs its own Alloy, so it monitors itself too.
-```
+
-# COMPOSE_FILE="$COMPOSE_FILE:compose.expose-ports.yml"
+1. Configure DNS: `monitoring.example.org`, plus `prometheus.`/`loki.`/`pushgateway.` subdomains for whichever of those you enable below
+2. Traefik on this node needs basic auth configured (`BASIC_AUTH=1`, see the Traefik recipe's "Configuring basic auth" section) — Prometheus/Loki/Pushgateway route through its `basicauth@file` middleware, so without it those endpoints won't work. Use the same username (default `admin`, see `WRITE_BASIC_AUTH_USERNAME`) and password you'll insert as the `basic_auth` secret below when generating Traefik's `usersfile`.
+3. `abra app config monitoring.example.org` Uncomment `compose.prometheus.yml` (metrics), `compose.loki.yml` (logs) and `compose.grafana.yml` (dashboard)
+4. `abra app secret insert monitoring.example.org basic_auth v1 <password>` — this is the password every gathering host's Alloy (including this instance's own) authenticates with; also what Traefik's basicauth expects on the public Prometheus/Loki/Pushgateway endpoints
+5. `abra app secret insert monitoring.example.org gf_adminpasswd v1 <password>`
+6. `abra app deploy monitoring.example.org`
+
+### Post-setup guide
+
+- configure the SMTP mailer under `Alerting > Contact points`
+  - edit the default contact point, choose "Alertmanager" as type & `http://alertmanager:9093` as URL
+  - use the "Test" button to send a test mail. It should fire a request at the alertmanager & that should send a mail
+- from your dashboard panels, choose `Edit > Alert` to create alerts based on those panels
+
+## Additional features
+
+### Discovering metrics from other apps
+
+Alloy auto-discovers and scrapes other Docker Swarm services running on the same host, on the `proxy` network, that opt in via labels. No manual scrape config needed. On the app's `compose.yml`:
+
+```yaml
+deploy:
+  labels:
+    - "prometheus.io/scrape=true"       # required: opt in
+    # - "prometheus.io/port=8082"       # optional: defaults to 80
+    # - "prometheus.io/path=/metrics"   # optional: defaults to /metrics
+    # - "prometheus.io/auth=basic"      # optional: basic auth, see below
+    # - "prometheus.io/auth=bearer"     # optional: bearer token auth
 ```
 
-## Setup Metrics Browser
+Each scraped target gets `instance` (`<service_name>.<slot>`, stable across redeploys), `domain` (the service's stack namespace with underscores converted back to dots, e.g. `traefik.example.com`) and `task_slot` labels attached automatically.
 
+If the target needs authentication, configure the monitoring-ng instance with a scrape-auth secret that will be used for targets having the auth label set:
+```
+COMPOSE_FILE="$COMPOSE_FILE:compose.scrape-auth.yml"
+SECRET_SCRAPE_AUTH_VERSION=v1
+```
+`abra app secret insert <domain> scrape_auth v1 <password-or-token>`, then set the scraped app's `prometheus.io/auth` label to `basic` or `bearer` to match how it checks the secret.
 
-1. Configure DNS
+Check discovered targets via `alloy.example.org` (needs `compose.alloy-webui.yml`, see below), or query the central Prometheus for `up{job="<service_name>"}`.
-   - monitoring.example.org
-   - prometheus.monitoring.example.org
-   - loki.monitoring.example.org
-2. Setup monitoring stack
-   - `abra app new monitoring-ng`
-   - `abra app config monitoring.example.org`
-      Uncomment all the stuff
-   - `abra app secret insert monitoring.example.org basic_auth v1 <secret>`
-      this needs the plaintext traefik basic-auth secret, not the hashed one!
-   - `abra app secret ls monitoring.example.org`
-   - `abra app deploy monitoring.example.org`
-3. Add scrape config to prometheus
-  - `abra app cmd monitoring.example.org prometheus gathering.org`
-  - or manually
-    ```
-    cp scrape-config.example.yml gathering.org.yml
-    # adjust domain
-    # mkdir scrape_configs
-    abra app cp monitoring.dev.local-it.cloud gathering.org.yml prometheus:/prometheus/scrape_configs/
-    ```
 
-* check that all configured targets are up:
+### Manual scraping
-  https://prometheus.monitoring.example.org/targets
 
+For targets where auto-discovery doesn't work (e.g. not a Docker Swarm service on the `proxy` network, or missing labels): add them directly to Prometheus instead. Create a scrape config file:
+```yaml
+- targets:
+    - 'metrics.something-external.example.org'
+    - 'app-without-labels.example.org'
+```
+and copy it into Prometheus' scrape config directory:
+```
+abra app cp monitoring.gathering.org targets.yml prometheus:/prometheus/scrape_configs/
+```
+Prometheus picks up files there automatically.
 
+### Alloy Web UI / Live Debugging
 
-| Service       | Authentication     | Domain                            |
+Alloy's own web UI isn't exposed by default, its HTTP server only listens
-| ------------- | ------------------ | --------------------------------- |
+on localhost inside its own container. To reach it from outside (e.g. to
-| Grafana       | Email / SSO        | monitoring.example.org            |
+browse its component graph or use live debugging), add:
-| Prometheus    | traefik basic-auth | prometheus.monitoring.example.org |
+```
-| loki          | traefik basic-auth | loki.monitoring.example.org       |
+COMPOSE_FILE="$COMPOSE_FILE:compose.alloy-webui.yml"
-| Cadvisor      | traefik basic-auth | cadvisor.monitoring.example.org   |
+```
-| Node Exporter | traefik basic-auth | node.monitoring.example.org       |
+This exposes it (behind the same basic-auth) at `alloy.$DOMAIN`.
+
+To actually see what's being collected (live-tailing the metrics/logs
+flowing through each component, not just their config) also set
+`LIVE_DEBUGGING=true`. Only enable this while troubleshooting.
 
 ### Logging from a docker host to loki server without anything else
 
@@ -107,36 +122,25 @@ $ echo '{
 $ systemctl restart docker.service
 ```
 
-## Setup Push Gateway
+### Setup Push Gateway
 
-1. Enable in the env fiöle by uncommenting the following lines:
+1. Enable in the env file by uncommenting the following lines:
 ```
 ## Prometheus Pushgateway
 # COMPOSE_FILE="$COMPOSE_FILE:compose.pushgateway.yml"
 ```
 2. `abra app deploy monitoring.example.org`
 
-This will expose the pushgateway at `https://pushgateway.${DOMAIN}`.
+This will expose the pushgateway at `https://pushgateway.${DOMAIN}`, secured behind the same basic auth as the other services.
-It is secured behind the same basic auth as the other services.
+After that you need to add the `pushgateway.${DOMAIN}` to the scrape config of Prometheus.
-After that you need to add the `pushgateway.${DOMAIN}` to the scare config.
 
-## Post-setup guide
-
-- configure prometheus/loki/alertmanager as data sources in grafana under `Configuration > Data sources`
-  - for loki, you need to set a "Custom HTTP Header": `X-Scope-OrgID: fake`
-- configure the SMTP mailer under `Alerting > Contact points`
-  - edit the default contact point, choose "Alertmanager" as type & `http://alertmanager:9093` as URL
-  - use the "Test" button to send a test mail. It should fire a request at the alertmanager & that should send a mail
-- `abra app cp` your `scrap_configs: ...` into `/prometheus/scrape_configs` & log into your prometheus web UI to ensure they're working
-- load your dashboards in manually under `Create > Dashboard`
-- from your dashboard panels, choose `Edit > Alert` to create alerts based on those panels
 
 
 ---
 
 THX to the previous work of @decentral1se @knooflok @3wc @cellarspoon @mirsal
 
-## Adding Matrix as Alert Contact point
+### Adding Matrix as Alert Contact point
 
 1. Enable the [matrix-alertmanager-receiver](https://github.com/metio/matrix-alertmanager-receiver/):
 ```
@@ -156,13 +160,9 @@ GF_MATRIX_HOME_SERVER_URL=
 ```
 4. Configure Alertmanager webhook and set the url to `http://matrix-alertmanager-receiver:12345/alerts/<room-id>`
 
-## alerts
+### Alerts
-
-It is possible to enable the following alerts, by setting the corresponding env variable to `true`:
-- backupbot failed: `ALERT_BACKUP_FAILED_ENABLED`
-- backupbot missing: `ALERT_BACKUP_MISSING_ENABLED`
-- backupbot not successfull: `ALERT_BACKUP_NOT_SUCCESSFULL_ENABLED`
-- node disk space: `ALERT_NODE_DISK_SPACE_ENABLED`
-- node memory usage: `ALERT_NODE_MEMORY_USAGE_ENABLED`
 
+It is possible to enable the following alerts, by uncommenting the corresponding env variable:
 
+- node disk space: `ALERT_NODE_DISK_SPACE_LEFT`
+- node memory usage: `ALERT_NODE_MEMORY_USAGE`

abra.sh

@@ -1,26 +1,16 @@
-export ENTRYPOINT_VERSION=v1
 export GF_DATASOURCES_VERSION=v1
-export GF_DASHBOARDS_VERSION=v2
+export GF_DASHBOARDS_VERSION=v3
-export GF_SWARM_DASH_VERSION=v2
+export GF_SWARM_DASH_VERSION=v3
-export GF_STACKS_DASH_VERSION=v2
+export GF_STACKS_DASH_VERSION=v3
-export GF_TRAEFIK_DASH_VERSION=v2
+export GF_TRAEFIK_DASH_VERSION=v3
 export GF_BACKUP_DASH_VERSION=v1
 export GF_CUSTOM_INI_VERSION=v4
-export PROMTAIL_YML_VERSION=v3
 export LOKI_YML_VERSION=v3
 export PROMETHEUS_YML_VERSION=v2
-export MATRIX_ALERTMANAGER_CONFIG_VERSION=e
+export MATRIX_ALERTMANAGER_CONFIG_VERSION=v1
-export MATRIX_ALERTMANAGER_ENTRYPOINT_VERSION=a
+export MATRIX_ALERTMANAGER_ENTRYPOINT_VERSION=v1
-export GRAFANA_ALERTS_NODE_VERSION=v1c
+export GF_ALERTS_NODE_VERSION=v3
-
+export CONFIG_ALLOY_VERSION=v2
-# creates a default prometheus scrape config for a given node
-add_node(){
-  name=$1
-  add_domain "$name" "metrics.traefik.$name"
-  add_domain "$name" "node.monitoring.$name"
-  add_domain "$name" "cadvisor.monitoring.$name"
-  cat "/prometheus/scrape_configs/$name.yml"
-}
 
 # migrates secrets from old names to new names by reading values from the
 # running containers on the server and re-inserting them under the new names.
@@ -37,7 +27,7 @@ migrate_secret_names() {
   # Hardcoded migration mappings: old_secret_name|new_secret_name
   MIGRATIONS="
 grafana_admin_password|gf_adminpasswd
-grafana_smtp_password|gf_smtp_passwd
+grafana_smtp_password|gf_smtp_pass
 grafana_oidc_client_secret|gf_oidc_secret
 matrix_access_token|matrix_token
 loki_aws_secret_access_key|loki_aws_key
@@ -116,18 +106,3 @@ loki_aws_secret_access_key|loki_aws_key
   echo ""
   echo "Done."
 }
-
-# adds a domain to a scrape config or creates a new one
-add_domain(){
-  name=$1
-  domain=$2
-  if [ ! -d "/prometheus/scrape_configs/" ]; then
-    mkdir -p /prometheus/scrape_configs/
-  fi
-  cd /prometheus/scrape_configs/ || exit 1
-  if [ ! -f "$name.yml" ]; then
-    echo -e "- targets:\n  - '$domain'" > "$name.yml"
-  else
-    echo "  - '$domain'"  >> "$name.yml"
-  fi
-}

alerts/node.yml.tmpl

@@ -2,13 +2,13 @@ apiVersion: 1
 
 # List of alert rule UIDs that should be deleted
 deleteRules:
-  {{ if ne (env "ALERT_NODE_DISK_SPACE_ENABLED") "true" }}
+  {{ if not (env "ALERT_NODE_DISK_SPACE_LEFT") }}
   - orgId: 1
-    uid: bds8bhxu97pxca
+    uid: coopcloud_node_disk_space_left
   {{ end }}
-  {{ if ne (env "ALERT_NODE_MEMORY_USAGE_ENABLED") "true" }}
+  {{ if not (env "ALERT_NODE_MEMORY_USAGE") }}
   - orgId: 1
-    uid: ads8cswmly96oa
+    uid: coopcloud_node_memory_usage
   {{ end }}
 
 groups:
@@ -17,8 +17,8 @@ groups:
   folder: node
   interval: 5m
   rules:
-    {{ if eq (env "ALERT_NODE_DISK_SPACE_ENABLED") "true" }}
+    {{ if (env "ALERT_NODE_DISK_SPACE_LEFT") }}
-    - uid: bds8bhxu97pxca
+    - uid: coopcloud_node_disk_space_left
       title: Node Disk Space
       condition: C
       data:
@@ -29,7 +29,7 @@ groups:
           datasourceUid: PBFA97CFB590B2093
           model:
             editorMode: code
-            expr: (node_filesystem_free_bytes{fstype="ext4"} / node_filesystem_size_bytes{fstype="ext4"}) * 100
+            expr: (node_filesystem_free_bytes{fstype=~"ext4|xfs"} / node_filesystem_size_bytes{fstype=~"ext4|xfs"}) * 100
             instant: true
             intervalMs: 1000
             legendFormat: __auto
@@ -45,7 +45,7 @@ groups:
             conditions:
                 - evaluator:
                     params:
-                        - 10
+                        - {{ env "ALERT_NODE_DISK_SPACE_LEFT" }}
                     type: lt
                   operator:
                     type: and
@@ -70,13 +70,13 @@ groups:
       annotations:
         description: ""
         runbook_url: ""
-        summary: Less than 10% disk space left on {{`{{ $labels.instance }}`}} ({{`{{ (index $values "A").Value }}`}}% left)
+        summary: Less than {{ env "ALERT_NODE_DISK_SPACE_LEFT" }}% disk space left on {{`{{ $labels.instance }}`}} ({{`{{ (index $values "A").Value }}`}}% left)
       labels:
         "": ""
       isPaused: false
     {{ end }}
-    {{ if eq (env "ALERT_NODE_MEMORY_USAGE_ENABLED") "true" }}
+    {{ if (env "ALERT_NODE_MEMORY_USAGE")  }}
-    - uid: ads8cswmly96oa
+    - uid: coopcloud_node_memory_usage
       title: Node Memory Usage
       condition: C
       data:
@@ -103,7 +103,7 @@ groups:
             conditions:
                 - evaluator:
                     params:
-                        - 85
+                        - {{ env "ALERT_NODE_MEMORY_USAGE"  }}
                     type: gt
                   operator:
                     type: and
@@ -126,6 +126,6 @@ groups:
       execErrState: Error
       for: 5m
       annotations:
-        summary: Memory usage is above 85% on {{`{{ $labels.instance }}`}}  ({{`{{ printf "%.2f" (index $values "A").Value }}`}}%  usage)
+        summary: Memory usage is above {{ env "ALERT_NODE_MEMORY_USAGE"  }}% on {{`{{ $labels.instance }}`}}  ({{`{{ printf "%.2f" (index $values "A").Value }}`}}%  usage)
       isPaused: false
     {{ end }}

compose.alloy-webui.yml

@@ -0,0 +1,16 @@
+version: "3.8"
+
+services:
+  app:
+    environment:
+      - ALLOY_HTTP_LISTEN_ADDR=0.0.0.0
+    deploy:
+      labels:
+        - "traefik.enable=true"
+        - "traefik.swarm.network=proxy"
+        - "traefik.http.services.${STACK_NAME}-alloy.loadbalancer.server.port=12345"
+        - "traefik.http.routers.${STACK_NAME}-alloy.rule=Host(`alloy.${DOMAIN}`)"
+        - "traefik.http.routers.${STACK_NAME}-alloy.entrypoints=web-secure"
+        - "traefik.http.routers.${STACK_NAME}-alloy.tls=true"
+        - "traefik.http.routers.${STACK_NAME}-alloy.tls.certresolver=${LETS_ENCRYPT_ENV}"
+        - "traefik.http.routers.${STACK_NAME}-alloy.middlewares=basicauth@file"

compose.expose-ports.yml

@@ -1,13 +0,0 @@
----
-version: "3.8"
-
-services:
-  app:
-    ports:
-      - "9100:9100"
-    deploy:
-
-  cadvisor:
-    ports:
-      - "9101:8080"
-    deploy:

compose.grafana-smtp.yml

@@ -3,16 +3,16 @@ version: '3.8'
 services:
   grafana:
     secrets:
-      - gf_smtp_passwd
+      - gf_smtp_pass
     environment:
       - GF_SMTP_HOST
       - GF_SMTP_USER
-      - GF_SMTP_PASSWORD__FILE=/run/secrets/gf_smtp_passwd
+      - GF_SMTP_PASSWORD__FILE=/run/secrets/gf_smtp_pass
       - GF_SMTP_ENABLED
       - GF_SMTP_FROM_ADDRESS
       - GF_SMTP_SKIP_VERIFY
 
 secrets:
-  gf_smtp_passwd:
+  gf_smtp_pass:
     external: true
-    name: ${STACK_NAME}_gf_smtp_passwd_${SECRET_GF_SMTP_PASSWD_VERSION}
+    name: ${STACK_NAME}_gf_smtp_pass_${SECRET_GF_SMTP_PASS_VERSION}

compose.grafana.yml

@@ -2,7 +2,7 @@ version: '3.8'
 
 services:
   grafana:
-    image: grafana/grafana:12.4.0
+    image: grafana/grafana:13.0.6
     volumes:
       - grafana-data:/var/lib/grafana:rw
     secrets:
@@ -32,19 +32,19 @@ services:
       - GF_SECURITY_ADMIN_PASSWORD__FILE=/run/secrets/gf_adminpasswd
       - GF_SECURITY_ALLOW_EMBEDDING
       - GF_INSTALL_PLUGINS
-      - ALERT_NODE_DISK_SPACE_ENABLED
+      - ALERT_NODE_DISK_SPACE_LEFT
-      - ALERT_NODE_MEMORY_USAGE_ENABLED
+      - ALERT_NODE_MEMORY_USAGE
     deploy:
       labels:
         - "traefik.enable=true"
-        - "traefik.docker.network=proxy"
+        - "traefik.swarm.network=proxy"
         - "traefik.http.services.${STACK_NAME}-grafana.loadbalancer.server.port=3000"
         - "traefik.http.routers.${STACK_NAME}-grafana.rule=Host(`${GRAFANA_DOMAIN:-$DOMAIN}`)"
         - "traefik.http.routers.${STACK_NAME}-grafana.entrypoints=web-secure"
         - "traefik.http.routers.${STACK_NAME}-grafana.tls=true"
         - "traefik.http.routers.${STACK_NAME}-grafana.tls.certresolver=${LETS_ENCRYPT_ENV}"
     healthcheck:
-      test: "wget -q http://localhost:3000/ -O/dev/null"
+      test: "wget -q http://localhost:3000/healthz -O/dev/null"
       interval: 5s
       timeout: 10s
       retries: 3
@@ -75,7 +75,7 @@ configs:
     file: grafana-backup-dashboard.json
   gf_alerts_node:
     template_driver: golang
-    name: ${STACK_NAME}_gf_alerts_node_${GRAFANA_ALERTS_NODE_VERSION}
+    name: ${STACK_NAME}_gf_alerts_node_${GF_ALERTS_NODE_VERSION}
     file: alerts/node.yml.tmpl
 
 volumes:

compose.loki.yml

@@ -2,7 +2,7 @@ version: '3.8'
 
 services:
   loki:
-    image: grafana/loki:3.6.7
+    image: grafana/loki:3.7.2
     command: -config.file=/etc/loki/local-config.yaml
     networks:
       - proxy
@@ -27,7 +27,7 @@ services:
         condition: on-failure
       labels:
         - "traefik.enable=true"
-        - "traefik.docker.network=proxy"
+        - "traefik.swarm.network=proxy"
         - "traefik.http.services.${STACK_NAME}-loki.loadbalancer.server.port=3100"
         - "traefik.http.routers.${STACK_NAME}-loki.rule=Host(`loki.${DOMAIN}`)"
         - "traefik.http.routers.${STACK_NAME}-loki.entrypoints=web-secure"

compose.prometheus.yml

@@ -2,7 +2,7 @@ version: '3.8'
 
 services:
   prometheus:
-    image: prom/prometheus:v3.10.0
+    image: prom/prometheus:v3.12.0
     secrets:
       - basic_auth
     volumes:
@@ -16,20 +16,28 @@ services:
       - "--web.console.libraries=/usr/share/prometheus/console_libraries"
       - "--web.console.templates=/usr/share/prometheus/consoles"
       - "--storage.tsdb.retention.time=${PROMETHEUS_RETENTION_TIME}"
+      - "--web.enable-remote-write-receiver"
     networks:
       - proxy
       - internal
+    healthcheck:
+      test: "wget -q --spider http://localhost:9090/-/healthy || exit 1"
+      interval: 5s
+      timeout: 10s
+      retries: 3
+      start_period: 30s
     deploy:
       restart_policy:
         condition: on-failure
       labels:
         - "traefik.enable=true"
-        - "traefik.docker.network=proxy"
+        - "traefik.swarm.network=proxy"
         - "traefik.http.services.${STACK_NAME}-prometheus.loadbalancer.server.port=9090"
         - "traefik.http.routers.${STACK_NAME}-prometheus.rule=Host(`prometheus.${DOMAIN}`)"
         - "traefik.http.routers.${STACK_NAME}-prometheus.entrypoints=web-secure"
         - "traefik.http.routers.${STACK_NAME}-prometheus.tls=true"
         - "traefik.http.routers.${STACK_NAME}-prometheus.tls.certresolver=${LETS_ENCRYPT_ENV}"
+        - "traefik.http.routers.${STACK_NAME}-prometheus.middlewares=basicauth@file"
 
 configs:
   prometheus_yml:

compose.promtail.yml

@@ -1,30 +0,0 @@
-version: "3.8"
-
-services:
-  promtail:
-    image: grafana/promtail:3.6.7
-    volumes:
-      - /var/log:/var/log:ro
-      - /var/run/docker.sock:/var/run/docker.sock
-    command: -config.file=/etc/promtail/config.yml
-    configs:
-      - source: promtail_yml
-        target: /etc/promtail/config.yml
-    networks:
-      - internal
-    secrets:
-      - basic_auth
-    environment:
-      - DOMAIN
-      - LOKI_PUSH_URL
-
-configs:
-  promtail_yml:
-    name: ${STACK_NAME}_promtail_yml_${PROMTAIL_YML_VERSION}
-    file: promtail.yml.tmpl
-    template_driver: golang
-
-secrets:
-  basic_auth:
-    external: true
-    name: ${STACK_NAME}_basic_auth_${SECRET_BASIC_AUTH_VERSION}

compose.pushgateway.yml

@@ -12,12 +12,18 @@ services:
     networks:
       - internal
       - proxy
+    healthcheck:
+      test: "wget -q --spider http://localhost:9191/-/healthy || exit 1"
+      interval: 5s
+      timeout: 10s
+      retries: 3
+      start_period: 10s
     deploy:
       restart_policy:
         condition: on-failure
       labels:
         - "traefik.enable=true"
-        - "traefik.docker.network=proxy"
+        - "traefik.swarm.network=proxy"
         - "traefik.http.services.${STACK_NAME}-pushgateway.loadbalancer.server.port=9191"
         - "traefik.http.routers.${STACK_NAME}-pushgateway.rule=Host(`pushgateway.${DOMAIN}`)"
         - "traefik.http.routers.${STACK_NAME}-pushgateway.entrypoints=web-secure"

compose.scrape-auth.yml

@@ -0,0 +1,13 @@
+---
+version: "3.8"
+
+services:
+  app:
+    secrets:
+      - source: scrape_auth
+        target: scrape_auth
+
+secrets:
+  scrape_auth:
+    external: true
+    name: ${STACK_NAME}_scrape_auth_${SECRET_SCRAPE_AUTH_VERSION}

compose.syslog.yml

@@ -0,0 +1,6 @@
+---
+version: "3.8"
+services:
+  app:
+    ports:
+    - "514:514"

compose.yml

@@ -3,89 +3,69 @@ version: "3.8"
 
 services:
   app:
-    image: prom/node-exporter:v1.10.2
+    image: grafana/alloy:v1.18.1
-    user: root
+    hostname: "${DOMAIN}"
-    environment:
-      - NODE_ID={{.Node.ID}}
-    volumes:
-      - /proc:/host/proc:ro
-      - /sys:/host/sys:ro
-      - /:/rootfs:ro
-      - /etc/hostname:/etc/nodename:ro
-    command:
-      - "--path.sysfs=/host/sys"
-      - "--path.procfs=/host/proc"
-      - "--path.rootfs=/rootfs"
-      - "--collector.textfile.directory=/etc/node-exporter/"
-      - "--collector.filesystem.ignored-mount-points=^/(sys|proc|dev|host|etc)($$|/)"
-      - "--no-collector.ipvs"
     configs:
-      - source: entrypoint
+      - source: config_alloy
-        target: /entrypoint.sh
+        target: /etc/alloy/config.alloy
+    volumes:
+      - /:/rootfs:ro
+      - /var/run/docker.sock:/var/run/docker.sock:ro
+      - /sys:/sys:ro
+      - /var/lib/docker:/var/lib/docker:ro
+      # long-form avoids toolshed/abra#900
+      - type: bind
+        source: "${CONTAINERD_SOCKET:-/run/containerd/containerd.sock}"
+        target: /run/containerd/containerd.sock
+        read_only: true
+      - alloy-data:/var/lib/alloy/data
+    # runs through a shell so ALLOY_HTTP_LISTEN_ADDR (set by
+    # compose.alloy-webui.yml) is resolved from the container's own
+    # environment at startup, not by compose at deploy time.
+    # $$ escapes it from compose's own interpolation
+    entrypoint: ["/bin/sh", "-c"]
+    command:
+      - >-
+        exec alloy run
+        --storage.path=/var/lib/alloy/data
+        --server.http.listen-addr=$${ALLOY_HTTP_LISTEN_ADDR:-127.0.0.1}:12345
+        /etc/alloy/config.alloy
     networks:
-      - internal
       - proxy
-    entrypoint: [ "/bin/sh", "-e", "/entrypoint.sh" ]
+      - internal
+    environment:
+      - SCRAPE_BASIC_AUTH_USERNAME=${SCRAPE_BASIC_AUTH_USERNAME:-alloy}
+      - WRITE_BASIC_AUTH_USERNAME=${WRITE_BASIC_AUTH_USERNAME:-admin}
+      - LIVE_DEBUGGING=${LIVE_DEBUGGING:-false}
+      - NODE_ID={{.Node.ID}}
+    secrets:
+      - basic_auth
+    # no wget/curl in this image; bash's /dev/tcp is used instead. Works
+    # against localhost regardless of ALLOY_HTTP_LISTEN_ADDR
+    healthcheck:
+      test: ["CMD", "bash", "-c", "exec 3<>/dev/tcp/localhost/12345 && printf 'GET /-/ready HTTP/1.0\r\nHost: localhost\r\n\r\n' >&3 && head -1 <&3 | grep -q 200"]
+      interval: 5s
+      timeout: 10s
+      retries: 3
+      start_period: 10s
     deploy:
       restart_policy:
         condition: on-failure
       labels:
         - "backupbot.backup=${ENABLE_BACKUPS:-true}"
-        - "traefik.enable=true"
-        - "traefik.docker.network=proxy"
-        - "traefik.http.services.${STACK_NAME}-node.loadbalancer.server.port=9100"
-        - "traefik.http.routers.${STACK_NAME}-node.rule=Host(`node.${DOMAIN}`)"
-        - "traefik.http.routers.${STACK_NAME}-node.entrypoints=web-secure"
-        - "traefik.http.routers.${STACK_NAME}-node.tls=true"
-        - "traefik.http.routers.${STACK_NAME}-node.tls.certresolver=${LETS_ENCRYPT_ENV}"
-        - "traefik.http.routers.${STACK_NAME}-node.middlewares=basicauth@file"
         - "coop-cloud.${STACK_NAME}.version=1.6.0+v1.8.1"
-        - "coop-cloud.${STACK_NAME}.timeout=${TIMEOUT}"
-
-  cadvisor:
-    image: gcr.io/cadvisor/cadvisor:v0.55.1
-    command:
-      - "-logtostderr"
-      - "--enable_metrics=cpu,cpuLoad,disk,diskIO,process,memory,network"
-      # all possible metrics: advtcp,app,cpu,cpuLoad,cpu_topology,cpuset,disk,diskIO,hugetlb,memory,memory_numa,network,oom_event,percpu,perf_event,process,referenced_memory,resctrl,sched,tcp,udp.
-      - "--housekeeping_interval=120s"
-      - "--docker_only=true"
-    volumes:
-      - /var/lib/docker/:/var/lib/docker:ro
-      - /dev/disk/:/dev/disk:ro
-      - /sys:/sys:ro
-      - /var/run:/var/run:ro
-      - /:/rootfs:ro
-    networks:
-      - internal
-      - proxy
-    deploy:
-      restart_policy:
-        condition: on-failure
-      labels:
-        - "traefik.enable=true"
-        - "traefik.docker.network=proxy"
-        - "traefik.http.services.${STACK_NAME}-cadvisor.loadbalancer.server.port=8080"
-        - "traefik.http.routers.${STACK_NAME}-cadvisor.rule=Host(`cadvisor.${DOMAIN}`)"
-        - "traefik.http.routers.${STACK_NAME}-cadvisor.entrypoints=web-secure"
-        - "traefik.http.routers.${STACK_NAME}-cadvisor.tls=true"
-        - "traefik.http.routers.${STACK_NAME}-cadvisor.tls.certresolver=${LETS_ENCRYPT_ENV}"
-        - "traefik.http.routers.${STACK_NAME}-cadvisor.middlewares=basicauth@file"
-    healthcheck:
-      test: wget --quiet --tries=1 --spider http://localhost:8080/healthz || exit 1
-      interval: 15s
-      timeout: 15s
-      retries: 5
-      start_period: 30s
-
 configs:
-  entrypoint:
+  config_alloy:
-    name: ${STACK_NAME}_entrypoint_${ENTRYPOINT_VERSION}
+    template_driver: golang
-    file: node-exporter-entrypoint.sh
+    name: ${STACK_NAME}_config_alloy_${CONFIG_ALLOY_VERSION}
-
+    file: config.alloy.tmpl
-
-
 networks:
   proxy:
     external: true
   internal:
+volumes:
+  alloy-data:
+secrets:
+  basic_auth:
+    external: true
+    name: ${STACK_NAME}_basic_auth_${SECRET_BASIC_AUTH_VERSION}

config.alloy.tmpl

@@ -0,0 +1,355 @@
+logging {
+  level = "info"
+  format = "logfmt"
+}
+
+livedebugging {
+  enabled = {{ env "LIVE_DEBUGGING" }}
+}
+
+discovery.docker "linux" {
+  host = "unix:///var/run/docker.sock"
+}
+
+{{ if ne (env "PROMETHEUS_REMOTE_WRITE_URL") "" }}
+prometheus.exporter.cadvisor "docker" {
+  docker_only = true
+  {{ if eq (env "CADVISOR_DISK_USAGE") "1" }}
+  enabled_metrics = ["cpu", "cpuLoad", "disk", "diskIO", "memory", "network"]
+  {{ else }}
+  enabled_metrics = ["cpu", "cpuLoad", "diskIO", "memory", "network"]
+  {{ end }}
+  // host-wide totals already come from prometheus.exporter.unix
+  disable_root_cgroup_stats = true
+}
+
+prometheus.exporter.unix "default" {
+  include_exporter_metrics = true
+  rootfs_path = "/rootfs"
+  procfs_path = "/rootfs/proc"
+  sysfs_path = "/rootfs/sys"
+
+  disable_collectors = ["ipvs"]
+
+  filesystem {
+    fs_types_exclude     = "^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|tmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$"
+    mount_points_exclude = "^/(sys|proc|dev|host|etc)($|/)"
+    mount_timeout        = "5s"
+  }
+
+  netclass { ignored_devices = "^(veth.*)$" }
+  netdev   { device_exclude  = "^(veth.*)$" }
+}
+
+prometheus.exporter.self "alloy" {}
+
+prometheus.scrape "default" {
+  scrape_interval = "120s"
+
+  targets = array.concat(
+    prometheus.exporter.self.alloy.targets,
+    prometheus.exporter.unix.default.targets,
+    prometheus.exporter.cadvisor.docker.targets,
+  )
+
+  forward_to = [prometheus.remote_write.prometheus.receiver]
+}
+
+prometheus.remote_write "prometheus" {
+  endpoint {
+    url = "{{ env "PROMETHEUS_REMOTE_WRITE_URL" }}"
+
+    basic_auth {
+      username = "{{ env "WRITE_BASIC_AUTH_USERNAME" }}"
+      password = "{{ secret "basic_auth" }}"
+    }
+  }
+}
+
+// Scrape Prometheus metrics from other containers on this host.
+// Containers opt in via Docker labels:
+//   prometheus.io/scrape=true        required: enable scraping
+//   prometheus.io/port=9090          optional: port exposing /metrics (defaults to 80 if not set)
+//   prometheus.io/path=/metrics      optional: path to metrics endpoint (default: /metrics)
+//   prometheus.io/auth=basic         optional: use basic auth with the scrape_auth secret (see compose.scrape-auth.yml)
+//   prometheus.io/auth=bearer        optional: use bearer auth with the scrape_auth secret (see compose.scrape-auth.yml)
+discovery.dockerswarm "swarm" {
+  host = "unix:///var/run/docker.sock"
+  // "tasks" not "services": dnsrr-mode services (e.g. traefik) have no VIP
+  // and are invisible to the "services" role
+  role = "tasks"
+}
+
+discovery.relabel "metrics" {
+  targets = discovery.dockerswarm.swarm.targets
+
+  // skip old task history, only scrape currently-running tasks
+  rule {
+    source_labels = ["__meta_dockerswarm_task_desired_state"]
+    regex         = "running"
+    action        = "keep"
+  }
+
+  // only scrape hosts running on this host within the swam
+  rule {
+    source_labels = ["__meta_dockerswarm_node_id"]
+    regex         = "{{ env "NODE_ID" }}"
+    action        = "keep"
+  }
+
+  rule {
+    source_labels = ["__meta_dockerswarm_network_name"]
+    regex         = "proxy"
+    action        = "keep"
+  }
+
+  rule {
+    source_labels = ["__meta_dockerswarm_service_label_prometheus_io_scrape"]
+    regex         = "true"
+    action        = "keep"
+  }
+
+  // default to port 80 when prometheus.io/port isn't set
+  rule {
+    source_labels = ["__meta_dockerswarm_service_label_prometheus_io_port"]
+    regex         = "^$"
+    target_label  = "__meta_dockerswarm_service_label_prometheus_io_port"
+    replacement   = "80"
+  }
+
+  // a task with multiple published ports produces one target per port;
+  // this unifies all of them to the single port above, so duplicates
+  // collapse at scrape time instead of scraping every port
+  rule {
+    source_labels = ["__address__", "__meta_dockerswarm_service_label_prometheus_io_port"]
+    regex         = `(.+):\d+;(\d+)`
+    target_label  = "__address__"
+    replacement   = "$1:$2"
+  }
+
+  rule {
+    source_labels = ["__meta_dockerswarm_service_label_prometheus_io_path"]
+    regex         = `(.+)`
+    target_label  = "__metrics_path__"
+  }
+
+  rule {
+    source_labels = ["__meta_dockerswarm_service_name"]
+    target_label  = "job"
+  }
+
+  // task IDs (and the default address-derived instance label) change on
+  // every redeploy; service+slot is stable across redeploys of the same
+  // replica, so data stays continuous instead of restarting each deploy
+  rule {
+    source_labels = ["__meta_dockerswarm_service_name", "__meta_dockerswarm_task_slot"]
+    separator     = "."
+    target_label  = "instance"
+  }
+
+  rule {
+    source_labels = ["__meta_dockerswarm_task_slot"]
+    target_label  = "task_slot"
+  }
+
+  // coop-cloud's STACK_NAME is the domain with "." replaced by "_"
+  // Derive a readable dotted domain label from it. RE2 has no global
+  // replace, so this is done by chaining multiple replacements. Each
+  // rule swaps the first remaining _ for a "." until none are left.
+  rule {
+    source_labels = ["__meta_dockerswarm_service_label_com_docker_stack_namespace"]
+    target_label  = "domain"
+  }
+  rule {
+    source_labels = ["domain"]
+    regex         = `([^_]*)_(.*)`
+    target_label  = "domain"
+    replacement   = "$1.$2"
+  }
+  rule {
+    source_labels = ["domain"]
+    regex         = `([^_]*)_(.*)`
+    target_label  = "domain"
+    replacement   = "$1.$2"
+  }
+  rule {
+    source_labels = ["domain"]
+    regex         = `([^_]*)_(.*)`
+    target_label  = "domain"
+    replacement   = "$1.$2"
+  }
+  rule {
+    source_labels = ["domain"]
+    regex         = `([^_]*)_(.*)`
+    target_label  = "domain"
+    replacement   = "$1.$2"
+  }
+  rule {
+    source_labels = ["domain"]
+    regex         = `([^_]*)_(.*)`
+    target_label  = "domain"
+    replacement   = "$1.$2"
+  }
+  rule {
+    source_labels = ["domain"]
+    regex         = `([^_]*)_(.*)`
+    target_label  = "domain"
+    replacement   = "$1.$2"
+  }
+}
+
+discovery.relabel "metrics_noauth" {
+  targets = discovery.relabel.metrics.output
+  rule {
+    source_labels = ["__meta_dockerswarm_service_label_prometheus_io_auth"]
+    regex         = "^$"
+    action        = "keep"
+  }
+}
+
+discovery.relabel "metrics_basicauth" {
+  targets = discovery.relabel.metrics.output
+  rule {
+    source_labels = ["__meta_dockerswarm_service_label_prometheus_io_auth"]
+    regex         = "basic"
+    action        = "keep"
+  }
+}
+
+discovery.relabel "metrics_bearerauth" {
+  targets = discovery.relabel.metrics.output
+  rule {
+    source_labels = ["__meta_dockerswarm_service_label_prometheus_io_auth"]
+    regex         = "bearer"
+    action        = "keep"
+  }
+}
+
+prometheus.scrape "containers" {
+  scrape_interval = "120s"
+  targets         = discovery.relabel.metrics_noauth.output
+  forward_to      = [prometheus.remote_write.prometheus.receiver]
+}
+
+{{ if ne (env "SECRET_SCRAPE_AUTH_VERSION") "" }}
+prometheus.scrape "containers_basicauth" {
+  scrape_interval = "120s"
+  targets         = discovery.relabel.metrics_basicauth.output
+  forward_to      = [prometheus.remote_write.prometheus.receiver]
+  basic_auth {
+    username = "{{ env "SCRAPE_BASIC_AUTH_USERNAME" }}"
+    password = "{{ secret "scrape_auth" }}"
+  }
+}
+
+prometheus.scrape "containers_bearerauth" {
+  scrape_interval = "120s"
+  targets         = discovery.relabel.metrics_bearerauth.output
+  forward_to      = [prometheus.remote_write.prometheus.receiver]
+  bearer_token    = "{{ secret "scrape_auth" }}"
+}
+{{ end }}
+{{ end }}
+
+{{ if ne (env "LOKI_PUSH_URL") "" }}
+discovery.relabel "docker" {
+  targets = discovery.docker.linux.targets
+
+  rule {
+    source_labels = ["__meta_docker_container_name"]
+    target_label  = "container_name"
+  }
+  rule {
+    source_labels = ["__meta_docker_container_id"]
+    target_label  = "container_id"
+  }
+  rule {
+    source_labels = ["__meta_docker_container_label_com_docker_stack_namespace"]
+    target_label  = "stack_namespace"
+  }
+  rule {
+    source_labels = ["__meta_docker_container_label_com_docker_swarm_service_name"]
+    target_label  = "service_name"
+  }
+  rule {
+    source_labels = ["__meta_docker_container_log_stream"]
+    target_label  = "stream"
+  }
+}
+
+loki.source.docker "docker" {
+  host       = "unix:///var/run/docker.sock"
+  targets    = discovery.relabel.docker.output
+  labels     = {"app" = "docker"}
+  forward_to = [loki.write.loki.receiver]
+}
+
+// JOURNALD: reads the systemd journal binary log directly.
+// Use on systemd hosts (most modern Linux distros). Requires no syslogd.
+{{ if eq (env "JOURNALD") "1" }}
+loki.source.journal "journal" {
+  path = "/rootfs/var/log/journal"
+  labels = { job = "{{ env "DOMAIN" }}" }
+  forward_to = [loki.write.loki.receiver]
+}
+{{ end }}
+
+// SYSLOG_FILES: tails all /var/log/*log files (syslog, auth.log, kern.log, etc.).
+// Use on non-systemd hosts where a syslogd writes to /var/log.
+{{ if eq (env "SYSLOG_FILES") "1" }}
+local.file_match "syslog_files" {
+  path_targets = [{ __path__ = "/rootfs/var/log/*log" }]
+}
+
+loki.source.file "syslog_files" {
+  targets    = local.file_match.syslog_files.targets
+  forward_to = [loki.process.syslog_files.receiver]
+}
+
+loki.process "syslog_files" {
+  stage.static_labels {
+    values = { job = "syslog" }
+  }
+  forward_to = [loki.write.loki.receiver]
+}
+{{ end }}
+
+// SYSLOG: opens a network syslog listener on port 514.
+// Use when a remote device or a local syslogd configured to 
+// forward over the network sends logs to this host.
+// Requires compose.syslog.yml to publish port 514 to the host.
+// This is NOT needed for reading local log files — use SYSLOG_FILES instead.
+{{ if eq (env "SYSLOG") "1" }}
+loki.relabel "syslog" {
+  rule {
+    action = "labelmap"
+    regex = "__syslog_(.+)"
+  }
+
+  forward_to = []
+}
+
+loki.source.syslog "syslog" {
+  listener {
+    address = "[::]:514"
+    label_structured_data = true
+    labels = { component = "loki.source.syslog" }
+  }
+
+  relabel_rules = loki.relabel.syslog.rules
+  forward_to = [loki.write.loki.receiver]
+}
+{{ end }}
+
+loki.write "loki" {
+  endpoint {
+    url = "{{ env "LOKI_PUSH_URL" }}"
+
+    basic_auth {
+      username = "{{ env "WRITE_BASIC_AUTH_USERNAME" }}"
+      password = "{{ secret "basic_auth" }}"
+    }
+  }
+  external_labels = { hostname = "{{ env "DOMAIN" }}" }
+}
+{{ end }}

grafana-alerts.json.tmpl

@@ -1,315 +0,0 @@
-{
-  "apiVersion": 1,
-  "groups": [
-    {
-      "orgId": 1,
-      "name": "backupbot",
-      "folder": "node",
-      "interval": "1m",
-      "rules": [
-        {{ if eq (env "ALERT_BACKUP_FAILED_ENABLED") "true" }}
-        {
-          "uid": "de8e5xxup7t34a",
-          "title": "Backup Failed",
-          "condition": "C",
-          "data": [
-            {
-              "refId": "A",
-              "relativeTimeRange": { "from": 600, "to": 0 },
-              "datasourceUid": "PBFA97CFB590B2093",
-              "model": {
-                "disableTextWrap": false,
-                "editorMode": "builder",
-                "expr": "backup",
-                "fullMetaSearch": false,
-                "includeNullMetadata": true,
-                "instant": true,
-                "intervalMs": 1000,
-                "legendFormat": "__auto",
-                "maxDataPoints": 43200,
-                "range": false,
-                "refId": "A",
-                "useBackend": false
-              }
-            },
-            {
-              "refId": "C",
-              "relativeTimeRange": { "from": 600, "to": 0 },
-              "datasourceUid": "__expr__",
-              "model": {
-                "conditions": [
-                  {
-                    "evaluator": { "params": [0], "type": "lt" },
-                    "operator": { "type": "and" },
-                    "query": { "params": ["C"] },
-                    "reducer": { "params": [], "type": "last" },
-                    "type": "query"
-                  }
-                ],
-                "datasource": { "type": "__expr__", "uid": "__expr__" },
-                "expression": "A",
-                "intervalMs": 1000,
-                "maxDataPoints": 43200,
-                "refId": "C",
-                "type": "threshold"
-              }
-            }
-          ],
-          "noDataState": "NoData",
-          "execErrState": "Error",
-          "for": "1m",
-          "isPaused": false
-        },
-        {{ end }}
-        {{ if eq (env "ALERT_BACKUP_MISSING_ENABLED") "true" }}
-        {
-          "uid": "ce8e65uddcwe8d",
-          "title": "Backup Missing",
-          "condition": "B",
-          "data": [
-            {
-              "refId": "A",
-              "relativeTimeRange": { "from": 600, "to": 0 },
-              "datasourceUid": "PBFA97CFB590B2093",
-              "model": {
-                "disableTextWrap": false,
-                "editorMode": "builder",
-                "expr": "rate(backup[24h])",
-                "fullMetaSearch": false,
-                "includeNullMetadata": true,
-                "instant": true,
-                "intervalMs": 1000,
-                "legendFormat": "__auto",
-                "maxDataPoints": 43200,
-                "range": false,
-                "refId": "A",
-                "useBackend": false
-              }
-            },
-            {
-              "refId": "B",
-              "relativeTimeRange": { "from": 600, "to": 0 },
-              "datasourceUid": "__expr__",
-              "model": {
-                "conditions": [
-                  {
-                    "evaluator": { "params": [0, 0], "type": "within_range" },
-                    "operator": { "type": "and" },
-                    "query": { "params": ["C"] },
-                    "reducer": { "params": [], "type": "last" },
-                    "type": "query"
-                  }
-                ],
-                "datasource": { "type": "__expr__", "uid": "__expr__" },
-                "expression": "A",
-                "intervalMs": 1000,
-                "maxDataPoints": 43200,
-                "refId": "B",
-                "type": "threshold"
-              }
-            }
-          ],
-          "noDataState": "NoData",
-          "execErrState": "Error",
-          "for": "5m",
-          "isPaused": false
-        },
-        {{ end }}
-        {{ if eq (env "ALERT_BACKUP_NOT_SUCCESSFULL_ENABLED") "true" }}
-        {
-            "uid": "de8e6bc92a8lcc",
-            "title": "Backup Not Successfull",
-            "condition": "B",
-            "data": [
-                {
-                    "refId": "A",
-                    "relativeTimeRange": {
-                        "from": 60,
-                        "to": 0
-                    },
-                    "datasourceUid": "PBFA97CFB590B2093",
-                    "model": {
-                        "disableTextWrap": false,
-                        "editorMode": "builder",
-                        "expr": "backup",
-                        "fullMetaSearch": false,
-                        "includeNullMetadata": true,
-                        "instant": true,
-                        "intervalMs": 1000,
-                        "legendFormat": "__auto",
-                        "maxDataPoints": 43200,
-                        "range": false,
-                        "refId": "A",
-                        "useBackend": false
-                    }
-                },
-                {
-                    "refId": "B",
-                    "relativeTimeRange": {
-                        "from": 60,
-                        "to": 0
-                    },
-                    "datasourceUid": "__expr__",
-                    "model": {
-                        "conditions": [
-                            {
-                                "evaluator": {
-                                    "params": [
-                                        0
-                                    ],
-                                    "type": "gt"
-                                },
-                                "operator": {
-                                    "type": "and"
-                                },
-                                "query": {
-                                    "params": [
-                                        "C"
-                                    ]
-                                },
-                                "reducer": {
-                                    "params": [],
-                                    "type": "last"
-                                },
-                                "type": "query"
-                            }
-                        ],
-                        "datasource": {
-                            "type": "__expr__",
-                            "uid": "__expr__"
-                        },
-                        "expression": "A",
-                        "intervalMs": 1000,
-                        "maxDataPoints": 43200,
-                        "refId": "B",
-                        "type": "threshold"
-                    }
-                }
-            ],
-            "noDataState": "NoData",
-            "execErrState": "Error",
-            "for": "20m",
-            "annotations": {
-                "summary": "Backup did not finish within 20 minutes"
-            },
-            "labels": {},
-            "isPaused": false
-        }
-        {{ end }}
-      ]
-    },
-    {
-      "orgId": 1,
-      "name": "node",
-      "folder": "node",
-      "interval": "5m",
-      "rules": [
-        {{ if eq (env "ALERT_NODE_DISK_SPACE_ENABLED") "true" }}
-        {
-          "uid": "bds8bhxu97pxca",
-          "title": "Node Disk Space",
-          "condition": "C",
-          "data": [
-            {
-              "refId": "A",
-              "relativeTimeRange": { "from": 600, "to": 0 },
-              "datasourceUid": "PBFA97CFB590B2093",
-              "model": {
-                "editorMode": "code",
-                "expr": "(node_filesystem_free_bytes{fstype=\"ext4\"} / node_filesystem_size_bytes{fstype=\"ext4\"}) * 100",
-                "instant": true,
-                "intervalMs": 1000,
-                "legendFormat": "__auto",
-                "maxDataPoints": 43200,
-                "range": false,
-                "refId": "A"
-              }
-            },
-            {
-              "refId": "C",
-              "relativeTimeRange": { "from": 600, "to": 0 },
-              "datasourceUid": "__expr__",
-              "model": {
-                "conditions": [
-                  {
-                    "evaluator": { "params": [10], "type": "lt" },
-                    "operator": { "type": "and" },
-                    "query": { "params": ["C"] },
-                    "reducer": { "params": [], "type": "last" },
-                    "type": "query"
-                  }
-                ],
-                "datasource": { "type": "__expr__", "uid": "__expr__" },
-                "expression": "A",
-                "intervalMs": 1000,
-                "maxDataPoints": 43200,
-                "refId": "C",
-                "type": "threshold"
-              }
-            }
-          ],
-          "noDataState": "NoData",
-          "execErrState": "Error",
-          "for": "5m",
-          "annotations": {},
-          "labels": {},
-          "isPaused": false
-        },
-        {{ end }}
-        {{ if eq (env "ALERT_NODE_MEMORY_USAGE_ENABLED") "true" }}
-        {
-          "uid": "ads8cswmly96oa",
-          "title": "Node Memory Usage",
-          "condition": "C",
-          "data": [
-            {
-              "refId": "A",
-              "relativeTimeRange": { "from": 600, "to": 0 },
-              "datasourceUid": "PBFA97CFB590B2093",
-              "model": {
-                "editorMode": "code",
-                "expr": "(node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100",
-                "instant": true,
-                "intervalMs": 1000,
-                "legendFormat": "__auto",
-                "maxDataPoints": 43200,
-                "range": false,
-                "refId": "A"
-              }
-            },
-            {
-              "refId": "C",
-              "relativeTimeRange": { "from": 600, "to": 0 },
-              "datasourceUid": "__expr__",
-              "model": {
-                "conditions": [
-                  {
-                    "evaluator": { "params": [90], "type": "gt" },
-                    "operator": { "type": "and" },
-                    "query": { "params": ["C"] },
-                    "reducer": { "params": [], "type": "last" },
-                    "type": "query"
-                  }
-                ],
-                "datasource": { "type": "__expr__", "uid": "__expr__" },
-                "expression": "A",
-                "intervalMs": 1000,
-                "maxDataPoints": 43200,
-                "refId": "C",
-                "type": "threshold"
-              }
-            }
-          ],
-          "noDataState": "NoData",
-          "execErrState": "Error",
-          "for": "5m",
-          "annotations": {},
-          "labels": {},
-          "isPaused": false
-        }
-        {{ end }}
-      ]
-    }
-  ]
-}
-

grafana-dashboards.yml

@@ -11,13 +11,3 @@ providers:
     options:
       path: /var/lib/grafana/dashboards
       foldersFromFilesStructure: true
-  - name: 'default-alert-provider'
-    orgId: 1
-    folder: 'default-alerts'
-    type: file
-    disableDeletion: false
-    updateIntervalSeconds: 10
-    allowUiUpdates: true
-    options:
-      path: /var/lib/grafana/alerts
-      foldersFromFilesStructure: true

grafana-stacks-dashboard.json

@@ -110,7 +110,7 @@
             "uid": "PBFA97CFB590B2093"
           },
           "editorMode": "code",
-          "expr": "(time() - min(container_start_time_seconds{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}))",
+          "expr": "(time() - min(container_start_time_seconds{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}))",
           "format": "time_series",
           "intervalFactor": 1,
           "legendFormat": "",
@@ -215,7 +215,7 @@
             "uid": "PBFA97CFB590B2093"
           },
           "editorMode": "code",
-          "expr": "sort(\n    sum(\n        rate(\n            container_cpu_usage_seconds_total{\n                container_label_com_docker_stack_namespace=~\"$stack\", \n                instance=~\"cadvisor.monitoring.$instance.*\"}[$interval]\n            )\n        ) \n    by (container_label_com_docker_swarm_service_name)\n)",
+          "expr": "sort(\n    sum(\n        rate(\n            container_cpu_usage_seconds_total{\n                container_label_com_docker_stack_namespace=~\"$stack\", \n                instance=~\".*$node_id.*\"}[$interval]\n            )\n        ) \n    by (container_label_com_docker_swarm_service_name)\n)",
           "format": "time_series",
           "intervalFactor": 2,
           "legendFormat": "{{ container_label_com_docker_swarm_service_name }}",
@@ -283,7 +283,7 @@
             "uid": "PBFA97CFB590B2093"
           },
           "editorMode": "code",
-          "expr": "count(rate(container_last_seen{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}[$interval]))",
+          "expr": "count(rate(container_last_seen{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}[$interval]))",
           "format": "time_series",
           "intervalFactor": 2,
           "range": true,
@@ -386,7 +386,7 @@
             "uid": "PBFA97CFB590B2093"
           },
           "editorMode": "code",
-          "expr": "sum(\n    container_memory_rss{\n        container_label_com_docker_stack_namespace=~\"$stack\", \n        instance=~\"cadvisor.monitoring.$instance.*\"}\n    ) \nby (container_label_com_docker_swarm_service_name)",
+          "expr": "sum(\n    container_memory_rss{\n        container_label_com_docker_stack_namespace=~\"$stack\", \n        instance=~\".*$node_id.*\"}\n    ) \nby (container_label_com_docker_swarm_service_name)",
           "format": "time_series",
           "hide": false,
           "intervalFactor": 2,
@@ -493,7 +493,7 @@
             "uid": "PBFA97CFB590B2093"
           },
           "editorMode": "code",
-          "expr": "sum(rate(container_network_receive_bytes_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
+          "expr": "sum(rate(container_network_receive_bytes_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
           "format": "time_series",
           "intervalFactor": 2,
           "legendFormat": "RX: {{ container_label_com_docker_swarm_service_name }}",
@@ -507,7 +507,7 @@
             "uid": "PBFA97CFB590B2093"
           },
           "editorMode": "code",
-          "expr": "sum(rate(container_network_transmit_bytes_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}[$interval])) by (container_label_com_docker_swarm_service_name) * -1",
+          "expr": "sum(rate(container_network_transmit_bytes_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}[$interval])) by (container_label_com_docker_swarm_service_name) * -1",
           "hide": false,
           "legendFormat": "TX: {{container_label_com_docker_swarm_service_name}}",
           "range": true,
@@ -634,7 +634,7 @@
             "uid": "PBFA97CFB590B2093"
           },
           "editorMode": "code",
-          "expr": "sum(container_fs_usage_bytes{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}) by (container_label_com_docker_swarm_service_name)",
+          "expr": "sum(container_fs_usage_bytes{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}) by (container_label_com_docker_swarm_service_name)",
           "format": "time_series",
           "hide": false,
           "intervalFactor": 2,
@@ -742,7 +742,7 @@
             "uid": "PBFA97CFB590B2093"
           },
           "editorMode": "code",
-          "expr": "sum(rate(container_fs_usage_bytes{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
+          "expr": "sum(rate(container_fs_usage_bytes{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
           "format": "time_series",
           "hide": false,
           "intervalFactor": 2,
@@ -865,7 +865,7 @@
             "uid": "PBFA97CFB590B2093"
           },
           "editorMode": "code",
-          "expr": "sum(rate(container_fs_io_time_seconds_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
+          "expr": "sum(rate(container_fs_io_time_seconds_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
           "format": "time_series",
           "hide": false,
           "intervalFactor": 2,
@@ -973,7 +973,7 @@
             "uid": "PBFA97CFB590B2093"
           },
           "editorMode": "code",
-          "expr": "container_fs_io_current{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}",
+          "expr": "container_fs_io_current{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}",
           "format": "time_series",
           "hide": false,
           "intervalFactor": 2,
@@ -1081,7 +1081,7 @@
             "uid": "PBFA97CFB590B2093"
           },
           "editorMode": "code",
-          "expr": "sum(rate(container_fs_reads_bytes_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
+          "expr": "sum(rate(container_fs_reads_bytes_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
           "format": "time_series",
           "intervalFactor": 2,
           "legendFormat": "{{ container_label_com_docker_swarm_service_name }}",
@@ -1188,7 +1188,7 @@
             "uid": "PBFA97CFB590B2093"
           },
           "editorMode": "code",
-          "expr": "sum(rate(container_fs_read_seconds_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
+          "expr": "sum(rate(container_fs_read_seconds_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
           "format": "time_series",
           "intervalFactor": 2,
           "legendFormat": "{{ container_label_com_docker_swarm_service_name }}",
@@ -1295,7 +1295,7 @@
             "uid": "PBFA97CFB590B2093"
           },
           "editorMode": "code",
-          "expr": "sum(rate(container_fs_writes_bytes_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
+          "expr": "sum(rate(container_fs_writes_bytes_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
           "format": "time_series",
           "intervalFactor": 2,
           "legendFormat": "{{ container_label_com_docker_swarm_service_name }}",
@@ -1402,7 +1402,7 @@
             "uid": "PBFA97CFB590B2093"
           },
           "editorMode": "code",
-          "expr": "sum(rate(container_fs_write_seconds_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
+          "expr": "sum(rate(container_fs_write_seconds_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
           "format": "time_series",
           "intervalFactor": 2,
           "legendFormat": "{{ container_label_com_docker_swarm_service_name }}",
@@ -1453,7 +1453,7 @@
           "query": "label_values(container_label_com_docker_stack_namespace)",
           "refId": "PrometheusVariableQueryEditor-VariableQuery"
         },
-        "refresh": 1,
+        "refresh": 2,
         "regex": "",
         "skipUrlSync": false,
         "sort": 2,
@@ -1567,19 +1567,19 @@
           "type": "prometheus",
           "uid": "PBFA97CFB590B2093"
         },
-        "definition": "label_values(instance)",
+        "definition": "label_values(node_uname_info, instance)",
         "hide": 0,
         "includeAll": true,
-        "label": "instance",
+        "label": "Swarm Node",
         "multi": true,
-        "name": "instance",
+        "name": "node_id",
         "options": [],
         "query": {
-          "query": "label_values(instance)",
+          "query": "label_values(node_uname_info, instance)",
           "refId": "PrometheusVariableQueryEditor-VariableQuery"
         },
-        "refresh": 1,
+        "refresh": 2,
-        "regex": "/.*cadvisor.monitoring.(?<instance>.*):80/",
+        "regex": "/^(?:(?:node|cadvisor|monitoring).)*(.*)$/",
         "skipUrlSync": false,
         "sort": 0,
         "type": "query"
@@ -1620,4 +1620,4 @@
   "uid": "KdVoGQm7z",
   "version": 36,
   "weekStart": ""
-}
+}

grafana-swarm-dashboard.json

@@ -118,7 +118,7 @@
             "type": "prometheus",
             "uid": "PBFA97CFB590B2093"
           },
-          "expr": "topk(1, sum((node_time_seconds - node_boot_time_seconds) * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"}) by (node_name))",
+          "expr": "topk(1, sum(label_replace(node_time_seconds{instance=~\".*$node_id.*\"} - node_boot_time_seconds{instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) by (instance))",
           "format": "time_series",
           "intervalFactor": 2,
           "legendFormat": "",
@@ -198,7 +198,7 @@
             "type": "prometheus",
             "uid": "PBFA97CFB590B2093"
           },
-          "expr": "count(node_meta * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"})",
+          "expr": "count(label_replace(node_uname_info{instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\"))",
           "format": "time_series",
           "intervalFactor": 2,
           "legendFormat": "",
@@ -278,7 +278,7 @@
             "type": "prometheus",
             "uid": "PBFA97CFB590B2093"
           },
-          "expr": "count(node_cpu_seconds_total{mode=\"idle\"} * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"})",
+          "expr": "count(label_replace(node_cpu_seconds_total{mode=\"idle\", instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\"))",
           "format": "time_series",
           "intervalFactor": 2,
           "legendFormat": "",
@@ -363,7 +363,7 @@
             "uid": "PBFA97CFB590B2093"
           },
           "editorMode": "code",
-          "expr": "sum((node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"} * 100) / count(node_meta * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"})",
+          "expr": "sum(label_replace((node_memory_MemAvailable_bytes{instance=~\".*$node_id.*\"} / node_memory_MemTotal_bytes{instance=~\".*$node_id.*\"}) * 100, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) / count(label_replace(node_uname_info{instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\"))",
           "format": "time_series",
           "intervalFactor": 2,
           "legendFormat": "",
@@ -429,11 +429,11 @@
             "type": "prometheus",
             "uid": "PBFA97CFB590B2093"
           },
-          "expr": "node_load5 * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"}",
+          "expr": "label_replace(node_load5{instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")",
           "format": "time_series",
           "interval": "",
           "intervalFactor": 2,
-          "legendFormat": "load5 {{node_name}}",
+          "legendFormat": "load5 {{instance}}",
           "refId": "A",
           "step": 2
         }
@@ -536,7 +536,7 @@
             "type": "prometheus",
             "uid": "PBFA97CFB590B2093"
           },
-          "expr": "sum(node_memory_MemTotal_bytes * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"})",
+          "expr": "sum(label_replace(node_memory_MemTotal_bytes{instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\"))",
           "format": "time_series",
           "intervalFactor": 2,
           "legendFormat": "",
@@ -616,7 +616,7 @@
             "uid": "PBFA97CFB590B2093"
           },
           "exemplar": true,
-          "expr": "sum(node_filesystem_size_bytes{fstype=\"ext4\",mountpoint=~\"(/$)|(/media.*)\"} * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"})",
+          "expr": "sum(label_replace(node_filesystem_size_bytes{fstype=~\"ext4|xfs\",mountpoint=~\"(/$)|(/media.*)\", instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\"))",
           "format": "time_series",
           "interval": "",
           "intervalFactor": 2,
@@ -701,7 +701,7 @@
             "type": "prometheus",
             "uid": "PBFA97CFB590B2093"
           },
-          "expr": "sum(irate(node_cpu_seconds_total{mode=\"idle\"}[$interval]) * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"}) * 100 / count(node_cpu_seconds_total{mode=\"user\"} * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"}) ",
+          "expr": "sum(label_replace(irate(node_cpu_seconds_total{mode=\"idle\", instance=~\".*$node_id.*\"}[$interval]), \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) * 100 / count(label_replace(node_cpu_seconds_total{mode=\"user\", instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\"))",
           "format": "time_series",
           "intervalFactor": 2,
           "legendFormat": "",
@@ -833,7 +833,7 @@
           },
           "editorMode": "code",
           "exemplar": true,
-          "expr": "node_filesystem_free_bytes{fstype=\"ext4\"}  / node_filesystem_size_bytes{fstype=\"ext4\"} * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"} * 100",
+          "expr": "label_replace(node_filesystem_free_bytes{fstype=~\"ext4|xfs\", instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\") / label_replace(node_filesystem_size_bytes{fstype=~\"ext4|xfs\", instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\") * 100",
           "format": "time_series",
           "interval": "",
           "intervalFactor": 2,
@@ -900,10 +900,10 @@
             "type": "prometheus",
             "uid": "PBFA97CFB590B2093"
           },
-          "expr": "100 - (avg(irate(node_cpu_seconds_total{mode=\"idle\"}[$interval])  * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"} * 100) by (node_name))",
+          "expr": "100 - (avg(label_replace(irate(node_cpu_seconds_total{mode=\"idle\", instance=~\".*$node_id.*\"}[$interval]), \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\") * 100) by (instance))",
           "format": "time_series",
           "intervalFactor": 2,
-          "legendFormat": "{{node_name}}",
+          "legendFormat": "{{instance}}",
           "refId": "A",
           "step": 2
         }
@@ -1045,12 +1045,12 @@
             "uid": "PBFA97CFB590B2093"
           },
           "editorMode": "code",
-          "expr": "sum((node_memory_MemFree_bytes) * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"}) by (node_name)",
+          "expr": "sum(label_replace(node_memory_MemFree_bytes{instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) by (instance)",
           "format": "time_series",
           "hide": false,
           "interval": "",
           "intervalFactor": 2,
-          "legendFormat": "Free  {{node_name}}",
+          "legendFormat": "Free  {{instance}}",
           "range": true,
           "refId": "free",
           "step": 2
@@ -1061,12 +1061,12 @@
             "uid": "PBFA97CFB590B2093"
           },
           "editorMode": "code",
-          "expr": "sum((node_memory_Cached_bytes + node_memory_Buffers_bytes + node_memory_Slab_bytes) * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"}) by (node_name)",
+          "expr": "sum(label_replace(node_memory_Cached_bytes{instance=~\".*$node_id.*\"} + node_memory_Buffers_bytes{instance=~\".*$node_id.*\"} + node_memory_Slab_bytes{instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) by (instance)",
           "format": "time_series",
           "hide": false,
           "interval": "",
           "intervalFactor": 2,
-          "legendFormat": "cache,buffer,slab  {{node_name}}",
+          "legendFormat": "cache,buffer,slab  {{instance}}",
           "range": true,
           "refId": "cache,buffer,slab",
           "step": 2
@@ -1077,12 +1077,12 @@
             "uid": "PBFA97CFB590B2093"
           },
           "editorMode": "code",
-          "expr": "sum((node_memory_MemTotal_bytes - node_memory_MemFree_bytes - node_memory_Cached_bytes - node_memory_Buffers_bytes - node_memory_Slab_bytes) * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"}) by (node_name)",
+          "expr": "sum(label_replace(node_memory_MemTotal_bytes{instance=~\".*$node_id.*\"} - node_memory_MemFree_bytes{instance=~\".*$node_id.*\"} - node_memory_Cached_bytes{instance=~\".*$node_id.*\"} - node_memory_Buffers_bytes{instance=~\".*$node_id.*\"} - node_memory_Slab_bytes{instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) by (instance)",
           "format": "time_series",
           "hide": false,
           "interval": "",
           "intervalFactor": 2,
-          "legendFormat": "Used {{node_name}}",
+          "legendFormat": "Used {{instance}}",
           "range": true,
           "refId": "Used",
           "step": 2
@@ -1094,13 +1094,13 @@
           },
           "editorMode": "code",
           "exemplar": false,
-          "expr": "sum((node_memory_MemTotal_bytes) * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"}) by (node_name)",
+          "expr": "sum(label_replace(node_memory_MemTotal_bytes{instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) by (instance)",
           "format": "time_series",
           "hide": false,
           "instant": false,
           "interval": "",
           "intervalFactor": 2,
-          "legendFormat": "Total {{node_name}}",
+          "legendFormat": "Total {{instance}}",
           "range": true,
           "refId": "total",
           "step": 2
@@ -1161,11 +1161,11 @@
             "type": "prometheus",
             "uid": "PBFA97CFB590B2093"
           },
-          "expr": "sum(irate(node_disk_read_bytes_total[$interval]) * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"}) by (node_name)",
+          "expr": "sum(label_replace(irate(node_disk_read_bytes_total{instance=~\".*$node_id.*\"}[$interval]), \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) by (instance)",
           "format": "time_series",
           "interval": "",
           "intervalFactor": 2,
-          "legendFormat": "Read {{node_name}}",
+          "legendFormat": "Read {{instance}}",
           "refId": "A",
           "step": 2
         },
@@ -1174,10 +1174,10 @@
             "type": "prometheus",
             "uid": "PBFA97CFB590B2093"
           },
-          "expr": "sum(irate(node_disk_written_bytes_total[$interval]) * on(instance) group_left(node_name) node_meta{node_id=~\"$node_id\"}) by (node_name)",
+          "expr": "sum(label_replace(irate(node_disk_written_bytes_total{instance=~\".*$node_id.*\"}[$interval]), \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) by (instance)",
           "format": "time_series",
           "intervalFactor": 2,
-          "legendFormat": "Written {{node_name}}",
+          "legendFormat": "Written {{instance}}",
           "refId": "B",
           "step": 2
         }
@@ -1264,10 +1264,10 @@
             "type": "prometheus",
             "uid": "PBFA97CFB590B2093"
           },
-          "expr": "sum(irate(node_disk_reads_completed_total[$interval]) * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"}) by (node_name)",
+          "expr": "sum(label_replace(irate(node_disk_reads_completed_total{instance=~\".*$node_id.*\"}[$interval]), \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) by (instance)",
           "format": "time_series",
           "intervalFactor": 2,
-          "legendFormat": "Reads {{node_name}}",
+          "legendFormat": "Reads {{instance}}",
           "refId": "A",
           "step": 2
         },
@@ -1276,10 +1276,10 @@
             "type": "prometheus",
             "uid": "PBFA97CFB590B2093"
           },
-          "expr": "sum(irate(node_disk_writes_completed_total[$interval]) * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"}) by (node_name)",
+          "expr": "sum(label_replace(irate(node_disk_writes_completed_total{instance=~\".*$node_id.*\"}[$interval]), \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) by (instance)",
           "format": "time_series",
           "intervalFactor": 2,
-          "legendFormat": "Writes {{node_name}}",
+          "legendFormat": "Writes {{instance}}",
           "refId": "B",
           "step": 2
         }
@@ -1368,10 +1368,10 @@
             "type": "prometheus",
             "uid": "PBFA97CFB590B2093"
           },
-          "expr": "(avg(irate(node_cpu_seconds_total{mode=\"iowait\"}[$interval])  * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"} * 100) by (node_name))",
+          "expr": "(avg(label_replace(irate(node_cpu_seconds_total{mode=\"iowait\", instance=~\".*$node_id.*\"}[$interval]), \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\") * 100) by (instance))",
           "format": "time_series",
           "intervalFactor": 2,
-          "legendFormat": "{{node_name}}",
+          "legendFormat": "{{instance}}",
           "refId": "A",
           "step": 2
         }
@@ -1464,7 +1464,7 @@
             "type": "prometheus",
             "uid": "PBFA97CFB590B2093"
           },
-          "expr": "sum(rate(container_last_seen[5m]) * on(container_label_com_docker_swarm_node_id) group_left(node_name) node_meta{node_name=~\"${node_id}\"}) by (container_label_com_docker_swarm_service_name)",
+          "expr": "sum(rate(container_last_seen{instance=~\".*$node_id.*\"}[5m])) by (container_label_com_docker_swarm_service_name)",
           "format": "time_series",
           "hide": false,
           "intervalFactor": 10,
@@ -1573,7 +1573,7 @@
             "uid": "PBFA97CFB590B2093"
           },
           "exemplar": true,
-          "expr": "count(rate(container_last_seen[5m]) * on(container_label_com_docker_swarm_node_id) group_left(node_name) node_meta{node_name=~\"$node_id\"})",
+          "expr": "count(rate(container_last_seen{instance=~\".*$node_id.*\"}[5m]))",
           "format": "time_series",
           "interval": "",
           "intervalFactor": 2,
@@ -1668,11 +1668,11 @@
             "type": "prometheus",
             "uid": "PBFA97CFB590B2093"
           },
-          "expr": "sum(rate(container_network_receive_bytes_total[$interval]) * on(container_label_com_docker_swarm_node_id) group_left(node_name) node_meta{node_name=~\"${node_id}\"}) by (node_name)",
+          "expr": "sum(label_replace(rate(container_network_receive_bytes_total{instance=~\".*$node_id.*\"}[$interval]), \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) by (instance)",
           "format": "time_series",
           "interval": "",
           "intervalFactor": 2,
-          "legendFormat": "IN {{node_name}}",
+          "legendFormat": "IN {{instance}}",
           "refId": "A",
           "step": 2
         },
@@ -1681,11 +1681,11 @@
             "type": "prometheus",
             "uid": "PBFA97CFB590B2093"
           },
-          "expr": "- sum(rate(container_network_transmit_bytes_total[$interval]) * on(container_label_com_docker_swarm_node_id) group_left(node_name) node_meta{node_name=~\"${node_id}\"}) by (node_name)",
+          "expr": "- sum(label_replace(rate(container_network_transmit_bytes_total{instance=~\".*$node_id.*\"}[$interval]), \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) by (instance)",
           "format": "time_series",
           "hide": false,
           "intervalFactor": 2,
-          "legendFormat": "OUT {{node_name}}",
+          "legendFormat": "OUT {{instance}}",
           "metric": "",
           "refId": "B",
           "step": 2
@@ -1762,11 +1762,11 @@
         "name": "node_id",
         "options": [],
         "query": {
-          "query": "node_meta",
+          "query": "label_values(node_uname_info, instance)",
           "refId": "Prometheus-node_id-Variable-Query"
         },
-        "refresh": 1,
+        "refresh": 2,
-        "regex": "/node_name=\"([^\"]+)\"/",
+        "regex": "/^(?:(?:node|cadvisor|monitoring).)*(.*)$/",
         "skipUrlSync": false,
         "sort": 0,
         "type": "query"

grafana-traefik-dashboard.json

@@ -116,12 +116,12 @@
             "uid": "PBFA97CFB590B2093"
           },
           "exemplar": false,
-          "expr": "time() - process_start_time_seconds{job=\"$job\"}",
+          "expr": "time() - label_replace(process_start_time_seconds{instance=~\".*$domain.*\"} or process_start_time_seconds{domain=\"$domain\"}, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)",
           "format": "time_series",
           "instant": true,
           "interval": "",
           "intervalFactor": 2,
-          "legendFormat": "{{ instance }}",
+          "legendFormat": "{{ domain }}",
           "refId": "A"
         }
       ],
@@ -200,7 +200,7 @@
             "uid": "PBFA97CFB590B2093"
           },
           "exemplar": true,
-          "expr": "sum(increase(traefik_service_requests_total{code=\"499\", instance=\"$instance\"}[$interval]))",
+          "expr": "sum(label_replace(increase(traefik_service_requests_total{code=\"499\", instance=~\".*$domain.*\"}[$interval]) or increase(traefik_service_requests_total{code=\"499\", domain=\"$domain\"}[$interval]), \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`))",
           "interval": "",
           "legendFormat": "",
           "refId": "A"
@@ -280,7 +280,7 @@
             "uid": "PBFA97CFB590B2093"
           },
           "exemplar": true,
-          "expr": "sum(increase(traefik_service_requests_total{code=\"404\",protocol=~\"$protocol\", instance=\"$instance\"}[$interval]))",
+          "expr": "sum(label_replace(increase(traefik_service_requests_total{code=\"404\",protocol=~\"$protocol\", instance=~\".*$domain.*\"}[$interval]) or increase(traefik_service_requests_total{code=\"404\",protocol=~\"$protocol\", domain=\"$domain\"}[$interval]), \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`))",
           "format": "time_series",
           "interval": "",
           "intervalFactor": 2,
@@ -365,7 +365,7 @@
             "uid": "PBFA97CFB590B2093"
           },
           "exemplar": true,
-          "expr": "sum(increase(traefik_service_requests_total{code=\"200\",protocol=~\"$protocol\", instance=\"$instance\"}[$interval]))",
+          "expr": "sum(label_replace(increase(traefik_service_requests_total{code=\"200\",protocol=~\"$protocol\", instance=~\".*$domain.*\"}[$interval]) or increase(traefik_service_requests_total{code=\"200\",protocol=~\"$protocol\", domain=\"$domain\"}[$interval]), \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`))",
           "format": "time_series",
           "interval": "",
           "intervalFactor": 2,
@@ -417,7 +417,7 @@
             "uid": "PBFA97CFB590B2093"
           },
           "exemplar": true,
-          "expr": "topk(5, sum(traefik_service_requests_total{protocol=~\"$protocol\", instance=\"$instance\"}) by (code))",
+          "expr": "topk(5, sum(label_replace(traefik_service_requests_total{protocol=~\"$protocol\", instance=~\".*$domain.*\"} or traefik_service_requests_total{protocol=~\"$protocol\", domain=\"$domain\"}, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) by (code))",
           "format": "time_series",
           "interval": "",
           "intervalFactor": 2,
@@ -489,7 +489,7 @@
             "uid": "PBFA97CFB590B2093"
           },
           "exemplar": true,
-          "expr": "sum(increase(traefik_service_requests_total{code=\"499\",method=\"GET\", instance=\"$instance\"}[$interval])) by (service)",
+          "expr": "sum(label_replace(increase(traefik_service_requests_total{code=\"499\",method=\"GET\", instance=~\".*$domain.*\"}[$interval]) or increase(traefik_service_requests_total{code=\"499\",method=\"GET\", domain=\"$domain\"}[$interval]), \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) by (service)",
           "format": "time_series",
           "interval": "",
           "intervalFactor": 2,
@@ -590,7 +590,7 @@
             "uid": "PBFA97CFB590B2093"
           },
           "exemplar": true,
-          "expr": "(sum(traefik_entrypoint_request_duration_seconds_sum{protocol=~\"$protocol\", instance=\"$instance\"}) / sum(traefik_entrypoint_requests_total{protocol=~\"$protocol\", instance=\"$instance\"}) * 1000) - (sum(traefik_service_request_duration_seconds_sum{protocol=~\"$protocol\", instance=\"$instance\"}) / sum(traefik_service_requests_total{protocol=~\"$protocol\", instance=\"$instance\"}) * 1000)",
+          "expr": "(sum(label_replace(traefik_entrypoint_request_duration_seconds_sum{protocol=~\"$protocol\", instance=~\".*$domain.*\"} or traefik_entrypoint_request_duration_seconds_sum{protocol=~\"$protocol\", domain=\"$domain\"}, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) / sum(label_replace(traefik_entrypoint_requests_total{protocol=~\"$protocol\", instance=~\".*$domain.*\"} or traefik_entrypoint_requests_total{protocol=~\"$protocol\", domain=\"$domain\"}, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) * 1000) - (sum(label_replace(traefik_service_request_duration_seconds_sum{protocol=~\"$protocol\", instance=~\".*$domain.*\"} or traefik_service_request_duration_seconds_sum{protocol=~\"$protocol\", domain=\"$domain\"}, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) / sum(label_replace(traefik_service_requests_total{protocol=~\"$protocol\", instance=~\".*$domain.*\"} or traefik_service_requests_total{protocol=~\"$protocol\", domain=\"$domain\"}, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) * 1000)",
           "format": "time_series",
           "instant": false,
           "interval": "",
@@ -709,7 +709,7 @@
             "uid": "PBFA97CFB590B2093"
           },
           "exemplar": true,
-          "expr": "sum(traefik_entrypoint_request_duration_seconds_sum{instance=\"$instance\"}) / sum(traefik_entrypoint_requests_total{instance=\"$instance\"}) * 1000",
+          "expr": "sum(label_replace(traefik_entrypoint_request_duration_seconds_sum{instance=~\".*$domain.*\"} or traefik_entrypoint_request_duration_seconds_sum{domain=\"$domain\"}, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) / sum(label_replace(traefik_entrypoint_requests_total{instance=~\".*$domain.*\"} or traefik_entrypoint_requests_total{domain=\"$domain\"}, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) * 1000",
           "format": "time_series",
           "interval": "",
           "intervalFactor": 2,
@@ -781,7 +781,7 @@
           },
           "editorMode": "code",
           "exemplar": true,
-          "expr": "sum(delta(traefik_service_requests_total{instance=\"${instance:raw}\"}[$interval]))",
+          "expr": "sum(label_replace(delta(traefik_service_requests_total{instance=~\".*$domain.*\"}[$interval]) or delta(traefik_service_requests_total{domain=\"$domain\"}[$interval]), \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`))",
           "format": "time_series",
           "interval": "",
           "intervalFactor": 2,
@@ -879,7 +879,7 @@
             "uid": "PBFA97CFB590B2093"
           },
           "exemplar": true,
-          "expr": "sum(rate(traefik_service_requests_total{instance=\"$instance\"}[$interval]))",
+          "expr": "sum(label_replace(rate(traefik_service_requests_total{instance=~\".*$domain.*\"}[$interval]) or rate(traefik_service_requests_total{domain=\"$domain\"}[$interval]), \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`))",
           "format": "time_series",
           "interval": "",
           "intervalFactor": 2,
@@ -984,7 +984,7 @@
           },
           "editorMode": "code",
           "exemplar": true,
-          "expr": "sum(rate(traefik_service_request_duration_seconds_sum{ instance=\"$instance\" }[5m])) by(service)",
+          "expr": "sum(label_replace(rate(traefik_service_request_duration_seconds_sum{instance=~\".*$domain.*\"}[5m]) or rate(traefik_service_request_duration_seconds_sum{domain=\"$domain\"}[5m]), \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) by(service)",
           "format": "time_series",
           "interval": "",
           "intervalFactor": 2,
@@ -1053,11 +1053,11 @@
             "uid": "PBFA97CFB590B2093"
           },
           "exemplar": true,
-          "expr": "process_open_fds{job=~\"$job\", instance=\"$instance\"}",
+          "expr": "label_replace(process_open_fds{instance=~\".*$domain.*\"} or process_open_fds{domain=\"$domain\"}, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)",
           "format": "time_series",
           "interval": "",
           "intervalFactor": 2,
-          "legendFormat": "{{ instance }}",
+          "legendFormat": "{{ domain }}",
           "refId": "A",
           "step": 240
         }
@@ -1154,7 +1154,7 @@
             "uid": "PBFA97CFB590B2093"
           },
           "exemplar": true,
-          "expr": "sum(rate(traefik_service_requests_total{protocol=~\"http|https\", instance=\"$instance\"}[$interval])) by (service)",
+          "expr": "sum(label_replace(rate(traefik_service_requests_total{protocol=~\"http|https\", instance=~\".*$domain.*\"}[$interval]) or rate(traefik_service_requests_total{protocol=~\"http|https\", domain=\"$domain\"}[$interval]), \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) by (service)",
           "format": "time_series",
           "interval": "",
           "intervalFactor": 2,
@@ -1255,7 +1255,7 @@
             "uid": "PBFA97CFB590B2093"
           },
           "exemplar": true,
-          "expr": "sum(traefik_entrypoint_open_connections{instance=\"$instance\"}) by (method)",
+          "expr": "sum(label_replace(traefik_entrypoint_open_connections{instance=~\".*$domain.*\"} or traefik_entrypoint_open_connections{domain=\"$domain\"}, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) by (method)",
           "format": "time_series",
           "interval": "",
           "intervalFactor": 1,
@@ -1355,7 +1355,7 @@
             "uid": "PBFA97CFB590B2093"
           },
           "exemplar": true,
-          "expr": "sum(traefik_service_open_connections{instance=\"$instance\"}) by (method)",
+          "expr": "sum(label_replace(traefik_service_open_connections{instance=~\".*$domain.*\"} or traefik_service_open_connections{domain=\"$domain\"}, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) by (method)",
           "format": "time_series",
           "interval": "",
           "intervalFactor": 1,
@@ -1459,7 +1459,7 @@
             "uid": "PBFA97CFB590B2093"
           },
           "exemplar": true,
-          "expr": "sum(increase(traefik_service_requests_total{protocol=~\"$protocol\", instance=\"$instance\"}[$interval])) by (code)",
+          "expr": "sum(label_replace(increase(traefik_service_requests_total{protocol=~\"$protocol\", instance=~\".*$domain.*\"}[$interval]) or increase(traefik_service_requests_total{protocol=~\"$protocol\", domain=\"$domain\"}[$interval]), \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) by (code)",
           "format": "time_series",
           "interval": "",
           "intervalFactor": 2,
@@ -1514,36 +1514,6 @@
   ],
   "templating": {
     "list": [
-      {
-        "current": {
-          "selected": false,
-          "text": "default",
-          "value": "default"
-        },
-        "datasource": {
-          "type": "prometheus",
-          "uid": "PBFA97CFB590B2093"
-        },
-        "definition": "",
-        "hide": 0,
-        "includeAll": false,
-        "label": "Job:",
-        "multi": false,
-        "name": "job",
-        "options": [],
-        "query": {
-          "query": "label_values(job)",
-          "refId": "Prometheus-job-Variable-Query"
-        },
-        "refresh": 1,
-        "regex": "",
-        "skipUrlSync": false,
-        "sort": 2,
-        "tagValuesQuery": "",
-        "tagsQuery": "",
-        "type": "query",
-        "useTags": false
-      },
       {
         "current": {
           "selected": true,
@@ -1569,7 +1539,7 @@
           "query": "label_values(traefik_service_requests_total, protocol)",
           "refId": "Prometheus-protocol-Variable-Query"
         },
-        "refresh": 1,
+        "refresh": 2,
         "regex": "",
         "skipUrlSync": false,
         "sort": 0,
@@ -1663,19 +1633,19 @@
           "type": "prometheus",
           "uid": "PBFA97CFB590B2093"
         },
-        "definition": "label_values(instance)",
+        "definition": "query_result(label_replace(traefik_config_reloads_total, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`))",
         "hide": 0,
         "includeAll": false,
-        "label": "Instance:",
+        "label": "Domain:",
         "multi": false,
-        "name": "instance",
+        "name": "domain",
         "options": [],
         "query": {
-          "query": "label_values(instance)",
+          "query": "query_result(label_replace(traefik_config_reloads_total, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`))",
           "refId": "PrometheusVariableQueryEditor-VariableQuery"
         },
-        "refresh": 1,
+        "refresh": 2,
-        "regex": ".*8082",
+        "regex": "/domain=\"([^\"]+)\"/",
         "skipUrlSync": false,
         "sort": 1,
         "tagValuesQuery": "",

node-exporter-entrypoint.sh

@@ -1,11 +0,0 @@
-#!/bin/sh -e
-
-NODE_NAME=$(cat /etc/nodename)
-
-mkdir -p /etc/node-exporter
-
-echo "node_meta{node_id=\"$NODE_ID\", container_label_com_docker_swarm_node_id=\"$NODE_ID\", node_name=\"$NODE_NAME\"} 1" > /etc/node-exporter/node-meta.prom
-
-set -- /bin/node_exporter "$@"
-
-exec "$@"

promtail.yml.tmpl

@@ -1,37 +0,0 @@
-server:
-  http_listen_port: 9080
-  grpc_listen_port: 0
-
-positions:
-  filename: /tmp/positions.yaml
-
-clients:
-  - url: {{ env "LOKI_PUSH_URL" }}
-    basic_auth:
-      username: admin
-      password: {{ secret "basic_auth" }}
-    external_labels:
-      hostname: {{ env "DOMAIN" }}
-
-scrape_configs:
-- job_name: system
-  static_configs:
-  - targets:
-      - localhost
-    labels:
-      job: varlogs
-      __path__: /var/log/*log
-
-- job_name: "docker"
-  docker_sd_configs:
-    - host: "unix:///var/run/docker.sock"
-      refresh_interval: "10s"
-  relabel_configs:
-    - source_labels: ['__meta_docker_container_name']
-      target_label: "container_name"
-    - source_labels: ['__meta_docker_container_id']
-      target_label: "container_id"
-    - source_labels: ['__meta_docker_container_label_com_docker_stack_namespace']
-      target_label: "stack_namespace"
-    - source_labels: ['__meta_docker_container_label_com_docker_swarm_service_name']
-      target_label: "service_name"

release/next

@@ -1,12 +1,64 @@
-1. OIDC was moved into a seperate compose file. If you have oidc configured you need to add the following line to you .env file:
+BREAKING CHANGE
+Migration plan for upgrading from 1.6.0+v1.8.1.
 
-COMPOSE_FILE="$COMPOSE_FILE:compose.grafana-oidc.yml"
+## 1. Reinsert secrets with shortened names
 
-2. SMTP was moved into a seperate compose file. If you have smtp configured you need to add the following line to you .env file:
+Secret and config names were shortened to max 14 characters to prevent going over Docker's 64 character
+limit when STACK_NAME and VERSION are added to it.
 
-COMPOSE_FILE="$COMPOSE_FILE:compose.grafana-smtp.yml"
+- `abra app secret list <domain>` to see which secrets are missing under their new name
+- `abra app cmd --local <domain> migrate_secret_names` to reinsert all of them automatically
+      (or manually: `abra app secret insert <domain> <secret_name> v1 <value>` per secret)
 
-3. The scrape-config.example.yml file and add_node() command were updated to use a secure endpoint for the traefik metrics instead of http. This requires an updated Traefik recipe that publishes the metrics on https.
+## 2. If you use OIDC (moved to seperate compose file)
 
-4. Secret and config names were shortened to max 14 characters to prevent going over Docker's 64 character limit when STACK_NAME and VERSION are added to it.
+- Add to your .env: `COMPOSE_FILE="$COMPOSE_FILE:compose.grafana-oidc.yml"`
-When upgrading, you need to reinsert the secrets with their shorter names. Run `abra app secret list <domain>` to see which secrets aren't created on the server (because their name was shortened) and run `abra app secret insert <domain> <secret_name> v1 <value>` to reinsert them with the shorter name. Or you can use the migrate_secret_names function in abra.sh to reinsert all existing secrets with their shorter name automatically: `abra app cmd --local <domain> migrate_secret_names`
+
+## 3. If you use SMTP (moved to a seperate compose file)
+
+- Add to your .env: `COMPOSE_FILE="$COMPOSE_FILE:compose.grafana-smtp.yml"`
+
+## 4. node_exporter/cadvisor/promtail replaced by Grafana Alloy
+
+Metrics collection changed from Prometheus scraping endpoints
+to Alloy pushing via `remote_write`/`loki push`.
+
+- Remove `compose.promtail.yml`, `compose.expose-ports.yml` and
+      `compose.basic-auth.yml` from your .env if present. They no
+      longer exist. `compose.yml` now declares the `basic_auth` secret directly, so
+      `SECRET_BASIC_AUTH_VERSION` is always required.
+- Add `PROMETHEUS_REMOTE_WRITE_URL=https://prometheus.$DOMAIN/api/v1/write`
+      (`$DOMAIN` if this host also runs `compose.prometheus.yml`, otherwise a remote
+      Prometheus' URL). Without this, Alloy collects no metrics at all.
+- Add `LOKI_PUSH_URL` (existing var, still used) and pick a log source:
+      `JOURNALD=1` (systemd hosts), `SYSLOG_FILES=1` (non-systemd, tails
+      `/var/log/*log`), or `SYSLOG=1` + `compose.syslog.yml` (network syslog listener).
+
+- If this host had its own `node.$DOMAIN`/`cadvisor.$DOMAIN` scrape target
+      configured on a central Prometheus, remove it. Those endpoints are gone.
+- `scrape-config.example.yml` and the `add_node`/`add_domain` abra.sh commands
+      are gone. Replaced by label-based auto-discovery (see README).
+- `docker stack deploy` doesn't prune removed services, so old `cadvisor`/
+      `promtail` containers keep running after a normal `abra app deploy`. Run
+      `abra app undeploy <domain>` then `abra app deploy <domain>` to clear them out.
+- Diff your `.env` against the current `.env.sample`, to verify any other changes.
+
+### New: label-based metrics auto-discovery
+
+Alloy now auto-discovers and scrapes other Docker Swarm services on the same
+host/`proxy` network that opt in via `prometheus.io/scrape=true` deploy labels.
+See the README's "Auto-discovering metrics from other apps" section.
+
+- If you scrape Traefik metrics: the old `metrics.traefik.$domain` pull-based
+      endpoint still works if you keep the scrape config in Prometheus and
+      existing dashboards keep showing its data, but it's recommended to get
+      Traefik onto the new label-based discovery.
+
+### Dashboards
+
+The Swarm, Stacks and Traefik dashboards were reworked to show old (pull-model)
+and new (Alloy push-model) data as one continuous line, so you don't lose history
+across the migration.
+
+The disk usage per container was disabled per default because of the CPU-expensive 
+filesystem scan (30% cpu increase). Enable by uncommenting env CADVISOR_DISK_USAGE=1

scrape-config.example.yml

@@ -1,4 +0,0 @@
-- targets:
-  - 'metrics.traefik.example.org'
-  - 'node.monitoring.example.org'
-  - 'cadvisor.monitoring.example.org'