Merge branch 'alloy' into smartctl

fix: scrape once per hour
feat: systemd timer
2026-07-25 14:00:49 -03:00 · 2026-06-20 13:09:39 -03:00 · 2026-06-20 12:54:19 -03:00 · 2026-06-19 20:34:28 -03:00
22 changed files with 374 additions and 435 deletions
@@ -5,34 +5,9 @@ DOMAIN=monitoring-ng.example.com
 #TIMEOUT=120
 ENABLE_BACKUPS=true

-# Secret Alloy authenticates with when writing metrics/logs to Prometheus/
-# Loki and what Traefik's basicauth middleware expects for that
 SECRET_BASIC_AUTH_VERSION=v1
-# Username sent along with SECRET_BASIC_AUTH_VERSION above (default: admin)
-# WRITE_BASIC_AUTH_USERNAME=admin
-
-# Expose Alloy's web UI publicly (behind basic-auth) at alloy.$DOMAIN.
-# COMPOSE_FILE="$COMPOSE_FILE:compose.alloy-webui.yml"
-# Enable Live Debugging in web ui
-# LIVE_DEBUGGING=false
-
-# Enable this to send metrics to a Prometheus server, adapt DOMAIN if
-# server is remote
-# PROMETHEUS_REMOTE_WRITE_URL=https://prometheus.$DOMAIN/api/v1/write
-
-# Enable authenticated scraping of containers that opt in via
-# prometheus.io/auth=basic or prometheus.io/auth=bearer labels (used as
-# password/bearer token respectively). Insert it with:
-# abra app secret insert <domain> scrape_auth v1 <password>
-# COMPOSE_FILE="$COMPOSE_FILE:compose.scrape-auth.yml"
-# SECRET_SCRAPE_AUTH_VERSION=v1
-#
-# Username sent along with it (default: alloy)
-# SCRAPE_BASIC_AUTH_USERNAME=alloy
-
-# Edit this if your distribution sets this socket to another location
-# CONTAINERD_SOCKET=/var/run/containerd/containerd.sock
-
+# Enable Live Debugging
+LIVE_DEBUGGING=false
 # Enable this to send logs to a Loki server, adapt DOMAIN if server is
 # remote
 # LOKI_PUSH_URL=https://loki.$DOMAIN/loki/api/v1/push
@@ -51,6 +26,13 @@ SECRET_BASIC_AUTH_VERSION=v1
 # SYSLOG=1
 # COMPOSE_FILE="$COMPOSE_FILE:compose.syslog.yml"

+# Enable this to send metrics to a Prometheus server, adapt DOMAIN if
+# server is remote
+# PROMETHEUS_REMOTE_WRITE_URL=https://prometheus.$DOMAIN/api/v1/write
+
+# Monitor physical disks health
+# COMPOSE_FILE="$COMPOSE_FILE:compose.smartctl.yml"
+
 # Monitoring Server
 #
 ## Prometheus
@@ -82,8 +64,6 @@ SECRET_BASIC_AUTH_VERSION=v1
 # COMPOSE_FILE="$COMPOSE_FILE:compose.grafana.yml"
 # GF_SERVER_ROOT_URL=https://monitoring.example.com
 # SECRET_GF_ADMINPASSWD_VERSION=v1
-## Seperate domain for Grafana
-#GRAFANA_DOMAIN=grafana.example.com
 #
 ## Single-Sign-On with OIDC
 # COMPOSE_FILE="$COMPOSE_FILE:compose.grafana-oidc.yml"
@@ -105,7 +85,7 @@ SECRET_BASIC_AUTH_VERSION=v1
 # GF_SMTP_ENABLED=true
 # GF_SMTP_FROM_ADDRESS=grafana@example.com
 # GF_SMTP_SKIP_VERIFY=false
-# SECRET_GF_SMTP_PASS_VERSION=v1
+# SECRET_GF_SMTP_PASSWD_VERSION=v1
 #

 ## Grafana Matrix Contact Point (optional)
@@ -4,7 +4,6 @@ Yet another monitoring stack ...
 This time its a all-in-one grafana/prometheus/loki/alloy stack.
 It's based heavily on the [monitoring-lite](https://git.coopcloud.tech/coop-cloud/monitoring-lite) stack, but has everything in one recipe included now. So you can deploy monitoring instances to only gather metrics / logs (alloy) and also deploy instances with the full monitoring stack (grafana/prometheus/loki) with the same recipe and just different .env configuration.

-Metrics and logs are collected by [Grafana Alloy](https://grafana.com/docs/alloy/latest/) and pushed to a central Prometheus/Loki (via `remote_write`/`loki push`). Every `monitoring-ng` instance runs its own Alloy, whether or not it also runs the central stack.

 <!-- metadata -->

@@ -19,93 +18,55 @@ Metrics and logs are collected by [Grafana Alloy](https://grafana.com/docs/alloy

 <!-- endmetadata -->

-## Setup: gathering-only host
+## Setup Metrics Gathering

-Deploys just Alloy, pushing this host's own node/container metrics and logs to a central instance. Use this on every host you want metrics/logs from.
+Where gathering.org is the node you want to gather metrics from.

-1. `abra app new monitoring-ng --server gathering.org`
-2. `abra app config monitoring.gathering.org`
-3. Point it at your central instance:
-   ```
-   PROMETHEUS_REMOTE_WRITE_URL=https://prometheus.example.org/api/v1/write
-   LOKI_PUSH_URL=https://loki.example.org/loki/api/v1/push
-   JOURNALD=1   # or SYSLOG_FILES=1 / SYSLOG=1, see .env.sample
-   ```
-4. `abra app secret insert monitoring.gathering.org basic_auth v1 <password>`. Same username/password as the `usersfile` credential configured for Traefik's basicauth on the central node (see below). This is what Alloy authenticates with when pushing metrics/logs. Alloy defaults to username `admin` for this. Uf the Traefik `usersfile` uses a different username, set `WRITE_BASIC_AUTH_USERNAME` in this recipe to match.
+1. Configure DNS
+  - cadvisor.monitoring.gathering.org
+  - node.monitoring.gathering.org
+2. [Configure Traefik to use BasicAuth](https://git.coopcloud.tech/coop-cloud/traefik#configuring-wildcard-ssl-using-dns)
+3. `abra app new monitoring-ng`
+4. `abra app config monitoring.gathering.org` (for gathering only the main `compose.yml` is needed, nothing more.)
 5. `abra app deploy monitoring.gathering.org`
+6. check that endpoints are up and basic-auth works
+  - cadvisor.monitoring.gathering.org
+  - node.monitoring.gathering.org

-## Setup: full monitoring stack (metrics/logs browser)
+## Setup Metrics Browser

-This is what a gathering host pushes into. It also runs its own Alloy, so it monitors itself too.
+This builds upon [Setup Metrics Gathering](#setup-metrics-grathering) so make sure you did that first.

-1. Configure DNS: `monitoring.example.org`, plus `prometheus.`/`loki.`/`pushgateway.` subdomains for whichever of those you enable below
-2. Traefik on this node needs basic auth configured (`BASIC_AUTH=1`, see the Traefik recipe's "Configuring basic auth" section) — Prometheus/Loki/Pushgateway route through its `basicauth@file` middleware, so without it those endpoints won't work. Use the same username (default `admin`, see `WRITE_BASIC_AUTH_USERNAME`) and password you'll insert as the `basic_auth` secret below when generating Traefik's `usersfile`.
-3. `abra app config monitoring.example.org` Uncomment `compose.prometheus.yml` (metrics), `compose.loki.yml` (logs) and `compose.grafana.yml` (dashboard)
-4. `abra app secret insert monitoring.example.org basic_auth v1 <password>` — this is the password every gathering host's Alloy (including this instance's own) authenticates with; also what Traefik's basicauth expects on the public Prometheus/Loki/Pushgateway endpoints
-5. `abra app secret insert monitoring.example.org gf_adminpasswd v1 <password>`
-6. `abra app deploy monitoring.example.org`
+1. Configure DNS
+   - monitoring.example.org
+2. Setup monitoring stack
+   - `abra app config monitoring.example.org` Uncomment prometheus, loki and grafana
+   - `abra app secret insert monitoring.example.org basic_auth v1 <password>`
+      this needs the plaintext traefik basic-auth secret, not the hashed one!
+   - `abra app secret ls monitoring.example.org`
+   - `abra app deploy monitoring.example.org`
+3. Add scrape config to prometheus
+  - `abra app cmd monitoring.example.org prometheus gathering.org`
+  - or manually
+    ```
+    cp scrape-config.example.yml gathering.org.yml
+    # adjust domain
+    # mkdir scrape_configs
+    abra app cp monitoring.dev.local-it.cloud gathering.org.yml prometheus:/prometheus/scrape_configs/
+    ```

-### Post-setup guide
+* check that all configured targets are up:
+  https://prometheus.monitoring.example.org/targets

- configure the SMTP mailer under `Alerting > Contact points`
-  - edit the default contact point, choose "Alertmanager" as type & `http://alertmanager:9093` as URL
-  - use the "Test" button to send a test mail. It should fire a request at the alertmanager & that should send a mail
- from your dashboard panels, choose `Edit > Alert` to create alerts based on those panels

-## Additional features

-### Discovering metrics from other apps
-
-Alloy auto-discovers and scrapes other Docker Swarm services running on the same host, on the `proxy` network, that opt in via labels. No manual scrape config needed. On the app's `compose.yml`:
-
-```yaml
-deploy:
-  labels:
-    - "prometheus.io/scrape=true"       # required: opt in
-    # - "prometheus.io/port=8082"       # optional: defaults to 80
-    # - "prometheus.io/path=/metrics"   # optional: defaults to /metrics
-    # - "prometheus.io/auth=basic"      # optional: basic auth, see below
-    # - "prometheus.io/auth=bearer"     # optional: bearer token auth
-```
-
-Each scraped target gets `instance` (`<service_name>.<slot>`, stable across redeploys), `domain` (the service's stack namespace with underscores converted back to dots, e.g. `traefik.example.com`) and `task_slot` labels attached automatically.
-
-If the target needs authentication, configure the monitoring-ng instance with a scrape-auth secret that will be used for targets having the auth label set:
-```
-COMPOSE_FILE="$COMPOSE_FILE:compose.scrape-auth.yml"
-SECRET_SCRAPE_AUTH_VERSION=v1
-```
-`abra app secret insert <domain> scrape_auth v1 <password-or-token>`, then set the scraped app's `prometheus.io/auth` label to `basic` or `bearer` to match how it checks the secret.
-
-Check discovered targets via `alloy.example.org` (needs `compose.alloy-webui.yml`, see below), or query the central Prometheus for `up{job="<service_name>"}`.
-
-### Manual scraping
-
-For targets where auto-discovery doesn't work (e.g. not a Docker Swarm service on the `proxy` network, or missing labels): add them directly to Prometheus instead. Create a scrape config file:
-```yaml
- targets:
-    - 'metrics.something-external.example.org'
-    - 'app-without-labels.example.org'
-```
-and copy it into Prometheus' scrape config directory:
-```
-abra app cp monitoring.gathering.org targets.yml prometheus:/prometheus/scrape_configs/
-```
-Prometheus picks up files there automatically.
-
-### Alloy Web UI / Live Debugging
-
-Alloy's own web UI isn't exposed by default, its HTTP server only listens
-on localhost inside its own container. To reach it from outside (e.g. to
-browse its component graph or use live debugging), add:
-```
-COMPOSE_FILE="$COMPOSE_FILE:compose.alloy-webui.yml"
-```
-This exposes it (behind the same basic-auth) at `alloy.$DOMAIN`.
-
-To actually see what's being collected (live-tailing the metrics/logs
-flowing through each component, not just their config) also set
-`LIVE_DEBUGGING=true`. Only enable this while troubleshooting.
+| Service       | Authentication     | Domain                            |
+| ------------- | ------------------ | --------------------------------- |
+| Grafana       | Email / SSO        | monitoring.example.org            |
+| Prometheus    | traefik basic-auth | prometheus.monitoring.example.org |
+| loki          | traefik basic-auth | loki.monitoring.example.org       |
+| Cadvisor      | traefik basic-auth | cadvisor.monitoring.example.org   |
+| Node Exporter | traefik basic-auth | node.monitoring.example.org       |

 ### Logging from a docker host to loki server without anything else

@@ -122,25 +83,36 @@ $ echo '{
 $ systemctl restart docker.service
 ```

-### Setup Push Gateway
+## Setup Push Gateway

-1. Enable in the env file by uncommenting the following lines:
+1. Enable in the env fiöle by uncommenting the following lines:
 ```
 ## Prometheus Pushgateway
 # COMPOSE_FILE="$COMPOSE_FILE:compose.pushgateway.yml"
 ```
 2. `abra app deploy monitoring.example.org`

-This will expose the pushgateway at `https://pushgateway.${DOMAIN}`, secured behind the same basic auth as the other services.
-After that you need to add the `pushgateway.${DOMAIN}` to the scrape config of Prometheus.
+This will expose the pushgateway at `https://pushgateway.${DOMAIN}`.
+It is secured behind the same basic auth as the other services.
+After that you need to add the `pushgateway.${DOMAIN}` to the scare config.

+## Post-setup guide
+
+- configure prometheus/loki/alertmanager as data sources in grafana under `Configuration > Data sources`
+  - for loki, you need to set a "Custom HTTP Header": `X-Scope-OrgID: fake`
+- configure the SMTP mailer under `Alerting > Contact points`
+  - edit the default contact point, choose "Alertmanager" as type & `http://alertmanager:9093` as URL
+  - use the "Test" button to send a test mail. It should fire a request at the alertmanager & that should send a mail
+- `abra app cp` your `scrap_configs: ...` into `/prometheus/scrape_configs` & log into your prometheus web UI to ensure they're working
+- load your dashboards in manually under `Create > Dashboard`
+- from your dashboard panels, choose `Edit > Alert` to create alerts based on those panels


 ---

 THX to the previous work of @decentral1se @knooflok @3wc @cellarspoon @mirsal

-### Adding Matrix as Alert Contact point
+## Adding Matrix as Alert Contact point

 1. Enable the [matrix-alertmanager-receiver](https://github.com/metio/matrix-alertmanager-receiver/):
 ```
@@ -160,9 +132,17 @@ GF_MATRIX_HOME_SERVER_URL=
 ```
 4. Configure Alertmanager webhook and set the url to `http://matrix-alertmanager-receiver:12345/alerts/<room-id>`

-### Alerts
+## Alerts

 It is possible to enable the following alerts, by uncommenting the corresponding env variable:

 - node disk space: `ALERT_NODE_DISK_SPACE_LEFT`
 - node memory usage: `ALERT_NODE_MEMORY_USAGE`
+
+## smart monitoring
+
+To be able monitor hard drive health data, you need to configure
+`smartd` to run on the host system, and also the
+`collect-smartctl-json.sh` script provided here (via cronjob or as
+a `smartd` hook).  This is a limitation on Docker Swarm, which prevents
+the `smartctl_exporter` from running on privileged mode.
@@ -1,16 +1,26 @@
+export ENTRYPOINT_VERSION=v1
 export GF_DATASOURCES_VERSION=v1
 export GF_DASHBOARDS_VERSION=v2
 export GF_SWARM_DASH_VERSION=v2
 export GF_STACKS_DASH_VERSION=v2
-export GF_TRAEFIK_DASH_VERSION=v3
+export GF_TRAEFIK_DASH_VERSION=v2
 export GF_BACKUP_DASH_VERSION=v1
 export GF_CUSTOM_INI_VERSION=v4
 export LOKI_YML_VERSION=v3
 export PROMETHEUS_YML_VERSION=v2
 export MATRIX_ALERTMANAGER_CONFIG_VERSION=v1
 export MATRIX_ALERTMANAGER_ENTRYPOINT_VERSION=v1
-export GF_ALERTS_NODE_VERSION=v2
-export CONFIG_ALLOY_VERSION=v11
+export GRAFANA_ALERTS_NODE_VERSION=v2
+export CONFIG_ALLOY_VERSION=v10
+
+# creates a default prometheus scrape config for a given node
+add_node(){
+  name=$1
+  add_domain "$name" "metrics.traefik.$name"
+  add_domain "$name" "node.monitoring.$name"
+  add_domain "$name" "cadvisor.monitoring.$name"
+  cat "/prometheus/scrape_configs/$name.yml"
+}

 # migrates secrets from old names to new names by reading values from the
 # running containers on the server and re-inserting them under the new names.
@@ -27,7 +37,7 @@ migrate_secret_names() {
  # Hardcoded migration mappings: old_secret_name|new_secret_name
  MIGRATIONS="
 grafana_admin_password|gf_adminpasswd
-grafana_smtp_password|gf_smtp_pass
+grafana_smtp_password|gf_smtp_passwd
 grafana_oidc_client_secret|gf_oidc_secret
 matrix_access_token|matrix_token
 loki_aws_secret_access_key|loki_aws_key
@@ -106,3 +116,18 @@ loki_aws_secret_access_key|loki_aws_key
  echo ""
  echo "Done."
 }
+
+# adds a domain to a scrape config or creates a new one
+add_domain(){
+  name=$1
+  domain=$2
+  if [ ! -d "/prometheus/scrape_configs/" ]; then
+    mkdir -p /prometheus/scrape_configs/
+  fi
+  cd /prometheus/scrape_configs/ || exit 1
+  if [ ! -f "$name.yml" ]; then
+    echo -e "- targets:\n  - '$domain'" > "$name.yml"
+  else
+    echo "  - '$domain'"  >> "$name.yml"
+  fi
+}
@@ -29,7 +29,7 @@ groups:
          datasourceUid: PBFA97CFB590B2093
          model:
            editorMode: code
-            expr: (node_filesystem_free_bytes{fstype=~"ext4|xfs"} / node_filesystem_size_bytes{fstype=~"ext4|xfs"}) * 100
+            expr: (node_filesystem_free_bytes{fstype="ext4"} / node_filesystem_size_bytes{fstype="ext4"}) * 100
            instant: true
            intervalMs: 1000
            legendFormat: __auto
@@ -0,0 +1,6 @@
+[Unit]
+Description=Collect SMART data
+
+[Service]
+Type=oneshot
+ExecStart=/usr/local/bin/collect-smartctl-json.sh
@@ -0,0 +1,69 @@
+#! /bin/bash
+# Adapted from https://github.com/prometheus-community/smartctl_exporter/blob/master/collect-smartctl-json.sh
+
+script_dir=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )
+
+# Data directory to dump smartctl output
+# This directory will be created if it doesn't exist
+data_dir="/var/lib/smartmontools/json"
+
+# The original script used --xall but that doesn't work
+# This matches the command in readSMARTctl()
+smartctl_args="--json --info --health --attributes --tolerance=verypermissive \
+--nocheck=standby --format=brief --log=error"
+
+# Ignore this devices
+smartctl_ignore_dev_regex="^(/dev/bus)"
+
+# Determine the json query tool to use
+if command -v jq >/dev/null; then
+	json_tool="jq"
+	json_args="--raw-output"
+elif command -v yq >/dev/null; then
+	json_tool="yq"
+	json_args="--unwrapScalar"
+else
+	echo -e "One of 'yq' or 'jq' is required. Please try again after \
+installing one of them"
+	exit 1
+fi
+
+if [[ ! "${UID}" -eq 0 ]] && ! command -v sudo >/dev/null; then
+	# Not root and sudo doesn't exist
+	echo "sudo does not exist. Please run this as root"
+	exit 1
+fi
+
+SUDO="sudo"
+if [[ "${UID}" -eq 0 ]]; then
+	# Don't use sudo if root
+	SUDO=""
+fi
+
+[[ ! -d "${data_dir}" ]] && mkdir --parents "${data_dir}"
+
+if [[ $# -ne 0 ]]; then
+	devices="${1}"
+else
+	devices="$(smartctl --scan --json | "${json_tool}" "${json_args}" \
+".devices[].name | select(test(\"${smartctl_ignore_dev_regex}\") | not)")"
+  mapfile -t devices <<< "${devices[@]}"
+fi
+
+for device in "${devices[@]}"
+  do
+	echo -n "Collecting data for '${device}'..."
+	# shellcheck disable=SC2086
+	data="$($SUDO smartctl ${smartctl_args} ${device})"
+	# Accommodate a smartmontools pre-7.3 bug
+	data=${data#"  Pending defect count:"}
+	type="$(echo "${data}" | "${json_tool}" "${json_args}" '.device.type')"
+	family="$(echo "${data}" | "${json_tool}" "${json_args}" \
+'select(.model_family != null) | .model_family | sub(" |/" ; "_" ; "g")
+ | sub("\"|\\(|\\)" ; "" ; "g")')"
+	model="$(echo "${data}" | "${json_tool}" "${json_args}" \
+'.model_name | sub(" |/" ; "_" ; "g") | sub("\"|\\(|\\)" ; "" ; "g")')"
+	device_name="$(basename "${device}")"
+	echo -e "\tSaving to ${device_name}.json"
+	echo "${data}" > "${data_dir}/${device_name}.json"
+done
@@ -0,0 +1,9 @@
+[Unit]
+Description=Collect SMART data
+
+[Timer]
+OnCalendar=hourly
+Persistent=true
+
+[Install]
+WantedBy=timers.target
@@ -1,16 +0,0 @@
-version: "3.8"
-
-services:
-  app:
-    environment:
-      - ALLOY_HTTP_LISTEN_ADDR=0.0.0.0
-    deploy:
-      labels:
-        - "traefik.enable=true"
-        - "traefik.swarm.network=proxy"
-        - "traefik.http.services.${STACK_NAME}-alloy.loadbalancer.server.port=12345"
-        - "traefik.http.routers.${STACK_NAME}-alloy.rule=Host(`alloy.${DOMAIN}`)"
-        - "traefik.http.routers.${STACK_NAME}-alloy.entrypoints=web-secure"
-        - "traefik.http.routers.${STACK_NAME}-alloy.tls=true"
-        - "traefik.http.routers.${STACK_NAME}-alloy.tls.certresolver=${LETS_ENCRYPT_ENV}"
-        - "traefik.http.routers.${STACK_NAME}-alloy.middlewares=basicauth@file"
@@ -0,0 +1,7 @@
+---
+version: "3.8"
+
+secrets:
+  basic_auth:
+    external: true
+    name: ${STACK_NAME}_basic_auth_${SECRET_BASIC_AUTH_VERSION}
@@ -3,16 +3,16 @@ version: '3.8'
 services:
  grafana:
    secrets:
-      - gf_smtp_pass
+      - gf_smtp_passwd
    environment:
      - GF_SMTP_HOST
      - GF_SMTP_USER
-      - GF_SMTP_PASSWORD__FILE=/run/secrets/gf_smtp_pass
+      - GF_SMTP_PASSWORD__FILE=/run/secrets/gf_smtp_passwd
      - GF_SMTP_ENABLED
      - GF_SMTP_FROM_ADDRESS
      - GF_SMTP_SKIP_VERIFY

 secrets:
-  gf_smtp_pass:
+  gf_smtp_passwd:
    external: true
-    name: ${STACK_NAME}_gf_smtp_pass_${SECRET_GF_SMTP_PASS_VERSION}
+    name: ${STACK_NAME}_gf_smtp_passwd_${SECRET_GF_SMTP_PASSWD_VERSION}
@@ -2,7 +2,7 @@ version: '3.8'

 services:
  grafana:
-    image: grafana/grafana:13.0.6
+    image: grafana/grafana:13.0.3
    volumes:
      - grafana-data:/var/lib/grafana:rw
    secrets:
@@ -39,7 +39,7 @@ services:
        - "traefik.enable=true"
        - "traefik.swarm.network=proxy"
        - "traefik.http.services.${STACK_NAME}-grafana.loadbalancer.server.port=3000"
-        - "traefik.http.routers.${STACK_NAME}-grafana.rule=Host(`${GRAFANA_DOMAIN:-$DOMAIN}`)"
+        - "traefik.http.routers.${STACK_NAME}-grafana.rule=Host(`${DOMAIN}`)"
        - "traefik.http.routers.${STACK_NAME}-grafana.entrypoints=web-secure"
        - "traefik.http.routers.${STACK_NAME}-grafana.tls=true"
        - "traefik.http.routers.${STACK_NAME}-grafana.tls.certresolver=${LETS_ENCRYPT_ENV}"
@@ -75,7 +75,7 @@ configs:
    file: grafana-backup-dashboard.json
  gf_alerts_node:
    template_driver: golang
-    name: ${STACK_NAME}_gf_alerts_node_${GF_ALERTS_NODE_VERSION}
+    name: ${STACK_NAME}_gf_alerts_node_${GRAFANA_ALERTS_NODE_VERSION}
    file: alerts/node.yml.tmpl

 volumes:
@@ -20,12 +20,6 @@ services:
    networks:
      - proxy
      - internal
-    healthcheck:
-      test: "wget -q --spider http://localhost:9090/-/healthy || exit 1"
-      interval: 5s
-      timeout: 10s
-      retries: 3
-      start_period: 30s
    deploy:
      restart_policy:
        condition: on-failure
@@ -12,12 +12,6 @@ services:
    networks:
      - internal
      - proxy
-    healthcheck:
-      test: "wget -q --spider http://localhost:9191/-/healthy || exit 1"
-      interval: 5s
-      timeout: 10s
-      retries: 3
-      start_period: 10s
    deploy:
      restart_policy:
        condition: on-failure
@@ -1,13 +0,0 @@
---
-version: "3.8"
-
-services:
-  app:
-    secrets:
-      - source: scrape_auth
-        target: scrape_auth
-
-secrets:
-  scrape_auth:
-    external: true
-    name: ${STACK_NAME}_scrape_auth_${SECRET_SCRAPE_AUTH_VERSION}
@@ -0,0 +1,18 @@
+---
+version: "3.8"
+services:
+  smartctl:
+    image: "prometheuscommunity/smartctl-exporter:v0.14.0"
+    volumes:
+    - "/dev:/dev"
+    - "/var/lib/smartmontools/json:/debug"
+    command:
+    - "--smartctl.fake-data"
+    - "--smartctl.interval=1h"
+    networks:
+    - "proxy"
+    deploy:
+      labels:
+        - "prometheus.io/scrape=true"
+        - "prometheus.io/port=9633"
+        - "prometheus.io/path=/metrics"
@@ -3,7 +3,7 @@ version: "3.8"

 services:
  app:
-    image: grafana/alloy:v1.18.1
+    image: grafana/alloy:v1.17.0
    hostname: "${DOMAIN}"
    configs:
      - source: config_alloy
@@ -13,47 +13,31 @@ services:
      - /var/run/docker.sock:/var/run/docker.sock:ro
      - /sys:/sys:ro
      - /var/lib/docker:/var/lib/docker:ro
-      # long-form avoids toolshed/abra#900
-      - type: bind
-        source: "${CONTAINERD_SOCKET:-/run/containerd/containerd.sock}"
-        target: /run/containerd/containerd.sock
-        read_only: true
      - alloy-data:/var/lib/alloy/data
-    # runs through a shell so ALLOY_HTTP_LISTEN_ADDR (set by
-    # compose.alloy-webui.yml) is resolved from the container's own
-    # environment at startup, not by compose at deploy time.
-    # $$ escapes it from compose's own interpolation
-    entrypoint: ["/bin/sh", "-c"]
    command:
-      - >-
-        exec alloy run
-        --storage.path=/var/lib/alloy/data
-        --server.http.listen-addr=$${ALLOY_HTTP_LISTEN_ADDR:-127.0.0.1}:12345
-        /etc/alloy/config.alloy
+      - "run"
+      - "--storage.path=/var/lib/alloy/data"
+      - "--server.http.listen-addr=0.0.0.0:12345"
+      - "/etc/alloy/config.alloy"
    networks:
      - proxy
      - internal
-    environment:
-      - SCRAPE_BASIC_AUTH_USERNAME=${SCRAPE_BASIC_AUTH_USERNAME:-alloy}
-      - WRITE_BASIC_AUTH_USERNAME=${WRITE_BASIC_AUTH_USERNAME:-admin}
-      - LIVE_DEBUGGING=${LIVE_DEBUGGING:-false}
-      - NODE_ID={{.Node.ID}}
    secrets:
      - basic_auth
-    # no wget/curl in this image; bash's /dev/tcp is used instead. Works
-    # against localhost regardless of ALLOY_HTTP_LISTEN_ADDR
-    healthcheck:
-      test: ["CMD", "bash", "-c", "exec 3<>/dev/tcp/localhost/12345 && printf 'GET /-/ready HTTP/1.0\r\nHost: localhost\r\n\r\n' >&3 && head -1 <&3 | grep -q 200"]
-      interval: 5s
-      timeout: 10s
-      retries: 3
-      start_period: 10s
    deploy:
      restart_policy:
        condition: on-failure
      labels:
        - "backupbot.backup=${ENABLE_BACKUPS:-true}"
        - "coop-cloud.${STACK_NAME}.version=1.6.0+v1.8.1"
+        - "traefik.enable=true"
+        - "traefik.swarm.network=proxy"
+        - "traefik.http.services.${STACK_NAME}-alloy.loadbalancer.server.port=12345"
+        - "traefik.http.routers.${STACK_NAME}-alloy.rule=Host(`alloy.${DOMAIN}`)"
+        - "traefik.http.routers.${STACK_NAME}-alloy.entrypoints=web-secure"
+        - "traefik.http.routers.${STACK_NAME}-alloy.tls=true"
+        - "traefik.http.routers.${STACK_NAME}-alloy.tls.certresolver=${LETS_ENCRYPT_ENV}"
+        - "traefik.http.routers.${STACK_NAME}-alloy.middlewares=basicauth@file"
 configs:
  config_alloy:
    template_driver: golang
@@ -44,6 +44,7 @@ prometheus.scrape "default" {
    prometheus.exporter.self.alloy.targets,
    prometheus.exporter.unix.default.targets,
    prometheus.exporter.cadvisor.docker.targets,
+    discovery.docker.containers.targets,
  )

  forward_to = [prometheus.remote_write.prometheus.receiver]
@@ -54,43 +55,31 @@ prometheus.remote_write "prometheus" {
    url = "{{ env "PROMETHEUS_REMOTE_WRITE_URL" }}"

    basic_auth {
-      username = "{{ env "WRITE_BASIC_AUTH_USERNAME" }}"
+      username = "admin"
      password = "{{ secret "basic_auth" }}"
    }
  }
 }

+discovery.docker "containers" {
+  host                = "unix:///var/run/docker.sock"
+  match_first_network = false
+}
+
 // Scrape Prometheus metrics from other containers on this host.
 // Containers opt in via Docker labels:
 //   prometheus.io/scrape=true        required: enable scraping
-//   prometheus.io/port=9090          optional: port exposing /metrics (defaults to 80 if not set)
+//   prometheus.io/port=9090          optional: port exposing /metrics (defaults to first exposed port)
 //   prometheus.io/path=/metrics      optional: path to metrics endpoint (default: /metrics)
-//   prometheus.io/auth=basic         optional: use basic auth with the scrape_auth secret (see compose.scrape-auth.yml)
-//   prometheus.io/auth=bearer        optional: use bearer auth with the scrape_auth secret (see compose.scrape-auth.yml)
+//   prometheus.io/auth=basic         optional: use basic auth with the shared basic_auth secret
 discovery.dockerswarm "swarm" {
  host = "unix:///var/run/docker.sock"
-  // "tasks" not "services": dnsrr-mode services (e.g. traefik) have no VIP
-  // and are invisible to the "services" role
-  role = "tasks"
+  role = "services"
 }

 discovery.relabel "metrics" {
  targets = discovery.dockerswarm.swarm.targets

-  // skip old task history, only scrape currently-running tasks
-  rule {
-    source_labels = ["__meta_dockerswarm_task_desired_state"]
-    regex         = "running"
-    action        = "keep"
-  }
-
-  // only scrape hosts running on this host within the swam
-  rule {
-    source_labels = ["__meta_dockerswarm_node_id"]
-    regex         = "{{ env "NODE_ID" }}"
-    action        = "keep"
-  }
-
  rule {
    source_labels = ["__meta_dockerswarm_network_name"]
    regex         = "proxy"
@@ -103,17 +92,6 @@ discovery.relabel "metrics" {
    action        = "keep"
  }

-  // default to port 80 when prometheus.io/port isn't set
-  rule {
-    source_labels = ["__meta_dockerswarm_service_label_prometheus_io_port"]
-    regex         = "^$"
-    target_label  = "__meta_dockerswarm_service_label_prometheus_io_port"
-    replacement   = "80"
-  }
-
-  // a task with multiple published ports produces one target per port;
-  // this unifies all of them to the single port above, so duplicates
-  // collapse at scrape time instead of scraping every port
  rule {
    source_labels = ["__address__", "__meta_dockerswarm_service_label_prometheus_io_port"]
    regex         = `(.+):\d+;(\d+)`
@@ -131,65 +109,6 @@ discovery.relabel "metrics" {
    source_labels = ["__meta_dockerswarm_service_name"]
    target_label  = "job"
  }
-
-  // task IDs (and the default address-derived instance label) change on
-  // every redeploy; service+slot is stable across redeploys of the same
-  // replica, so data stays continuous instead of restarting each deploy
-  rule {
-    source_labels = ["__meta_dockerswarm_service_name", "__meta_dockerswarm_task_slot"]
-    separator     = "."
-    target_label  = "instance"
-  }
-
-  rule {
-    source_labels = ["__meta_dockerswarm_task_slot"]
-    target_label  = "task_slot"
-  }
-
-  // coop-cloud's STACK_NAME is the domain with "." replaced by "_"
-  // Derive a readable dotted domain label from it. RE2 has no global
-  // replace, so this is done by chaining multiple replacements. Each
-  // rule swaps the first remaining _ for a "." until none are left.
-  rule {
-    source_labels = ["__meta_dockerswarm_service_label_com_docker_stack_namespace"]
-    target_label  = "domain"
-  }
-  rule {
-    source_labels = ["domain"]
-    regex         = `([^_]*)_(.*)`
-    target_label  = "domain"
-    replacement   = "$1.$2"
-  }
-  rule {
-    source_labels = ["domain"]
-    regex         = `([^_]*)_(.*)`
-    target_label  = "domain"
-    replacement   = "$1.$2"
-  }
-  rule {
-    source_labels = ["domain"]
-    regex         = `([^_]*)_(.*)`
-    target_label  = "domain"
-    replacement   = "$1.$2"
-  }
-  rule {
-    source_labels = ["domain"]
-    regex         = `([^_]*)_(.*)`
-    target_label  = "domain"
-    replacement   = "$1.$2"
-  }
-  rule {
-    source_labels = ["domain"]
-    regex         = `([^_]*)_(.*)`
-    target_label  = "domain"
-    replacement   = "$1.$2"
-  }
-  rule {
-    source_labels = ["domain"]
-    regex         = `([^_]*)_(.*)`
-    target_label  = "domain"
-    replacement   = "$1.$2"
-  }
 }

 discovery.relabel "metrics_noauth" {
@@ -225,14 +144,13 @@ prometheus.scrape "containers" {
  forward_to      = [prometheus.remote_write.prometheus.receiver]
 }

-{{ if ne (env "SECRET_SCRAPE_AUTH_VERSION") "" }}
 prometheus.scrape "containers_basicauth" {
  scrape_interval = "120s"
  targets         = discovery.relabel.metrics_basicauth.output
  forward_to      = [prometheus.remote_write.prometheus.receiver]
  basic_auth {
-    username = "{{ env "SCRAPE_BASIC_AUTH_USERNAME" }}"
-    password = "{{ secret "scrape_auth" }}"
+    username = "admin"
+    password = "{{ secret "basic_auth" }}"
  }
 }

@@ -240,10 +158,9 @@ prometheus.scrape "containers_bearerauth" {
  scrape_interval = "120s"
  targets         = discovery.relabel.metrics_bearerauth.output
  forward_to      = [prometheus.remote_write.prometheus.receiver]
-  bearer_token    = "{{ secret "scrape_auth" }}"
+  bearer_token    = "{{ secret "basic_auth" }}"
 }
 {{ end }}
-{{ end }}

 {{ if ne (env "LOKI_PUSH_URL") "" }}
 discovery.relabel "docker" {
@@ -340,7 +257,7 @@ loki.write "loki" {
    url = "{{ env "LOKI_PUSH_URL" }}"

    basic_auth {
-      username = "{{ env "WRITE_BASIC_AUTH_USERNAME" }}"
+      username = "admin"
      password = "{{ secret "basic_auth" }}"
    }
  }
@@ -110,7 +110,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "editorMode": "code",
-          "expr": "(time() - min(container_start_time_seconds{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}))",
+          "expr": "(time() - min(container_start_time_seconds{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "",
@@ -215,7 +215,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "editorMode": "code",
-          "expr": "sort(\n    sum(\n        rate(\n            container_cpu_usage_seconds_total{\n                container_label_com_docker_stack_namespace=~\"$stack\", \n                instance=~\".*$node_id.*\"}[$interval]\n            )\n        ) \n    by (container_label_com_docker_swarm_service_name)\n)",
+          "expr": "sort(\n    sum(\n        rate(\n            container_cpu_usage_seconds_total{\n                container_label_com_docker_stack_namespace=~\"$stack\", \n                instance=~\"cadvisor.monitoring.$instance.*\"}[$interval]\n            )\n        ) \n    by (container_label_com_docker_swarm_service_name)\n)",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "{{ container_label_com_docker_swarm_service_name }}",
@@ -283,7 +283,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "editorMode": "code",
-          "expr": "count(rate(container_last_seen{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}[$interval]))",
+          "expr": "count(rate(container_last_seen{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}[$interval]))",
          "format": "time_series",
          "intervalFactor": 2,
          "range": true,
@@ -386,7 +386,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "editorMode": "code",
-          "expr": "sum(\n    container_memory_rss{\n        container_label_com_docker_stack_namespace=~\"$stack\", \n        instance=~\".*$node_id.*\"}\n    ) \nby (container_label_com_docker_swarm_service_name)",
+          "expr": "sum(\n    container_memory_rss{\n        container_label_com_docker_stack_namespace=~\"$stack\", \n        instance=~\"cadvisor.monitoring.$instance.*\"}\n    ) \nby (container_label_com_docker_swarm_service_name)",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 2,
@@ -493,7 +493,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "editorMode": "code",
-          "expr": "sum(rate(container_network_receive_bytes_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
+          "expr": "sum(rate(container_network_receive_bytes_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "RX: {{ container_label_com_docker_swarm_service_name }}",
@@ -507,7 +507,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "editorMode": "code",
-          "expr": "sum(rate(container_network_transmit_bytes_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}[$interval])) by (container_label_com_docker_swarm_service_name) * -1",
+          "expr": "sum(rate(container_network_transmit_bytes_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}[$interval])) by (container_label_com_docker_swarm_service_name) * -1",
          "hide": false,
          "legendFormat": "TX: {{container_label_com_docker_swarm_service_name}}",
          "range": true,
@@ -634,7 +634,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "editorMode": "code",
-          "expr": "sum(container_fs_usage_bytes{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}) by (container_label_com_docker_swarm_service_name)",
+          "expr": "sum(container_fs_usage_bytes{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}) by (container_label_com_docker_swarm_service_name)",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 2,
@@ -742,7 +742,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "editorMode": "code",
-          "expr": "sum(rate(container_fs_usage_bytes{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
+          "expr": "sum(rate(container_fs_usage_bytes{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 2,
@@ -865,7 +865,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "editorMode": "code",
-          "expr": "sum(rate(container_fs_io_time_seconds_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
+          "expr": "sum(rate(container_fs_io_time_seconds_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 2,
@@ -973,7 +973,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "editorMode": "code",
-          "expr": "container_fs_io_current{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}",
+          "expr": "container_fs_io_current{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 2,
@@ -1081,7 +1081,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "editorMode": "code",
-          "expr": "sum(rate(container_fs_reads_bytes_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
+          "expr": "sum(rate(container_fs_reads_bytes_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "{{ container_label_com_docker_swarm_service_name }}",
@@ -1188,7 +1188,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "editorMode": "code",
-          "expr": "sum(rate(container_fs_read_seconds_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
+          "expr": "sum(rate(container_fs_read_seconds_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "{{ container_label_com_docker_swarm_service_name }}",
@@ -1295,7 +1295,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "editorMode": "code",
-          "expr": "sum(rate(container_fs_writes_bytes_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
+          "expr": "sum(rate(container_fs_writes_bytes_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "{{ container_label_com_docker_swarm_service_name }}",
@@ -1402,7 +1402,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "editorMode": "code",
-          "expr": "sum(rate(container_fs_write_seconds_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
+          "expr": "sum(rate(container_fs_write_seconds_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "{{ container_label_com_docker_swarm_service_name }}",
@@ -1453,7 +1453,7 @@
          "query": "label_values(container_label_com_docker_stack_namespace)",
          "refId": "PrometheusVariableQueryEditor-VariableQuery"
        },
-        "refresh": 2,
+        "refresh": 1,
        "regex": "",
        "skipUrlSync": false,
        "sort": 2,
@@ -1567,19 +1567,19 @@
          "type": "prometheus",
          "uid": "PBFA97CFB590B2093"
        },
-        "definition": "label_values(node_uname_info, instance)",
+        "definition": "label_values(instance)",
        "hide": 0,
        "includeAll": true,
-        "label": "Swarm Node",
+        "label": "instance",
        "multi": true,
-        "name": "node_id",
+        "name": "instance",
        "options": [],
        "query": {
-          "query": "label_values(node_uname_info, instance)",
+          "query": "label_values(instance)",
          "refId": "PrometheusVariableQueryEditor-VariableQuery"
        },
-        "refresh": 2,
-        "regex": "/^(?:(?:node|cadvisor|monitoring).)*(.*)$/",
+        "refresh": 1,
+        "regex": "/.*cadvisor.monitoring.(?<instance>.*):80/",
        "skipUrlSync": false,
        "sort": 0,
        "type": "query"
@@ -1620,4 +1620,4 @@
  "uid": "KdVoGQm7z",
  "version": 36,
  "weekStart": ""
-}
+}
@@ -118,7 +118,7 @@
            "type": "prometheus",
            "uid": "PBFA97CFB590B2093"
          },
-          "expr": "topk(1, sum(label_replace(node_time_seconds{instance=~\".*$node_id.*\"} - node_boot_time_seconds{instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) by (instance))",
+          "expr": "topk(1, sum((node_time_seconds - node_boot_time_seconds) * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"}) by (node_name))",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "",
@@ -198,7 +198,7 @@
            "type": "prometheus",
            "uid": "PBFA97CFB590B2093"
          },
-          "expr": "count(label_replace(node_uname_info{instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\"))",
+          "expr": "count(node_meta * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"})",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "",
@@ -278,7 +278,7 @@
            "type": "prometheus",
            "uid": "PBFA97CFB590B2093"
          },
-          "expr": "count(label_replace(node_cpu_seconds_total{mode=\"idle\", instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\"))",
+          "expr": "count(node_cpu_seconds_total{mode=\"idle\"} * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"})",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "",
@@ -363,7 +363,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "editorMode": "code",
-          "expr": "sum(label_replace((node_memory_MemAvailable_bytes{instance=~\".*$node_id.*\"} / node_memory_MemTotal_bytes{instance=~\".*$node_id.*\"}) * 100, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) / count(label_replace(node_uname_info{instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\"))",
+          "expr": "sum((node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"} * 100) / count(node_meta * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"})",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "",
@@ -429,11 +429,11 @@
            "type": "prometheus",
            "uid": "PBFA97CFB590B2093"
          },
-          "expr": "label_replace(node_load5{instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")",
+          "expr": "node_load5 * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"}",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 2,
-          "legendFormat": "load5 {{instance}}",
+          "legendFormat": "load5 {{node_name}}",
          "refId": "A",
          "step": 2
        }
@@ -536,7 +536,7 @@
            "type": "prometheus",
            "uid": "PBFA97CFB590B2093"
          },
-          "expr": "sum(label_replace(node_memory_MemTotal_bytes{instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\"))",
+          "expr": "sum(node_memory_MemTotal_bytes * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"})",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "",
@@ -616,7 +616,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "exemplar": true,
-          "expr": "sum(label_replace(node_filesystem_size_bytes{fstype=~\"ext4|xfs\",mountpoint=~\"(/$)|(/media.*)\", instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\"))",
+          "expr": "sum(node_filesystem_size_bytes{fstype=\"ext4\",mountpoint=~\"(/$)|(/media.*)\"} * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"})",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 2,
@@ -701,7 +701,7 @@
            "type": "prometheus",
            "uid": "PBFA97CFB590B2093"
          },
-          "expr": "sum(label_replace(irate(node_cpu_seconds_total{mode=\"idle\", instance=~\".*$node_id.*\"}[$interval]), \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) * 100 / count(label_replace(node_cpu_seconds_total{mode=\"user\", instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\"))",
+          "expr": "sum(irate(node_cpu_seconds_total{mode=\"idle\"}[$interval]) * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"}) * 100 / count(node_cpu_seconds_total{mode=\"user\"} * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"}) ",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "",
@@ -833,7 +833,7 @@
          },
          "editorMode": "code",
          "exemplar": true,
-          "expr": "label_replace(node_filesystem_free_bytes{fstype=~\"ext4|xfs\", instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\") / label_replace(node_filesystem_size_bytes{fstype=~\"ext4|xfs\", instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\") * 100",
+          "expr": "node_filesystem_free_bytes{fstype=\"ext4\"}  / node_filesystem_size_bytes{fstype=\"ext4\"} * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"} * 100",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 2,
@@ -900,10 +900,10 @@
            "type": "prometheus",
            "uid": "PBFA97CFB590B2093"
          },
-          "expr": "100 - (avg(label_replace(irate(node_cpu_seconds_total{mode=\"idle\", instance=~\".*$node_id.*\"}[$interval]), \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\") * 100) by (instance))",
+          "expr": "100 - (avg(irate(node_cpu_seconds_total{mode=\"idle\"}[$interval])  * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"} * 100) by (node_name))",
          "format": "time_series",
          "intervalFactor": 2,
-          "legendFormat": "{{instance}}",
+          "legendFormat": "{{node_name}}",
          "refId": "A",
          "step": 2
        }
@@ -1045,12 +1045,12 @@
            "uid": "PBFA97CFB590B2093"
          },
          "editorMode": "code",
-          "expr": "sum(label_replace(node_memory_MemFree_bytes{instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) by (instance)",
+          "expr": "sum((node_memory_MemFree_bytes) * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"}) by (node_name)",
          "format": "time_series",
          "hide": false,
          "interval": "",
          "intervalFactor": 2,
-          "legendFormat": "Free  {{instance}}",
+          "legendFormat": "Free  {{node_name}}",
          "range": true,
          "refId": "free",
          "step": 2
@@ -1061,12 +1061,12 @@
            "uid": "PBFA97CFB590B2093"
          },
          "editorMode": "code",
-          "expr": "sum(label_replace(node_memory_Cached_bytes{instance=~\".*$node_id.*\"} + node_memory_Buffers_bytes{instance=~\".*$node_id.*\"} + node_memory_Slab_bytes{instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) by (instance)",
+          "expr": "sum((node_memory_Cached_bytes + node_memory_Buffers_bytes + node_memory_Slab_bytes) * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"}) by (node_name)",
          "format": "time_series",
          "hide": false,
          "interval": "",
          "intervalFactor": 2,
-          "legendFormat": "cache,buffer,slab  {{instance}}",
+          "legendFormat": "cache,buffer,slab  {{node_name}}",
          "range": true,
          "refId": "cache,buffer,slab",
          "step": 2
@@ -1077,12 +1077,12 @@
            "uid": "PBFA97CFB590B2093"
          },
          "editorMode": "code",
-          "expr": "sum(label_replace(node_memory_MemTotal_bytes{instance=~\".*$node_id.*\"} - node_memory_MemFree_bytes{instance=~\".*$node_id.*\"} - node_memory_Cached_bytes{instance=~\".*$node_id.*\"} - node_memory_Buffers_bytes{instance=~\".*$node_id.*\"} - node_memory_Slab_bytes{instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) by (instance)",
+          "expr": "sum((node_memory_MemTotal_bytes - node_memory_MemFree_bytes - node_memory_Cached_bytes - node_memory_Buffers_bytes - node_memory_Slab_bytes) * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"}) by (node_name)",
          "format": "time_series",
          "hide": false,
          "interval": "",
          "intervalFactor": 2,
-          "legendFormat": "Used {{instance}}",
+          "legendFormat": "Used {{node_name}}",
          "range": true,
          "refId": "Used",
          "step": 2
@@ -1094,13 +1094,13 @@
          },
          "editorMode": "code",
          "exemplar": false,
-          "expr": "sum(label_replace(node_memory_MemTotal_bytes{instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) by (instance)",
+          "expr": "sum((node_memory_MemTotal_bytes) * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"}) by (node_name)",
          "format": "time_series",
          "hide": false,
          "instant": false,
          "interval": "",
          "intervalFactor": 2,
-          "legendFormat": "Total {{instance}}",
+          "legendFormat": "Total {{node_name}}",
          "range": true,
          "refId": "total",
          "step": 2
@@ -1161,11 +1161,11 @@
            "type": "prometheus",
            "uid": "PBFA97CFB590B2093"
          },
-          "expr": "sum(label_replace(irate(node_disk_read_bytes_total{instance=~\".*$node_id.*\"}[$interval]), \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) by (instance)",
+          "expr": "sum(irate(node_disk_read_bytes_total[$interval]) * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"}) by (node_name)",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 2,
-          "legendFormat": "Read {{instance}}",
+          "legendFormat": "Read {{node_name}}",
          "refId": "A",
          "step": 2
        },
@@ -1174,10 +1174,10 @@
            "type": "prometheus",
            "uid": "PBFA97CFB590B2093"
          },
-          "expr": "sum(label_replace(irate(node_disk_written_bytes_total{instance=~\".*$node_id.*\"}[$interval]), \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) by (instance)",
+          "expr": "sum(irate(node_disk_written_bytes_total[$interval]) * on(instance) group_left(node_name) node_meta{node_id=~\"$node_id\"}) by (node_name)",
          "format": "time_series",
          "intervalFactor": 2,
-          "legendFormat": "Written {{instance}}",
+          "legendFormat": "Written {{node_name}}",
          "refId": "B",
          "step": 2
        }
@@ -1264,10 +1264,10 @@
            "type": "prometheus",
            "uid": "PBFA97CFB590B2093"
          },
-          "expr": "sum(label_replace(irate(node_disk_reads_completed_total{instance=~\".*$node_id.*\"}[$interval]), \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) by (instance)",
+          "expr": "sum(irate(node_disk_reads_completed_total[$interval]) * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"}) by (node_name)",
          "format": "time_series",
          "intervalFactor": 2,
-          "legendFormat": "Reads {{instance}}",
+          "legendFormat": "Reads {{node_name}}",
          "refId": "A",
          "step": 2
        },
@@ -1276,10 +1276,10 @@
            "type": "prometheus",
            "uid": "PBFA97CFB590B2093"
          },
-          "expr": "sum(label_replace(irate(node_disk_writes_completed_total{instance=~\".*$node_id.*\"}[$interval]), \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) by (instance)",
+          "expr": "sum(irate(node_disk_writes_completed_total[$interval]) * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"}) by (node_name)",
          "format": "time_series",
          "intervalFactor": 2,
-          "legendFormat": "Writes {{instance}}",
+          "legendFormat": "Writes {{node_name}}",
          "refId": "B",
          "step": 2
        }
@@ -1368,10 +1368,10 @@
            "type": "prometheus",
            "uid": "PBFA97CFB590B2093"
          },
-          "expr": "(avg(label_replace(irate(node_cpu_seconds_total{mode=\"iowait\", instance=~\".*$node_id.*\"}[$interval]), \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\") * 100) by (instance))",
+          "expr": "(avg(irate(node_cpu_seconds_total{mode=\"iowait\"}[$interval])  * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"} * 100) by (node_name))",
          "format": "time_series",
          "intervalFactor": 2,
-          "legendFormat": "{{instance}}",
+          "legendFormat": "{{node_name}}",
          "refId": "A",
          "step": 2
        }
@@ -1464,7 +1464,7 @@
            "type": "prometheus",
            "uid": "PBFA97CFB590B2093"
          },
-          "expr": "sum(rate(container_last_seen{instance=~\".*$node_id.*\"}[5m])) by (container_label_com_docker_swarm_service_name)",
+          "expr": "sum(rate(container_last_seen[5m]) * on(container_label_com_docker_swarm_node_id) group_left(node_name) node_meta{node_name=~\"${node_id}\"}) by (container_label_com_docker_swarm_service_name)",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 10,
@@ -1573,7 +1573,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "exemplar": true,
-          "expr": "count(rate(container_last_seen{instance=~\".*$node_id.*\"}[5m]))",
+          "expr": "count(rate(container_last_seen[5m]) * on(container_label_com_docker_swarm_node_id) group_left(node_name) node_meta{node_name=~\"$node_id\"})",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 2,
@@ -1668,11 +1668,11 @@
            "type": "prometheus",
            "uid": "PBFA97CFB590B2093"
          },
-          "expr": "sum(label_replace(rate(container_network_receive_bytes_total{instance=~\".*$node_id.*\"}[$interval]), \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) by (instance)",
+          "expr": "sum(rate(container_network_receive_bytes_total[$interval]) * on(container_label_com_docker_swarm_node_id) group_left(node_name) node_meta{node_name=~\"${node_id}\"}) by (node_name)",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 2,
-          "legendFormat": "IN {{instance}}",
+          "legendFormat": "IN {{node_name}}",
          "refId": "A",
          "step": 2
        },
@@ -1681,11 +1681,11 @@
            "type": "prometheus",
            "uid": "PBFA97CFB590B2093"
          },
-          "expr": "- sum(label_replace(rate(container_network_transmit_bytes_total{instance=~\".*$node_id.*\"}[$interval]), \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) by (instance)",
+          "expr": "- sum(rate(container_network_transmit_bytes_total[$interval]) * on(container_label_com_docker_swarm_node_id) group_left(node_name) node_meta{node_name=~\"${node_id}\"}) by (node_name)",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 2,
-          "legendFormat": "OUT {{instance}}",
+          "legendFormat": "OUT {{node_name}}",
          "metric": "",
          "refId": "B",
          "step": 2
@@ -1762,11 +1762,11 @@
        "name": "node_id",
        "options": [],
        "query": {
-          "query": "label_values(node_uname_info, instance)",
+          "query": "node_meta",
          "refId": "Prometheus-node_id-Variable-Query"
        },
-        "refresh": 2,
-        "regex": "/^(?:(?:node|cadvisor|monitoring).)*(.*)$/",
+        "refresh": 1,
+        "regex": "/node_name=\"([^\"]+)\"/",
        "skipUrlSync": false,
        "sort": 0,
        "type": "query"
@@ -116,12 +116,12 @@
            "uid": "PBFA97CFB590B2093"
          },
          "exemplar": false,
-          "expr": "time() - label_replace(process_start_time_seconds{instance=~\".*$domain.*\"} or process_start_time_seconds{domain=\"$domain\"}, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)",
+          "expr": "time() - process_start_time_seconds{job=\"$job\"}",
          "format": "time_series",
          "instant": true,
          "interval": "",
          "intervalFactor": 2,
-          "legendFormat": "{{ domain }}",
+          "legendFormat": "{{ instance }}",
          "refId": "A"
        }
      ],
@@ -200,7 +200,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "exemplar": true,
-          "expr": "sum(label_replace(increase(traefik_service_requests_total{code=\"499\", instance=~\".*$domain.*\"}[$interval]) or increase(traefik_service_requests_total{code=\"499\", domain=\"$domain\"}[$interval]), \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`))",
+          "expr": "sum(increase(traefik_service_requests_total{code=\"499\", instance=\"$instance\"}[$interval]))",
          "interval": "",
          "legendFormat": "",
          "refId": "A"
@@ -280,7 +280,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "exemplar": true,
-          "expr": "sum(label_replace(increase(traefik_service_requests_total{code=\"404\",protocol=~\"$protocol\", instance=~\".*$domain.*\"}[$interval]) or increase(traefik_service_requests_total{code=\"404\",protocol=~\"$protocol\", domain=\"$domain\"}[$interval]), \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`))",
+          "expr": "sum(increase(traefik_service_requests_total{code=\"404\",protocol=~\"$protocol\", instance=\"$instance\"}[$interval]))",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 2,
@@ -365,7 +365,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "exemplar": true,
-          "expr": "sum(label_replace(increase(traefik_service_requests_total{code=\"200\",protocol=~\"$protocol\", instance=~\".*$domain.*\"}[$interval]) or increase(traefik_service_requests_total{code=\"200\",protocol=~\"$protocol\", domain=\"$domain\"}[$interval]), \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`))",
+          "expr": "sum(increase(traefik_service_requests_total{code=\"200\",protocol=~\"$protocol\", instance=\"$instance\"}[$interval]))",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 2,
@@ -417,7 +417,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "exemplar": true,
-          "expr": "topk(5, sum(label_replace(traefik_service_requests_total{protocol=~\"$protocol\", instance=~\".*$domain.*\"} or traefik_service_requests_total{protocol=~\"$protocol\", domain=\"$domain\"}, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) by (code))",
+          "expr": "topk(5, sum(traefik_service_requests_total{protocol=~\"$protocol\", instance=\"$instance\"}) by (code))",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 2,
@@ -489,7 +489,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "exemplar": true,
-          "expr": "sum(label_replace(increase(traefik_service_requests_total{code=\"499\",method=\"GET\", instance=~\".*$domain.*\"}[$interval]) or increase(traefik_service_requests_total{code=\"499\",method=\"GET\", domain=\"$domain\"}[$interval]), \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) by (service)",
+          "expr": "sum(increase(traefik_service_requests_total{code=\"499\",method=\"GET\", instance=\"$instance\"}[$interval])) by (service)",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 2,
@@ -590,7 +590,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "exemplar": true,
-          "expr": "(sum(label_replace(traefik_entrypoint_request_duration_seconds_sum{protocol=~\"$protocol\", instance=~\".*$domain.*\"} or traefik_entrypoint_request_duration_seconds_sum{protocol=~\"$protocol\", domain=\"$domain\"}, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) / sum(label_replace(traefik_entrypoint_requests_total{protocol=~\"$protocol\", instance=~\".*$domain.*\"} or traefik_entrypoint_requests_total{protocol=~\"$protocol\", domain=\"$domain\"}, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) * 1000) - (sum(label_replace(traefik_service_request_duration_seconds_sum{protocol=~\"$protocol\", instance=~\".*$domain.*\"} or traefik_service_request_duration_seconds_sum{protocol=~\"$protocol\", domain=\"$domain\"}, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) / sum(label_replace(traefik_service_requests_total{protocol=~\"$protocol\", instance=~\".*$domain.*\"} or traefik_service_requests_total{protocol=~\"$protocol\", domain=\"$domain\"}, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) * 1000)",
+          "expr": "(sum(traefik_entrypoint_request_duration_seconds_sum{protocol=~\"$protocol\", instance=\"$instance\"}) / sum(traefik_entrypoint_requests_total{protocol=~\"$protocol\", instance=\"$instance\"}) * 1000) - (sum(traefik_service_request_duration_seconds_sum{protocol=~\"$protocol\", instance=\"$instance\"}) / sum(traefik_service_requests_total{protocol=~\"$protocol\", instance=\"$instance\"}) * 1000)",
          "format": "time_series",
          "instant": false,
          "interval": "",
@@ -709,7 +709,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "exemplar": true,
-          "expr": "sum(label_replace(traefik_entrypoint_request_duration_seconds_sum{instance=~\".*$domain.*\"} or traefik_entrypoint_request_duration_seconds_sum{domain=\"$domain\"}, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) / sum(label_replace(traefik_entrypoint_requests_total{instance=~\".*$domain.*\"} or traefik_entrypoint_requests_total{domain=\"$domain\"}, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) * 1000",
+          "expr": "sum(traefik_entrypoint_request_duration_seconds_sum{instance=\"$instance\"}) / sum(traefik_entrypoint_requests_total{instance=\"$instance\"}) * 1000",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 2,
@@ -781,7 +781,7 @@
          },
          "editorMode": "code",
          "exemplar": true,
-          "expr": "sum(label_replace(delta(traefik_service_requests_total{instance=~\".*$domain.*\"}[$interval]) or delta(traefik_service_requests_total{domain=\"$domain\"}[$interval]), \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`))",
+          "expr": "sum(delta(traefik_service_requests_total{instance=\"${instance:raw}\"}[$interval]))",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 2,
@@ -879,7 +879,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "exemplar": true,
-          "expr": "sum(label_replace(rate(traefik_service_requests_total{instance=~\".*$domain.*\"}[$interval]) or rate(traefik_service_requests_total{domain=\"$domain\"}[$interval]), \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`))",
+          "expr": "sum(rate(traefik_service_requests_total{instance=\"$instance\"}[$interval]))",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 2,
@@ -984,7 +984,7 @@
          },
          "editorMode": "code",
          "exemplar": true,
-          "expr": "sum(label_replace(rate(traefik_service_request_duration_seconds_sum{instance=~\".*$domain.*\"}[5m]) or rate(traefik_service_request_duration_seconds_sum{domain=\"$domain\"}[5m]), \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) by(service)",
+          "expr": "sum(rate(traefik_service_request_duration_seconds_sum{ instance=\"$instance\" }[5m])) by(service)",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 2,
@@ -1053,11 +1053,11 @@
            "uid": "PBFA97CFB590B2093"
          },
          "exemplar": true,
-          "expr": "label_replace(process_open_fds{instance=~\".*$domain.*\"} or process_open_fds{domain=\"$domain\"}, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)",
+          "expr": "process_open_fds{job=~\"$job\", instance=\"$instance\"}",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 2,
-          "legendFormat": "{{ domain }}",
+          "legendFormat": "{{ instance }}",
          "refId": "A",
          "step": 240
        }
@@ -1154,7 +1154,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "exemplar": true,
-          "expr": "sum(label_replace(rate(traefik_service_requests_total{protocol=~\"http|https\", instance=~\".*$domain.*\"}[$interval]) or rate(traefik_service_requests_total{protocol=~\"http|https\", domain=\"$domain\"}[$interval]), \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) by (service)",
+          "expr": "sum(rate(traefik_service_requests_total{protocol=~\"http|https\", instance=\"$instance\"}[$interval])) by (service)",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 2,
@@ -1255,7 +1255,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "exemplar": true,
-          "expr": "sum(label_replace(traefik_entrypoint_open_connections{instance=~\".*$domain.*\"} or traefik_entrypoint_open_connections{domain=\"$domain\"}, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) by (method)",
+          "expr": "sum(traefik_entrypoint_open_connections{instance=\"$instance\"}) by (method)",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 1,
@@ -1355,7 +1355,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "exemplar": true,
-          "expr": "sum(label_replace(traefik_service_open_connections{instance=~\".*$domain.*\"} or traefik_service_open_connections{domain=\"$domain\"}, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) by (method)",
+          "expr": "sum(traefik_service_open_connections{instance=\"$instance\"}) by (method)",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 1,
@@ -1459,7 +1459,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "exemplar": true,
-          "expr": "sum(label_replace(increase(traefik_service_requests_total{protocol=~\"$protocol\", instance=~\".*$domain.*\"}[$interval]) or increase(traefik_service_requests_total{protocol=~\"$protocol\", domain=\"$domain\"}[$interval]), \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) by (code)",
+          "expr": "sum(increase(traefik_service_requests_total{protocol=~\"$protocol\", instance=\"$instance\"}[$interval])) by (code)",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 2,
@@ -1514,6 +1514,36 @@
  ],
  "templating": {
    "list": [
+      {
+        "current": {
+          "selected": false,
+          "text": "default",
+          "value": "default"
+        },
+        "datasource": {
+          "type": "prometheus",
+          "uid": "PBFA97CFB590B2093"
+        },
+        "definition": "",
+        "hide": 0,
+        "includeAll": false,
+        "label": "Job:",
+        "multi": false,
+        "name": "job",
+        "options": [],
+        "query": {
+          "query": "label_values(job)",
+          "refId": "Prometheus-job-Variable-Query"
+        },
+        "refresh": 1,
+        "regex": "",
+        "skipUrlSync": false,
+        "sort": 2,
+        "tagValuesQuery": "",
+        "tagsQuery": "",
+        "type": "query",
+        "useTags": false
+      },
      {
        "current": {
          "selected": true,
@@ -1539,7 +1569,7 @@
          "query": "label_values(traefik_service_requests_total, protocol)",
          "refId": "Prometheus-protocol-Variable-Query"
        },
-        "refresh": 2,
+        "refresh": 1,
        "regex": "",
        "skipUrlSync": false,
        "sort": 0,
@@ -1633,19 +1663,19 @@
          "type": "prometheus",
          "uid": "PBFA97CFB590B2093"
        },
-        "definition": "query_result(label_replace(traefik_config_reloads_total, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`))",
+        "definition": "label_values(instance)",
        "hide": 0,
        "includeAll": false,
-        "label": "Domain:",
+        "label": "Instance:",
        "multi": false,
-        "name": "domain",
+        "name": "instance",
        "options": [],
        "query": {
-          "query": "query_result(label_replace(traefik_config_reloads_total, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`))",
+          "query": "label_values(instance)",
          "refId": "PrometheusVariableQueryEditor-VariableQuery"
        },
-        "refresh": 2,
-        "regex": "/domain=\"([^\"]+)\"/",
+        "refresh": 1,
+        "regex": ".*8082",
        "skipUrlSync": false,
        "sort": 1,
        "tagValuesQuery": "",
@@ -1,61 +1,12 @@
-BREAKING CHANGE
-Migration plan for upgrading from 1.6.0+v1.8.1.
+1. OIDC was moved into a seperate compose file. If you have oidc configured you need to add the following line to you .env file:

-## 1. Reinsert secrets with shortened names
+COMPOSE_FILE="$COMPOSE_FILE:compose.grafana-oidc.yml"

-Secret and config names were shortened to max 14 characters to prevent going over Docker's 64 character
-limit when STACK_NAME and VERSION are added to it.
+2. SMTP was moved into a seperate compose file. If you have smtp configured you need to add the following line to you .env file:

- `abra app secret list <domain>` to see which secrets are missing under their new name
- `abra app cmd --local <domain> migrate_secret_names` to reinsert all of them automatically
-      (or manually: `abra app secret insert <domain> <secret_name> v1 <value>` per secret)
+COMPOSE_FILE="$COMPOSE_FILE:compose.grafana-smtp.yml"

-## 2. If you use OIDC (moved to seperate compose file)
+3. The scrape-config.example.yml file and add_node() command were updated to use a secure endpoint for the traefik metrics instead of http. This requires an updated Traefik recipe that publishes the metrics on https.

- Add to your .env: `COMPOSE_FILE="$COMPOSE_FILE:compose.grafana-oidc.yml"`
-
-## 3. If you use SMTP (moved to a seperate compose file)
-
- Add to your .env: `COMPOSE_FILE="$COMPOSE_FILE:compose.grafana-smtp.yml"`
-
-## 4. node_exporter/cadvisor/promtail replaced by Grafana Alloy
-
-Metrics collection changed from Prometheus scraping endpoints
-to Alloy pushing via `remote_write`/`loki push`.
-
- Remove `compose.promtail.yml`, `compose.expose-ports.yml` and
-      `compose.basic-auth.yml` from your .env if present. They no
-      longer exist. `compose.yml` now declares the `basic_auth` secret directly, so
-      `SECRET_BASIC_AUTH_VERSION` is always required.
- Add `PROMETHEUS_REMOTE_WRITE_URL=https://prometheus.$DOMAIN/api/v1/write`
-      (`$DOMAIN` if this host also runs `compose.prometheus.yml`, otherwise a remote
-      Prometheus' URL). Without this, Alloy collects no metrics at all.
- Add `LOKI_PUSH_URL` (existing var, still used) and pick a log source:
-      `JOURNALD=1` (systemd hosts), `SYSLOG_FILES=1` (non-systemd, tails
-      `/var/log/*log`), or `SYSLOG=1` + `compose.syslog.yml` (network syslog listener).
-
- If this host had its own `node.$DOMAIN`/`cadvisor.$DOMAIN` scrape target
-      configured on a central Prometheus, remove it. Those endpoints are gone.
- `scrape-config.example.yml` and the `add_node`/`add_domain` abra.sh commands
-      are gone. Replaced by label-based auto-discovery (see README).
- `docker stack deploy` doesn't prune removed services, so old `cadvisor`/
-      `promtail` containers keep running after a normal `abra app deploy`. Run
-      `abra app undeploy <domain>` then `abra app deploy <domain>` to clear them out.
- Diff your `.env` against the current `.env.sample`, to verify any other changes.
-
-### New: label-based metrics auto-discovery
-
-Alloy now auto-discovers and scrapes other Docker Swarm services on the same
-host/`proxy` network that opt in via `prometheus.io/scrape=true` deploy labels.
-See the README's "Auto-discovering metrics from other apps" section.
-
- If you scrape Traefik metrics: the old `metrics.traefik.$domain` pull-based
-      endpoint still works if you keep the scrape config in Prometheus and
-      existing dashboards keep showing its data, but it's recommended to get
-      Traefik onto the new label-based discovery.
-
-### Dashboards
-
-The Swarm, Stacks and Traefik dashboards were reworked to show old (pull-model)
-and new (Alloy push-model) data as one continuous line, so you don't lose history
-across the migration.
+4. Secret and config names were shortened to max 14 characters to prevent going over Docker's 64 character limit when STACK_NAME and VERSION are added to it.
+When upgrading, you need to reinsert the secrets with their shorter names. Run `abra app secret list <domain>` to see which secrets aren't created on the server (because their name was shortened) and run `abra app secret insert <domain> <secret_name> v1 <value>` to reinsert them with the shorter name. Or you can use the migrate_secret_names function in abra.sh to reinsert all existing secrets with their shorter name automatically: `abra app cmd --local <domain> migrate_secret_names`
@@ -0,0 +1,4 @@
+- targets:
+  - 'metrics.traefik.example.org'
+  - 'node.monitoring.example.org'
+  - 'cadvisor.monitoring.example.org'
Author	SHA1	Message	Date
fauno	1581ae6430	Merge branch 'alloy' into smartctl	2026-07-25 14:00:49 -03:00
fauno	2972272303	fix: scrape once per hour	2026-06-20 13:09:39 -03:00
fauno	4c69cf97ab	feat: systemd timer	2026-06-20 12:54:19 -03:00
fauno	1110786179	feat: smart monitoring	2026-06-19 20:34:28 -03:00