fix backup missing alert & bump Not successful alert to 60m

2025-11-04 19:34:03 +01:00
29 changed files with 908 additions and 1182 deletions
@@ -1,55 +1,19 @@
 TYPE=monitoring-ng
 LETS_ENCRYPT_ENV=production
 COMPOSE_FILE=compose.yml
-DOMAIN=monitoring-ng.example.com
-#TIMEOUT=120
+DOMAIN=monitoring.example.com
+TIMEOUT=120
 ENABLE_BACKUPS=true

-# Secret Alloy authenticates with when writing metrics/logs to Prometheus/
-# Loki and what Traefik's basicauth middleware expects for that
-SECRET_BASIC_AUTH_VERSION=v1
-# Username sent along with SECRET_BASIC_AUTH_VERSION above (default: admin)
-# WRITE_BASIC_AUTH_USERNAME=admin
-
-# Expose Alloy's web UI publicly (behind basic-auth) at alloy.$DOMAIN.
-# COMPOSE_FILE="$COMPOSE_FILE:compose.alloy-webui.yml"
-# Enable Live Debugging in web ui
-# LIVE_DEBUGGING=false
-
-# Enable this to send metrics to a Prometheus server, adapt DOMAIN if
-# server is remote
-# PROMETHEUS_REMOTE_WRITE_URL=https://prometheus.$DOMAIN/api/v1/write
-
-# Enable authenticated scraping of containers that opt in via
-# prometheus.io/auth=basic or prometheus.io/auth=bearer labels (used as
-# password/bearer token respectively). Insert it with:
-# abra app secret insert <domain> scrape_auth v1 <password>
-# COMPOSE_FILE="$COMPOSE_FILE:compose.scrape-auth.yml"
-# SECRET_SCRAPE_AUTH_VERSION=v1
+## Enable this secret for Promtail / Prometheus
+# SECRET_BASIC_AUTH_VERSION=v1
 #
-# Username sent along with it (default: alloy)
-# SCRAPE_BASIC_AUTH_USERNAME=alloy
+# Promtail (Gathering Logs)
+# COMPOSE_FILE="$COMPOSE_FILE:compose.promtail.yml"
+# LOKI_PUSH_URL=https://loki.monitoring.example.org/loki/api/v1/push

-# Edit this if your distribution sets this socket to another location
-# CONTAINERD_SOCKET=/var/run/containerd/containerd.sock
-
-# Enable this to send logs to a Loki server, adapt DOMAIN if server is
-# remote
-# LOKI_PUSH_URL=https://loki.$DOMAIN/loki/api/v1/push
-# Enable on systemd hosts to read logs from the journal
-# JOURNALD=1
-#
-# Enable on non-systemd hosts (Alpine, older Debian/Ubuntu) to tail
-# /var/log/*log files (syslog, auth.log, kern.log, etc.) that a local
-# syslogd writes. No syslogd reconfiguration needed.
-# SYSLOG_FILES=1
-#
-# Enable to receive syslog messages over the network on port 514/tcp.
-# Use for remote devices that push syslog to this host, or for a 
-# local syslogd configured to forward over the network.
-# Not needed if you just want to read local log files — use SYSLOG_FILES instead.
-# SYSLOG=1
-# COMPOSE_FILE="$COMPOSE_FILE:compose.syslog.yml"
+## Expose node and cadvisor ports instead of traefik
+# COMPOSE_FILE="$COMPOSE_FILE:compose.expose-ports.yml"

 # Monitoring Server
 #
@@ -75,20 +39,19 @@ SECRET_BASIC_AUTH_VERSION=v1
 # LOKI_AWS_REGION=eu-west-1
 # LOKI_ACCESS_KEY_ID=bush-debrief-approval-robust-scraggly-molecule
 # LOKI_BUCKET_NAMES=loki
-# SECRET_LOKI_AWS_KEY_VERSION=v1
+# SECRET_LOKI_AWS_SECRET_ACCESS_KEY_VERSION=v1
 #
 ## Grafana
 #
 # COMPOSE_FILE="$COMPOSE_FILE:compose.grafana.yml"
 # GF_SERVER_ROOT_URL=https://monitoring.example.com
-# SECRET_GF_ADMINPASSWD_VERSION=v1
+# SECRET_GRAFANA_ADMIN_PASSWORD_VERSION=v1
 ## Seperate domain for Grafana
 #GRAFANA_DOMAIN=grafana.example.com
 #
 ## Single-Sign-On with OIDC
-# COMPOSE_FILE="$COMPOSE_FILE:compose.grafana-oidc.yml"
 # OIDC_ENABLED=1
-# SECRET_GF_OIDC_SECRET_VERSION=v1
+# SECRET_GRAFANA_OIDC_CLIENT_SECRET_VERSION=v1
 # OIDC_CLIENT_ID=grafana
 # OIDC_AUTH_URL="https://authentik.example.com/application/o/authorize/"
 # OIDC_API_URL="https://authentik.example.com/application/o/userinfo/"
@@ -99,26 +62,24 @@ SECRET_BASIC_AUTH_VERSION=v1
 # GF_INSTALL_PLUGINS=grafana-piechart-panel
 #
 ## grafana SMTP configuration (optional)
-# COMPOSE_FILE="$COMPOSE_FILE:compose.grafana-smtp.yml"
 # GF_SMTP_HOST=changeme
 # GF_SMTP_USER=changme
 # GF_SMTP_ENABLED=true
 # GF_SMTP_FROM_ADDRESS=grafana@example.com
 # GF_SMTP_SKIP_VERIFY=false
-# SECRET_GF_SMTP_PASS_VERSION=v1
+# SECRET_GRAFANA_SMTP_PASSWORD_VERSION=v1
 #

 ## Grafana Matrix Contact Point (optional)
 #COMPOSE_FILE="$COMPOSE_FILE:compose.matrix-alertmanager-receiver.yml"
-#SECRET_MATRIX_TOKEN_VERSION=v1
+#SECRET_MATRIX_ACCESS_TOKEN_VERSION=v1
 #GF_MATRIX_USER_ID="<user-id>"
 #GF_MATRIX_ROOM_ID="<room-id>"
 #GF_MATRIX_HOMESERVER_URL="<homeserver-url>"

-## ALerts
-
-# Node disk space alert will trigger when free disk space left is below the given number in percent
-#ALERT_NODE_DISK_SPACE_LEFT=10
-
-# Node memory usage alert will trigger when memory usage is above the given number in percent
-#ALERT_NODE_MEMORY_USAGE=85
+# ALerts
+#ALERT_BACKUP_FAILED_ENABLED=true
+#ALERT_BACKUP_MISSING_ENABLED=true
+#ALERT_BACKUP_NOT_SUCCESSFULL_ENABLED=true
+#ALERT_NODE_DISK_SPACE_ENABLED=true
+#ALERT_NODE_MEMORY_USAGE_ENABLED=true
@@ -1,10 +1,9 @@
 # monitoring-ng

 Yet another monitoring stack ...
-This time its a all-in-one grafana/prometheus/loki/alloy stack.
-It's based heavily on the [monitoring-lite](https://git.coopcloud.tech/coop-cloud/monitoring-lite) stack, but has everything in one recipe included now. So you can deploy monitoring instances to only gather metrics / logs (alloy) and also deploy instances with the full monitoring stack (grafana/prometheus/loki) with the same recipe and just different .env configuration.
+This time its a all-in-one grafana/prometheus/loki/node_exporter/cadvisor/promtail stack.
+It's based heavily on the [monitoring-lite](https://git.coopcloud.tech/coop-cloud/monitoring-lite) stack, but has everything in one recipe included now. So you can deploy monitoring instances to only gather metrics / logs (node_exporter/cadvisor/promtail) and also deploy instances with the full monitoring stack (grafana/prometheus/loki) with the same recipe and just different .env configuration.

-Metrics and logs are collected by [Grafana Alloy](https://grafana.com/docs/alloy/latest/) and pushed to a central Prometheus/Loki (via `remote_write`/`loki push`). Every `monitoring-ng` instance runs its own Alloy, whether or not it also runs the central stack.

 <!-- metadata -->

@@ -19,93 +18,79 @@ Metrics and logs are collected by [Grafana Alloy](https://grafana.com/docs/alloy

 <!-- endmetadata -->

-## Setup: gathering-only host
+## Setup a Metrics Gathering

-Deploys just Alloy, pushing this host's own node/container metrics and logs to a central instance. Use this on every host you want metrics/logs from.
+Where gathering.org is the node you want to gather metrics from.

-1. `abra app new monitoring-ng --server gathering.org`
-2. `abra app config monitoring.gathering.org`
-3. Point it at your central instance:
-   ```
-   PROMETHEUS_REMOTE_WRITE_URL=https://prometheus.example.org/api/v1/write
-   LOKI_PUSH_URL=https://loki.example.org/loki/api/v1/push
-   JOURNALD=1   # or SYSLOG_FILES=1 / SYSLOG=1, see .env.sample
-   ```
-4. `abra app secret insert monitoring.gathering.org basic_auth v1 <password>`. Same username/password as the `usersfile` credential configured for Traefik's basicauth on the central node (see below). This is what Alloy authenticates with when pushing metrics/logs. Alloy defaults to username `admin` for this. Uf the Traefik `usersfile` uses a different username, set `WRITE_BASIC_AUTH_USERNAME` in this recipe to match.
-5. `abra app deploy monitoring.gathering.org`
+1. Configure DNS
+  - monitoring.gathering.org
+  - cadvisor.monitoring.gathering.org
+  - node.monitoring.gathering.org
+1. Configure Traefik to use BasicAuth
+  * `abra app config traefik.gathering.org`
+  uncomment
+  ```
+  # BASIC_AUTH
+  COMPOSE_FILE="$COMPOSE_FILE:compose.basicauth.yml"
+  BASIC_AUTH=1
+  SECRET_USERSFILE_VERSION=v1
+  ```
+  - Generate userslist with httpasswd hashed password
+    `abra app secret insert traefik.gathering.org usersfile v1 'admin:<hashed-secret>'`
+    make sure there is no whitespace in between `admin:<hashed-secret>`, it seems to break stuff...
+  - `abra app deploy -f traefik`
+1. `abra app new monitoring-ng`
+1. `abra app config monitoring.gathering.org`
+   for gathering only the main `compose.yml` is needed, nothing more.
+1. `abra app deploy monitoring.gathering.org`
+1. check that endpoints are up and basic-auth works
+  - cadvisor.monitoring.gathering.org
+  - node.monitoring.gathering.org

-## Setup: full monitoring stack (metrics/logs browser)
+### Expose node and cadvisor via ports instead of traefik

-This is what a gathering host pushes into. It also runs its own Alloy, so it monitors itself too.
-
-1. Configure DNS: `monitoring.example.org`, plus `prometheus.`/`loki.`/`pushgateway.` subdomains for whichever of those you enable below
-2. Traefik on this node needs basic auth configured (`BASIC_AUTH=1`, see the Traefik recipe's "Configuring basic auth" section) — Prometheus/Loki/Pushgateway route through its `basicauth@file` middleware, so without it those endpoints won't work. Use the same username (default `admin`, see `WRITE_BASIC_AUTH_USERNAME`) and password you'll insert as the `basic_auth` secret below when generating Traefik's `usersfile`.
-3. `abra app config monitoring.example.org` Uncomment `compose.prometheus.yml` (metrics), `compose.loki.yml` (logs) and `compose.grafana.yml` (dashboard)
-4. `abra app secret insert monitoring.example.org basic_auth v1 <password>` — this is the password every gathering host's Alloy (including this instance's own) authenticates with; also what Traefik's basicauth expects on the public Prometheus/Loki/Pushgateway endpoints
-5. `abra app secret insert monitoring.example.org gf_adminpasswd v1 <password>`
-6. `abra app deploy monitoring.example.org`
-
-### Post-setup guide
-
- configure the SMTP mailer under `Alerting > Contact points`
-  - edit the default contact point, choose "Alertmanager" as type & `http://alertmanager:9093` as URL
-  - use the "Test" button to send a test mail. It should fire a request at the alertmanager & that should send a mail
- from your dashboard panels, choose `Edit > Alert` to create alerts based on those panels
-
-## Additional features
-
-### Discovering metrics from other apps
-
-Alloy auto-discovers and scrapes other Docker Swarm services running on the same host, on the `proxy` network, that opt in via labels. No manual scrape config needed. On the app's `compose.yml`:
-
-```yaml
-deploy:
-  labels:
-    - "prometheus.io/scrape=true"       # required: opt in
-    # - "prometheus.io/port=8082"       # optional: defaults to 80
-    # - "prometheus.io/path=/metrics"   # optional: defaults to /metrics
-    # - "prometheus.io/auth=basic"      # optional: basic auth, see below
-    # - "prometheus.io/auth=bearer"     # optional: bearer token auth
+In case you have no traefik running on the machine, you can expose the ports directly by uncommenting the following line:
+```
+# COMPOSE_FILE="$COMPOSE_FILE:compose.expose-ports.yml"
 ```

-Each scraped target gets `instance` (`<service_name>.<slot>`, stable across redeploys), `domain` (the service's stack namespace with underscores converted back to dots, e.g. `traefik.example.com`) and `task_slot` labels attached automatically.
+## Setup Metrics Browser

-If the target needs authentication, configure the monitoring-ng instance with a scrape-auth secret that will be used for targets having the auth label set:
-```
-COMPOSE_FILE="$COMPOSE_FILE:compose.scrape-auth.yml"
-SECRET_SCRAPE_AUTH_VERSION=v1
-```
-`abra app secret insert <domain> scrape_auth v1 <password-or-token>`, then set the scraped app's `prometheus.io/auth` label to `basic` or `bearer` to match how it checks the secret.

-Check discovered targets via `alloy.example.org` (needs `compose.alloy-webui.yml`, see below), or query the central Prometheus for `up{job="<service_name>"}`.
+1. Configure DNS
+   - monitoring.example.org
+   - prometheus.monitoring.example.org
+   - loki.monitoring.example.org
+2. Setup monitoring stack
+   - `abra app new monitoring-ng`
+   - `abra app config monitoring.example.org`
+      Uncomment all the stuff
+   - `abra app secret insert monitoring.example.org basic_auth v1 <secret>`
+      this needs the plaintext traefik basic-auth secret, not the hashed one!
+   - `abra app secret ls monitoring.example.org`
+   - `abra app deploy monitoring.example.org`
+3. Add scrape config to prometheus
+  - `abra app cmd monitoring.example.org prometheus gathering.org`
+  - or manually
+    ```
+    cp scrape-config.example.yml gathering.org.yml
+    # adjust domain
+    # mkdir scrape_configs
+    abra app cp monitoring.dev.local-it.cloud gathering.org.yml prometheus:/prometheus/scrape_configs/
+    ```

-### Manual scraping
+* check that all configured targets are up:
+  https://prometheus.monitoring.example.org/targets

-For targets where auto-discovery doesn't work (e.g. not a Docker Swarm service on the `proxy` network, or missing labels): add them directly to Prometheus instead. Create a scrape config file:
-```yaml
- targets:
-    - 'metrics.something-external.example.org'
-    - 'app-without-labels.example.org'
-```
-and copy it into Prometheus' scrape config directory:
-```
-abra app cp monitoring.gathering.org targets.yml prometheus:/prometheus/scrape_configs/
-```
-Prometheus picks up files there automatically.

-### Alloy Web UI / Live Debugging

-Alloy's own web UI isn't exposed by default, its HTTP server only listens
-on localhost inside its own container. To reach it from outside (e.g. to
-browse its component graph or use live debugging), add:
-```
-COMPOSE_FILE="$COMPOSE_FILE:compose.alloy-webui.yml"
-```
-This exposes it (behind the same basic-auth) at `alloy.$DOMAIN`.
-
-To actually see what's being collected (live-tailing the metrics/logs
-flowing through each component, not just their config) also set
-`LIVE_DEBUGGING=true`. Only enable this while troubleshooting.
+| Service       | Authentication     | Domain                            |
+| ------------- | ------------------ | --------------------------------- |
+| Grafana       | Email / SSO        | monitoring.example.org            |
+| Prometheus    | traefik basic-auth | prometheus.monitoring.example.org |
+| loki          | traefik basic-auth | loki.monitoring.example.org       |
+| Cadvisor      | traefik basic-auth | cadvisor.monitoring.example.org   |
+| Node Exporter | traefik basic-auth | node.monitoring.example.org       |

 ### Logging from a docker host to loki server without anything else

@@ -122,25 +107,36 @@ $ echo '{
 $ systemctl restart docker.service
 ```

-### Setup Push Gateway
+## Setup Push Gateway

-1. Enable in the env file by uncommenting the following lines:
+1. Enable in the env fiöle by uncommenting the following lines:
 ```
 ## Prometheus Pushgateway
 # COMPOSE_FILE="$COMPOSE_FILE:compose.pushgateway.yml"
 ```
 2. `abra app deploy monitoring.example.org`

-This will expose the pushgateway at `https://pushgateway.${DOMAIN}`, secured behind the same basic auth as the other services.
-After that you need to add the `pushgateway.${DOMAIN}` to the scrape config of Prometheus.
+This will expose the pushgateway at `https://pushgateway.${DOMAIN}`.
+It is secured behind the same basic auth as the other services.
+After that you need to add the `pushgateway.${DOMAIN}` to the scare config.

+## Post-setup guide
+
+- configure prometheus/loki/alertmanager as data sources in grafana under `Configuration > Data sources`
+  - for loki, you need to set a "Custom HTTP Header": `X-Scope-OrgID: fake`
+- configure the SMTP mailer under `Alerting > Contact points`
+  - edit the default contact point, choose "Alertmanager" as type & `http://alertmanager:9093` as URL
+  - use the "Test" button to send a test mail. It should fire a request at the alertmanager & that should send a mail
+- `abra app cp` your `scrap_configs: ...` into `/prometheus/scrape_configs` & log into your prometheus web UI to ensure they're working
+- load your dashboards in manually under `Create > Dashboard`
+- from your dashboard panels, choose `Edit > Alert` to create alerts based on those panels


 ---

 THX to the previous work of @decentral1se @knooflok @3wc @cellarspoon @mirsal

-### Adding Matrix as Alert Contact point
+## Adding Matrix as Alert Contact point

 1. Enable the [matrix-alertmanager-receiver](https://github.com/metio/matrix-alertmanager-receiver/):
 ```
@@ -149,7 +145,7 @@ COMPOSE_FILE="$COMPOSE_FILE:compose.matrix-alertmanager-receiver.yml"

 2. Insert the matrix access token secret:
 ```
-abra app secret insert monitoring.marx.klasse-methode.it matrix_token v1
+abra app secret insert monitoring.marx.klasse-methode.it matrix_access_token v1
 ```

 3. Set required configurations:
@@ -160,9 +156,13 @@ GF_MATRIX_HOME_SERVER_URL=
 ```
 4. Configure Alertmanager webhook and set the url to `http://matrix-alertmanager-receiver:12345/alerts/<room-id>`

-### Alerts
+## alerts
+
+It is possible to enable the following alerts, by setting the corresponding env variable to `true`:
+- backupbot failed: `ALERT_BACKUP_FAILED_ENABLED`
+- backupbot missing: `ALERT_BACKUP_MISSING_ENABLED`
+- backupbot not successfull: `ALERT_BACKUP_NOT_SUCCESSFULL_ENABLED`
+- node disk space: `ALERT_NODE_DISK_SPACE_ENABLED`
+- node memory usage: `ALERT_NODE_MEMORY_USAGE_ENABLED`

-It is possible to enable the following alerts, by uncommenting the corresponding env variable:

- node disk space: `ALERT_NODE_DISK_SPACE_LEFT`
- node memory usage: `ALERT_NODE_MEMORY_USAGE`
@@ -1,108 +1,38 @@
-export GF_DATASOURCES_VERSION=v1
-export GF_DASHBOARDS_VERSION=v2
-export GF_SWARM_DASH_VERSION=v2
-export GF_STACKS_DASH_VERSION=v2
-export GF_TRAEFIK_DASH_VERSION=v3
-export GF_BACKUP_DASH_VERSION=v1
-export GF_CUSTOM_INI_VERSION=v4
-export LOKI_YML_VERSION=v3
+export ENTRYPOINT_VERSION=v1
+export GRAFANA_DATASOURCES_YML_VERSION=v1
+export GRAFANA_DASHBOARDS_YML_VERSION=v2
+export GRAFANA_SWARM_DASHBOARD_JSON_VERSION=v2
+export GRAFANA_STACKS_DASHBOARD_JSON_VERSION=v2
+export GRAFANA_TRAEFIK_DASHBOARD_JSON_VERSION=v2
+export GRAFANA_BACKUP_DASHBOARD_JSON_VERSION=v1
+export GRAFANA_ALERTS_JSON_VERSION=v3
+export GRAFANA_CUSTOM_INI_VERSION=v4
+export PROMTAIL_YML_VERSION=v3
+export LOKI_YML_VERSION=v2
 export PROMETHEUS_YML_VERSION=v2
-export MATRIX_ALERTMANAGER_CONFIG_VERSION=v1
-export MATRIX_ALERTMANAGER_ENTRYPOINT_VERSION=v1
-export GF_ALERTS_NODE_VERSION=v2
-export CONFIG_ALLOY_VERSION=v11
+export MATRIX_ALERTMANAGER_CONFIG_VERSION=e
+export MATRIX_ALERTMANAGER_ENTRYPOINT_VERSION=a

-# migrates secrets from old names to new names by reading values from the
-# running containers on the server and re-inserting them under the new names.
-# preview changes: abra app cmd --local <app> migrate_secret_names
-# execute changes: abra app cmd --local <app> migrate_secret_names execute
-migrate_secret_names() {
-  if ! command -v jq &> /dev/null; then
-    echo "jq is required on your local machine to migrate secret names"
-    echo "It could not be found in your PATH, please install jq to proceed."
-    echo "For example: On a debian/ubuntu system, run `apt install jq`"
-    exit 1
+# creates a default prometheus scrape config for a given node
+add_node(){
+  name=$1
+  add_domain "$name" "$name:8082"
+  add_domain "$name" "node.monitoring.$name"
+  add_domain "$name" "cadvisor.monitoring.$name"
+  cat "/prometheus/scrape_configs/$name.yml"
+}
+
+# adds a domain to a scrape config or creates a new one
+add_domain(){
+  name=$1
+  domain=$2
+  if [ ! -d "/prometheus/scrape_configs/" ]; then
+    mkdir -p /prometheus/scrape_configs/
  fi
-
-  # Hardcoded migration mappings: old_secret_name|new_secret_name
-  MIGRATIONS="
-grafana_admin_password|gf_adminpasswd
-grafana_smtp_password|gf_smtp_pass
-grafana_oidc_client_secret|gf_oidc_secret
-matrix_access_token|matrix_token
-loki_aws_secret_access_key|loki_aws_key
-"
-
-  # Determine which server the app is deployed on
-  SERVER=$(abra app ls -m | jq -r --arg domain "$APP_NAME" '[.[].apps[] | select(.domain == $domain) | .server] | first' 2>/dev/null)
-
-  if [ -z "$SERVER" ]; then
-    echo "Error: could not determine server for app '$APP_NAME'"
-    exit 1
-  fi
-
-  # Build a lookup table of all secrets currently mounted in this stack.
-  # Each line: <secretID> <containerID> <secretName>
-  LOOKUP=$(ssh "$SERVER" "
-    docker stack services ${STACK_NAME} --format '{{.Name}}' | while read svc; do
-      CID=\$(docker ps --no-trunc -q --filter \"name=\${svc}\" | head -1)
-      docker service inspect \"\$svc\" --format '{{json .Spec.TaskTemplate.ContainerSpec.Secrets}}' | \
-        jq -r --arg cid \"\$CID\" '.[]? | .SecretID + \" \" + \$cid + \" \" + .SecretName'
-    done | sort -k3 -r
-  " 2>/dev/null)
-
-  echo "Secret migration plan for: $APP_NAME (server: $SERVER)"
-  echo ""
-  printf "  %-24s %-8s %s\n" "OLD NAME" "FOUND" "ACTION"
-  printf "  %-24s %-8s %s\n" "--------" "-----" "------"
-
-  # Check each old name against the lookup table and display the plan
-  ANY_FOUND=false
-  while IFS='|' read -r OLD_NAME NEW_NAME; do
-    [ -z "$OLD_NAME" ] && continue
-    MATCH=$(echo "$LOOKUP" | grep " ${STACK_NAME}_${OLD_NAME}_" | head -1)
-    if [ -n "$MATCH" ]; then
-      printf "  %-24s %-8s %s\n" "$OLD_NAME" "yes" "recreate as '$NEW_NAME' version V1"
-      ANY_FOUND=true
-    else
-      printf "  %-24s %-8s %s\n" "$OLD_NAME" "no" "nothing (not found on server)"
-    fi
-  done <<< "$MIGRATIONS"
-
-  echo ""
-
-  if [ "$ANY_FOUND" = false ]; then
-    echo "No old secrets found on server. Nothing to migrate."
-    return 0
+  cd /prometheus/scrape_configs/ || exit 1
+  if [ ! -f "$name.yml" ]; then
+    echo -e "- targets:\n  - '$domain'" > "$name.yml"
+  else
+    echo "  - '$domain'"  >> "$name.yml"
  fi
-
-  if [ "$1" != "execute" ]; then
-    echo "To apply the above changes, run:"
-    echo "  abra app cmd --local $APP_NAME migrate_secret_names execute"
-    return 0
-  fi
-
-  # read each found secret from its container and re-insert with the new name
-  while IFS='|' read -r OLD_NAME NEW_NAME; do
-    [ -z "$OLD_NAME" ] && continue
-
-    MATCH=$(echo "$LOOKUP" | grep " ${STACK_NAME}_${OLD_NAME}_" | head -1)
-    [ -z "$MATCH" ] && continue
-
-    SECRET_ID=$(echo "$MATCH" | awk '{print $1}')
-    CID=$(echo "$MATCH" | awk '{print $2}')
-    SECRET_VALUE=$(ssh "$SERVER" "cat /var/lib/docker/containers/${CID}/mounts/secrets/${SECRET_ID} 2>/dev/null || sudo cat /var/lib/docker/containers/${CID}/mounts/secrets/${SECRET_ID} 2>/dev/null")
-
-    if [ -z "$SECRET_VALUE" ]; then
-      echo "Error: could not read value for '$OLD_NAME', skipping"
-      continue
-    fi
-
-    echo "Migrating: '$OLD_NAME' -> '$NEW_NAME' (v1)"
-    printf '%s' "$SECRET_VALUE" | abra app secret insert -C "$APP_NAME" "$NEW_NAME" v1
-    
-  done <<< "$MIGRATIONS"
-
-  echo ""
-  echo "Done."
 }
@@ -12,7 +12,7 @@ http:
 matrix:
  homeserver-url: "{{ env "GF_MATRIX_HOMESERVER_URL" }}"
  user-id: "{{ env "GF_MATRIX_USER_ID" }}"
-  access-token: "{{ secret "matrix_token" }}"
+  access-token: "{{ secret "matrix_access_token" }}"
  room-mapping:
    matrixroom: "{{ env "GF_MATRIX_ROOM_ID" }}"

@@ -1,131 +0,0 @@
-apiVersion: 1
-
-# List of alert rule UIDs that should be deleted
-deleteRules:
-  {{ if not (env "ALERT_NODE_DISK_SPACE_LEFT") }}
-  - orgId: 1
-    uid: coopcloud_node_disk_space_left
-  {{ end }}
-  {{ if not (env "ALERT_NODE_MEMORY_USAGE") }}
-  - orgId: 1
-    uid: coopcloud_node_memory_usage
-  {{ end }}
-
-groups:
- orgId: 1
-  name: node
-  folder: node
-  interval: 5m
-  rules:
-    {{ if (env "ALERT_NODE_DISK_SPACE_LEFT") }}
-    - uid: coopcloud_node_disk_space_left
-      title: Node Disk Space
-      condition: C
-      data:
-        - refId: A
-          relativeTimeRange:
-            from: 600
-            to: 0
-          datasourceUid: PBFA97CFB590B2093
-          model:
-            editorMode: code
-            expr: (node_filesystem_free_bytes{fstype=~"ext4|xfs"} / node_filesystem_size_bytes{fstype=~"ext4|xfs"}) * 100
-            instant: true
-            intervalMs: 1000
-            legendFormat: __auto
-            maxDataPoints: 43200
-            range: false
-            refId: A
-        - refId: C
-          relativeTimeRange:
-            from: 600
-            to: 0
-          datasourceUid: __expr__
-          model:
-            conditions:
-                - evaluator:
-                    params:
-                        - {{ env "ALERT_NODE_DISK_SPACE_LEFT" }}
-                    type: lt
-                  operator:
-                    type: and
-                  query:
-                    params:
-                        - C
-                  reducer:
-                    params: []
-                    type: last
-                  type: query
-            datasource:
-                type: __expr__
-                uid: __expr__
-            expression: A
-            intervalMs: 1000
-            maxDataPoints: 43200
-            refId: C
-            type: threshold
-      noDataState: NoData
-      execErrState: Error
-      for: 5m
-      annotations:
-        description: ""
-        runbook_url: ""
-        summary: Less than {{ env "ALERT_NODE_DISK_SPACE_LEFT" }}% disk space left on {{`{{ $labels.instance }}`}} ({{`{{ (index $values "A").Value }}`}}% left)
-      labels:
-        "": ""
-      isPaused: false
-    {{ end }}
-    {{ if (env "ALERT_NODE_MEMORY_USAGE")  }}
-    - uid: coopcloud_node_memory_usage
-      title: Node Memory Usage
-      condition: C
-      data:
-        - refId: A
-          relativeTimeRange:
-            from: 600
-            to: 0
-          datasourceUid: PBFA97CFB590B2093
-          model:
-            editorMode: code
-            expr: (node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100
-            instant: true
-            intervalMs: 1000
-            legendFormat: __auto
-            maxDataPoints: 43200
-            range: false
-            refId: A
-        - refId: C
-          relativeTimeRange:
-            from: 600
-            to: 0
-          datasourceUid: __expr__
-          model:
-            conditions:
-                - evaluator:
-                    params:
-                        - {{ env "ALERT_NODE_MEMORY_USAGE"  }}
-                    type: gt
-                  operator:
-                    type: and
-                  query:
-                    params:
-                        - C
-                  reducer:
-                    params: []
-                    type: last
-                  type: query
-            datasource:
-                type: __expr__
-                uid: __expr__
-            expression: A
-            intervalMs: 1000
-            maxDataPoints: 43200
-            refId: C
-            type: threshold
-      noDataState: NoData
-      execErrState: Error
-      for: 5m
-      annotations:
-        summary: Memory usage is above {{ env "ALERT_NODE_MEMORY_USAGE"  }}% on {{`{{ $labels.instance }}`}}  ({{`{{ printf "%.2f" (index $values "A").Value }}`}}%  usage)
-      isPaused: false
-    {{ end }}
@@ -1,16 +0,0 @@
-version: "3.8"
-
-services:
-  app:
-    environment:
-      - ALLOY_HTTP_LISTEN_ADDR=0.0.0.0
-    deploy:
-      labels:
-        - "traefik.enable=true"
-        - "traefik.swarm.network=proxy"
-        - "traefik.http.services.${STACK_NAME}-alloy.loadbalancer.server.port=12345"
-        - "traefik.http.routers.${STACK_NAME}-alloy.rule=Host(`alloy.${DOMAIN}`)"
-        - "traefik.http.routers.${STACK_NAME}-alloy.entrypoints=web-secure"
-        - "traefik.http.routers.${STACK_NAME}-alloy.tls=true"
-        - "traefik.http.routers.${STACK_NAME}-alloy.tls.certresolver=${LETS_ENCRYPT_ENV}"
-        - "traefik.http.routers.${STACK_NAME}-alloy.middlewares=basicauth@file"
@@ -0,0 +1,13 @@
+---
+version: "3.8"
+
+services:
+  app:
+    ports:
+      - "9100:9100"
+    deploy:
+
+  cadvisor:
+    ports:
+      - "9101:8080"
+    deploy:
@@ -1,17 +0,0 @@
-version: '3.8'
-
-services:
-  grafana:
-    secrets:
-      - gf_oidc_secret
-    environment:
-      - OIDC_API_URL
-      - OIDC_AUTH_URL
-      - OIDC_CLIENT_ID
-      - OIDC_ENABLED
-      - OIDC_TOKEN_URL
-
-secrets:
-  gf_oidc_secret:
-    external: true
-    name: ${STACK_NAME}_gf_oidc_secret_${SECRET_GF_OIDC_SECRET_VERSION}
@@ -1,18 +0,0 @@
-version: '3.8'
-
-services:
-  grafana:
-    secrets:
-      - gf_smtp_pass
-    environment:
-      - GF_SMTP_HOST
-      - GF_SMTP_USER
-      - GF_SMTP_PASSWORD__FILE=/run/secrets/gf_smtp_pass
-      - GF_SMTP_ENABLED
-      - GF_SMTP_FROM_ADDRESS
-      - GF_SMTP_SKIP_VERIFY
-
-secrets:
-  gf_smtp_pass:
-    external: true
-    name: ${STACK_NAME}_gf_smtp_pass_${SECRET_GF_SMTP_PASS_VERSION}
@@ -2,87 +2,104 @@ version: '3.8'

 services:
  grafana:
-    image: grafana/grafana:13.0.6
+    image: grafana/grafana:10.4.14
    volumes:
      - grafana-data:/var/lib/grafana:rw
    secrets:
-      - gf_adminpasswd
+      - grafana_admin_password
+      - grafana_oidc_client_secret
+      - grafana_smtp_password
    configs:
-      - source: gf_custom_ini
+      - source: grafana_custom_ini
        target: /etc/grafana/grafana.ini
-      - source: gf_datasources
+      - source: grafana_datasources_yml
        target: /etc/grafana/provisioning/datasources/datasources.yml
-      - source: gf_dashboards
+      - source: grafana_dashboards_yml
        target: /etc/grafana/provisioning/dashboards/dashboards.yml
-      - source: gf_swarm_dash
+      - source: grafana_swarm_dashboard_json
        target: /var/lib/grafana/dashboards/docker-swarm-nodes.json
-      - source: gf_stacks_dash
+      - source: grafana_stacks_dashboard_json
        target: /var/lib/grafana/dashboards/docker-swarm-stacks.json
-      - source: gf_traefik_dash
+      - source: grafana_traefik_dashboard_json
        target: /var/lib/grafana/dashboards/traefik.json
-      - source: gf_backup_dash
+      - source: grafana_backup_dashboard_json
        target: /var/lib/grafana/dashboards/backup.json
-      - source: gf_alerts_node
-        target: /etc/grafana/provisioning/alerting/node.yml
+      - source: grafana_alerts_json
+        target: /var/lib/grafana/alerts/alerts.json
    networks:
      - proxy
      - internal
    environment:
      - GF_SERVER_ROOT_URL
-      - GF_SECURITY_ADMIN_PASSWORD__FILE=/run/secrets/gf_adminpasswd
+      - GF_SECURITY_ADMIN_PASSWORD__FILE=/run/secrets/grafana_admin_password
+      - GF_SMTP_HOST
+      - GF_SMTP_USER
+      - GF_SMTP_PASSWORD__FILE=/run/secrets/grafana_smtp_password
+      - GF_SMTP_ENABLED
+      - GF_SMTP_FROM_ADDRESS
+      - GF_SMTP_SKIP_VERIFY
      - GF_SECURITY_ALLOW_EMBEDDING
      - GF_INSTALL_PLUGINS
-      - ALERT_NODE_DISK_SPACE_LEFT
-      - ALERT_NODE_MEMORY_USAGE
+      - OIDC_API_URL
+      - OIDC_AUTH_URL
+      - OIDC_CLIENT_ID
+      - OIDC_ENABLED
+      - OIDC_TOKEN_URL
    deploy:
      labels:
        - "traefik.enable=true"
-        - "traefik.swarm.network=proxy"
+        - "traefik.docker.network=proxy"
        - "traefik.http.services.${STACK_NAME}-grafana.loadbalancer.server.port=3000"
        - "traefik.http.routers.${STACK_NAME}-grafana.rule=Host(`${GRAFANA_DOMAIN:-$DOMAIN}`)"
        - "traefik.http.routers.${STACK_NAME}-grafana.entrypoints=web-secure"
        - "traefik.http.routers.${STACK_NAME}-grafana.tls=true"
        - "traefik.http.routers.${STACK_NAME}-grafana.tls.certresolver=${LETS_ENCRYPT_ENV}"
    healthcheck:
-      test: "wget -q http://localhost:3000/healthz -O/dev/null"
+      test: "wget -q http://localhost:3000/ -O/dev/null"
      interval: 5s
      timeout: 10s
      retries: 3
      start_period: 10s

 configs:
-  gf_custom_ini:
+  grafana_custom_ini:
    template_driver: golang
-    name: ${STACK_NAME}_gf_custom_ini_${GF_CUSTOM_INI_VERSION}
+    name: ${STACK_NAME}_grafana_custom_ini_${GRAFANA_CUSTOM_INI_VERSION}
    file: grafana_custom.ini
-  gf_datasources:
-    name: ${STACK_NAME}_gf_datasources_${GF_DATASOURCES_VERSION}
+  grafana_datasources_yml:
+    name: ${STACK_NAME}_g_datasources_yml_${GRAFANA_DATASOURCES_YML_VERSION}
    file: grafana-datasources.yml
-  gf_dashboards:
-    name: ${STACK_NAME}_gf_dashboards_${GF_DASHBOARDS_VERSION}
+  grafana_dashboards_yml:
+    name: ${STACK_NAME}_g_dashboards_yml_${GRAFANA_DASHBOARDS_YML_VERSION}
    file: grafana-dashboards.yml
-  gf_swarm_dash:
-    name: ${STACK_NAME}_gf_swarm_dash_${GF_SWARM_DASH_VERSION}
+  grafana_swarm_dashboard_json:
+    name: ${STACK_NAME}_g_swarm_dashboard_json_${GRAFANA_SWARM_DASHBOARD_JSON_VERSION}
    file: grafana-swarm-dashboard.json
-  gf_stacks_dash:
-    name: ${STACK_NAME}_gf_stacks_dash_${GF_STACKS_DASH_VERSION}
+  grafana_stacks_dashboard_json:
+    name: ${STACK_NAME}_g_stacks_dashboard_json_${GRAFANA_STACKS_DASHBOARD_JSON_VERSION}
    file: grafana-stacks-dashboard.json
-  gf_traefik_dash:
-    name: ${STACK_NAME}_gf_traefik_dash_${GF_TRAEFIK_DASH_VERSION}
+  grafana_traefik_dashboard_json:
+    name: ${STACK_NAME}_g_traefik_dashboard_json_${GRAFANA_TRAEFIK_DASHBOARD_JSON_VERSION}
    file: grafana-traefik-dashboard.json
-  gf_backup_dash:
-    name: ${STACK_NAME}_gf_backup_dash_${GF_BACKUP_DASH_VERSION}
+  grafana_backup_dashboard_json:
+    name: ${STACK_NAME}_g_backup_dashboard_json_${GRAFANA_BACKUP_DASHBOARD_JSON_VERSION}
    file: grafana-backup-dashboard.json
-  gf_alerts_node:
+  grafana_alerts_json:
    template_driver: golang
-    name: ${STACK_NAME}_gf_alerts_node_${GF_ALERTS_NODE_VERSION}
-    file: alerts/node.yml.tmpl
+    name: ${STACK_NAME}_g_alerts_json_${GRAFANA_ALERTS_JSON_VERSION}
+    file: grafana-alerts.json.tmpl

 volumes:
  grafana-data:


 secrets:
-  gf_adminpasswd:
+  grafana_admin_password:
    external: true
-    name: ${STACK_NAME}_gf_adminpasswd_${SECRET_GF_ADMINPASSWD_VERSION}
+    name: ${STACK_NAME}_grafana_admin_password_${SECRET_GRAFANA_ADMIN_PASSWORD_VERSION}
+  grafana_oidc_client_secret:
+    external: true
+    name: ${STACK_NAME}_grafana_oidc_client_secret_${SECRET_GRAFANA_OIDC_CLIENT_SECRET_VERSION}
+  grafana_smtp_password:
+    external: true
+    name: ${STACK_NAME}_grafana_smtp_password_${SECRET_GRAFANA_SMTP_PASSWORD_VERSION}
@@ -2,7 +2,7 @@ version: '3.8'

 services:
  loki:
-    image: grafana/loki:3.7.2
+    image: grafana/loki:2.9.11
    command: -config.file=/etc/loki/local-config.yaml
    networks:
      - proxy
@@ -12,7 +12,7 @@ services:
    volumes:
      - loki-data:/loki
    # secrets:
-    #   - loki_aws_key
+    #   - loki_aws_secret_access_key
    environment:
      - LOKI_ACCESS_KEY_ID
      - LOKI_AWS_ENDPOINT
@@ -27,7 +27,7 @@ services:
        condition: on-failure
      labels:
        - "traefik.enable=true"
-        - "traefik.swarm.network=proxy"
+        - "traefik.docker.network=proxy"
        - "traefik.http.services.${STACK_NAME}-loki.loadbalancer.server.port=3100"
        - "traefik.http.routers.${STACK_NAME}-loki.rule=Host(`loki.${DOMAIN}`)"
        - "traefik.http.routers.${STACK_NAME}-loki.entrypoints=web-secure"
@@ -47,6 +47,6 @@ volumes:
  loki-data:

 # secrets:
-#   loki_aws_key:
+#   loki_aws_secret_access_key:
 #     external: true
-#     name: ${STACK_NAME}_loki_aws_key_${SECRET_LOKI_AWS_KEY_VERSION}
+#     name: ${STACK_NAME}_loki_aws_secret_access_key_${SECRET_LOKI_AWS_SECRET_ACCESS_KEY_VERSION}
@@ -2,9 +2,9 @@ version: '3.8'

 services:
  matrix-alertmanager-receiver:
-    image: metio/matrix-alertmanager-receiver:2026.2.25
+    image: metio/matrix-alertmanager-receiver:2025.2.9
    secrets:
-      - matrix_token
+      - matrix_access_token
    configs:
      - source: matrix-alertmanager-receiver-config
        target: /etc/matrix-alertmanager-receiver/config.yml
@@ -23,6 +23,6 @@ configs:
    file: alertmanager-matrix-config.yml.tmpl

 secrets:
-  matrix_token:
+  matrix_access_token:
    external: true
-    name: ${STACK_NAME}_matrix_token_${SECRET_MATRIX_TOKEN_VERSION}
+    name: ${STACK_NAME}_matrix_access_token_${SECRET_MATRIX_ACCESS_TOKEN_VERSION}
@@ -2,7 +2,7 @@ version: '3.8'

 services:
  prometheus:
-    image: prom/prometheus:v3.12.0
+    image: prom/prometheus:v2.55.1
    secrets:
      - basic_auth
    volumes:
@@ -16,28 +16,20 @@ services:
      - "--web.console.libraries=/usr/share/prometheus/console_libraries"
      - "--web.console.templates=/usr/share/prometheus/consoles"
      - "--storage.tsdb.retention.time=${PROMETHEUS_RETENTION_TIME}"
-      - "--web.enable-remote-write-receiver"
    networks:
      - proxy
      - internal
-    healthcheck:
-      test: "wget -q --spider http://localhost:9090/-/healthy || exit 1"
-      interval: 5s
-      timeout: 10s
-      retries: 3
-      start_period: 30s
    deploy:
      restart_policy:
        condition: on-failure
      labels:
        - "traefik.enable=true"
-        - "traefik.swarm.network=proxy"
+        - "traefik.docker.network=proxy"
        - "traefik.http.services.${STACK_NAME}-prometheus.loadbalancer.server.port=9090"
        - "traefik.http.routers.${STACK_NAME}-prometheus.rule=Host(`prometheus.${DOMAIN}`)"
        - "traefik.http.routers.${STACK_NAME}-prometheus.entrypoints=web-secure"
        - "traefik.http.routers.${STACK_NAME}-prometheus.tls=true"
        - "traefik.http.routers.${STACK_NAME}-prometheus.tls.certresolver=${LETS_ENCRYPT_ENV}"
-        - "traefik.http.routers.${STACK_NAME}-prometheus.middlewares=basicauth@file"

 configs:
  prometheus_yml:
@@ -0,0 +1,30 @@
+version: "3.8"
+
+services:
+  promtail:
+    image: grafana/promtail:2.9.11
+    volumes:
+      - /var/log:/var/log:ro
+      - /var/run/docker.sock:/var/run/docker.sock
+    command: -config.file=/etc/promtail/config.yml
+    configs:
+      - source: promtail_yml
+        target: /etc/promtail/config.yml
+    networks:
+      - internal
+    secrets:
+      - basic_auth
+    environment:
+      - DOMAIN
+      - LOKI_PUSH_URL
+
+configs:
+  promtail_yml:
+    name: ${STACK_NAME}_promtail_yml_${PROMTAIL_YML_VERSION}
+    file: promtail.yml.tmpl
+    template_driver: golang
+
+secrets:
+  basic_auth:
+    external: true
+    name: ${STACK_NAME}_basic_auth_${SECRET_BASIC_AUTH_VERSION}
@@ -2,7 +2,7 @@ version: '3.8'

 services:
  pushgateway:
-    image: prom/pushgateway:v1.11.2
+    image: prom/pushgateway:v1.10.0
    command:
      - '--web.listen-address=:9191'
      - '--push.disable-consistency-check'
@@ -12,18 +12,12 @@ services:
    networks:
      - internal
      - proxy
-    healthcheck:
-      test: "wget -q --spider http://localhost:9191/-/healthy || exit 1"
-      interval: 5s
-      timeout: 10s
-      retries: 3
-      start_period: 10s
    deploy:
      restart_policy:
        condition: on-failure
      labels:
        - "traefik.enable=true"
-        - "traefik.swarm.network=proxy"
+        - "traefik.docker.network=proxy"
        - "traefik.http.services.${STACK_NAME}-pushgateway.loadbalancer.server.port=9191"
        - "traefik.http.routers.${STACK_NAME}-pushgateway.rule=Host(`pushgateway.${DOMAIN}`)"
        - "traefik.http.routers.${STACK_NAME}-pushgateway.entrypoints=web-secure"
@@ -1,13 +0,0 @@
---
-version: "3.8"
-
-services:
-  app:
-    secrets:
-      - source: scrape_auth
-        target: scrape_auth
-
-secrets:
-  scrape_auth:
-    external: true
-    name: ${STACK_NAME}_scrape_auth_${SECRET_SCRAPE_AUTH_VERSION}
@@ -1,6 +0,0 @@
---
-version: "3.8"
-services:
-  app:
-    ports:
-    - "514:514"
@@ -3,69 +3,89 @@ version: "3.8"

 services:
  app:
-    image: grafana/alloy:v1.18.1
-    hostname: "${DOMAIN}"
-    configs:
-      - source: config_alloy
-        target: /etc/alloy/config.alloy
-    volumes:
-      - /:/rootfs:ro
-      - /var/run/docker.sock:/var/run/docker.sock:ro
-      - /sys:/sys:ro
-      - /var/lib/docker:/var/lib/docker:ro
-      # long-form avoids toolshed/abra#900
-      - type: bind
-        source: "${CONTAINERD_SOCKET:-/run/containerd/containerd.sock}"
-        target: /run/containerd/containerd.sock
-        read_only: true
-      - alloy-data:/var/lib/alloy/data
-    # runs through a shell so ALLOY_HTTP_LISTEN_ADDR (set by
-    # compose.alloy-webui.yml) is resolved from the container's own
-    # environment at startup, not by compose at deploy time.
-    # $$ escapes it from compose's own interpolation
-    entrypoint: ["/bin/sh", "-c"]
-    command:
-      - >-
-        exec alloy run
-        --storage.path=/var/lib/alloy/data
-        --server.http.listen-addr=$${ALLOY_HTTP_LISTEN_ADDR:-127.0.0.1}:12345
-        /etc/alloy/config.alloy
-    networks:
-      - proxy
-      - internal
+    image: prom/node-exporter:v1.8.1
+    user: root
    environment:
-      - SCRAPE_BASIC_AUTH_USERNAME=${SCRAPE_BASIC_AUTH_USERNAME:-alloy}
-      - WRITE_BASIC_AUTH_USERNAME=${WRITE_BASIC_AUTH_USERNAME:-admin}
-      - LIVE_DEBUGGING=${LIVE_DEBUGGING:-false}
      - NODE_ID={{.Node.ID}}
-    secrets:
-      - basic_auth
-    # no wget/curl in this image; bash's /dev/tcp is used instead. Works
-    # against localhost regardless of ALLOY_HTTP_LISTEN_ADDR
-    healthcheck:
-      test: ["CMD", "bash", "-c", "exec 3<>/dev/tcp/localhost/12345 && printf 'GET /-/ready HTTP/1.0\r\nHost: localhost\r\n\r\n' >&3 && head -1 <&3 | grep -q 200"]
-      interval: 5s
-      timeout: 10s
-      retries: 3
-      start_period: 10s
+    volumes:
+      - /proc:/host/proc:ro
+      - /sys:/host/sys:ro
+      - /:/rootfs:ro
+      - /etc/hostname:/etc/nodename:ro
+    command:
+      - "--path.sysfs=/host/sys"
+      - "--path.procfs=/host/proc"
+      - "--path.rootfs=/rootfs"
+      - "--collector.textfile.directory=/etc/node-exporter/"
+      - "--collector.filesystem.ignored-mount-points=^/(sys|proc|dev|host|etc)($$|/)"
+      - "--no-collector.ipvs"
+    configs:
+      - source: entrypoint
+        target: /entrypoint.sh
+    networks:
+      - internal
+      - proxy
+    entrypoint: [ "/bin/sh", "-e", "/entrypoint.sh" ]
    deploy:
      restart_policy:
        condition: on-failure
      labels:
        - "backupbot.backup=${ENABLE_BACKUPS:-true}"
+        - "traefik.enable=true"
+        - "traefik.docker.network=proxy"
+        - "traefik.http.services.${STACK_NAME}-node.loadbalancer.server.port=9100"
+        - "traefik.http.routers.${STACK_NAME}-node.rule=Host(`node.${DOMAIN}`)"
+        - "traefik.http.routers.${STACK_NAME}-node.entrypoints=web-secure"
+        - "traefik.http.routers.${STACK_NAME}-node.tls=true"
+        - "traefik.http.routers.${STACK_NAME}-node.tls.certresolver=${LETS_ENCRYPT_ENV}"
+        - "traefik.http.routers.${STACK_NAME}-node.middlewares=basicauth@file"
        - "coop-cloud.${STACK_NAME}.version=1.6.0+v1.8.1"
+        - "coop-cloud.${STACK_NAME}.timeout=${TIMEOUT:-120}"
+
+  cadvisor:
+    image: gcr.io/cadvisor/cadvisor:v0.49.2
+    command:
+      - "-logtostderr"
+      - "--enable_metrics=cpu,cpuLoad,disk,diskIO,process,memory,network"
+      # all possible metrics: advtcp,app,cpu,cpuLoad,cpu_topology,cpuset,disk,diskIO,hugetlb,memory,memory_numa,network,oom_event,percpu,perf_event,process,referenced_memory,resctrl,sched,tcp,udp.
+      - "--housekeeping_interval=120s"
+      - "--docker_only=true"
+    volumes:
+      - /var/lib/docker/:/var/lib/docker:ro
+      - /dev/disk/:/dev/disk:ro
+      - /sys:/sys:ro
+      - /var/run:/var/run:ro
+      - /:/rootfs:ro
+    networks:
+      - internal
+      - proxy
+    deploy:
+      restart_policy:
+        condition: on-failure
+      labels:
+        - "traefik.enable=true"
+        - "traefik.docker.network=proxy"
+        - "traefik.http.services.${STACK_NAME}-cadvisor.loadbalancer.server.port=8080"
+        - "traefik.http.routers.${STACK_NAME}-cadvisor.rule=Host(`cadvisor.${DOMAIN}`)"
+        - "traefik.http.routers.${STACK_NAME}-cadvisor.entrypoints=web-secure"
+        - "traefik.http.routers.${STACK_NAME}-cadvisor.tls=true"
+        - "traefik.http.routers.${STACK_NAME}-cadvisor.tls.certresolver=${LETS_ENCRYPT_ENV}"
+        - "traefik.http.routers.${STACK_NAME}-cadvisor.middlewares=basicauth@file"
+    healthcheck:
+      test: wget --quiet --tries=1 --spider http://localhost:8080/healthz || exit 1
+      interval: 15s
+      timeout: 15s
+      retries: 5
+      start_period: 30s
+
 configs:
-  config_alloy:
-    template_driver: golang
-    name: ${STACK_NAME}_config_alloy_${CONFIG_ALLOY_VERSION}
-    file: config.alloy.tmpl
+  entrypoint:
+    name: ${STACK_NAME}_entrypoint_${ENTRYPOINT_VERSION}
+    file: node-exporter-entrypoint.sh
+
+
+
 networks:
  proxy:
    external: true
  internal:
-volumes:
-  alloy-data:
-secrets:
-  basic_auth:
-    external: true
-    name: ${STACK_NAME}_basic_auth_${SECRET_BASIC_AUTH_VERSION}
@@ -1,349 +0,0 @@
-logging {
-  level = "info"
-  format = "logfmt"
-}
-
-livedebugging {
-  enabled = {{ env "LIVE_DEBUGGING" }}
-}
-
-discovery.docker "linux" {
-  host = "unix:///var/run/docker.sock"
-}
-
-{{ if ne (env "PROMETHEUS_REMOTE_WRITE_URL") "" }}
-prometheus.exporter.cadvisor "docker" {
-  docker_only = true
-  enabled_metrics = ["cpu", "cpuLoad", "disk", "diskIO", "memory", "network", "process"]
-}
-
-prometheus.exporter.unix "default" {
-  include_exporter_metrics = true
-  rootfs_path = "/rootfs"
-  procfs_path = "/rootfs/proc"
-  sysfs_path = "/rootfs/sys"
-
-  disable_collectors = ["ipvs"]
-
-  filesystem {
-    fs_types_exclude     = "^(autofs|binfmt_misc|bpf|cgroup2?|configfs|debugfs|devpts|devtmpfs|tmpfs|fusectl|hugetlbfs|iso9660|mqueue|nsfs|overlay|proc|procfs|pstore|rpc_pipefs|securityfs|selinuxfs|squashfs|sysfs|tracefs)$"
-    mount_points_exclude = "^/(sys|proc|dev|host|etc)($|/)"
-    mount_timeout        = "5s"
-  }
-
-  netclass { ignored_devices = "^(veth.*)$" }
-  netdev   { device_exclude  = "^(veth.*)$" }
-}
-
-prometheus.exporter.self "alloy" {}
-
-prometheus.scrape "default" {
-  scrape_interval = "120s"
-
-  targets = array.concat(
-    prometheus.exporter.self.alloy.targets,
-    prometheus.exporter.unix.default.targets,
-    prometheus.exporter.cadvisor.docker.targets,
-  )
-
-  forward_to = [prometheus.remote_write.prometheus.receiver]
-}
-
-prometheus.remote_write "prometheus" {
-  endpoint {
-    url = "{{ env "PROMETHEUS_REMOTE_WRITE_URL" }}"
-
-    basic_auth {
-      username = "{{ env "WRITE_BASIC_AUTH_USERNAME" }}"
-      password = "{{ secret "basic_auth" }}"
-    }
-  }
-}
-
-// Scrape Prometheus metrics from other containers on this host.
-// Containers opt in via Docker labels:
-//   prometheus.io/scrape=true        required: enable scraping
-//   prometheus.io/port=9090          optional: port exposing /metrics (defaults to 80 if not set)
-//   prometheus.io/path=/metrics      optional: path to metrics endpoint (default: /metrics)
-//   prometheus.io/auth=basic         optional: use basic auth with the scrape_auth secret (see compose.scrape-auth.yml)
-//   prometheus.io/auth=bearer        optional: use bearer auth with the scrape_auth secret (see compose.scrape-auth.yml)
-discovery.dockerswarm "swarm" {
-  host = "unix:///var/run/docker.sock"
-  // "tasks" not "services": dnsrr-mode services (e.g. traefik) have no VIP
-  // and are invisible to the "services" role
-  role = "tasks"
-}
-
-discovery.relabel "metrics" {
-  targets = discovery.dockerswarm.swarm.targets
-
-  // skip old task history, only scrape currently-running tasks
-  rule {
-    source_labels = ["__meta_dockerswarm_task_desired_state"]
-    regex         = "running"
-    action        = "keep"
-  }
-
-  // only scrape hosts running on this host within the swam
-  rule {
-    source_labels = ["__meta_dockerswarm_node_id"]
-    regex         = "{{ env "NODE_ID" }}"
-    action        = "keep"
-  }
-
-  rule {
-    source_labels = ["__meta_dockerswarm_network_name"]
-    regex         = "proxy"
-    action        = "keep"
-  }
-
-  rule {
-    source_labels = ["__meta_dockerswarm_service_label_prometheus_io_scrape"]
-    regex         = "true"
-    action        = "keep"
-  }
-
-  // default to port 80 when prometheus.io/port isn't set
-  rule {
-    source_labels = ["__meta_dockerswarm_service_label_prometheus_io_port"]
-    regex         = "^$"
-    target_label  = "__meta_dockerswarm_service_label_prometheus_io_port"
-    replacement   = "80"
-  }
-
-  // a task with multiple published ports produces one target per port;
-  // this unifies all of them to the single port above, so duplicates
-  // collapse at scrape time instead of scraping every port
-  rule {
-    source_labels = ["__address__", "__meta_dockerswarm_service_label_prometheus_io_port"]
-    regex         = `(.+):\d+;(\d+)`
-    target_label  = "__address__"
-    replacement   = "$1:$2"
-  }
-
-  rule {
-    source_labels = ["__meta_dockerswarm_service_label_prometheus_io_path"]
-    regex         = `(.+)`
-    target_label  = "__metrics_path__"
-  }
-
-  rule {
-    source_labels = ["__meta_dockerswarm_service_name"]
-    target_label  = "job"
-  }
-
-  // task IDs (and the default address-derived instance label) change on
-  // every redeploy; service+slot is stable across redeploys of the same
-  // replica, so data stays continuous instead of restarting each deploy
-  rule {
-    source_labels = ["__meta_dockerswarm_service_name", "__meta_dockerswarm_task_slot"]
-    separator     = "."
-    target_label  = "instance"
-  }
-
-  rule {
-    source_labels = ["__meta_dockerswarm_task_slot"]
-    target_label  = "task_slot"
-  }
-
-  // coop-cloud's STACK_NAME is the domain with "." replaced by "_"
-  // Derive a readable dotted domain label from it. RE2 has no global
-  // replace, so this is done by chaining multiple replacements. Each
-  // rule swaps the first remaining _ for a "." until none are left.
-  rule {
-    source_labels = ["__meta_dockerswarm_service_label_com_docker_stack_namespace"]
-    target_label  = "domain"
-  }
-  rule {
-    source_labels = ["domain"]
-    regex         = `([^_]*)_(.*)`
-    target_label  = "domain"
-    replacement   = "$1.$2"
-  }
-  rule {
-    source_labels = ["domain"]
-    regex         = `([^_]*)_(.*)`
-    target_label  = "domain"
-    replacement   = "$1.$2"
-  }
-  rule {
-    source_labels = ["domain"]
-    regex         = `([^_]*)_(.*)`
-    target_label  = "domain"
-    replacement   = "$1.$2"
-  }
-  rule {
-    source_labels = ["domain"]
-    regex         = `([^_]*)_(.*)`
-    target_label  = "domain"
-    replacement   = "$1.$2"
-  }
-  rule {
-    source_labels = ["domain"]
-    regex         = `([^_]*)_(.*)`
-    target_label  = "domain"
-    replacement   = "$1.$2"
-  }
-  rule {
-    source_labels = ["domain"]
-    regex         = `([^_]*)_(.*)`
-    target_label  = "domain"
-    replacement   = "$1.$2"
-  }
-}
-
-discovery.relabel "metrics_noauth" {
-  targets = discovery.relabel.metrics.output
-  rule {
-    source_labels = ["__meta_dockerswarm_service_label_prometheus_io_auth"]
-    regex         = "^$"
-    action        = "keep"
-  }
-}
-
-discovery.relabel "metrics_basicauth" {
-  targets = discovery.relabel.metrics.output
-  rule {
-    source_labels = ["__meta_dockerswarm_service_label_prometheus_io_auth"]
-    regex         = "basic"
-    action        = "keep"
-  }
-}
-
-discovery.relabel "metrics_bearerauth" {
-  targets = discovery.relabel.metrics.output
-  rule {
-    source_labels = ["__meta_dockerswarm_service_label_prometheus_io_auth"]
-    regex         = "bearer"
-    action        = "keep"
-  }
-}
-
-prometheus.scrape "containers" {
-  scrape_interval = "120s"
-  targets         = discovery.relabel.metrics_noauth.output
-  forward_to      = [prometheus.remote_write.prometheus.receiver]
-}
-
-{{ if ne (env "SECRET_SCRAPE_AUTH_VERSION") "" }}
-prometheus.scrape "containers_basicauth" {
-  scrape_interval = "120s"
-  targets         = discovery.relabel.metrics_basicauth.output
-  forward_to      = [prometheus.remote_write.prometheus.receiver]
-  basic_auth {
-    username = "{{ env "SCRAPE_BASIC_AUTH_USERNAME" }}"
-    password = "{{ secret "scrape_auth" }}"
-  }
-}
-
-prometheus.scrape "containers_bearerauth" {
-  scrape_interval = "120s"
-  targets         = discovery.relabel.metrics_bearerauth.output
-  forward_to      = [prometheus.remote_write.prometheus.receiver]
-  bearer_token    = "{{ secret "scrape_auth" }}"
-}
-{{ end }}
-{{ end }}
-
-{{ if ne (env "LOKI_PUSH_URL") "" }}
-discovery.relabel "docker" {
-  targets = discovery.docker.linux.targets
-
-  rule {
-    source_labels = ["__meta_docker_container_name"]
-    target_label  = "container_name"
-  }
-  rule {
-    source_labels = ["__meta_docker_container_id"]
-    target_label  = "container_id"
-  }
-  rule {
-    source_labels = ["__meta_docker_container_label_com_docker_stack_namespace"]
-    target_label  = "stack_namespace"
-  }
-  rule {
-    source_labels = ["__meta_docker_container_label_com_docker_swarm_service_name"]
-    target_label  = "service_name"
-  }
-  rule {
-    source_labels = ["__meta_docker_container_log_stream"]
-    target_label  = "stream"
-  }
-}
-
-loki.source.docker "docker" {
-  host       = "unix:///var/run/docker.sock"
-  targets    = discovery.relabel.docker.output
-  labels     = {"app" = "docker"}
-  forward_to = [loki.write.loki.receiver]
-}
-
-// JOURNALD: reads the systemd journal binary log directly.
-// Use on systemd hosts (most modern Linux distros). Requires no syslogd.
-{{ if eq (env "JOURNALD") "1" }}
-loki.source.journal "journal" {
-  path = "/rootfs/var/log/journal"
-  labels = { job = "{{ env "DOMAIN" }}" }
-  forward_to = [loki.write.loki.receiver]
-}
-{{ end }}
-
-// SYSLOG_FILES: tails all /var/log/*log files (syslog, auth.log, kern.log, etc.).
-// Use on non-systemd hosts where a syslogd writes to /var/log.
-{{ if eq (env "SYSLOG_FILES") "1" }}
-local.file_match "syslog_files" {
-  path_targets = [{ __path__ = "/rootfs/var/log/*log" }]
-}
-
-loki.source.file "syslog_files" {
-  targets    = local.file_match.syslog_files.targets
-  forward_to = [loki.process.syslog_files.receiver]
-}
-
-loki.process "syslog_files" {
-  stage.static_labels {
-    values = { job = "syslog" }
-  }
-  forward_to = [loki.write.loki.receiver]
-}
-{{ end }}
-
-// SYSLOG: opens a network syslog listener on port 514.
-// Use when a remote device or a local syslogd configured to 
-// forward over the network sends logs to this host.
-// Requires compose.syslog.yml to publish port 514 to the host.
-// This is NOT needed for reading local log files — use SYSLOG_FILES instead.
-{{ if eq (env "SYSLOG") "1" }}
-loki.relabel "syslog" {
-  rule {
-    action = "labelmap"
-    regex = "__syslog_(.+)"
-  }
-
-  forward_to = []
-}
-
-loki.source.syslog "syslog" {
-  listener {
-    address = "[::]:514"
-    label_structured_data = true
-    labels = { component = "loki.source.syslog" }
-  }
-
-  relabel_rules = loki.relabel.syslog.rules
-  forward_to = [loki.write.loki.receiver]
-}
-{{ end }}
-
-loki.write "loki" {
-  endpoint {
-    url = "{{ env "LOKI_PUSH_URL" }}"
-
-    basic_auth {
-      username = "{{ env "WRITE_BASIC_AUTH_USERNAME" }}"
-      password = "{{ secret "basic_auth" }}"
-    }
-  }
-  external_labels = { hostname = "{{ env "DOMAIN" }}" }
-}
-{{ end }}
@@ -0,0 +1,341 @@
+{
+  "apiVersion": 1,
+  "groups": [
+    {
+      "orgId": 1,
+      "name": "backupbot",
+      "folder": "node",
+      "interval": "1m",
+      "rules": [
+        {{ if eq (env "ALERT_BACKUP_FAILED_ENABLED") "true" }}
+        {
+          "uid": "de8e5xxup7t34a",
+          "title": "Backup Failed",
+          "condition": "C",
+          "data": [
+            {
+              "refId": "A",
+              "relativeTimeRange": { "from": 600, "to": 0 },
+              "datasourceUid": "PBFA97CFB590B2093",
+              "model": {
+                "disableTextWrap": false,
+                "editorMode": "builder",
+                "expr": "backup",
+                "fullMetaSearch": false,
+                "includeNullMetadata": true,
+                "instant": true,
+                "intervalMs": 1000,
+                "legendFormat": "__auto",
+                "maxDataPoints": 43200,
+                "range": false,
+                "refId": "A",
+                "useBackend": false
+              }
+            },
+            {
+              "refId": "C",
+              "relativeTimeRange": { "from": 600, "to": 0 },
+              "datasourceUid": "__expr__",
+              "model": {
+                "conditions": [
+                  {
+                    "evaluator": { "params": [0], "type": "lt" },
+                    "operator": { "type": "and" },
+                    "query": { "params": ["C"] },
+                    "reducer": { "params": [], "type": "last" },
+                    "type": "query"
+                  }
+                ],
+                "datasource": { "type": "__expr__", "uid": "__expr__" },
+                "expression": "A",
+                "intervalMs": 1000,
+                "maxDataPoints": 43200,
+                "refId": "C",
+                "type": "threshold"
+              }
+            }
+          ],
+          "noDataState": "NoData",
+          "execErrState": "Error",
+          "for": "1m",
+          "isPaused": false
+        },
+        {{ end }}
+        {{ if eq (env "ALERT_BACKUP_MISSING_ENABLED") "true" }}
+        {
+          "uid": "ce8e65uddcwe8d",
+          "title": "Backup Missing",
+          "condition": "B",
+          "data": [
+            {
+                "refId": "A",
+                "relativeTimeRange": {
+                    "from": 28800,
+                    "to": 0
+                },
+                "datasourceUid": "PBFA97CFB590B2093",
+                "model": {
+                    "disableTextWrap": false,
+                    "editorMode": "code",
+                    "expr": "rate(backup[25h]) == 0",
+                    "fullMetaSearch": false,
+                    "includeNullMetadata": true,
+                    "instant": true,
+                    "intervalMs": 1000,
+                    "legendFormat": "__auto",
+                    "maxDataPoints": 43200,
+                    "range": false,
+                    "refId": "A",
+                    "useBackend": false
+                }
+            },
+            {
+                "refId": "B",
+                "relativeTimeRange": {
+                    "from": 28800,
+                    "to": 0
+                },
+                "datasourceUid": "__expr__",
+                "model": {
+                    "conditions": [
+                        {
+                            "evaluator": {
+                                "params": [
+                                    -1,
+                                    0
+                                ],
+                                "type": "gt"
+                            },
+                            "operator": {
+                                "type": "and"
+                            },
+                            "query": {
+                                "params": []
+                            },
+                            "reducer": {
+                                "params": [],
+                                "type": "avg"
+                            },
+                            "type": "query"
+                        }
+                    ],
+                    "datasource": {
+                        "name": "Expression",
+                        "type": "__expr__",
+                        "uid": "__expr__"
+                    },
+                    "expression": "A",
+                    "hide": false,
+                    "intervalMs": 1000,
+                    "maxDataPoints": 43200,
+                    "refId": "B",
+                    "type": "threshold"
+                }
+            }
+          ],
+          "noDataState": "NoData",
+          "execErrState": "Error",
+          "for": "5m",
+          "annotations": {},
+          "labels": {},
+          "isPaused": false
+        },
+        {{ end }}
+        {{ if eq (env "ALERT_BACKUP_NOT_SUCCESSFULL_ENABLED") "true" }}
+        {
+            "uid": "de8e6bc92a8lcc",
+            "title": "Backup Not Successfull",
+            "condition": "B",
+            "data": [
+                {
+                    "refId": "A",
+                    "relativeTimeRange": {
+                        "from": 60,
+                        "to": 0
+                    },
+                    "datasourceUid": "PBFA97CFB590B2093",
+                    "model": {
+                        "disableTextWrap": false,
+                        "editorMode": "builder",
+                        "expr": "backup",
+                        "fullMetaSearch": false,
+                        "includeNullMetadata": true,
+                        "instant": true,
+                        "intervalMs": 1000,
+                        "legendFormat": "__auto",
+                        "maxDataPoints": 43200,
+                        "range": false,
+                        "refId": "A",
+                        "useBackend": false
+                    }
+                },
+                {
+                    "refId": "B",
+                    "relativeTimeRange": {
+                        "from": 60,
+                        "to": 0
+                    },
+                    "datasourceUid": "__expr__",
+                    "model": {
+                        "conditions": [
+                            {
+                                "evaluator": {
+                                    "params": [
+                                        0
+                                    ],
+                                    "type": "gt"
+                                },
+                                "operator": {
+                                    "type": "and"
+                                },
+                                "query": {
+                                    "params": [
+                                        "C"
+                                    ]
+                                },
+                                "reducer": {
+                                    "params": [],
+                                    "type": "last"
+                                },
+                                "type": "query"
+                            }
+                        ],
+                        "datasource": {
+                            "type": "__expr__",
+                            "uid": "__expr__"
+                        },
+                        "expression": "A",
+                        "intervalMs": 1000,
+                        "maxDataPoints": 43200,
+                        "refId": "B",
+                        "type": "threshold"
+                    }
+                }
+            ],
+            "noDataState": "NoData",
+            "execErrState": "Error",
+            "for": "60m",
+            "annotations": {
+                "summary": "Backup did not finish within 60 minutes"
+            },
+            "labels": {},
+            "isPaused": false
+        }
+        {{ end }}
+      ]
+    },
+    {
+      "orgId": 1,
+      "name": "node",
+      "folder": "node",
+      "interval": "5m",
+      "rules": [
+        {{ if eq (env "ALERT_NODE_DISK_SPACE_ENABLED") "true" }}
+        {
+          "uid": "bds8bhxu97pxca",
+          "title": "Node Disk Space",
+          "condition": "C",
+          "data": [
+            {
+              "refId": "A",
+              "relativeTimeRange": { "from": 600, "to": 0 },
+              "datasourceUid": "PBFA97CFB590B2093",
+              "model": {
+                "editorMode": "code",
+                "expr": "(node_filesystem_free_bytes{fstype=\"ext4\",mountpoint=~\"(/$)|(/media.*)\"} / node_filesystem_size_bytes{fstype=\"ext4\",mountpoint=~\"(/$)|(/media.*)\"}) * 100",
+                "instant": true,
+                "intervalMs": 1000,
+                "legendFormat": "__auto",
+                "maxDataPoints": 43200,
+                "range": false,
+                "refId": "A"
+              }
+            },
+            {
+              "refId": "C",
+              "relativeTimeRange": { "from": 600, "to": 0 },
+              "datasourceUid": "__expr__",
+              "model": {
+                "conditions": [
+                  {
+                    "evaluator": { "params": [10], "type": "lt" },
+                    "operator": { "type": "and" },
+                    "query": { "params": ["C"] },
+                    "reducer": { "params": [], "type": "last" },
+                    "type": "query"
+                  }
+                ],
+                "datasource": { "type": "__expr__", "uid": "__expr__" },
+                "expression": "A",
+                "intervalMs": 1000,
+                "maxDataPoints": 43200,
+                "refId": "C",
+                "type": "threshold"
+              }
+            }
+          ],
+          "noDataState": "NoData",
+          "execErrState": "Error",
+          "for": "5m",
+          "annotations": {},
+          "labels": {},
+          "isPaused": false
+        },
+        {{ end }}
+        {{ if eq (env "ALERT_NODE_MEMORY_USAGE_ENABLED") "true" }}
+        {
+          "uid": "ads8cswmly96oa",
+          "title": "Node Memory Usage",
+          "condition": "C",
+          "data": [
+            {
+              "refId": "A",
+              "relativeTimeRange": { "from": 600, "to": 0 },
+              "datasourceUid": "PBFA97CFB590B2093",
+              "model": {
+                "editorMode": "code",
+                "expr": "(node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * 100",
+                "instant": true,
+                "intervalMs": 1000,
+                "legendFormat": "__auto",
+                "maxDataPoints": 43200,
+                "range": false,
+                "refId": "A"
+              }
+            },
+            {
+              "refId": "C",
+              "relativeTimeRange": { "from": 600, "to": 0 },
+              "datasourceUid": "__expr__",
+              "model": {
+                "conditions": [
+                  {
+                    "evaluator": { "params": [90], "type": "gt" },
+                    "operator": { "type": "and" },
+                    "query": { "params": ["C"] },
+                    "reducer": { "params": [], "type": "last" },
+                    "type": "query"
+                  }
+                ],
+                "datasource": { "type": "__expr__", "uid": "__expr__" },
+                "expression": "A",
+                "intervalMs": 1000,
+                "maxDataPoints": 43200,
+                "refId": "C",
+                "type": "threshold"
+              }
+            }
+          ],
+          "noDataState": "NoData",
+          "execErrState": "Error",
+          "for": "5m",
+          "annotations": {},
+          "labels": {},
+          "isPaused": false
+        }
+        {{ end }}
+      ]
+    }
+  ]
+}
+
@@ -110,7 +110,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "editorMode": "code",
-          "expr": "(time() - min(container_start_time_seconds{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}))",
+          "expr": "(time() - min(container_start_time_seconds{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}))",
          "format": "time_series",
          "intervalFactor": 1,
          "legendFormat": "",
@@ -215,7 +215,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "editorMode": "code",
-          "expr": "sort(\n    sum(\n        rate(\n            container_cpu_usage_seconds_total{\n                container_label_com_docker_stack_namespace=~\"$stack\", \n                instance=~\".*$node_id.*\"}[$interval]\n            )\n        ) \n    by (container_label_com_docker_swarm_service_name)\n)",
+          "expr": "sort(\n    sum(\n        rate(\n            container_cpu_usage_seconds_total{\n                container_label_com_docker_stack_namespace=~\"$stack\", \n                instance=~\"cadvisor.monitoring.$instance.*\"}[$interval]\n            )\n        ) \n    by (container_label_com_docker_swarm_service_name)\n)",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "{{ container_label_com_docker_swarm_service_name }}",
@@ -283,7 +283,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "editorMode": "code",
-          "expr": "count(rate(container_last_seen{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}[$interval]))",
+          "expr": "count(rate(container_last_seen{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}[$interval]))",
          "format": "time_series",
          "intervalFactor": 2,
          "range": true,
@@ -386,7 +386,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "editorMode": "code",
-          "expr": "sum(\n    container_memory_rss{\n        container_label_com_docker_stack_namespace=~\"$stack\", \n        instance=~\".*$node_id.*\"}\n    ) \nby (container_label_com_docker_swarm_service_name)",
+          "expr": "sum(\n    container_memory_rss{\n        container_label_com_docker_stack_namespace=~\"$stack\", \n        instance=~\"cadvisor.monitoring.$instance.*\"}\n    ) \nby (container_label_com_docker_swarm_service_name)",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 2,
@@ -493,7 +493,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "editorMode": "code",
-          "expr": "sum(rate(container_network_receive_bytes_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
+          "expr": "sum(rate(container_network_receive_bytes_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "RX: {{ container_label_com_docker_swarm_service_name }}",
@@ -507,7 +507,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "editorMode": "code",
-          "expr": "sum(rate(container_network_transmit_bytes_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}[$interval])) by (container_label_com_docker_swarm_service_name) * -1",
+          "expr": "sum(rate(container_network_transmit_bytes_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}[$interval])) by (container_label_com_docker_swarm_service_name) * -1",
          "hide": false,
          "legendFormat": "TX: {{container_label_com_docker_swarm_service_name}}",
          "range": true,
@@ -634,7 +634,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "editorMode": "code",
-          "expr": "sum(container_fs_usage_bytes{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}) by (container_label_com_docker_swarm_service_name)",
+          "expr": "sum(container_fs_usage_bytes{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}) by (container_label_com_docker_swarm_service_name)",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 2,
@@ -742,7 +742,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "editorMode": "code",
-          "expr": "sum(rate(container_fs_usage_bytes{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
+          "expr": "sum(rate(container_fs_usage_bytes{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 2,
@@ -865,7 +865,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "editorMode": "code",
-          "expr": "sum(rate(container_fs_io_time_seconds_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
+          "expr": "sum(rate(container_fs_io_time_seconds_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 2,
@@ -973,7 +973,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "editorMode": "code",
-          "expr": "container_fs_io_current{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}",
+          "expr": "container_fs_io_current{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 2,
@@ -1081,7 +1081,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "editorMode": "code",
-          "expr": "sum(rate(container_fs_reads_bytes_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
+          "expr": "sum(rate(container_fs_reads_bytes_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "{{ container_label_com_docker_swarm_service_name }}",
@@ -1188,7 +1188,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "editorMode": "code",
-          "expr": "sum(rate(container_fs_read_seconds_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
+          "expr": "sum(rate(container_fs_read_seconds_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "{{ container_label_com_docker_swarm_service_name }}",
@@ -1295,7 +1295,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "editorMode": "code",
-          "expr": "sum(rate(container_fs_writes_bytes_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
+          "expr": "sum(rate(container_fs_writes_bytes_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "{{ container_label_com_docker_swarm_service_name }}",
@@ -1402,7 +1402,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "editorMode": "code",
-          "expr": "sum(rate(container_fs_write_seconds_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\".*$node_id.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
+          "expr": "sum(rate(container_fs_write_seconds_total{container_label_com_docker_stack_namespace=~\"$stack\", instance=~\"cadvisor.monitoring.$instance.*\"}[$interval])) by (container_label_com_docker_swarm_service_name)",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "{{ container_label_com_docker_swarm_service_name }}",
@@ -1453,7 +1453,7 @@
          "query": "label_values(container_label_com_docker_stack_namespace)",
          "refId": "PrometheusVariableQueryEditor-VariableQuery"
        },
-        "refresh": 2,
+        "refresh": 1,
        "regex": "",
        "skipUrlSync": false,
        "sort": 2,
@@ -1567,19 +1567,19 @@
          "type": "prometheus",
          "uid": "PBFA97CFB590B2093"
        },
-        "definition": "label_values(node_uname_info, instance)",
+        "definition": "label_values(instance)",
        "hide": 0,
        "includeAll": true,
-        "label": "Swarm Node",
+        "label": "instance",
        "multi": true,
-        "name": "node_id",
+        "name": "instance",
        "options": [],
        "query": {
-          "query": "label_values(node_uname_info, instance)",
+          "query": "label_values(instance)",
          "refId": "PrometheusVariableQueryEditor-VariableQuery"
        },
-        "refresh": 2,
-        "regex": "/^(?:(?:node|cadvisor|monitoring).)*(.*)$/",
+        "refresh": 1,
+        "regex": "/.*cadvisor.monitoring.(?<instance>.*):80/",
        "skipUrlSync": false,
        "sort": 0,
        "type": "query"
@@ -1620,4 +1620,4 @@
  "uid": "KdVoGQm7z",
  "version": 36,
  "weekStart": ""
-}
+}
@@ -93,6 +93,7 @@
      },
      "hideTimeOverride": true,
      "id": 2,
+      "links": [],
      "maxDataPoints": 100,
      "options": {
        "colorMode": "value",
@@ -106,19 +107,17 @@
          "fields": "",
          "values": false
        },
-        "showPercentChange": false,
        "text": {},
-        "textMode": "auto",
-        "wideLayout": true
+        "textMode": "auto"
      },
-      "pluginVersion": "10.4.14",
+      "pluginVersion": "10.0.2",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "PBFA97CFB590B2093"
          },
-          "expr": "topk(1, sum(label_replace(node_time_seconds{instance=~\".*$node_id.*\"} - node_boot_time_seconds{instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) by (instance))",
+          "expr": "topk(1, sum((node_time_seconds - node_boot_time_seconds) * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"}) by (node_name))",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "",
@@ -173,6 +172,7 @@
        "y": 0
      },
      "id": 1,
+      "links": [],
      "maxDataPoints": 100,
      "options": {
        "colorMode": "value",
@@ -186,19 +186,17 @@
          "fields": "",
          "values": false
        },
-        "showPercentChange": false,
        "text": {},
-        "textMode": "auto",
-        "wideLayout": true
+        "textMode": "auto"
      },
-      "pluginVersion": "10.4.14",
+      "pluginVersion": "10.0.2",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "PBFA97CFB590B2093"
          },
-          "expr": "count(label_replace(node_uname_info{instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\"))",
+          "expr": "count(node_meta * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"})",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "",
@@ -253,6 +251,7 @@
      },
      "hideTimeOverride": true,
      "id": 4,
+      "links": [],
      "maxDataPoints": 100,
      "options": {
        "colorMode": "value",
@@ -266,19 +265,17 @@
          "fields": "",
          "values": false
        },
-        "showPercentChange": false,
        "text": {},
-        "textMode": "auto",
-        "wideLayout": true
+        "textMode": "auto"
      },
-      "pluginVersion": "10.4.14",
+      "pluginVersion": "10.0.2",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "PBFA97CFB590B2093"
          },
-          "expr": "count(label_replace(node_cpu_seconds_total{mode=\"idle\", instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\"))",
+          "expr": "count(node_cpu_seconds_total{mode=\"idle\"} * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"})",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "",
@@ -338,10 +335,9 @@
        "y": 0
      },
      "id": 8,
+      "links": [],
      "maxDataPoints": 100,
      "options": {
-        "minVizHeight": 75,
-        "minVizWidth": 75,
        "orientation": "horizontal",
        "reduceOptions": {
          "calcs": [
@@ -352,10 +348,9 @@
        },
        "showThresholdLabels": false,
        "showThresholdMarkers": true,
-        "sizing": "auto",
        "text": {}
      },
-      "pluginVersion": "10.4.14",
+      "pluginVersion": "10.0.2",
      "targets": [
        {
          "datasource": {
@@ -363,7 +358,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "editorMode": "code",
-          "expr": "sum(label_replace((node_memory_MemAvailable_bytes{instance=~\".*$node_id.*\"} / node_memory_MemTotal_bytes{instance=~\".*$node_id.*\"}) * 100, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) / count(label_replace(node_uname_info{instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\"))",
+          "expr": "sum((node_memory_MemAvailable_bytes / node_memory_MemTotal_bytes) * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"} * 100) / count(node_meta * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"})",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "",
@@ -410,12 +405,13 @@
      },
      "lines": true,
      "linewidth": 1,
+      "links": [],
      "nullPointMode": "null",
      "options": {
        "alertThreshold": true
      },
      "percentage": false,
-      "pluginVersion": "10.4.14",
+      "pluginVersion": "10.0.2",
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
@@ -429,11 +425,11 @@
            "type": "prometheus",
            "uid": "PBFA97CFB590B2093"
          },
-          "expr": "label_replace(node_load5{instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")",
+          "expr": "node_load5 * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"}",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 2,
-          "legendFormat": "load5 {{instance}}",
+          "legendFormat": "load5 {{node_name}}",
          "refId": "A",
          "step": 2
        }
@@ -511,6 +507,7 @@
      },
      "hideTimeOverride": true,
      "id": 3,
+      "links": [],
      "maxDataPoints": 100,
      "options": {
        "colorMode": "value",
@@ -524,19 +521,17 @@
          "fields": "",
          "values": false
        },
-        "showPercentChange": false,
        "text": {},
-        "textMode": "auto",
-        "wideLayout": true
+        "textMode": "auto"
      },
-      "pluginVersion": "10.4.14",
+      "pluginVersion": "10.0.2",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "PBFA97CFB590B2093"
          },
-          "expr": "sum(label_replace(node_memory_MemTotal_bytes{instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\"))",
+          "expr": "sum(node_memory_MemTotal_bytes * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"})",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "",
@@ -590,6 +585,7 @@
      },
      "hideTimeOverride": true,
      "id": 9,
+      "links": [],
      "maxDataPoints": 100,
      "options": {
        "colorMode": "value",
@@ -603,12 +599,10 @@
          "fields": "",
          "values": false
        },
-        "showPercentChange": false,
        "text": {},
-        "textMode": "auto",
-        "wideLayout": true
+        "textMode": "auto"
      },
-      "pluginVersion": "10.4.14",
+      "pluginVersion": "10.0.2",
      "targets": [
        {
          "datasource": {
@@ -616,7 +610,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "exemplar": true,
-          "expr": "sum(label_replace(node_filesystem_size_bytes{fstype=~\"ext4|xfs\",mountpoint=~\"(/$)|(/media.*)\", instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\"))",
+          "expr": "sum(node_filesystem_size_bytes{fstype=\"ext4\",mountpoint=~\"(/$)|(/media.*)\"} * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"})",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 2,
@@ -677,10 +671,9 @@
      },
      "hideTimeOverride": true,
      "id": 11,
+      "links": [],
      "maxDataPoints": 100,
      "options": {
-        "minVizHeight": 75,
-        "minVizWidth": 75,
        "orientation": "horizontal",
        "reduceOptions": {
          "calcs": [
@@ -691,17 +684,16 @@
        },
        "showThresholdLabels": false,
        "showThresholdMarkers": true,
-        "sizing": "auto",
        "text": {}
      },
-      "pluginVersion": "10.4.14",
+      "pluginVersion": "10.0.2",
      "targets": [
        {
          "datasource": {
            "type": "prometheus",
            "uid": "PBFA97CFB590B2093"
          },
-          "expr": "sum(label_replace(irate(node_cpu_seconds_total{mode=\"idle\", instance=~\".*$node_id.*\"}[$interval]), \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) * 100 / count(label_replace(node_cpu_seconds_total{mode=\"user\", instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\"))",
+          "expr": "sum(irate(node_cpu_seconds_total{mode=\"idle\"}[$interval]) * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"}) * 100 / count(node_cpu_seconds_total{mode=\"user\"} * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"}) ",
          "format": "time_series",
          "intervalFactor": 2,
          "legendFormat": "",
@@ -721,39 +713,7 @@
      "fieldConfig": {
        "defaults": {
          "color": {
-            "mode": "palette-classic"
-          },
-          "custom": {
-            "axisBorderShow": false,
-            "axisCenteredZero": false,
-            "axisColorMode": "text",
-            "axisLabel": "",
-            "axisPlacement": "left",
-            "barAlignment": 0,
-            "drawStyle": "line",
-            "fillOpacity": 0,
-            "gradientMode": "none",
-            "hideFrom": {
-              "legend": false,
-              "tooltip": false,
-              "viz": false
-            },
-            "insertNulls": false,
-            "lineInterpolation": "linear",
-            "lineWidth": 1,
-            "pointSize": 5,
-            "scaleDistribution": {
-              "type": "linear"
-            },
-            "showPoints": "auto",
-            "spanNulls": false,
-            "stacking": {
-              "group": "A",
-              "mode": "none"
-            },
-            "thresholdsStyle": {
-              "mode": "off"
-            }
+            "mode": "thresholds"
          },
          "mappings": [
            {
@@ -787,42 +747,33 @@
          },
          "unit": "percent"
        },
-        "overrides": [
-          {
-            "matcher": {
-              "id": "byType",
-              "options": "time"
-            },
-            "properties": [
-              {
-                "id": "custom.axisPlacement",
-                "value": "hidden"
-              }
-            ]
-          }
-        ]
+        "overrides": []
      },
      "gridPos": {
-        "h": 6,
-        "w": 6,
+        "h": 4,
+        "w": 2.6666666666666665,
        "x": 0,
        "y": 8
      },
      "id": 10,
+      "links": [],
      "maxDataPoints": 100,
+      "maxPerRow": 12,
      "options": {
-        "legend": {
-          "calcs": [],
-          "displayMode": "list",
-          "placement": "bottom",
-          "showLegend": true
+        "colorMode": "value",
+        "graphMode": "area",
+        "justifyMode": "auto",
+        "orientation": "horizontal",
+        "reduceOptions": {
+          "calcs": [
+            "last"
+          ],
+          "fields": "",
+          "values": false
        },
-        "tooltip": {
-          "mode": "single",
-          "sort": "none"
-        }
+        "textMode": "auto"
      },
-      "pluginVersion": "10.4.14",
+      "pluginVersion": "10.0.2",
      "repeat": "node_id",
      "repeatDirection": "h",
      "targets": [
@@ -831,20 +782,18 @@
            "type": "prometheus",
            "uid": "PBFA97CFB590B2093"
          },
-          "editorMode": "code",
          "exemplar": true,
-          "expr": "label_replace(node_filesystem_free_bytes{fstype=~\"ext4|xfs\", instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\") / label_replace(node_filesystem_size_bytes{fstype=~\"ext4|xfs\", instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\") * 100",
+          "expr": "sum((node_filesystem_free_bytes{fstype=\"ext4\",mountpoint=~\"(/$)|(/media.*)\"} / node_filesystem_size_bytes{fstype=\"ext4\",mountpoint=~\"(/$)|(/media.*)\"}) * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"} * 100) / count(node_meta * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"})",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 2,
-          "legendFormat": "{{mountpoint}}",
-          "range": true,
+          "legendFormat": "",
          "refId": "A",
          "step": 20
        }
      ],
      "title": "Available Disk Space $node_id",
-      "type": "timeseries"
+      "type": "stat"
    },
    {
      "aliasColors": {},
@@ -862,7 +811,7 @@
        "h": 7,
        "w": 24,
        "x": 0,
-        "y": 14
+        "y": 12
      },
      "hiddenSeries": false,
      "id": 14,
@@ -881,12 +830,13 @@
      },
      "lines": true,
      "linewidth": 1,
+      "links": [],
      "nullPointMode": "null as zero",
      "options": {
        "alertThreshold": true
      },
      "percentage": false,
-      "pluginVersion": "10.4.14",
+      "pluginVersion": "10.0.2",
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
@@ -900,10 +850,10 @@
            "type": "prometheus",
            "uid": "PBFA97CFB590B2093"
          },
-          "expr": "100 - (avg(label_replace(irate(node_cpu_seconds_total{mode=\"idle\", instance=~\".*$node_id.*\"}[$interval]), \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\") * 100) by (instance))",
+          "expr": "100 - (avg(irate(node_cpu_seconds_total{mode=\"idle\"}[$interval])  * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"} * 100) by (node_name))",
          "format": "time_series",
          "intervalFactor": 2,
-          "legendFormat": "{{instance}}",
+          "legendFormat": "{{node_name}}",
          "refId": "A",
          "step": 2
        }
@@ -950,7 +900,6 @@
            "mode": "palette-classic"
          },
          "custom": {
-            "axisBorderShow": false,
            "axisCenteredZero": false,
            "axisColorMode": "text",
            "axisLabel": "",
@@ -964,7 +913,6 @@
              "tooltip": false,
              "viz": false
            },
-            "insertNulls": false,
            "lineInterpolation": "linear",
            "lineWidth": 1,
            "pointSize": 5,
@@ -1013,11 +961,12 @@
      },
      "gridPos": {
        "h": 7,
-        "w": 6,
+        "w": 2.6666666666666665,
        "x": 0,
-        "y": 21
+        "y": 19
      },
      "id": 15,
+      "links": [],
      "maxPerRow": 12,
      "options": {
        "legend": {
@@ -1045,12 +994,12 @@
            "uid": "PBFA97CFB590B2093"
          },
          "editorMode": "code",
-          "expr": "sum(label_replace(node_memory_MemFree_bytes{instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) by (instance)",
+          "expr": "sum((node_memory_MemFree_bytes) * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"}) by (node_name)",
          "format": "time_series",
          "hide": false,
          "interval": "",
          "intervalFactor": 2,
-          "legendFormat": "Free  {{instance}}",
+          "legendFormat": "Free  {{node_name}}",
          "range": true,
          "refId": "free",
          "step": 2
@@ -1061,12 +1010,12 @@
            "uid": "PBFA97CFB590B2093"
          },
          "editorMode": "code",
-          "expr": "sum(label_replace(node_memory_Cached_bytes{instance=~\".*$node_id.*\"} + node_memory_Buffers_bytes{instance=~\".*$node_id.*\"} + node_memory_Slab_bytes{instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) by (instance)",
+          "expr": "sum((node_memory_Cached_bytes + node_memory_Buffers_bytes + node_memory_Slab_bytes) * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"}) by (node_name)",
          "format": "time_series",
          "hide": false,
          "interval": "",
          "intervalFactor": 2,
-          "legendFormat": "cache,buffer,slab  {{instance}}",
+          "legendFormat": "cache,buffer,slab  {{node_name}}",
          "range": true,
          "refId": "cache,buffer,slab",
          "step": 2
@@ -1077,12 +1026,12 @@
            "uid": "PBFA97CFB590B2093"
          },
          "editorMode": "code",
-          "expr": "sum(label_replace(node_memory_MemTotal_bytes{instance=~\".*$node_id.*\"} - node_memory_MemFree_bytes{instance=~\".*$node_id.*\"} - node_memory_Cached_bytes{instance=~\".*$node_id.*\"} - node_memory_Buffers_bytes{instance=~\".*$node_id.*\"} - node_memory_Slab_bytes{instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) by (instance)",
+          "expr": "sum((node_memory_MemTotal_bytes - node_memory_MemFree_bytes - node_memory_Cached_bytes - node_memory_Buffers_bytes - node_memory_Slab_bytes) * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"}) by (node_name)",
          "format": "time_series",
          "hide": false,
          "interval": "",
          "intervalFactor": 2,
-          "legendFormat": "Used {{instance}}",
+          "legendFormat": "Used {{node_name}}",
          "range": true,
          "refId": "Used",
          "step": 2
@@ -1094,13 +1043,13 @@
          },
          "editorMode": "code",
          "exemplar": false,
-          "expr": "sum(label_replace(node_memory_MemTotal_bytes{instance=~\".*$node_id.*\"}, \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) by (instance)",
+          "expr": "sum((node_memory_MemTotal_bytes) * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"}) by (node_name)",
          "format": "time_series",
          "hide": false,
          "instant": false,
          "interval": "",
          "intervalFactor": 2,
-          "legendFormat": "Total {{instance}}",
+          "legendFormat": "Total {{node_name}}",
          "range": true,
          "refId": "total",
          "step": 2
@@ -1125,7 +1074,7 @@
        "h": 7,
        "w": 24,
        "x": 0,
-        "y": 28
+        "y": 26
      },
      "hiddenSeries": false,
      "id": 16,
@@ -1142,12 +1091,13 @@
      },
      "lines": true,
      "linewidth": 1,
+      "links": [],
      "nullPointMode": "null as zero",
      "options": {
        "alertThreshold": true
      },
      "percentage": false,
-      "pluginVersion": "10.4.14",
+      "pluginVersion": "10.0.2",
      "pointradius": 5,
      "points": false,
      "renderer": "flot",
@@ -1161,11 +1111,11 @@
            "type": "prometheus",
            "uid": "PBFA97CFB590B2093"
          },
-          "expr": "sum(label_replace(irate(node_disk_read_bytes_total{instance=~\".*$node_id.*\"}[$interval]), \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) by (instance)",
+          "expr": "sum(irate(node_disk_read_bytes_total[$interval]) * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"}) by (node_name)",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 2,
-          "legendFormat": "Read {{instance}}",
+          "legendFormat": "Read {{node_name}}",
          "refId": "A",
          "step": 2
        },
@@ -1174,10 +1124,10 @@
            "type": "prometheus",
            "uid": "PBFA97CFB590B2093"
          },
-          "expr": "sum(label_replace(irate(node_disk_written_bytes_total{instance=~\".*$node_id.*\"}[$interval]), \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) by (instance)",
+          "expr": "sum(irate(node_disk_written_bytes_total[$interval]) * on(instance) group_left(node_name) node_meta{node_id=~\"$node_id\"}) by (node_name)",
          "format": "time_series",
          "intervalFactor": 2,
-          "legendFormat": "Written {{instance}}",
+          "legendFormat": "Written {{node_name}}",
          "refId": "B",
          "step": 2
        }
@@ -1228,7 +1178,7 @@
        "h": 7,
        "w": 12,
        "x": 0,
-        "y": 35
+        "y": 33
      },
      "hiddenSeries": false,
      "id": 18,
@@ -1245,6 +1195,7 @@
      },
      "lines": true,
      "linewidth": 1,
+      "links": [],
      "nullPointMode": "null as zero",
      "options": {
        "alertThreshold": true
@@ -1264,10 +1215,10 @@
            "type": "prometheus",
            "uid": "PBFA97CFB590B2093"
          },
-          "expr": "sum(label_replace(irate(node_disk_reads_completed_total{instance=~\".*$node_id.*\"}[$interval]), \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) by (instance)",
+          "expr": "sum(irate(node_disk_reads_completed_total[$interval]) * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"}) by (node_name)",
          "format": "time_series",
          "intervalFactor": 2,
-          "legendFormat": "Reads {{instance}}",
+          "legendFormat": "Reads {{node_name}}",
          "refId": "A",
          "step": 2
        },
@@ -1276,10 +1227,10 @@
            "type": "prometheus",
            "uid": "PBFA97CFB590B2093"
          },
-          "expr": "sum(label_replace(irate(node_disk_writes_completed_total{instance=~\".*$node_id.*\"}[$interval]), \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) by (instance)",
+          "expr": "sum(irate(node_disk_writes_completed_total[$interval]) * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"}) by (node_name)",
          "format": "time_series",
          "intervalFactor": 2,
-          "legendFormat": "Writes {{instance}}",
+          "legendFormat": "Writes {{node_name}}",
          "refId": "B",
          "step": 2
        }
@@ -1330,7 +1281,7 @@
        "h": 7,
        "w": 12,
        "x": 12,
-        "y": 35
+        "y": 33
      },
      "hiddenSeries": false,
      "id": 19,
@@ -1349,6 +1300,7 @@
      },
      "lines": true,
      "linewidth": 1,
+      "links": [],
      "nullPointMode": "null as zero",
      "options": {
        "alertThreshold": true
@@ -1368,10 +1320,10 @@
            "type": "prometheus",
            "uid": "PBFA97CFB590B2093"
          },
-          "expr": "(avg(label_replace(irate(node_cpu_seconds_total{mode=\"iowait\", instance=~\".*$node_id.*\"}[$interval]), \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\") * 100) by (instance))",
+          "expr": "(avg(irate(node_cpu_seconds_total{mode=\"iowait\"}[$interval])  * on(instance) group_left(node_name) node_meta{node_name=~\"$node_id\"} * 100) by (node_name))",
          "format": "time_series",
          "intervalFactor": 2,
-          "legendFormat": "{{instance}}",
+          "legendFormat": "{{node_name}}",
          "refId": "A",
          "step": 2
        }
@@ -1424,7 +1376,7 @@
        "h": 7,
        "w": 18,
        "x": 0,
-        "y": 42
+        "y": 40
      },
      "hiddenSeries": false,
      "id": 12,
@@ -1445,6 +1397,7 @@
      },
      "lines": true,
      "linewidth": 1,
+      "links": [],
      "nullPointMode": "null",
      "options": {
        "alertThreshold": true
@@ -1464,7 +1417,7 @@
            "type": "prometheus",
            "uid": "PBFA97CFB590B2093"
          },
-          "expr": "sum(rate(container_last_seen{instance=~\".*$node_id.*\"}[5m])) by (container_label_com_docker_swarm_service_name)",
+          "expr": "sum(rate(container_last_seen[5m]) * on(container_label_com_docker_swarm_node_id) group_left(node_name) node_meta{node_name=~\"${node_id}\"}) by (container_label_com_docker_swarm_service_name)",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 10,
@@ -1546,9 +1499,10 @@
        "h": 7,
        "w": 6,
        "x": 18,
-        "y": 42
+        "y": 40
      },
      "id": 7,
+      "links": [],
      "maxDataPoints": 100,
      "options": {
        "colorMode": "none",
@@ -1573,7 +1527,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "exemplar": true,
-          "expr": "count(rate(container_last_seen{instance=~\".*$node_id.*\"}[5m]))",
+          "expr": "count(rate(container_last_seen[5m]) * on(container_label_com_docker_swarm_node_id) group_left(node_name) node_meta{node_name=~\"$node_id\"})",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 2,
@@ -1646,9 +1600,10 @@
        "h": 7,
        "w": 24,
        "x": 0,
-        "y": 49
+        "y": 47
      },
      "id": 17,
+      "links": [],
      "options": {
        "legend": {
          "calcs": [],
@@ -1668,11 +1623,11 @@
            "type": "prometheus",
            "uid": "PBFA97CFB590B2093"
          },
-          "expr": "sum(label_replace(rate(container_network_receive_bytes_total{instance=~\".*$node_id.*\"}[$interval]), \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) by (instance)",
+          "expr": "sum(rate(container_network_receive_bytes_total[$interval]) * on(container_label_com_docker_swarm_node_id) group_left(node_name) node_meta{node_name=~\"${node_id}\"}) by (node_name)",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 2,
-          "legendFormat": "IN {{instance}}",
+          "legendFormat": "IN {{node_name}}",
          "refId": "A",
          "step": 2
        },
@@ -1681,11 +1636,11 @@
            "type": "prometheus",
            "uid": "PBFA97CFB590B2093"
          },
-          "expr": "- sum(label_replace(rate(container_network_transmit_bytes_total{instance=~\".*$node_id.*\"}[$interval]), \"instance\", \"$1\", \"instance\", \"^(?:(?:node|cadvisor|monitoring).)*(.*)$\")) by (instance)",
+          "expr": "- sum(rate(container_network_transmit_bytes_total[$interval]) * on(container_label_com_docker_swarm_node_id) group_left(node_name) node_meta{node_name=~\"${node_id}\"}) by (node_name)",
          "format": "time_series",
          "hide": false,
          "intervalFactor": 2,
-          "legendFormat": "OUT {{instance}}",
+          "legendFormat": "OUT {{node_name}}",
          "metric": "",
          "refId": "B",
          "step": 2
@@ -1703,7 +1658,7 @@
        "h": 9,
        "w": 24,
        "x": 0,
-        "y": 56
+        "y": 54
      },
      "id": 30,
      "options": {
@@ -1733,7 +1688,8 @@
    }
  ],
  "refresh": "",
-  "schemaVersion": 39,
+  "schemaVersion": 38,
+  "style": "dark",
  "tags": [
    "swarmprom",
    "prometheus",
@@ -1762,11 +1718,11 @@
        "name": "node_id",
        "options": [],
        "query": {
-          "query": "label_values(node_uname_info, instance)",
+          "query": "node_meta",
          "refId": "Prometheus-node_id-Variable-Query"
        },
-        "refresh": 2,
-        "regex": "/^(?:(?:node|cadvisor|monitoring).)*(.*)$/",
+        "refresh": 1,
+        "regex": "/node_name=\"([^\"]+)\"/",
        "skipUrlSync": false,
        "sort": 0,
        "type": "query"
@@ -1880,6 +1836,6 @@
  "timezone": "",
  "title": "Docker Swarm Nodes",
  "uid": "BPlb-Sgik",
-  "version": 7,
+  "version": 24,
  "weekStart": ""
-}
+}
@@ -116,12 +116,12 @@
            "uid": "PBFA97CFB590B2093"
          },
          "exemplar": false,
-          "expr": "time() - label_replace(process_start_time_seconds{instance=~\".*$domain.*\"} or process_start_time_seconds{domain=\"$domain\"}, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)",
+          "expr": "time() - process_start_time_seconds{job=\"$job\"}",
          "format": "time_series",
          "instant": true,
          "interval": "",
          "intervalFactor": 2,
-          "legendFormat": "{{ domain }}",
+          "legendFormat": "{{ instance }}",
          "refId": "A"
        }
      ],
@@ -200,7 +200,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "exemplar": true,
-          "expr": "sum(label_replace(increase(traefik_service_requests_total{code=\"499\", instance=~\".*$domain.*\"}[$interval]) or increase(traefik_service_requests_total{code=\"499\", domain=\"$domain\"}[$interval]), \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`))",
+          "expr": "sum(increase(traefik_service_requests_total{code=\"499\", instance=\"$instance\"}[$interval]))",
          "interval": "",
          "legendFormat": "",
          "refId": "A"
@@ -280,7 +280,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "exemplar": true,
-          "expr": "sum(label_replace(increase(traefik_service_requests_total{code=\"404\",protocol=~\"$protocol\", instance=~\".*$domain.*\"}[$interval]) or increase(traefik_service_requests_total{code=\"404\",protocol=~\"$protocol\", domain=\"$domain\"}[$interval]), \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`))",
+          "expr": "sum(increase(traefik_service_requests_total{code=\"404\",protocol=~\"$protocol\", instance=\"$instance\"}[$interval]))",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 2,
@@ -365,7 +365,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "exemplar": true,
-          "expr": "sum(label_replace(increase(traefik_service_requests_total{code=\"200\",protocol=~\"$protocol\", instance=~\".*$domain.*\"}[$interval]) or increase(traefik_service_requests_total{code=\"200\",protocol=~\"$protocol\", domain=\"$domain\"}[$interval]), \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`))",
+          "expr": "sum(increase(traefik_service_requests_total{code=\"200\",protocol=~\"$protocol\", instance=\"$instance\"}[$interval]))",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 2,
@@ -417,7 +417,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "exemplar": true,
-          "expr": "topk(5, sum(label_replace(traefik_service_requests_total{protocol=~\"$protocol\", instance=~\".*$domain.*\"} or traefik_service_requests_total{protocol=~\"$protocol\", domain=\"$domain\"}, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) by (code))",
+          "expr": "topk(5, sum(traefik_service_requests_total{protocol=~\"$protocol\", instance=\"$instance\"}) by (code))",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 2,
@@ -489,7 +489,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "exemplar": true,
-          "expr": "sum(label_replace(increase(traefik_service_requests_total{code=\"499\",method=\"GET\", instance=~\".*$domain.*\"}[$interval]) or increase(traefik_service_requests_total{code=\"499\",method=\"GET\", domain=\"$domain\"}[$interval]), \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) by (service)",
+          "expr": "sum(increase(traefik_service_requests_total{code=\"499\",method=\"GET\", instance=\"$instance\"}[$interval])) by (service)",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 2,
@@ -590,7 +590,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "exemplar": true,
-          "expr": "(sum(label_replace(traefik_entrypoint_request_duration_seconds_sum{protocol=~\"$protocol\", instance=~\".*$domain.*\"} or traefik_entrypoint_request_duration_seconds_sum{protocol=~\"$protocol\", domain=\"$domain\"}, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) / sum(label_replace(traefik_entrypoint_requests_total{protocol=~\"$protocol\", instance=~\".*$domain.*\"} or traefik_entrypoint_requests_total{protocol=~\"$protocol\", domain=\"$domain\"}, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) * 1000) - (sum(label_replace(traefik_service_request_duration_seconds_sum{protocol=~\"$protocol\", instance=~\".*$domain.*\"} or traefik_service_request_duration_seconds_sum{protocol=~\"$protocol\", domain=\"$domain\"}, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) / sum(label_replace(traefik_service_requests_total{protocol=~\"$protocol\", instance=~\".*$domain.*\"} or traefik_service_requests_total{protocol=~\"$protocol\", domain=\"$domain\"}, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) * 1000)",
+          "expr": "(sum(traefik_entrypoint_request_duration_seconds_sum{protocol=~\"$protocol\", instance=\"$instance\"}) / sum(traefik_entrypoint_requests_total{protocol=~\"$protocol\", instance=\"$instance\"}) * 1000) - (sum(traefik_service_request_duration_seconds_sum{protocol=~\"$protocol\", instance=\"$instance\"}) / sum(traefik_service_requests_total{protocol=~\"$protocol\", instance=\"$instance\"}) * 1000)",
          "format": "time_series",
          "instant": false,
          "interval": "",
@@ -709,7 +709,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "exemplar": true,
-          "expr": "sum(label_replace(traefik_entrypoint_request_duration_seconds_sum{instance=~\".*$domain.*\"} or traefik_entrypoint_request_duration_seconds_sum{domain=\"$domain\"}, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) / sum(label_replace(traefik_entrypoint_requests_total{instance=~\".*$domain.*\"} or traefik_entrypoint_requests_total{domain=\"$domain\"}, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) * 1000",
+          "expr": "sum(traefik_entrypoint_request_duration_seconds_sum{instance=\"$instance\"}) / sum(traefik_entrypoint_requests_total{instance=\"$instance\"}) * 1000",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 2,
@@ -781,7 +781,7 @@
          },
          "editorMode": "code",
          "exemplar": true,
-          "expr": "sum(label_replace(delta(traefik_service_requests_total{instance=~\".*$domain.*\"}[$interval]) or delta(traefik_service_requests_total{domain=\"$domain\"}[$interval]), \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`))",
+          "expr": "sum(delta(traefik_service_requests_total{instance=\"${instance:raw}\"}[$interval]))",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 2,
@@ -879,7 +879,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "exemplar": true,
-          "expr": "sum(label_replace(rate(traefik_service_requests_total{instance=~\".*$domain.*\"}[$interval]) or rate(traefik_service_requests_total{domain=\"$domain\"}[$interval]), \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`))",
+          "expr": "sum(rate(traefik_service_requests_total{instance=\"$instance\"}[$interval]))",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 2,
@@ -984,7 +984,7 @@
          },
          "editorMode": "code",
          "exemplar": true,
-          "expr": "sum(label_replace(rate(traefik_service_request_duration_seconds_sum{instance=~\".*$domain.*\"}[5m]) or rate(traefik_service_request_duration_seconds_sum{domain=\"$domain\"}[5m]), \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) by(service)",
+          "expr": "sum(rate(traefik_service_request_duration_seconds_sum{ instance=\"$instance\" }[5m])) by(service)",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 2,
@@ -1053,11 +1053,11 @@
            "uid": "PBFA97CFB590B2093"
          },
          "exemplar": true,
-          "expr": "label_replace(process_open_fds{instance=~\".*$domain.*\"} or process_open_fds{domain=\"$domain\"}, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)",
+          "expr": "process_open_fds{job=~\"$job\", instance=\"$instance\"}",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 2,
-          "legendFormat": "{{ domain }}",
+          "legendFormat": "{{ instance }}",
          "refId": "A",
          "step": 240
        }
@@ -1154,7 +1154,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "exemplar": true,
-          "expr": "sum(label_replace(rate(traefik_service_requests_total{protocol=~\"http|https\", instance=~\".*$domain.*\"}[$interval]) or rate(traefik_service_requests_total{protocol=~\"http|https\", domain=\"$domain\"}[$interval]), \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) by (service)",
+          "expr": "sum(rate(traefik_service_requests_total{protocol=~\"http|https\", instance=\"$instance\"}[$interval])) by (service)",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 2,
@@ -1255,7 +1255,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "exemplar": true,
-          "expr": "sum(label_replace(traefik_entrypoint_open_connections{instance=~\".*$domain.*\"} or traefik_entrypoint_open_connections{domain=\"$domain\"}, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) by (method)",
+          "expr": "sum(traefik_entrypoint_open_connections{instance=\"$instance\"}) by (method)",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 1,
@@ -1355,7 +1355,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "exemplar": true,
-          "expr": "sum(label_replace(traefik_service_open_connections{instance=~\".*$domain.*\"} or traefik_service_open_connections{domain=\"$domain\"}, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) by (method)",
+          "expr": "sum(traefik_service_open_connections{instance=\"$instance\"}) by (method)",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 1,
@@ -1459,7 +1459,7 @@
            "uid": "PBFA97CFB590B2093"
          },
          "exemplar": true,
-          "expr": "sum(label_replace(increase(traefik_service_requests_total{protocol=~\"$protocol\", instance=~\".*$domain.*\"}[$interval]) or increase(traefik_service_requests_total{protocol=~\"$protocol\", domain=\"$domain\"}[$interval]), \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`)) by (code)",
+          "expr": "sum(increase(traefik_service_requests_total{protocol=~\"$protocol\", instance=\"$instance\"}[$interval])) by (code)",
          "format": "time_series",
          "interval": "",
          "intervalFactor": 2,
@@ -1514,6 +1514,36 @@
  ],
  "templating": {
    "list": [
+      {
+        "current": {
+          "selected": false,
+          "text": "default",
+          "value": "default"
+        },
+        "datasource": {
+          "type": "prometheus",
+          "uid": "PBFA97CFB590B2093"
+        },
+        "definition": "",
+        "hide": 0,
+        "includeAll": false,
+        "label": "Job:",
+        "multi": false,
+        "name": "job",
+        "options": [],
+        "query": {
+          "query": "label_values(job)",
+          "refId": "Prometheus-job-Variable-Query"
+        },
+        "refresh": 1,
+        "regex": "",
+        "skipUrlSync": false,
+        "sort": 2,
+        "tagValuesQuery": "",
+        "tagsQuery": "",
+        "type": "query",
+        "useTags": false
+      },
      {
        "current": {
          "selected": true,
@@ -1539,7 +1569,7 @@
          "query": "label_values(traefik_service_requests_total, protocol)",
          "refId": "Prometheus-protocol-Variable-Query"
        },
-        "refresh": 2,
+        "refresh": 1,
        "regex": "",
        "skipUrlSync": false,
        "sort": 0,
@@ -1633,19 +1663,19 @@
          "type": "prometheus",
          "uid": "PBFA97CFB590B2093"
        },
-        "definition": "query_result(label_replace(traefik_config_reloads_total, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`))",
+        "definition": "label_values(instance)",
        "hide": 0,
        "includeAll": false,
-        "label": "Domain:",
+        "label": "Instance:",
        "multi": false,
-        "name": "domain",
+        "name": "instance",
        "options": [],
        "query": {
-          "query": "query_result(label_replace(traefik_config_reloads_total, \"domain\", \"$1\", \"instance\", `^metrics\\.(.+)$`))",
+          "query": "label_values(instance)",
          "refId": "PrometheusVariableQueryEditor-VariableQuery"
        },
-        "refresh": 2,
-        "regex": "/domain=\"([^\"]+)\"/",
+        "refresh": 1,
+        "regex": ".*8082",
        "skipUrlSync": false,
        "sort": 1,
        "tagValuesQuery": "",
@@ -21,7 +21,7 @@ tls_skip_verify_insecure = false
 allow_sign_up = true
 auto_login = true
 client_id = {{ env "OIDC_CLIENT_ID" }}
-client_secret = {{ secret "gf_oidc_secret" }}
+client_secret = {{ secret "grafana_oidc_client_secret" }}
 auth_url = {{ env "OIDC_AUTH_URL" }}
 token_url = {{ env "OIDC_TOKEN_URL" }}
 api_url = {{ env "OIDC_API_URL" }}
@@ -34,6 +34,7 @@ ingester:
  max_chunk_age: 1h           # All chunks will be flushed when they hit this age, default is 1h
  chunk_target_size: 1048576  # Loki will attempt to build chunks up to 1.5MB, flushing first if chunk_idle_period or max_chunk_age is reached first
  chunk_retain_period: 30s    # Must be greater than index read cache TTL if using an index cache (Default index read cache TTL is 5m)
+  max_transfer_retries: 0     # Chunk transfers disabled
  wal:
    dir: "/tmp/wal"

@@ -52,7 +53,7 @@ schema_config:
    - from: 2020-10-24
      store: boltdb-shipper
      object_store: filesystem
-      schema: v13
+      schema: v11
      index:
        prefix: index_
        period: 24h
@@ -62,6 +63,7 @@ storage_config:
    active_index_directory: /loki/boltdb-shipper-active
    cache_location: /loki/boltdb-shipper-cache
    cache_ttl: 24h         # Can be increased for faster performance over longer query periods, uses more disk space
+    shared_store: filesystem
  filesystem:
    directory: /loki/chunks
 {{ end }}
@@ -70,6 +72,7 @@ schema_config:
  configs:
  - from: 2020-11-25
    store: boltdb-shipper
+    object_store: aws
    schema: v11
    index:
      prefix: index_
@@ -86,7 +89,7 @@ storage_config:
   endpoint: {{ env "LOKI_AWS_ENDPOINT" }}
   region: {{ env "LOKI_AWS_REGION" }}
   access_key_id: {{ env "LOKI_ACCESS_KEY_ID" }}
-   secret_access_key: {{ secret "loki_aws_key" }}
+   secret_access_key: {{ secret "loki_aws_secret_access_key" }}
   bucketnames: {{ env "LOKI_BUCKET_NAMES" }}
   insecure: false
   sse_encryption: false
@@ -100,24 +103,19 @@ storage_config:

 compactor:
  working_directory: /loki/boltdb-shipper-compactor
+  shared_store: filesystem
  compaction_interval: 10m
  retention_enabled: true
  retention_delete_delay: 2h
  retention_delete_worker_count: 150
-{{ if eq (env "LOKI_STORAGE_FILESYSTEM") "1" }}
-  delete_request_store: filesystem
-{{ end }}
-{{ if eq (env "LOKI_STORAGE_S3") "1" }}
-  delete_request_store: aws
-{{ end }}

 limits_config:
+  enforce_metric_name: false
  reject_old_samples: true
  reject_old_samples_max_age: 168h
  retention_period: {{ env "LOKI_RETENTION_PERIOD" }}
  split_queries_by_interval: 24h
  max_query_parallelism: 100
-  allow_structured_metadata: false

 query_scheduler:
  max_outstanding_requests_per_tenant: 4096
@@ -125,6 +123,9 @@ query_scheduler:
 frontend:
  max_outstanding_per_tenant: 4096

+chunk_store_config:
+  max_look_back_period: 0s
+
 table_manager:
  retention_deletes_enabled: false
  retention_period: 0s
@@ -0,0 +1,11 @@
+#!/bin/sh -e
+
+NODE_NAME=$(cat /etc/nodename)
+
+mkdir -p /etc/node-exporter
+
+echo "node_meta{node_id=\"$NODE_ID\", container_label_com_docker_swarm_node_id=\"$NODE_ID\", node_name=\"$NODE_NAME\"} 1" > /etc/node-exporter/node-meta.prom
+
+set -- /bin/node_exporter "$@"
+
+exec "$@"
@@ -0,0 +1,37 @@
+server:
+  http_listen_port: 9080
+  grpc_listen_port: 0
+
+positions:
+  filename: /tmp/positions.yaml
+
+clients:
+  - url: {{ env "LOKI_PUSH_URL" }}
+    basic_auth:
+      username: admin
+      password: {{ secret "basic_auth" }}
+    external_labels:
+      hostname: {{ env "DOMAIN" }}
+
+scrape_configs:
+- job_name: system
+  static_configs:
+  - targets:
+      - localhost
+    labels:
+      job: varlogs
+      __path__: /var/log/*log
+
+- job_name: "docker"
+  docker_sd_configs:
+    - host: "unix:///var/run/docker.sock"
+      refresh_interval: "10s"
+  relabel_configs:
+    - source_labels: ['__meta_docker_container_name']
+      target_label: "container_name"
+    - source_labels: ['__meta_docker_container_id']
+      target_label: "container_id"
+    - source_labels: ['__meta_docker_container_label_com_docker_stack_namespace']
+      target_label: "stack_namespace"
+    - source_labels: ['__meta_docker_container_label_com_docker_swarm_service_name']
+      target_label: "service_name"
@@ -1,61 +0,0 @@
-BREAKING CHANGE
-Migration plan for upgrading from 1.6.0+v1.8.1.
-
-## 1. Reinsert secrets with shortened names
-
-Secret and config names were shortened to max 14 characters to prevent going over Docker's 64 character
-limit when STACK_NAME and VERSION are added to it.
-
- `abra app secret list <domain>` to see which secrets are missing under their new name
- `abra app cmd --local <domain> migrate_secret_names` to reinsert all of them automatically
-      (or manually: `abra app secret insert <domain> <secret_name> v1 <value>` per secret)
-
-## 2. If you use OIDC (moved to seperate compose file)
-
- Add to your .env: `COMPOSE_FILE="$COMPOSE_FILE:compose.grafana-oidc.yml"`
-
-## 3. If you use SMTP (moved to a seperate compose file)
-
- Add to your .env: `COMPOSE_FILE="$COMPOSE_FILE:compose.grafana-smtp.yml"`
-
-## 4. node_exporter/cadvisor/promtail replaced by Grafana Alloy
-
-Metrics collection changed from Prometheus scraping endpoints
-to Alloy pushing via `remote_write`/`loki push`.
-
- Remove `compose.promtail.yml`, `compose.expose-ports.yml` and
-      `compose.basic-auth.yml` from your .env if present. They no
-      longer exist. `compose.yml` now declares the `basic_auth` secret directly, so
-      `SECRET_BASIC_AUTH_VERSION` is always required.
- Add `PROMETHEUS_REMOTE_WRITE_URL=https://prometheus.$DOMAIN/api/v1/write`
-      (`$DOMAIN` if this host also runs `compose.prometheus.yml`, otherwise a remote
-      Prometheus' URL). Without this, Alloy collects no metrics at all.
- Add `LOKI_PUSH_URL` (existing var, still used) and pick a log source:
-      `JOURNALD=1` (systemd hosts), `SYSLOG_FILES=1` (non-systemd, tails
-      `/var/log/*log`), or `SYSLOG=1` + `compose.syslog.yml` (network syslog listener).
-
- If this host had its own `node.$DOMAIN`/`cadvisor.$DOMAIN` scrape target
-      configured on a central Prometheus, remove it. Those endpoints are gone.
- `scrape-config.example.yml` and the `add_node`/`add_domain` abra.sh commands
-      are gone. Replaced by label-based auto-discovery (see README).
- `docker stack deploy` doesn't prune removed services, so old `cadvisor`/
-      `promtail` containers keep running after a normal `abra app deploy`. Run
-      `abra app undeploy <domain>` then `abra app deploy <domain>` to clear them out.
- Diff your `.env` against the current `.env.sample`, to verify any other changes.
-
-### New: label-based metrics auto-discovery
-
-Alloy now auto-discovers and scrapes other Docker Swarm services on the same
-host/`proxy` network that opt in via `prometheus.io/scrape=true` deploy labels.
-See the README's "Auto-discovering metrics from other apps" section.
-
- If you scrape Traefik metrics: the old `metrics.traefik.$domain` pull-based
-      endpoint still works if you keep the scrape config in Prometheus and
-      existing dashboards keep showing its data, but it's recommended to get
-      Traefik onto the new label-based discovery.
-
-### Dashboards
-
-The Swarm, Stacks and Traefik dashboards were reworked to show old (pull-model)
-and new (Alloy push-model) data as one continuous line, so you don't lose history
-across the migration.
@@ -0,0 +1,4 @@
+- targets:
+  - 'example.org:8082'
+  - 'node.monitoring.example.org'
+  - 'cadvisor.monitoring.example.org'