Yet another monitoring stack ... This time its a all-in-one grafana/prometheus/loki/alloy stack. It's based heavily on the monitoring-lite stack, but has everything in one recipe included now. So you can deploy monitoring instances to only gather metrics / logs (alloy) and also deploy instances with the full monitoring stack (grafana/prometheus/loki) with the same recipe and just different .env configuration.

Metrics and logs are collected by Grafana Alloy and pushed to a central Prometheus/Loki (via remote_write/loki push). Every monitoring-ng instance runs its own Alloy, whether or not it also runs the central stack.

Category: Apps
Status: 2, beta
Image: grafana/grafana, 4, upstream
Healthcheck: 3
Backups: 1
Email: 3
Tests: No
SSO: 1

Setup: gathering-only host

Deploys just Alloy, pushing this host's own node/container metrics and logs to a central instance. Use this on every host you want metrics/logs from.

abra app new monitoring-ng --server gathering.org
abra app config monitoring.gathering.org

Point it at your central instance:

PROMETHEUS_REMOTE_WRITE_URL=https://prometheus.example.org/api/v1/write
LOKI_PUSH_URL=https://loki.example.org/loki/api/v1/push
JOURNALD=1   # or SYSLOG_FILES=1 / SYSLOG=1, see .env.sample

abra app secret insert monitoring.gathering.org basic_auth v1 <password>. Same username/password as the usersfile credential configured for Traefik's basicauth on the central node (see below). This is what Alloy authenticates with when pushing metrics/logs. Alloy defaults to username admin for this. Uf the Traefik usersfile uses a different username, set WRITE_BASIC_AUTH_USERNAME in this recipe to match.
abra app deploy monitoring.gathering.org

Setup: full monitoring stack (metrics/logs browser)

This is what a gathering host pushes into. It also runs its own Alloy, so it monitors itself too.

Configure DNS: monitoring.example.org, plus prometheus./loki./pushgateway. subdomains for whichever of those you enable below
Traefik on this node needs basic auth configured (BASIC_AUTH=1, see the Traefik recipe's "Configuring basic auth" section) — Prometheus/Loki/Pushgateway route through its basicauth@file middleware, so without it those endpoints won't work. Use the same username (default admin, see WRITE_BASIC_AUTH_USERNAME) and password you'll insert as the basic_auth secret below when generating Traefik's usersfile.
abra app config monitoring.example.org Uncomment compose.prometheus.yml (metrics), compose.loki.yml (logs) and compose.grafana.yml (dashboard)
abra app secret insert monitoring.example.org basic_auth v1 <password> — this is the password every gathering host's Alloy (including this instance's own) authenticates with; also what Traefik's basicauth expects on the public Prometheus/Loki/Pushgateway endpoints
abra app secret insert monitoring.example.org gf_adminpasswd v1 <password>
abra app deploy monitoring.example.org

Post-setup guide

configure the SMTP mailer under Alerting > Contact points
- edit the default contact point, choose "Alertmanager" as type & http://alertmanager:9093 as URL
- use the "Test" button to send a test mail. It should fire a request at the alertmanager & that should send a mail
from your dashboard panels, choose Edit > Alert to create alerts based on those panels

Additional features

Discovering metrics from other apps

Alloy auto-discovers and scrapes other Docker Swarm services running on the same host, on the proxy network, that opt in via labels. No manual scrape config needed. On the app's compose.yml:

deploy:
  labels:
    - "prometheus.io/scrape=true"       # required: opt in
    # - "prometheus.io/port=8082"       # optional: defaults to 80
    # - "prometheus.io/path=/metrics"   # optional: defaults to /metrics
    # - "prometheus.io/auth=basic"      # optional: basic auth, see below
    # - "prometheus.io/auth=bearer"     # optional: bearer token auth

Each scraped target gets instance (<service_name>.<slot>, stable across redeploys), domain (the service's stack namespace with underscores converted back to dots, e.g. traefik.example.com) and task_slot labels attached automatically.

If the target needs authentication, configure the monitoring-ng instance with a scrape-auth secret that will be used for targets having the auth label set:

COMPOSE_FILE="$COMPOSE_FILE:compose.scrape-auth.yml"
SECRET_SCRAPE_AUTH_VERSION=v1

abra app secret insert <domain> scrape_auth v1 <password-or-token>, then set the scraped app's prometheus.io/auth label to basic or bearer to match how it checks the secret.

Check discovered targets via alloy.example.org (needs compose.alloy-webui.yml, see below), or query the central Prometheus for up{job="<service_name>"}.

Manual scraping

For targets where auto-discovery doesn't work (e.g. not a Docker Swarm service on the proxy network, or missing labels): add them directly to Prometheus instead. Create a scrape config file:

- targets:
    - 'metrics.something-external.example.org'
    - 'app-without-labels.example.org'

and copy it into Prometheus' scrape config directory:

abra app cp monitoring.gathering.org targets.yml prometheus:/prometheus/scrape_configs/

Prometheus picks up files there automatically.

Alloy Web UI / Live Debugging

Alloy's own web UI isn't exposed by default, its HTTP server only listens on localhost inside its own container. To reach it from outside (e.g. to browse its component graph or use live debugging), add:

COMPOSE_FILE="$COMPOSE_FILE:compose.alloy-webui.yml"

This exposes it (behind the same basic-auth) at alloy.$DOMAIN.

To actually see what's being collected (live-tailing the metrics/logs flowing through each component, not just their config) also set LIVE_DEBUGGING=true. Only enable this while troubleshooting.

Logging from a docker host to loki server without anything else

$ docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions
$ echo '{
    "debug" : true,
    "log-driver": "loki",
    "log-opts": {
        "loki-url": "https://<user>:<secret>@loki.monitoring.example.org/loki/api/v1/push",
        "loki-batch-size": "400"
    }
}' > /etc/docker/daemon.json
$ systemctl restart docker.service

Setup Push Gateway

Enable in the env file by uncommenting the following lines:

## Prometheus Pushgateway
# COMPOSE_FILE="$COMPOSE_FILE:compose.pushgateway.yml"

abra app deploy monitoring.example.org

This will expose the pushgateway at https://pushgateway.${DOMAIN}, secured behind the same basic auth as the other services. After that you need to add the pushgateway.${DOMAIN} to the scrape config of Prometheus.

THX to the previous work of @decentral1se @knooflok @3wc @cellarspoon @mirsal

Adding Matrix as Alert Contact point

Enable the matrix-alertmanager-receiver:

COMPOSE_FILE="$COMPOSE_FILE:compose.matrix-alertmanager-receiver.yml"

Insert the matrix access token secret:

abra app secret insert monitoring.marx.klasse-methode.it matrix_token v1

Set required configurations:

GF_MATRIX_USER_ID=
GF_MATRIX_ROOM_ID=
GF_MATRIX_HOME_SERVER_URL=

Configure Alertmanager webhook and set the url to http://matrix-alertmanager-receiver:12345/alerts/<room-id>

Alerts

It is possible to enable the following alerts, by uncommenting the corresponding env variable:

node disk space: ALERT_NODE_DISK_SPACE_LEFT
node memory usage: ALERT_NODE_MEMORY_USAGE