eCommons/monitoring-ng
0
0
Fork 0
forked from coop-cloud/monitoring-ng
Code Pull Requests Activity

docs: rewrite README and release notes for the Alloy migration

Browse Source
This commit is contained in:
dannygroenewegen
2026-08-17 15:58:04 +02:00
parent 50fc916107
commit 17c4f6237b
2 changed files with 129 additions and 76 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+84 -56
View File
@@ -4,6 +4,7 @@ Yet another monitoring stack ...
This time its a all-in-one grafana/prometheus/loki/alloy stack.
It's based heavily on the [monitoring-lite](https://git.coopcloud.tech/coop-cloud/monitoring-lite) stack, but has everything in one recipe included now. So you can deploy monitoring instances to only gather metrics / logs (alloy) and also deploy instances with the full monitoring stack (grafana/prometheus/loki) with the same recipe and just different .env configuration.
Metrics and logs are collected by [Grafana Alloy](https://grafana.com/docs/alloy/latest/) and pushed to a central Prometheus/Loki (via `remote_write`/`loki push`). Every `monitoring-ng` instance runs its own Alloy, whether or not it also runs the central stack.
<!-- metadata -->
@@ -18,55 +19,93 @@ It's based heavily on the [monitoring-lite](https://git.coopcloud.tech/coop-clou
<!-- endmetadata -->
## Setup Metrics Gathering
## Setup: gathering-only host
Where gathering.org is the node you want to gather metrics from.
Deploys just Alloy, pushing this host's own node/container metrics and logs to a central instance. Use this on every host you want metrics/logs from.
1. Configure DNS
- cadvisor.monitoring.gathering.org
- node.monitoring.gathering.org
2. [Configure Traefik to use BasicAuth](https://git.coopcloud.tech/coop-cloud/traefik#configuring-wildcard-ssl-using-dns)
3. `abra app new monitoring-ng`
4. `abra app config monitoring.gathering.org` (for gathering only the main `compose.yml` is needed, nothing more.)
1. `abra app new monitoring-ng --server gathering.org`
2. `abra app config monitoring.gathering.org`
3. Point it at your central instance:
```
PROMETHEUS_REMOTE_WRITE_URL=https://prometheus.example.org/api/v1/write
LOKI_PUSH_URL=https://loki.example.org/loki/api/v1/push
JOURNALD=1 # or SYSLOG_FILES=1 / SYSLOG=1, see .env.sample
```
4. `abra app secret insert monitoring.gathering.org basic_auth v1 <password>`. Same username/password as the `usersfile` credential configured for Traefik's basicauth on the central node (see below). This is what Alloy authenticates with when pushing metrics/logs. Alloy defaults to username `admin` for this. Uf the Traefik `usersfile` uses a different username, set `WRITE_BASIC_AUTH_USERNAME` in this recipe to match.
5. `abra app deploy monitoring.gathering.org`
6. check that endpoints are up and basic-auth works
- cadvisor.monitoring.gathering.org
- node.monitoring.gathering.org
## Setup Metrics Browser
## Setup: full monitoring stack (metrics/logs browser)
This builds upon [Setup Metrics Gathering](#setup-metrics-grathering) so make sure you did that first.
This is what a gathering host pushes into. It also runs its own Alloy, so it monitors itself too.
1. Configure DNS
- monitoring.example.org
2. Setup monitoring stack
- `abra app config monitoring.example.org` Uncomment prometheus, loki and grafana
- `abra app secret insert monitoring.example.org basic_auth v1 <password>`
this needs the plaintext traefik basic-auth secret, not the hashed one!
- `abra app secret ls monitoring.example.org`
- `abra app deploy monitoring.example.org`
3. Add scrape config to prometheus
- `abra app cmd monitoring.example.org prometheus gathering.org`
- or manually
```
cp scrape-config.example.yml gathering.org.yml
# adjust domain
# mkdir scrape_configs
abra app cp monitoring.dev.local-it.cloud gathering.org.yml prometheus:/prometheus/scrape_configs/
```
1. Configure DNS: `monitoring.example.org`, plus `prometheus.`/`loki.`/`pushgateway.` subdomains for whichever of those you enable below
2. Traefik on this node needs basic auth configured (`BASIC_AUTH=1`, see the Traefik recipe's "Configuring basic auth" section) — Prometheus/Loki/Pushgateway route through its `basicauth@file` middleware, so without it those endpoints won't work. Use the same username (default `admin`, see `WRITE_BASIC_AUTH_USERNAME`) and password you'll insert as the `basic_auth` secret below when generating Traefik's `usersfile`.
3. `abra app config monitoring.example.org` Uncomment `compose.prometheus.yml` (metrics), `compose.loki.yml` (logs) and `compose.grafana.yml` (dashboard)
4. `abra app secret insert monitoring.example.org basic_auth v1 <password>` — this is the password every gathering host's Alloy (including this instance's own) authenticates with; also what Traefik's basicauth expects on the public Prometheus/Loki/Pushgateway endpoints
5. `abra app secret insert monitoring.example.org gf_adminpasswd v1 <password>`
6. `abra app deploy monitoring.example.org`
* check that all configured targets are up:
https://prometheus.monitoring.example.org/targets
### Post-setup guide
- configure the SMTP mailer under `Alerting > Contact points`
- edit the default contact point, choose "Alertmanager" as type & `http://alertmanager:9093` as URL
- use the "Test" button to send a test mail. It should fire a request at the alertmanager & that should send a mail
- from your dashboard panels, choose `Edit > Alert` to create alerts based on those panels
## Additional features
| Service | Authentication | Domain |
| ------------- | ------------------ | --------------------------------- |
| Grafana | Email / SSO | monitoring.example.org |
| Prometheus | traefik basic-auth | prometheus.monitoring.example.org |
| loki | traefik basic-auth | loki.monitoring.example.org |
| Cadvisor | traefik basic-auth | cadvisor.monitoring.example.org |
| Node Exporter | traefik basic-auth | node.monitoring.example.org |
### Discovering metrics from other apps
Alloy auto-discovers and scrapes other Docker Swarm services running on the same host, on the `proxy` network, that opt in via labels. No manual scrape config needed. On the app's `compose.yml`:
```yaml
deploy:
labels:
- "prometheus.io/scrape=true" # required: opt in
# - "prometheus.io/port=8082" # optional: defaults to 80
# - "prometheus.io/path=/metrics" # optional: defaults to /metrics
# - "prometheus.io/auth=basic" # optional: basic auth, see below
# - "prometheus.io/auth=bearer" # optional: bearer token auth
```
Each scraped target gets `instance` (`<service_name>.<slot>`, stable across redeploys), `domain` (the service's stack namespace with underscores converted back to dots, e.g. `traefik.example.com`) and `task_slot` labels attached automatically.
If the target needs authentication, configure the monitoring-ng instance with a scrape-auth secret that will be used for targets having the auth label set:
```
COMPOSE_FILE="$COMPOSE_FILE:compose.scrape-auth.yml"
SECRET_SCRAPE_AUTH_VERSION=v1
```
`abra app secret insert <domain> scrape_auth v1 <password-or-token>`, then set the scraped app's `prometheus.io/auth` label to `basic` or `bearer` to match how it checks the secret.
Check discovered targets via `alloy.example.org` (needs `compose.alloy-webui.yml`, see below), or query the central Prometheus for `up{job="<service_name>"}`.
### Manual scraping
For targets where auto-discovery doesn't work (e.g. not a Docker Swarm service on the `proxy` network, or missing labels): add them directly to Prometheus instead. Create a scrape config file:
```yaml
- targets:
- 'metrics.something-external.example.org'
- 'app-without-labels.example.org'
```
and copy it into Prometheus' scrape config directory:
```
abra app cp monitoring.gathering.org targets.yml prometheus:/prometheus/scrape_configs/
```
Prometheus picks up files there automatically.
### Alloy Web UI / Live Debugging
Alloy's own web UI isn't exposed by default, its HTTP server only listens
on localhost inside its own container. To reach it from outside (e.g. to
browse its component graph or use live debugging), add:
```
COMPOSE_FILE="$COMPOSE_FILE:compose.alloy-webui.yml"
```
This exposes it (behind the same basic-auth) at `alloy.$DOMAIN`.
To actually see what's being collected (live-tailing the metrics/logs
flowing through each component, not just their config) also set
`LIVE_DEBUGGING=true`. Only enable this while troubleshooting.
### Logging from a docker host to loki server without anything else
@@ -83,36 +122,25 @@ $ echo '{
$ systemctl restart docker.service
```
## Setup Push Gateway
### Setup Push Gateway
1. Enable in the env fiöle by uncommenting the following lines:
1. Enable in the env file by uncommenting the following lines:
```
## Prometheus Pushgateway
# COMPOSE_FILE="$COMPOSE_FILE:compose.pushgateway.yml"
```
2. `abra app deploy monitoring.example.org`
This will expose the pushgateway at `https://pushgateway.${DOMAIN}`.
It is secured behind the same basic auth as the other services.
After that you need to add the `pushgateway.${DOMAIN}` to the scare config.
This will expose the pushgateway at `https://pushgateway.${DOMAIN}`, secured behind the same basic auth as the other services.
After that you need to add the `pushgateway.${DOMAIN}` to the scrape config of Prometheus.
## Post-setup guide
- configure prometheus/loki/alertmanager as data sources in grafana under `Configuration > Data sources`
- for loki, you need to set a "Custom HTTP Header": `X-Scope-OrgID: fake`
- configure the SMTP mailer under `Alerting > Contact points`
- edit the default contact point, choose "Alertmanager" as type & `http://alertmanager:9093` as URL
- use the "Test" button to send a test mail. It should fire a request at the alertmanager & that should send a mail
- `abra app cp` your `scrap_configs: ...` into `/prometheus/scrape_configs` & log into your prometheus web UI to ensure they're working
- load your dashboards in manually under `Create > Dashboard`
- from your dashboard panels, choose `Edit > Alert` to create alerts based on those panels
---
THX to the previous work of @decentral1se @knooflok @3wc @cellarspoon @mirsal
## Adding Matrix as Alert Contact point
### Adding Matrix as Alert Contact point
1. Enable the [matrix-alertmanager-receiver](https://github.com/metio/matrix-alertmanager-receiver/):
```
@@ -132,7 +160,7 @@ GF_MATRIX_HOME_SERVER_URL=
```
4. Configure Alertmanager webhook and set the url to `http://matrix-alertmanager-receiver:12345/alerts/<room-id>`
## Alerts
### Alerts
It is possible to enable the following alerts, by uncommenting the corresponding env variable:
release/next
+45 -20
View File
@@ -1,36 +1,61 @@
1. OIDC was moved into a seperate compose file. If you have oidc configured you need to add the following line to you .env file:
BREAKING CHANGE
Migration plan for upgrading from 1.6.0+v1.8.1.
COMPOSE_FILE="$COMPOSE_FILE:compose.grafana-oidc.yml"
## 1. Reinsert secrets with shortened names
2. SMTP was moved into a seperate compose file. If you have smtp configured you need to add the following line to you .env file:
Secret and config names were shortened to max 14 characters to prevent going over Docker's 64 character
limit when STACK_NAME and VERSION are added to it.
COMPOSE_FILE="$COMPOSE_FILE:compose.grafana-smtp.yml"
- `abra app secret list <domain>` to see which secrets are missing under their new name
- `abra app cmd --local <domain> migrate_secret_names` to reinsert all of them automatically
(or manually: `abra app secret insert <domain> <secret_name> v1 <value>` per secret)
3. The scrape-config.example.yml file and add_node() command were updated to use a secure endpoint for the traefik metrics instead of http. This requires an updated Traefik recipe that publishes the metrics on https.
## 2. If you use OIDC (moved to seperate compose file)
4. Secret and config names were shortened to max 14 characters to prevent going over Docker's 64 character limit when STACK_NAME and VERSION are added to it.
When upgrading, you need to reinsert the secrets with their shorter names. Run `abra app secret list <domain>` to see which secrets aren't created on the server (because their name was shortened) and run `abra app secret insert <domain> <secret_name> v1 <value>` to reinsert them with the shorter name. Or you can use the migrate_secret_names function in abra.sh to reinsert all existing secrets with their shorter name automatically: `abra app cmd --local <domain> migrate_secret_names`
- Add to your .env: `COMPOSE_FILE="$COMPOSE_FILE:compose.grafana-oidc.yml"`
5. BREAKING CHANGE: node_exporter and cadvisor were replaced by Grafana Alloy. This is a bigger migration, check your .env carefully:
## 3. If you use SMTP (moved to a seperate compose file)
- `compose.promtail.yml`, `compose.expose-ports.yml` and `compose.basic-auth.yml` no longer exist, remove them from your `COMPOSE_FILE` chain if present. `compose.yml` now declares the `basic_auth` secret directly, so `SECRET_BASIC_AUTH_VERSION` is required unconditionally (it's no longer gated behind `compose.basic-auth.yml`).
- Add to your .env: `COMPOSE_FILE="$COMPOSE_FILE:compose.grafana-smtp.yml"`
- Metrics collection flipped from Prometheus pulling node_exporter/cadvisor to Alloy pushing to Prometheus via remote_write. Add:
## 4. node_exporter/cadvisor/promtail replaced by Grafana Alloy
PROMETHEUS_REMOTE_WRITE_URL=https://prometheus.$DOMAIN/api/v1/write
Metrics collection changed from Prometheus scraping endpoints
to Alloy pushing via `remote_write`/`loki push`.
($DOMAIN if this host also runs its own compose.prometheus.yml, or a remote Prometheus' URL otherwise). Without this variable Alloy collects no node/cadvisor/self metrics at all.
- Remove `compose.promtail.yml`, `compose.expose-ports.yml` and
`compose.basic-auth.yml` from your .env if present. They no
longer exist. `compose.yml` now declares the `basic_auth` secret directly, so
`SECRET_BASIC_AUTH_VERSION` is always required.
- Add `PROMETHEUS_REMOTE_WRITE_URL=https://prometheus.$DOMAIN/api/v1/write`
(`$DOMAIN` if this host also runs `compose.prometheus.yml`, otherwise a remote
Prometheus' URL). Without this, Alloy collects no metrics at all.
- Add `LOKI_PUSH_URL` (existing var, still used) and pick a log source:
`JOURNALD=1` (systemd hosts), `SYSLOG_FILES=1` (non-systemd, tails
`/var/log/*log`), or `SYSLOG=1` + `compose.syslog.yml` (network syslog listener).
- Log shipping still uses the existing `LOKI_PUSH_URL` variable, but you now also need to pick a log source:
- If this host had its own `node.$DOMAIN`/`cadvisor.$DOMAIN` scrape target
configured on a central Prometheus, remove it. Those endpoints are gone.
- `scrape-config.example.yml` and the `add_node`/`add_domain` abra.sh commands
are gone. Replaced by label-based auto-discovery (see README).
- `docker stack deploy` doesn't prune removed services, so old `cadvisor`/
`promtail` containers keep running after a normal `abra app deploy`. Run
`abra app undeploy <domain>` then `abra app deploy <domain>` to clear them out.
- Diff your `.env` against the current `.env.sample`, to verify any other changes.
JOURNALD=1 # systemd hosts (most modern distros)
# SYSLOG_FILES=1 # non-systemd hosts tailing /var/log/*log
# SYSLOG=1 # network syslog listener on :514, also needs compose.syslog.yml
### New: label-based metrics auto-discovery
- `LIVE_DEBUGGING` has no default and is required: set `LIVE_DEBUGGING=false` unless you want Alloy's live debugging UI enabled, otherwise the Alloy config fails to parse.
Alloy now auto-discovers and scrapes other Docker Swarm services on the same
host/`proxy` network that opt in via `prometheus.io/scrape=true` deploy labels.
See the README's "Auto-discovering metrics from other apps" section.
- If this host had its own `node.$DOMAIN`/`cadvisor.$DOMAIN` scrape target configured on a central Prometheus, remove it — those endpoints are gone once Alloy takes over.
- If you scrape Traefik metrics: the old `metrics.traefik.$domain` pull-based
endpoint still works if you keep the scrape config in Prometheus and
existing dashboards keep showing its data, but it's recommended to get
Traefik onto the new label-based discovery.
- `docker stack deploy` doesn't prune services that were removed from the compose file, so the old `cadvisor`/`promtail` containers keep running after a normal `abra app deploy`. Run `abra app undeploy <domain>` followed by `abra app deploy <domain>` to clear them out.
### Dashboards
- Easiest way to catch anything else: diff your existing `.env` against the current `.env.sample`, several variables were renamed, removed or added in this migration.
The Swarm, Stacks and Traefik dashboards were reworked to show old (pull-model)
and new (Alloy push-model) data as one continuous line, so you don't lose history
across the migration.