diff --git a/README.md b/README.md index 276afc5..9b09925 100644 --- a/README.md +++ b/README.md @@ -4,6 +4,7 @@ Yet another monitoring stack ... This time its a all-in-one grafana/prometheus/loki/alloy stack. It's based heavily on the [monitoring-lite](https://git.coopcloud.tech/coop-cloud/monitoring-lite) stack, but has everything in one recipe included now. So you can deploy monitoring instances to only gather metrics / logs (alloy) and also deploy instances with the full monitoring stack (grafana/prometheus/loki) with the same recipe and just different .env configuration. +Metrics and logs are collected by [Grafana Alloy](https://grafana.com/docs/alloy/latest/) and pushed to a central Prometheus/Loki (via `remote_write`/`loki push`). Every `monitoring-ng` instance runs its own Alloy, whether or not it also runs the central stack. @@ -18,55 +19,93 @@ It's based heavily on the [monitoring-lite](https://git.coopcloud.tech/coop-clou -## Setup Metrics Gathering +## Setup: gathering-only host -Where gathering.org is the node you want to gather metrics from. +Deploys just Alloy, pushing this host's own node/container metrics and logs to a central instance. Use this on every host you want metrics/logs from. -1. Configure DNS - - cadvisor.monitoring.gathering.org - - node.monitoring.gathering.org -2. [Configure Traefik to use BasicAuth](https://git.coopcloud.tech/coop-cloud/traefik#configuring-wildcard-ssl-using-dns) -3. `abra app new monitoring-ng` -4. `abra app config monitoring.gathering.org` (for gathering only the main `compose.yml` is needed, nothing more.) +1. `abra app new monitoring-ng --server gathering.org` +2. `abra app config monitoring.gathering.org` +3. Point it at your central instance: + ``` + PROMETHEUS_REMOTE_WRITE_URL=https://prometheus.example.org/api/v1/write + LOKI_PUSH_URL=https://loki.example.org/loki/api/v1/push + JOURNALD=1 # or SYSLOG_FILES=1 / SYSLOG=1, see .env.sample + ``` +4. `abra app secret insert monitoring.gathering.org basic_auth v1 `. Same username/password as the `usersfile` credential configured for Traefik's basicauth on the central node (see below). This is what Alloy authenticates with when pushing metrics/logs. Alloy defaults to username `admin` for this. Uf the Traefik `usersfile` uses a different username, set `WRITE_BASIC_AUTH_USERNAME` in this recipe to match. 5. `abra app deploy monitoring.gathering.org` -6. check that endpoints are up and basic-auth works - - cadvisor.monitoring.gathering.org - - node.monitoring.gathering.org -## Setup Metrics Browser +## Setup: full monitoring stack (metrics/logs browser) -This builds upon [Setup Metrics Gathering](#setup-metrics-grathering) so make sure you did that first. +This is what a gathering host pushes into. It also runs its own Alloy, so it monitors itself too. -1. Configure DNS - - monitoring.example.org -2. Setup monitoring stack - - `abra app config monitoring.example.org` Uncomment prometheus, loki and grafana - - `abra app secret insert monitoring.example.org basic_auth v1 ` - this needs the plaintext traefik basic-auth secret, not the hashed one! - - `abra app secret ls monitoring.example.org` - - `abra app deploy monitoring.example.org` -3. Add scrape config to prometheus - - `abra app cmd monitoring.example.org prometheus gathering.org` - - or manually - ``` - cp scrape-config.example.yml gathering.org.yml - # adjust domain - # mkdir scrape_configs - abra app cp monitoring.dev.local-it.cloud gathering.org.yml prometheus:/prometheus/scrape_configs/ - ``` +1. Configure DNS: `monitoring.example.org`, plus `prometheus.`/`loki.`/`pushgateway.` subdomains for whichever of those you enable below +2. Traefik on this node needs basic auth configured (`BASIC_AUTH=1`, see the Traefik recipe's "Configuring basic auth" section) — Prometheus/Loki/Pushgateway route through its `basicauth@file` middleware, so without it those endpoints won't work. Use the same username (default `admin`, see `WRITE_BASIC_AUTH_USERNAME`) and password you'll insert as the `basic_auth` secret below when generating Traefik's `usersfile`. +3. `abra app config monitoring.example.org` Uncomment `compose.prometheus.yml` (metrics), `compose.loki.yml` (logs) and `compose.grafana.yml` (dashboard) +4. `abra app secret insert monitoring.example.org basic_auth v1 ` — this is the password every gathering host's Alloy (including this instance's own) authenticates with; also what Traefik's basicauth expects on the public Prometheus/Loki/Pushgateway endpoints +5. `abra app secret insert monitoring.example.org gf_adminpasswd v1 ` +6. `abra app deploy monitoring.example.org` -* check that all configured targets are up: - https://prometheus.monitoring.example.org/targets +### Post-setup guide +- configure the SMTP mailer under `Alerting > Contact points` + - edit the default contact point, choose "Alertmanager" as type & `http://alertmanager:9093` as URL + - use the "Test" button to send a test mail. It should fire a request at the alertmanager & that should send a mail +- from your dashboard panels, choose `Edit > Alert` to create alerts based on those panels +## Additional features -| Service | Authentication | Domain | -| ------------- | ------------------ | --------------------------------- | -| Grafana | Email / SSO | monitoring.example.org | -| Prometheus | traefik basic-auth | prometheus.monitoring.example.org | -| loki | traefik basic-auth | loki.monitoring.example.org | -| Cadvisor | traefik basic-auth | cadvisor.monitoring.example.org | -| Node Exporter | traefik basic-auth | node.monitoring.example.org | +### Discovering metrics from other apps + +Alloy auto-discovers and scrapes other Docker Swarm services running on the same host, on the `proxy` network, that opt in via labels. No manual scrape config needed. On the app's `compose.yml`: + +```yaml +deploy: + labels: + - "prometheus.io/scrape=true" # required: opt in + # - "prometheus.io/port=8082" # optional: defaults to 80 + # - "prometheus.io/path=/metrics" # optional: defaults to /metrics + # - "prometheus.io/auth=basic" # optional: basic auth, see below + # - "prometheus.io/auth=bearer" # optional: bearer token auth +``` + +Each scraped target gets `instance` (`.`, stable across redeploys), `domain` (the service's stack namespace with underscores converted back to dots, e.g. `traefik.example.com`) and `task_slot` labels attached automatically. + +If the target needs authentication, configure the monitoring-ng instance with a scrape-auth secret that will be used for targets having the auth label set: +``` +COMPOSE_FILE="$COMPOSE_FILE:compose.scrape-auth.yml" +SECRET_SCRAPE_AUTH_VERSION=v1 +``` +`abra app secret insert scrape_auth v1 `, then set the scraped app's `prometheus.io/auth` label to `basic` or `bearer` to match how it checks the secret. + +Check discovered targets via `alloy.example.org` (needs `compose.alloy-webui.yml`, see below), or query the central Prometheus for `up{job=""}`. + +### Manual scraping + +For targets where auto-discovery doesn't work (e.g. not a Docker Swarm service on the `proxy` network, or missing labels): add them directly to Prometheus instead. Create a scrape config file: +```yaml +- targets: + - 'metrics.something-external.example.org' + - 'app-without-labels.example.org' +``` +and copy it into Prometheus' scrape config directory: +``` +abra app cp monitoring.gathering.org targets.yml prometheus:/prometheus/scrape_configs/ +``` +Prometheus picks up files there automatically. + +### Alloy Web UI / Live Debugging + +Alloy's own web UI isn't exposed by default, its HTTP server only listens +on localhost inside its own container. To reach it from outside (e.g. to +browse its component graph or use live debugging), add: +``` +COMPOSE_FILE="$COMPOSE_FILE:compose.alloy-webui.yml" +``` +This exposes it (behind the same basic-auth) at `alloy.$DOMAIN`. + +To actually see what's being collected (live-tailing the metrics/logs +flowing through each component, not just their config) also set +`LIVE_DEBUGGING=true`. Only enable this while troubleshooting. ### Logging from a docker host to loki server without anything else @@ -83,36 +122,25 @@ $ echo '{ $ systemctl restart docker.service ``` -## Setup Push Gateway +### Setup Push Gateway -1. Enable in the env fiöle by uncommenting the following lines: +1. Enable in the env file by uncommenting the following lines: ``` ## Prometheus Pushgateway # COMPOSE_FILE="$COMPOSE_FILE:compose.pushgateway.yml" ``` 2. `abra app deploy monitoring.example.org` -This will expose the pushgateway at `https://pushgateway.${DOMAIN}`. -It is secured behind the same basic auth as the other services. -After that you need to add the `pushgateway.${DOMAIN}` to the scare config. +This will expose the pushgateway at `https://pushgateway.${DOMAIN}`, secured behind the same basic auth as the other services. +After that you need to add the `pushgateway.${DOMAIN}` to the scrape config of Prometheus. -## Post-setup guide - -- configure prometheus/loki/alertmanager as data sources in grafana under `Configuration > Data sources` - - for loki, you need to set a "Custom HTTP Header": `X-Scope-OrgID: fake` -- configure the SMTP mailer under `Alerting > Contact points` - - edit the default contact point, choose "Alertmanager" as type & `http://alertmanager:9093` as URL - - use the "Test" button to send a test mail. It should fire a request at the alertmanager & that should send a mail -- `abra app cp` your `scrap_configs: ...` into `/prometheus/scrape_configs` & log into your prometheus web UI to ensure they're working -- load your dashboards in manually under `Create > Dashboard` -- from your dashboard panels, choose `Edit > Alert` to create alerts based on those panels --- THX to the previous work of @decentral1se @knooflok @3wc @cellarspoon @mirsal -## Adding Matrix as Alert Contact point +### Adding Matrix as Alert Contact point 1. Enable the [matrix-alertmanager-receiver](https://github.com/metio/matrix-alertmanager-receiver/): ``` @@ -132,7 +160,7 @@ GF_MATRIX_HOME_SERVER_URL= ``` 4. Configure Alertmanager webhook and set the url to `http://matrix-alertmanager-receiver:12345/alerts/` -## Alerts +### Alerts It is possible to enable the following alerts, by uncommenting the corresponding env variable: diff --git a/release/next b/release/next index 1e23cc1..7226256 100644 --- a/release/next +++ b/release/next @@ -1,36 +1,61 @@ -1. OIDC was moved into a seperate compose file. If you have oidc configured you need to add the following line to you .env file: +BREAKING CHANGE +Migration plan for upgrading from 1.6.0+v1.8.1. -COMPOSE_FILE="$COMPOSE_FILE:compose.grafana-oidc.yml" +## 1. Reinsert secrets with shortened names -2. SMTP was moved into a seperate compose file. If you have smtp configured you need to add the following line to you .env file: +Secret and config names were shortened to max 14 characters to prevent going over Docker's 64 character +limit when STACK_NAME and VERSION are added to it. -COMPOSE_FILE="$COMPOSE_FILE:compose.grafana-smtp.yml" +- `abra app secret list ` to see which secrets are missing under their new name +- `abra app cmd --local migrate_secret_names` to reinsert all of them automatically + (or manually: `abra app secret insert v1 ` per secret) -3. The scrape-config.example.yml file and add_node() command were updated to use a secure endpoint for the traefik metrics instead of http. This requires an updated Traefik recipe that publishes the metrics on https. +## 2. If you use OIDC (moved to seperate compose file) -4. Secret and config names were shortened to max 14 characters to prevent going over Docker's 64 character limit when STACK_NAME and VERSION are added to it. -When upgrading, you need to reinsert the secrets with their shorter names. Run `abra app secret list ` to see which secrets aren't created on the server (because their name was shortened) and run `abra app secret insert v1 ` to reinsert them with the shorter name. Or you can use the migrate_secret_names function in abra.sh to reinsert all existing secrets with their shorter name automatically: `abra app cmd --local migrate_secret_names` +- Add to your .env: `COMPOSE_FILE="$COMPOSE_FILE:compose.grafana-oidc.yml"` -5. BREAKING CHANGE: node_exporter and cadvisor were replaced by Grafana Alloy. This is a bigger migration, check your .env carefully: +## 3. If you use SMTP (moved to a seperate compose file) -- `compose.promtail.yml`, `compose.expose-ports.yml` and `compose.basic-auth.yml` no longer exist, remove them from your `COMPOSE_FILE` chain if present. `compose.yml` now declares the `basic_auth` secret directly, so `SECRET_BASIC_AUTH_VERSION` is required unconditionally (it's no longer gated behind `compose.basic-auth.yml`). +- Add to your .env: `COMPOSE_FILE="$COMPOSE_FILE:compose.grafana-smtp.yml"` -- Metrics collection flipped from Prometheus pulling node_exporter/cadvisor to Alloy pushing to Prometheus via remote_write. Add: +## 4. node_exporter/cadvisor/promtail replaced by Grafana Alloy -PROMETHEUS_REMOTE_WRITE_URL=https://prometheus.$DOMAIN/api/v1/write +Metrics collection changed from Prometheus scraping endpoints +to Alloy pushing via `remote_write`/`loki push`. - ($DOMAIN if this host also runs its own compose.prometheus.yml, or a remote Prometheus' URL otherwise). Without this variable Alloy collects no node/cadvisor/self metrics at all. +- Remove `compose.promtail.yml`, `compose.expose-ports.yml` and + `compose.basic-auth.yml` from your .env if present. They no + longer exist. `compose.yml` now declares the `basic_auth` secret directly, so + `SECRET_BASIC_AUTH_VERSION` is always required. +- Add `PROMETHEUS_REMOTE_WRITE_URL=https://prometheus.$DOMAIN/api/v1/write` + (`$DOMAIN` if this host also runs `compose.prometheus.yml`, otherwise a remote + Prometheus' URL). Without this, Alloy collects no metrics at all. +- Add `LOKI_PUSH_URL` (existing var, still used) and pick a log source: + `JOURNALD=1` (systemd hosts), `SYSLOG_FILES=1` (non-systemd, tails + `/var/log/*log`), or `SYSLOG=1` + `compose.syslog.yml` (network syslog listener). -- Log shipping still uses the existing `LOKI_PUSH_URL` variable, but you now also need to pick a log source: +- If this host had its own `node.$DOMAIN`/`cadvisor.$DOMAIN` scrape target + configured on a central Prometheus, remove it. Those endpoints are gone. +- `scrape-config.example.yml` and the `add_node`/`add_domain` abra.sh commands + are gone. Replaced by label-based auto-discovery (see README). +- `docker stack deploy` doesn't prune removed services, so old `cadvisor`/ + `promtail` containers keep running after a normal `abra app deploy`. Run + `abra app undeploy ` then `abra app deploy ` to clear them out. +- Diff your `.env` against the current `.env.sample`, to verify any other changes. -JOURNALD=1 # systemd hosts (most modern distros) -# SYSLOG_FILES=1 # non-systemd hosts tailing /var/log/*log -# SYSLOG=1 # network syslog listener on :514, also needs compose.syslog.yml +### New: label-based metrics auto-discovery -- `LIVE_DEBUGGING` has no default and is required: set `LIVE_DEBUGGING=false` unless you want Alloy's live debugging UI enabled, otherwise the Alloy config fails to parse. +Alloy now auto-discovers and scrapes other Docker Swarm services on the same +host/`proxy` network that opt in via `prometheus.io/scrape=true` deploy labels. +See the README's "Auto-discovering metrics from other apps" section. -- If this host had its own `node.$DOMAIN`/`cadvisor.$DOMAIN` scrape target configured on a central Prometheus, remove it — those endpoints are gone once Alloy takes over. +- If you scrape Traefik metrics: the old `metrics.traefik.$domain` pull-based + endpoint still works if you keep the scrape config in Prometheus and + existing dashboards keep showing its data, but it's recommended to get + Traefik onto the new label-based discovery. -- `docker stack deploy` doesn't prune services that were removed from the compose file, so the old `cadvisor`/`promtail` containers keep running after a normal `abra app deploy`. Run `abra app undeploy ` followed by `abra app deploy ` to clear them out. +### Dashboards -- Easiest way to catch anything else: diff your existing `.env` against the current `.env.sample`, several variables were renamed, removed or added in this migration. +The Swarm, Stacks and Traefik dashboards were reworked to show old (pull-model) +and new (Alloy push-model) data as one continuous line, so you don't lose history +across the migration.