177 lines
8.1 KiB
Markdown
177 lines
8.1 KiB
Markdown
# monitoring-ng
|
|
|
|
Yet another monitoring stack ...
|
|
This time its a all-in-one grafana/prometheus/loki/alloy stack.
|
|
It's based heavily on the [monitoring-lite](https://git.coopcloud.tech/coop-cloud/monitoring-lite) stack, but has everything in one recipe included now. So you can deploy monitoring instances to only gather metrics / logs (alloy) and also deploy instances with the full monitoring stack (grafana/prometheus/loki) with the same recipe and just different .env configuration.
|
|
|
|
Metrics and logs are collected by [Grafana Alloy](https://grafana.com/docs/alloy/latest/) and pushed to a central Prometheus/Loki (via `remote_write`/`loki push`). Every `monitoring-ng` instance runs its own Alloy, whether or not it also runs the central stack.
|
|
|
|
<!-- metadata -->
|
|
- **Maintainer**: [@coop-cloud/monitoring-ng-maintainers](https://git.coopcloud.tech/org/coop-cloud/teams/monitoring-ng-maintainers)
|
|
- **Category**: Apps
|
|
- **Status**: 2, beta
|
|
- **Image**: [`grafana/grafana`](https://hub.docker.com/r/grafana/grafana), 4, upstream
|
|
- **Healthcheck**: 3
|
|
- **Backups**: 1
|
|
- **Email**: 3
|
|
- **Tests**: No
|
|
- **SSO**: 1
|
|
|
|
<!-- endmetadata -->
|
|
|
|
## Setup: gathering-only host
|
|
|
|
Deploys just Alloy, pushing this host's own node/container metrics and logs to a central instance. Use this on every host you want metrics/logs from.
|
|
|
|
1. `abra app new monitoring-ng --server gathering.org`
|
|
2. `abra app config monitoring.gathering.org`
|
|
3. Point it at your central instance:
|
|
```
|
|
PROMETHEUS_REMOTE_WRITE_URL=https://prometheus.example.org/api/v1/write
|
|
LOKI_PUSH_URL=https://loki.example.org/loki/api/v1/push
|
|
JOURNALD=1 # or SYSLOG_FILES=1 / SYSLOG=1, see .env.sample
|
|
```
|
|
4. `abra app secret insert monitoring.gathering.org basic_auth v1 <password>`. Same username/password as the `usersfile` credential configured for Traefik's basicauth on the central node (see below). This is what Alloy authenticates with when pushing metrics/logs. Alloy defaults to username `admin` for this. Uf the Traefik `usersfile` uses a different username, set `WRITE_BASIC_AUTH_USERNAME` in this recipe to match.
|
|
5. `abra app deploy monitoring.gathering.org`
|
|
|
|
## Setup: full monitoring stack (metrics/logs browser)
|
|
|
|
This is what a gathering host pushes into. It also runs its own Alloy, so it monitors itself too.
|
|
|
|
1. Configure DNS: `monitoring.example.org`, plus `prometheus.`/`loki.`/`pushgateway.` subdomains for whichever of those you enable below
|
|
2. Traefik on this node needs basic auth configured (`BASIC_AUTH=1`, see the Traefik recipe's "Configuring basic auth" section) — Prometheus/Loki/Pushgateway route through its `basicauth@file` middleware, so without it those endpoints won't work. Use the same username (default `admin`, see `WRITE_BASIC_AUTH_USERNAME`) and password you'll insert as the `basic_auth` secret below when generating Traefik's `usersfile`.
|
|
3. `abra app config monitoring.example.org` Uncomment `compose.prometheus.yml` (metrics), `compose.loki.yml` (logs) and `compose.grafana.yml` (dashboard)
|
|
4. `abra app secret insert monitoring.example.org basic_auth v1 <password>` — this is the password every gathering host's Alloy (including this instance's own) authenticates with; also what Traefik's basicauth expects on the public Prometheus/Loki/Pushgateway endpoints
|
|
5. `abra app secret insert monitoring.example.org gf_adminpasswd v1 <password>`
|
|
6. `abra app deploy monitoring.example.org`
|
|
|
|
### Post-setup guide
|
|
|
|
- configure the SMTP mailer under `Alerting > Contact points`
|
|
- edit the default contact point, choose "Alertmanager" as type & `http://alertmanager:9093` as URL
|
|
- use the "Test" button to send a test mail. It should fire a request at the alertmanager & that should send a mail
|
|
- from your dashboard panels, choose `Edit > Alert` to create alerts based on those panels
|
|
|
|
## Additional features
|
|
|
|
### Scrape docker events
|
|
|
|
Just uncomment the following line in your .env to collect docker events using
|
|
the official docker-cli image and the docker socket:
|
|
```
|
|
COMPOSE_FILE="$COMPOSE_FILE:compose.docker-events.yml"
|
|
```
|
|
|
|
### Discovering metrics from other apps
|
|
|
|
Alloy auto-discovers and scrapes other Docker Swarm services running on the same host, on the `proxy` network, that opt in via labels. No manual scrape config needed. On the app's `compose.yml`:
|
|
|
|
```yaml
|
|
deploy:
|
|
labels:
|
|
- "prometheus.io/scrape=true" # required: opt in
|
|
# - "prometheus.io/port=8082" # optional: defaults to 80
|
|
# - "prometheus.io/path=/metrics" # optional: defaults to /metrics
|
|
# - "prometheus.io/auth=basic" # optional: basic auth, see below
|
|
# - "prometheus.io/auth=bearer" # optional: bearer token auth
|
|
```
|
|
|
|
Each scraped target gets `instance` (`<service_name>.<slot>`, stable across redeploys), `domain` (the service's stack namespace with underscores converted back to dots, e.g. `traefik.example.com`) and `task_slot` labels attached automatically.
|
|
|
|
If the target needs authentication, configure the monitoring-ng instance with a scrape-auth secret that will be used for targets having the auth label set:
|
|
```
|
|
COMPOSE_FILE="$COMPOSE_FILE:compose.scrape-auth.yml"
|
|
SECRET_SCRAPE_AUTH_VERSION=v1
|
|
```
|
|
`abra app secret insert <domain> scrape_auth v1 <password-or-token>`, then set the scraped app's `prometheus.io/auth` label to `basic` or `bearer` to match how it checks the secret.
|
|
|
|
Check discovered targets via `alloy.example.org` (needs `compose.alloy-webui.yml`, see below), or query the central Prometheus for `up{job="<service_name>"}`.
|
|
|
|
### Manual scraping
|
|
|
|
For targets where auto-discovery doesn't work (e.g. not a Docker Swarm service on the `proxy` network, or missing labels): add them directly to Prometheus instead. Create a scrape config file:
|
|
```yaml
|
|
- targets:
|
|
- 'metrics.something-external.example.org'
|
|
- 'app-without-labels.example.org'
|
|
```
|
|
and copy it into Prometheus' scrape config directory:
|
|
```
|
|
abra app cp monitoring.gathering.org targets.yml prometheus:/prometheus/scrape_configs/
|
|
```
|
|
Prometheus picks up files there automatically.
|
|
|
|
### Alloy Web UI / Live Debugging
|
|
|
|
Alloy's own web UI isn't exposed by default, its HTTP server only listens
|
|
on localhost inside its own container. To reach it from outside (e.g. to
|
|
browse its component graph or use live debugging), add:
|
|
```
|
|
COMPOSE_FILE="$COMPOSE_FILE:compose.alloy-webui.yml"
|
|
```
|
|
This exposes it (behind the same basic-auth) at `alloy.$DOMAIN`.
|
|
|
|
To actually see what's being collected (live-tailing the metrics/logs
|
|
flowing through each component, not just their config) also set
|
|
`LIVE_DEBUGGING=true`. Only enable this while troubleshooting.
|
|
|
|
### Logging from a docker host to loki server without anything else
|
|
|
|
```
|
|
$ docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions
|
|
$ echo '{
|
|
"debug" : true,
|
|
"log-driver": "loki",
|
|
"log-opts": {
|
|
"loki-url": "https://<user>:<secret>@loki.monitoring.example.org/loki/api/v1/push",
|
|
"loki-batch-size": "400"
|
|
}
|
|
}' > /etc/docker/daemon.json
|
|
$ systemctl restart docker.service
|
|
```
|
|
|
|
### Setup Push Gateway
|
|
|
|
1. Enable in the env file by uncommenting the following lines:
|
|
```
|
|
## Prometheus Pushgateway
|
|
# COMPOSE_FILE="$COMPOSE_FILE:compose.pushgateway.yml"
|
|
```
|
|
2. `abra app deploy monitoring.example.org`
|
|
|
|
This will expose the pushgateway at `https://pushgateway.${DOMAIN}`, secured behind the same basic auth as the other services.
|
|
After that you need to add the `pushgateway.${DOMAIN}` to the scrape config of Prometheus.
|
|
|
|
|
|
|
|
---
|
|
|
|
THX to the previous work of @decentral1se @knooflok @3wc @cellarspoon @mirsal
|
|
|
|
### Adding Matrix as Alert Contact point
|
|
|
|
1. Enable the [matrix-alertmanager-receiver](https://github.com/metio/matrix-alertmanager-receiver/):
|
|
```
|
|
COMPOSE_FILE="$COMPOSE_FILE:compose.matrix-alertmanager-receiver.yml"
|
|
```
|
|
|
|
2. Insert the matrix access token secret:
|
|
```
|
|
abra app secret insert monitoring.marx.klasse-methode.it matrix_token v1
|
|
```
|
|
|
|
3. Set required configurations:
|
|
```
|
|
GF_MATRIX_USER_ID=
|
|
GF_MATRIX_ROOM_ID=
|
|
GF_MATRIX_HOME_SERVER_URL=
|
|
```
|
|
4. Configure Alertmanager webhook and set the url to `http://matrix-alertmanager-receiver:12345/alerts/<room-id>`
|
|
|
|
### Alerts
|
|
|
|
It is possible to enable the following alerts, by uncommenting the corresponding env variable:
|
|
|
|
- node disk space: `ALERT_NODE_DISK_SPACE_LEFT`
|
|
- node memory usage: `ALERT_NODE_MEMORY_USAGE`
|