yet another try on the monitoring stack
You can not select more than 25 topics Topics must start with a letter or number, can include dashes ('-') and can be up to 35 characters long.
Philipp Rothmann 8c82943289
fix: config version
4 days ago
.env.sample chore: remove junk 4 days ago
.gitignore chore: update readme 4 days ago fix: config version 4 days ago
compose.grafana.yml fix: grafana envs 4 days ago
compose.prometheus.yml chore: publish 120 release 5 days ago
compose.promtail.yml chore: publish 120 release 5 days ago
compose.yml chore: publish 0.1.0+v1.5.0 release 4 days ago
grafana_custom.ini fix: grafana envs 4 days ago


Yet another monitoring stack ... This time its a all-in-one grafana/prometheus/loki/node_exporter/cadvisor/promtail stack. It's based heavily on the monitoring-lite stack, but has everything in one recipe included now. So you can deploy monitoring instances to only gather metrics / logs (node_exporter/cadvisor/promtail) and also deploy instances with the full monitoring stack (grafana/prometheus/loki) with the same recipe and just different .env configuration.

  • Category: Apps
  • Status: 2, beta
  • Image: grafana/grafana, 4, upstream
  • Healthcheck: 3
  • Backups: 1
  • Email: 3
  • Tests: No
  • SSO: 1

Setup a Metrics Gathering

Where is the node you want to gather metrics from.

  1. Configure DNS
  1. Configure Traefik to use BasicAuth
  • abra app config uncomment
  • Generate userslist with httpasswd hashed password abra app secret insert userslist v1 'admin:<hashed-secret>' make sure there is no whitespace in between admin:<hashed-secret>, it seems to break stuff...
  • abra app deploy -f traefik
  1. abra app new monitoring-ng
  2. abra app config for gathering only the main compose.yml is needed, nothing more.
  3. abra app deploy
  4. check that endpoints are up and basic-auth works

Setup Metrics Browser

  1. Configure DNS
  2. Setup monitoring stack
    • abra app new monitoring-ng
    • abra app config Uncomment all the stuff
    • abra app secret insert basic_auth_admin_password v1 <secret> this needs the plaintext traefik basic-auth secret, not the hashed one!
    • abra app secret ls
    • abra app deploy
  3. add scrape config to prometheus
  • abra app cmd prometheus
  • or manually
    cp scrape-config.example.yml
    # adjust domain
    # mkdir scrape_configs
    abra app cp prometheus:/prometheus/scrape_configs/
Service Authentication Domain
Grafana Email / SSO
Prometheus traefik basic-auth
loki traefik basic-auth
Cadvisor traefik basic-auth
Node Exporter traefik basic-auth

Logging from a docker host to loki server without anything else

$ docker plugin install grafana/loki-docker-driver:latest --alias loki --grant-all-permissions
$ echo '{
    "debug" : true,
    "log-driver": "loki",
    "log-opts": {
        "loki-url": "https://<user>:<secret>",
        "loki-batch-size": "400"
}' > /etc/docker/daemon.json
$ systemctl restart docker.service

Post-setup guide

  • configure prometheus/loki/alertmanager as data sources in grafana under Configuration > Data sources
    • for loki, you need to set a "Custom HTTP Header": X-Scope-OrgID: fake
  • configure the SMTP mailer under Alerting > Contact points
    • edit the default contact point, choose "Alertmanager" as type & http://alertmanager:9093 as URL
    • use the "Test" button to send a test mail. It should fire a request at the alertmanager & that should send a mail
  • abra app cp your scrap_configs: ... into /prometheus/scrape_configs & log into your prometheus web UI to ensure they're working
  • load your dashboards in manually under Create > Dashboard
  • from your dashboard panels, choose Edit > Alert to create alerts based on those panels

THX to the previous work of @decentral1se @knooflok @3wc @cellarspoon @mirsal