BREAKING CHANGES: replace promtail and cadvisor for alloy #21

Open
fauno wants to merge 59 commits from alloy into main
59 Commits
Author SHA1 Message Date
dannygroenewegen 382f6353fc fix: strip monitoring. prefix from instance labels for cleaner dashboard views 2026-08-20 16:50:25 +02:00
dannygroenewegen c8f74e2a3c fix: refresh dashboard variables on time range change, not just page load 2026-08-20 14:07:10 +02:00
dannygroenewegen 6dbb6c5f55 fix: mount containerd socket read-only, avoid abra#900 2026-08-17 16:09:44 +02:00
dannygroenewegen 17c4f6237b docs: rewrite README and release notes for the Alloy migration 2026-08-17 15:58:04 +02:00
dannygroenewegen 50fc916107 feat: add healthchecks to Alloy, Prometheus and Pushgateway 2026-08-17 15:08:08 +02:00
dannygroenewegenandClaude Sonnet 5 7480a8d6ab fix abra linting secret length
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
2026-08-17 14:59:03 +02:00
dannygroenewegen 067f013fb7 fix: don't expose Alloy's web UI by default, but optionally with compose.alloy-webui.yml 2026-08-17 14:48:44 +02:00
dannygroenewegen 536fa7f37d fix Traefik dashboard 2026-08-17 13:37:27 +02:00
dannygroenewegen fafd1f89f0 fix: discover containers by task instead of service for scraping
Swarm service discovery misses dnsrr-mode services like traefik; switch
to swarm task discovery, scoped to this host, with a stable instance
label across redeploys and a default port when none is labeled.
2026-08-10 21:24:38 +02:00
dannygroenewegen 7ab6cc74be fix: make swarm/stacks dashboards work with old and new metric data
Replace node_meta-based joins with label_replace() of the instance
label, so old (pull-model) and new (Alloy push) series for the same
host render as one continuous series. Each rewritten query has a TODO
noting the simpler form to use once old data ages out of retention.
2026-08-07 16:57:39 +02:00
dannygroenewegen 0f989c4567 fix: drop unfiltered discovery.docker.containers.targets from default scrape
prometheus.exporter.cadvisor.docker.targets already reports resource
metrics for every container on the host. The unfiltered discovery just
added targets with internal overlay IPs, most of which failed to
scrape anything useful.
2026-08-07 15:52:31 +02:00
dannygroenewegen 402e022114 feat: separate scrape_auth secret, configurable usernames
Added an optional scrape_auth secret for authenticating scrapes of
containers that opt in via prometheus.io/auth labels, no longer
reusing the basic_auth secret meant for Prometheus/Loki writes.

Made usernames for both write endpoints and authenticated scraping
configurable in .env.
2026-08-07 15:52:19 +02:00
dannygroenewegen 7f3d98fd9f - Restore GRAFANA_DOMAIN
- Start documenting the migration in release/next
- Various cleanups
2026-08-07 14:47:22 +02:00
fauno 627d988902 fix: upgrade to grafana 13.0.6 2026-08-07 09:24:53 -03:00
fauno e75fa4487c fix: upgrade alloy to 1.18.1 2026-08-06 12:37:25 -03:00
fauno 5f3f9c957c fix: upgrade to grafana 13.0.5 2026-08-04 10:48:47 -03:00
fauno 7818c89ad7 fix: provide access to containerd socket toolshed/abra#900 2026-07-31 13:57:43 -03:00
fauno b47f1b021f fix: upgrade grafana to v13.0.3 2026-06-27 14:43:43 -03:00
fauno c6efc69859 fix: upgrade alloy to v1.17.0 2026-06-27 14:42:11 -03:00
fauno 6c529f3527 fix: upgrade to grafana v12.4.5 2026-06-27 14:37:58 -03:00
fauno 01b398ddea fix: unneeded flag 2026-06-27 14:24:59 -03:00
fauno 30ad8e54f6 fix: upgrade to alloy v1.16.3 2026-06-27 14:20:32 -03:00
fauno c9910eabf4 Merge branch 'main' of https://git.coopcloud.tech/coop-cloud/monitoring-ng into alloy 2026-06-20 23:59:20 -03:00
fauno 1d9eb10004 improve alloy config and match with main branch (#24)
Reviewed-on: #24
Reviewed-by: fauno <fauno@sutty.coop.ar>
2026-06-17 12:53:08 +00:00
fauno 23acf56637 fix: filter by proxy network 2026-06-16 22:01:50 -03:00
fauno 03227f1907 fix: send container metrics directly to prometheus 2026-06-16 22:01:05 -03:00
fauno d085c66d68 fix: needed labels come from docker swarm 2026-06-16 22:00:42 -03:00
fauno 1970061ff8 feat: live debugging alloy 2026-06-16 21:59:50 -03:00
fauno fa76179987 feat: enable alloy web ui 2026-06-16 21:20:29 -03:00
fauno 64cb07a4a2 feat: bearer auth support 2026-06-16 20:03:17 -03:00
dannygroenewegen e247677433 feat: scrape metrics from containers via Docker label discovery
Containers opt in with prometheus.io/scrape=true and optionally set
prometheus.io/port, prometheus.io/path, and prometheus.io/auth=basic.
2026-06-14 21:38:43 +02:00
dannygroenewegen f2310f2b86 improve alloy config and match with main branch
- Restrict Alloy UI to loopback
- Narrow volume mounts: drop /dev, reduce /var/run to docker.sock:ro
- Replace HTTP scrape of :12345 with prometheus.exporter.self
Match with main branch (node-exporter / promtail / cadvisor):
- Add docker_only and explicit enabled_metrics to cadvisor exporter
- Match node-exporter collector config
- Match promtail relabeling (container_name, container_id, stack_namespace,
  service_name) and external hostname label
- Add SYSLOG_FILES option to tail /var/log/*log (matches promtail)
- Fix journal path and syslog listener address
2026-06-13 22:07:55 +02:00
fauno f2711fa16e fix: upgrades 2026-06-03 00:19:41 -03:00
fauno 2870b9486c fix: use the actual health check path 2026-06-02 21:06:05 -03:00
fauno 3a1fabe4f9 fix: prevent redirections on health check 2026-06-02 21:04:56 -03:00
fauno a358837922 wip: relabel syslog according to docs 2026-06-02 21:00:34 -03:00
fauno dd0a0c1bb0 fixup! feat: read syslog 2026-06-02 20:19:45 -03:00
fauno 31cabc36ae fix: prevent traefik deprecation warnings 2026-06-02 19:16:49 -03:00
fauno d25986d5cb fix: README 2026-06-02 18:51:10 -03:00
fauno f8f8004445 feat: read syslog 2026-06-02 18:50:41 -03:00
fauno aa05d022da feat: optionally push to prometheus and loki 2026-06-02 18:50:20 -03:00
fauno fb52a76247 BREAKING CHANGE: deprecate node-exporter 2026-06-02 18:49:05 -03:00
fauno 2e2a52eae0 BREAKING CHANGE: deprecate promtail 2026-06-02 18:48:20 -03:00
fauno 48419d5afa fixup! BREAKING CHANGE: no need to expose exporters 2026-06-02 18:46:02 -03:00
fauno a0a6e2c509 fix: basic auth secret is always needed 2026-06-02 18:44:32 -03:00
fauno 024f2a8aec feat: send docker logs to loki 2026-06-02 18:39:24 -03:00
fauno 38095e23fa BREAKING CHANGE: no need to expose exporters 2026-06-02 18:37:56 -03:00
fauno 641161329e fix: grafana alternate domain doesn't work
the variable is not expanded and the domain name label ends up as a
literal "$DOMAIN".
2026-06-02 18:00:00 -03:00
fauno cdacfd035e fix: prometheus querying panel is accessible through basic auth 2026-06-02 17:52:25 -03:00
fauno b2d3901f61 fix: bind mounts recommended by docs 2026-06-02 13:24:28 -03:00
fauno 8becf1c1d6 fixup! feat: node exporter 2026-05-29 16:16:37 -03:00
fauno 777b1355dd fixup! feat: node exporter 2026-05-29 16:16:08 -03:00
fauno e83433cebd feat: node exporter 2026-05-29 16:04:19 -03:00
fauno a713f98ffb feat: instance name is domain 2026-05-29 16:03:59 -03:00
fauno 8dc84c591c fixup! feat: enable prometheus remote write receiver 2026-05-29 15:38:52 -03:00
fauno d9aa05a4b5 feat: send metrics to prometheus 2026-05-28 21:00:10 -03:00
fauno 349df12204 feat: enable prometheus remote write receiver 2026-05-28 20:44:00 -03:00
fauno 6c33089078 feat: cadvisor 2026-05-28 20:38:50 -03:00
fauno 4bedebfab1 BREAKING CHANGES: replace promtail and cadvisor for alloy 2026-05-28 20:33:36 -03:00