WIP: BREAKING CHANGES: replace promtail and cadvisor for alloy #21

2026-05-28T23:34:11Z

fauno commented

2026-05-28 23:34:11 +00:00

closes #20

fauno added 1 commit 2026-05-28 23:34:12 +00:00

BREAKING CHANGES: replace promtail and cadvisor for alloy 4bedebfab1

fauno added 1 commit 2026-05-28 23:39:01 +00:00

feat: cadvisor 6c33089078

fauno added 1 commit 2026-05-28 23:44:16 +00:00

feat: enable prometheus remote write receiver 349df12204

fauno added 1 commit 2026-05-29 00:00:25 +00:00

feat: send metrics to prometheus d9aa05a4b5

fauno commented

2026-05-29 00:01:22 +00:00

doing good progress but abyaya.la doesn't support two levels of subdomains yet so i can't keep testing! will continue soon

fauno commented

2026-05-29 00:02:40 +00:00

TODO: reset the CONFIG_ALLOY_VERSION back to v1 before merging

TODO: reset the `CONFIG_ALLOY_VERSION` back to `v1` before merging

dannygroenewegen commented

2026-05-29 06:45:49 +00:00

Thanks for this work! I also hope to do some testing with this soon.

fauno added 1 commit 2026-05-29 18:41:21 +00:00

fixup! feat: enable prometheus remote write receiver 8dc84c591c

fauno added 2 commits 2026-05-29 19:04:28 +00:00

feat: instance name is domain a713f98ffb

feat: node exporter e83433cebd

fauno added 1 commit 2026-05-29 19:16:14 +00:00

fixup! feat: node exporter 777b1355dd

fauno added 2 commits 2026-06-02 16:24:50 +00:00

fixup! feat: node exporter 8becf1c1d6

fix: bind mounts recommended by docs b2d3901f61

fauno added 2 commits 2026-06-02 21:01:07 +00:00

fix: prometheus querying panel is accessible through basic auth cdacfd035e

fix: grafana alternate domain doesn't work 641161329e

the variable is not expanded and the domain name label ends up as a
literal "$DOMAIN".

fauno added 8 commits 2026-06-02 21:50:56 +00:00

BREAKING CHANGE: no need to expose exporters 38095e23fa

feat: send docker logs to loki 024f2a8aec

fix: basic auth secret is always needed a0a6e2c509

fixup! BREAKING CHANGE: no need to expose exporters 48419d5afa

BREAKING CHANGE: deprecate promtail 2e2a52eae0

BREAKING CHANGE: deprecate node-exporter fb52a76247

feat: optionally push to prometheus and loki aa05d022da

feat: read syslog f8f8004445

fauno added 1 commit 2026-06-02 21:51:19 +00:00

fix: README d25986d5cb

fauno added 2 commits 2026-06-02 23:19:51 +00:00

fix: prevent traefik deprecation warnings 31cabc36ae

fixup! feat: read syslog dd0a0c1bb0

fauno added 1 commit 2026-06-03 00:00:54 +00:00

wip: relabel syslog according to docs a358837922

fauno added 2 commits 2026-06-03 00:06:21 +00:00

fix: prevent redirections on health check 3a1fabe4f9

fix: use the actual health check path 2870b9486c

fauno added 1 commit 2026-06-03 03:19:53 +00:00

fix: upgrades f2711fa16e

dannygroenewegen referenced this pull request

2026-06-14 07:32:11 +00:00

improve alloy config and match with main branch #24

fauno added 9 commits 2026-06-17 12:53:11 +00:00

improve alloy config and match with main branch f2310f2b86

- Restrict Alloy UI to loopback
- Narrow volume mounts: drop /dev, reduce /var/run to docker.sock:ro
- Replace HTTP scrape of :12345 with prometheus.exporter.self
Match with main branch (node-exporter / promtail / cadvisor):
- Add docker_only and explicit enabled_metrics to cadvisor exporter
- Match node-exporter collector config
- Match promtail relabeling (container_name, container_id, stack_namespace,
  service_name) and external hostname label
- Add SYSLOG_FILES option to tail /var/log/*log (matches promtail)
- Fix journal path and syslog listener address

feat: scrape metrics from containers via Docker label discovery e247677433

Containers opt in with prometheus.io/scrape=true and optionally set
prometheus.io/port, prometheus.io/path, and prometheus.io/auth=basic.

feat: bearer auth support 64cb07a4a2

feat: enable alloy web ui fa76179987

feat: live debugging alloy 1970061ff8

fix: needed labels come from docker swarm d085c66d68

fix: send container metrics directly to prometheus 03227f1907

fix: filter by proxy network 23acf56637

improve alloy config and match with main branch (#24 ) 1d9eb10004

Reviewed-on: #24
Reviewed-by: fauno <fauno@sutty.coop.ar>

fauno commented

2026-06-20 16:38:20 +00:00

i've been deploying this mainly for prometheus metrics and it's working great. i still have to test log ingestion. i'll need help with fixing the included grafana dashboards!

fauno added 1 commit 2026-06-21 02:59:29 +00:00

Merge branch 'main' of https://git.coopcloud.tech/coop-cloud/monitoring-ng into alloy c9910eabf4

fauno added 5 commits 2026-06-28 23:44:45 +00:00

fix: upgrade to alloy v1.16.3 30ad8e54f6

fix: unneeded flag 01b398ddea

fix: upgrade to grafana v12.4.5 6c529f3527

fix: upgrade alloy to v1.17.0 c6efc69859

fix: upgrade grafana to v13.0.3 b47f1b021f

fauno commented

2026-07-28 13:57:45 +00:00

TODOs:

update documentation
implement metrics on recipes
set instance label as hostname, now it's docker's internal ip address
fix included grafana dashboards
- node_name was set statically on node-exporter and missing from alloy

TODOs: - [ ] update documentation - [ ] implement metrics on recipes - [ ] set `instance` label as hostname, now it's docker's internal ip address - [ ] fix included grafana dashboards - [ ] `node_name` was set statically on `node-exporter` and missing from `alloy`

fauno commented

2026-07-31 13:06:46 +00:00

grafana-stacks-dashboard.json hardcodes the cadvisor.monitoring subdomain a lot to filter by instance, removing it from the dashboard makes the graphs i tried work again, but it breaks filtering by instance

- [ ] `grafana-stacks-dashboard.json` hardcodes the `cadvisor.monitoring` subdomain a lot to filter by instance, removing it from the dashboard makes the graphs i tried work again, but it breaks filtering by instance

dede commented

2026-07-31 15:31:55 +00:00

i checked with @p4u1 today. this is what we found out:

set instance label as hostname, now it's docker's internal ip address

in our test today this was the app domain instead

also we figured the containerd socket path needs to be included in the volumes section of the alloy service:

    volumes:
      # ...
      - /run/containerd/containerd.sock:/run/containerd/containerd.sock:ro

otherwise no containers/services are found.

i checked with @p4u1 today. this is what we found out: > set instance label as hostname, now it's docker's internal ip address in our test today this was the app domain instead also we figured the containerd socket path needs to be included in the volumes section of the alloy service: ``` volumes: # ... - /run/containerd/containerd.sock:/run/containerd/containerd.sock:ro ``` otherwise no containers/services are found.

fauno commented

2026-07-31 16:40:22 +00:00

thanks @dede ! on an alpine 3.21 server this socket seems to be located at /var/run/docker/containerd/containerd.sock while in ubuntu server 24.04 it's at /var/run/containerd/containerd.sock, maybe the location changes by distribution? or they're not the same socket?

i could do something like this i think:

volumes:
- "${CONTAINERD_SOCKET:-/var/run/containerd/containerd.sock}:/var/run/containerd/containerd.sock:ro"

thanks @dede ! on an alpine 3.21 server this socket seems to be located at `/var/run/docker/containerd/containerd.sock` while in ubuntu server 24.04 it's at `/var/run/containerd/containerd.sock`, maybe the location changes by distribution? or they're not the same socket? i could do something like this i think: ```yaml volumes: - "${CONTAINERD_SOCKET:-/var/run/containerd/containerd.sock}:/var/run/containerd/containerd.sock:ro" ```

fauno referenced this pull request from toolshed/abra

2026-07-31 16:56:00 +00:00

compose.yml decoding failed "too many colons" with dynamic volumes #900

fauno added 1 commit 2026-07-31 16:58:10 +00:00

fix: provide access to containerd socket toolshed/abra#900 7818c89ad7

This pull request is marked as a work in progress.

View command line instructions