coop-cloud/monitoring-ng
1
0
Fork 2
Code Issues 7 Pull Requests 5 Packages Projects Releases Wiki Activity

BREAKING CHANGES: replace promtail and cadvisor for alloy #21

Open
fauno wants to merge 59 commits from alloy into main
pull from: alloy
merge into: :main
Conversation 16 Commits 59 Files Changed 24
+737 -407
fauno commented 2026-05-28 23:34:11 +00:00
Owner
Copy Link
Copy Source

closes #20

closes #20
fauno added 1 commit 2026-05-28 23:34:12 +00:00
BREAKING CHANGES: replace promtail and cadvisor for alloy 4bedebfab1
fauno added 1 commit 2026-05-28 23:39:01 +00:00
feat: cadvisor 6c33089078
fauno added 1 commit 2026-05-28 23:44:16 +00:00
feat: enable prometheus remote write receiver 349df12204
fauno added 1 commit 2026-05-29 00:00:25 +00:00
feat: send metrics to prometheus d9aa05a4b5
fauno commented 2026-05-29 00:01:22 +00:00
Author
Owner
Copy Link
Copy Source

doing good progress but abyaya.la doesn't support two levels of subdomains yet so i can't keep testing! will continue soon

doing good progress but abyaya.la doesn't support two levels of subdomains yet so i can't keep testing! will continue soon
fauno commented 2026-05-29 00:02:40 +00:00
Author
Owner
Copy Link
Copy Source

TODO: reset the CONFIG_ALLOY_VERSION back to v1 before merging

TODO: reset the `CONFIG_ALLOY_VERSION` back to `v1` before merging
dannygroenewegen commented 2026-05-29 06:45:49 +00:00
Owner
Copy Link
Copy Source

Thanks for this work! I also hope to do some testing with this soon.

Thanks for this work! I also hope to do some testing with this soon.
fauno added 1 commit 2026-05-29 18:41:21 +00:00
fixup! feat: enable prometheus remote write receiver 8dc84c591c
fauno added 2 commits 2026-05-29 19:04:28 +00:00
feat: instance name is domain a713f98ffb
feat: node exporter e83433cebd
fauno added 1 commit 2026-05-29 19:16:14 +00:00
fixup! feat: node exporter 777b1355dd
fauno added 2 commits 2026-06-02 16:24:50 +00:00
fixup! feat: node exporter 8becf1c1d6
fix: bind mounts recommended by docs b2d3901f61
fauno added 2 commits 2026-06-02 21:01:07 +00:00
fix: prometheus querying panel is accessible through basic auth cdacfd035e
fix: grafana alternate domain doesn't work 641161329e 
the variable is not expanded and the domain name label ends up as a
literal "$DOMAIN".
fauno added 8 commits 2026-06-02 21:50:56 +00:00
BREAKING CHANGE: no need to expose exporters 38095e23fa
feat: send docker logs to loki 024f2a8aec
fix: basic auth secret is always needed a0a6e2c509
fixup! BREAKING CHANGE: no need to expose exporters 48419d5afa
BREAKING CHANGE: deprecate promtail 2e2a52eae0
BREAKING CHANGE: deprecate node-exporter fb52a76247
feat: optionally push to prometheus and loki aa05d022da
feat: read syslog f8f8004445
fauno added 1 commit 2026-06-02 21:51:19 +00:00
fix: README d25986d5cb
fauno added 2 commits 2026-06-02 23:19:51 +00:00
fix: prevent traefik deprecation warnings 31cabc36ae
fixup! feat: read syslog dd0a0c1bb0
fauno added 1 commit 2026-06-03 00:00:54 +00:00
wip: relabel syslog according to docs a358837922
fauno added 2 commits 2026-06-03 00:06:21 +00:00
fix: prevent redirections on health check 3a1fabe4f9
fix: use the actual health check path 2870b9486c
fauno added 1 commit 2026-06-03 03:19:53 +00:00
fix: upgrades f2711fa16e
fauno added 9 commits 2026-06-17 12:53:11 +00:00
improve alloy config and match with main branch f2310f2b86 
- Restrict Alloy UI to loopback
- Narrow volume mounts: drop /dev, reduce /var/run to docker.sock:ro
- Replace HTTP scrape of :12345 with prometheus.exporter.self
Match with main branch (node-exporter / promtail / cadvisor):
- Add docker_only and explicit enabled_metrics to cadvisor exporter
- Match node-exporter collector config
- Match promtail relabeling (container_name, container_id, stack_namespace,
  service_name) and external hostname label
- Add SYSLOG_FILES option to tail /var/log/*log (matches promtail)
- Fix journal path and syslog listener address
feat: scrape metrics from containers via Docker label discovery e247677433 
Containers opt in with prometheus.io/scrape=true and optionally set
prometheus.io/port, prometheus.io/path, and prometheus.io/auth=basic.
feat: bearer auth support 64cb07a4a2
feat: enable alloy web ui fa76179987
feat: live debugging alloy 1970061ff8
fix: needed labels come from docker swarm d085c66d68
fix: send container metrics directly to prometheus 03227f1907
fix: filter by proxy network 23acf56637
improve alloy config and match with main branch (#24) 1d9eb10004 
Reviewed-on: #24
Reviewed-by: fauno <fauno@sutty.coop.ar>
fauno commented 2026-06-20 16:38:20 +00:00
Author
Owner
Copy Link
Copy Source

i've been deploying this mainly for prometheus metrics and it's working great. i still have to test log ingestion. i'll need help with fixing the included grafana dashboards!

i've been deploying this mainly for prometheus metrics and it's working great. i still have to test log ingestion. i'll need help with fixing the included grafana dashboards!
fauno added 1 commit 2026-06-21 02:59:29 +00:00
Merge branch 'main' of https://git.coopcloud.tech/coop-cloud/monitoring-ng into alloy c9910eabf4
fauno added 5 commits 2026-06-28 23:44:45 +00:00
fix: upgrade to alloy v1.16.3 30ad8e54f6
fix: unneeded flag 01b398ddea
fix: upgrade to grafana v12.4.5 6c529f3527
fix: upgrade alloy to v1.17.0 c6efc69859
fix: upgrade grafana to v13.0.3 b47f1b021f
fauno commented 2026-07-28 13:57:45 +00:00
Author
Owner
Copy Link
Copy Source

TODOs:

  • update documentation
  • implement metrics on recipes
  • set instance label as hostname, now it's docker's internal ip address
  • fix included grafana dashboards
    • node_name was set statically on node-exporter and missing from alloy
TODOs: - [ ] update documentation - [ ] implement metrics on recipes - [ ] set `instance` label as hostname, now it's docker's internal ip address - [ ] fix included grafana dashboards - [ ] `node_name` was set statically on `node-exporter` and missing from `alloy`
fauno commented 2026-07-31 13:06:46 +00:00
Author
Owner
Copy Link
Copy Source
  • grafana-stacks-dashboard.json hardcodes the cadvisor.monitoring subdomain a lot to filter by instance, removing it from the dashboard makes the graphs i tried work again, but it breaks filtering by instance
- [ ] `grafana-stacks-dashboard.json` hardcodes the `cadvisor.monitoring` subdomain a lot to filter by instance, removing it from the dashboard makes the graphs i tried work again, but it breaks filtering by instance
dede commented 2026-07-31 15:31:55 +00:00
Member
Copy Link
Copy Source

i checked with @p4u1 today. this is what we found out:

set instance label as hostname, now it's docker's internal ip address

in our test today this was the app domain instead

also we figured the containerd socket path needs to be included in the volumes section of the alloy service:

    volumes:
      # ...
      - /run/containerd/containerd.sock:/run/containerd/containerd.sock:ro

otherwise no containers/services are found.

i checked with @p4u1 today. this is what we found out: > set instance label as hostname, now it's docker's internal ip address in our test today this was the app domain instead also we figured the containerd socket path needs to be included in the volumes section of the alloy service: ``` volumes: # ... - /run/containerd/containerd.sock:/run/containerd/containerd.sock:ro ``` otherwise no containers/services are found.
fauno commented 2026-07-31 16:40:22 +00:00
Author
Owner
Copy Link
Copy Source

thanks @dede ! on an alpine 3.21 server this socket seems to be located at /var/run/docker/containerd/containerd.sock while in ubuntu server 24.04 it's at /var/run/containerd/containerd.sock, maybe the location changes by distribution? or they're not the same socket?

i could do something like this i think:

volumes:
- "${CONTAINERD_SOCKET:-/var/run/containerd/containerd.sock}:/var/run/containerd/containerd.sock:ro"
thanks @dede ! on an alpine 3.21 server this socket seems to be located at `/var/run/docker/containerd/containerd.sock` while in ubuntu server 24.04 it's at `/var/run/containerd/containerd.sock`, maybe the location changes by distribution? or they're not the same socket? i could do something like this i think: ```yaml volumes: - "${CONTAINERD_SOCKET:-/var/run/containerd/containerd.sock}:/var/run/containerd/containerd.sock:ro" ```
👍 1
fauno added 1 commit 2026-07-31 16:58:10 +00:00
fix: provide access to containerd socket toolshed/abra#900 7818c89ad7
fauno added 1 commit 2026-08-04 13:49:04 +00:00
fix: upgrade to grafana 13.0.5 5f3f9c957c
fauno added 1 commit 2026-08-06 15:37:41 +00:00
fix: upgrade alloy to 1.18.1 e75fa4487c
fauno added 1 commit 2026-08-07 12:25:07 +00:00
fix: upgrade to grafana 13.0.6 627d988902
dannygroenewegen added 1 commit 2026-08-07 12:48:41 +00:00
- Restore GRAFANA_DOMAIN 7f3d98fd9f 
- Start documenting the migration in release/next
- Various cleanups
dannygroenewegen added 2 commits 2026-08-07 13:52:57 +00:00
feat: separate scrape_auth secret, configurable usernames 402e022114 
Added an optional scrape_auth secret for authenticating scrapes of
containers that opt in via prometheus.io/auth labels, no longer
reusing the basic_auth secret meant for Prometheus/Loki writes.

Made usernames for both write endpoints and authenticated scraping
configurable in .env.
fix: drop unfiltered discovery.docker.containers.targets from default scrape 0f989c4567 
prometheus.exporter.cadvisor.docker.targets already reports resource
metrics for every container on the host. The unfiltered discovery just
added targets with internal overlay IPs, most of which failed to
scrape anything useful.
dannygroenewegen force-pushed alloy from ccff913b5d to 0f989c4567 2026-08-07 13:52:57 +00:00 Compare
dannygroenewegen added 1 commit 2026-08-07 15:02:37 +00:00
fix: make swarm/stacks dashboards work with old and new metric data 7ab6cc74be 
Replace node_meta-based joins with label_replace() of the instance
label, so old (pull-model) and new (Alloy push) series for the same
host render as one continuous series. Each rewritten query has a TODO
noting the simpler form to use once old data ages out of retention.
dannygroenewegen added 1 commit 2026-08-10 19:25:43 +00:00
fix: discover containers by task instead of service for scraping fafd1f89f0 
Swarm service discovery misses dnsrr-mode services like traefik; switch
to swarm task discovery, scoped to this host, with a stable instance
label across redeploys and a default port when none is labeled.
dannygroenewegen commented 2026-08-10 19:41:25 +00:00
Owner
Copy Link
Copy Source

Created a PR for Traefik to add metric discovery labels. Will work on updating the README and short migration guide for the release notes this week. Almost ready!

Created a [PR for Traefik](https://git.coopcloud.tech/coop-cloud/traefik/pulls/132) to add metric discovery labels. Will work on updating the README and short migration guide for the release notes this week. Almost ready!
dannygroenewegen added 1 commit 2026-08-17 11:38:28 +00:00
fix Traefik dashboard 536fa7f37d
dannygroenewegen added 4 commits 2026-08-17 13:58:32 +00:00
fix: don't expose Alloy's web UI by default, but optionally with compose.alloy-webui.yml 067f013fb7
fix abra linting secret length 7480a8d6ab 
Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
feat: add healthchecks to Alloy, Prometheus and Pushgateway 50fc916107
docs: rewrite README and release notes for the Alloy migration 17c4f6237b
dannygroenewegen added 1 commit 2026-08-17 14:10:26 +00:00
fix: mount containerd socket read-only, avoid abra#900 6dbb6c5f55
dannygroenewegen commented 2026-08-17 14:20:57 +00:00
Owner
Copy Link
Copy Source

I think all open todos are resolved. Also updated the readme and added a migration guide to the release notes.
Anyone up for testing?

TODO: reset the CONFIG_ALLOY_VERSION back to v1 before merging

This still needs to be done before merging. And probably a squash commit for merging; don't think we need this whole kinda messy history?

I think all open todos are resolved. Also updated the readme and added a migration guide to the release notes. Anyone up for testing? > TODO: reset the `CONFIG_ALLOY_VERSION` back to `v1` before merging This still needs to be done before merging. And probably a squash commit for merging; don't think we need this whole kinda messy history?
dannygroenewegen requested review from monitoring-ng-maintainers 2026-08-17 14:21:08 +00:00
dannygroenewegen marked the pull request as ready for review 2026-08-17 14:22:11 +00:00
ammaratef45 commented 2026-08-18 03:42:38 +00:00
Owner
Copy Link
Copy Source

did a test deploy and looks like alloy works!image.png

did a test deploy and looks like alloy works!![image.png](/attachments/d52523e4-21e7-4509-bd29-62ecdb9fc586)
ammaratef45 commented 2026-08-18 03:43:07 +00:00
Owner
Copy Link
Copy Source

did a test deploy and looks like alloy works!image.png

did a test deploy and looks like alloy works!![image.png](/attachments/d52523e4-21e7-4509-bd29-62ecdb9fc586)
image.png
108 KiB
ammaratef45 commented 2026-08-18 03:45:54 +00:00
Owner
Copy Link
Copy Source

new readme is a lot more clean!

new readme is a lot more clean!
dannygroenewegen added 1 commit 2026-08-20 13:26:33 +00:00
fix: refresh dashboard variables on time range change, not just page load c8f74e2a3c
dannygroenewegen added 1 commit 2026-08-20 14:50:49 +00:00
fix: strip monitoring. prefix from instance labels for cleaner dashboard views 382f6353fc
simon commented 2026-08-21 13:27:07 +00:00
Member
Copy Link
Copy Source

Hey folks, thanks for the amazing work!

This:

monitoring-ng/compose.grafana.yml Line 42 in 382f6353fc
- "traefik.http.routers.${STACK_NAME}-grafana.rule=Host(`${GRAFANA_DOMAIN:-$DOMAIN}`)"

didn't work for me, needed to explicitly set the env as the nested replacement didn't work.

From docker service inspect:
traefik.http.routers.monitoring_kolli_cloud-grafana.rule=Host($DOMAIN)

Hey folks, thanks for the amazing work! This: https://git.coopcloud.tech/coop-cloud/monitoring-ng/src/commit/382f6353fc802abbddee19e571a6d2556c65371c/compose.grafana.yml#L42 didn't work for me, needed to explicitly set the env as the nested replacement didn't work. From docker service inspect: `traefik.http.routers.monitoring_kolli_cloud-grafana.rule=Host(`$DOMAIN`)`
simon commented 2026-08-21 15:34:50 +00:00
Member
Copy Link
Copy Source

I guess we could also remove this. I've seen your discussion in matrix about it:

monitoring-ng/grafana-dashboards.yml Lines 14 to 23 in 02b01e5c23
- name: 'default-alert-provider'
orgId: 1
folder: 'default-alerts'
type: file
disableDeletion: false
updateIntervalSeconds: 10
allowUiUpdates: true
options:
path: /var/lib/grafana/alerts
foldersFromFilesStructure: true

By default, that folder doesn't exist, resulting in continues errors:
2026-08-21T14:59:53.360890994Z logger=provisioning.dashboard type=file name=default-alert-provider t=2026-08-21T14:59:53.360693215Z level=error msg="failed to walk provisioned dashboards" error="stat /var/lib/grafana/alerts: no such file or directory"

And from my knowledge its never used anywhere anyway. Looks like @p4u1 added it 2 years ago, maybe he knows more? 0352a393de (diff-338a7a8dc1f47ef86aa7fe511610bc988be2e70c)

I guess we could also remove this. I've seen your discussion in matrix about it: https://git.coopcloud.tech/coop-cloud/monitoring-ng/src/commit/02b01e5c236e963ccdcfc422c3c099c6bbf41e8e/grafana-dashboards.yml#L14-L23 By default, that folder doesn't exist, resulting in continues errors: `2026-08-21T14:59:53.360890994Z logger=provisioning.dashboard type=file name=default-alert-provider t=2026-08-21T14:59:53.360693215Z level=error msg="failed to walk provisioned dashboards" error="stat /var/lib/grafana/alerts: no such file or directory"` And from my knowledge its never used anywhere anyway. Looks like @p4u1 added it 2 years ago, maybe he knows more? https://git.coopcloud.tech/coop-cloud/monitoring-ng/commit/0352a393de77d58c8462a69cf805f5cc7e3c8156#diff-338a7a8dc1f47ef86aa7fe511610bc988be2e70c
p4u1 commented 2026-08-21 16:20:06 +00:00
Owner
Copy Link
Copy Source

I guess we could also remove this. I've seen your discussion in matrix about it:

monitoring-ng/grafana-dashboards.yml Lines 14 to 23 in 02b01e5c23
- name: 'default-alert-provider'
orgId: 1
folder: 'default-alerts'
type: file
disableDeletion: false
updateIntervalSeconds: 10
allowUiUpdates: true
options:
path: /var/lib/grafana/alerts
foldersFromFilesStructure: true

By default, that folder doesn't exist, resulting in continues errors:
2026-08-21T14:59:53.360890994Z logger=provisioning.dashboard type=file name=default-alert-provider t=2026-08-21T14:59:53.360693215Z level=error msg="failed to walk provisioned dashboards" error="stat /var/lib/grafana/alerts: no such file or directory"

And from my knowledge its never used anywhere anyway. Looks like @p4u1 added it 2 years ago, maybe he knows more? 0352a393de (diff-338a7a8dc1f47ef86aa7fe511610bc988be2e70c)

Yeah, this can be removed. Alerts are now configured like this: 02b01e5c23/alerts/node.yml.tmpl

> I guess we could also remove this. I've seen your discussion in matrix about it: > > https://git.coopcloud.tech/coop-cloud/monitoring-ng/src/commit/02b01e5c236e963ccdcfc422c3c099c6bbf41e8e/grafana-dashboards.yml#L14-L23 > > By default, that folder doesn't exist, resulting in continues errors: > `2026-08-21T14:59:53.360890994Z logger=provisioning.dashboard type=file name=default-alert-provider t=2026-08-21T14:59:53.360693215Z level=error msg="failed to walk provisioned dashboards" error="stat /var/lib/grafana/alerts: no such file or directory"` > > And from my knowledge its never used anywhere anyway. Looks like @p4u1 added it 2 years ago, maybe he knows more? https://git.coopcloud.tech/coop-cloud/monitoring-ng/commit/0352a393de77d58c8462a69cf805f5cc7e3c8156#diff-338a7a8dc1f47ef86aa7fe511610bc988be2e70c Yeah, this can be removed. Alerts are now configured like this: https://git.coopcloud.tech/coop-cloud/monitoring-ng/src/commit/02b01e5c236e963ccdcfc422c3c099c6bbf41e8e/alerts/node.yml.tmpl
You are not authorized to merge this pull request.
This pull request can be merged automatically.
View command line instructions

Checkout

From your project repository, check out a new branch and test the changes.
git fetch -u origin alloy:alloy
git checkout alloy
Sign in to join this conversation.
Reviewers
No Reviewers
coop-cloud/monitoring-ng-maintainers
Labels
No items
No labels
Milestone
No items
No Milestone
Projects
Clear projects
No projects
Assignees
Clear assignees
3wordchant aadil (Aadil Ayub) abra-bot (Abra Bot) ammaratef45 amras (Sarma) BornDeleuze Brooke cas (Cassowary) coopcloud dannygroenewegen (Danny Groenewegen) decentral1se (d1) fauno (fauno) jjsfunhouse kawaiipunk (KawaiiPunk) knoflook moosemower moritz notplants oxaliq (sorrel) p4u1 renovate-bot (Comrade Renovate Bot) stevensting trav (Trav Fryer) yksflip
No Assignees
6 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: coop-cloud/monitoring-ng#21