improve alloy config and match with main branch #24

dannygroenewegen · 2026-06-14T07:32:11Z

dannygroenewegen commented

2026-06-14 07:32:11 +00:00

To be merged into alloy branch #21

dannygroenewegen added 1 commit 2026-06-14 07:32:12 +00:00

improve alloy config and match with main branch f2310f2b86

- Restrict Alloy UI to loopback
- Narrow volume mounts: drop /dev, reduce /var/run to docker.sock:ro
- Replace HTTP scrape of :12345 with prometheus.exporter.self
Match with main branch (node-exporter / promtail / cadvisor):
- Add docker_only and explicit enabled_metrics to cadvisor exporter
- Match node-exporter collector config
- Match promtail relabeling (container_name, container_id, stack_namespace,
  service_name) and external hostname label
- Add SYSLOG_FILES option to tail /var/log/*log (matches promtail)
- Fix journal path and syslog listener address

dannygroenewegen requested review from fauno 2026-06-14 07:32:12 +00:00

dannygroenewegen added 1 commit 2026-06-14 19:39:33 +00:00

feat: scrape metrics from containers via Docker label discovery e247677433

Containers opt in with prometheus.io/scrape=true and optionally set
prometheus.io/port, prometheus.io/path, and prometheus.io/auth=basic.

fauno commented

2026-06-15 14:56:33 +00:00

looks great, thanks! i'll test it this week :D

so adding bearer auth is just adapting the basic auth block? but the bearer auth is shared across all apps, like the basic auth. i mean, it's generated by this recipe.

looks great, thanks! i'll test it this week :D so adding bearer auth is just adapting the basic auth block? but the bearer auth is shared across all apps, like the basic auth. i mean, it's generated by this recipe.

dannygroenewegen commented

2026-06-16 06:56:54 +00:00

looks great, thanks! i'll test it this week :D

so adding bearer auth is just adapting the basic auth block? but the bearer auth is shared across all apps, like the basic auth. i mean, it's generated by this recipe.

Yes, one basic auth secret and one bearer secret that is used for every metric that requires it. But of course, we could extend this if needed. We could add a label like prometheus.io/authsecret=default to specify using another secret.

> looks great, thanks! i'll test it this week :D > > so adding bearer auth is just adapting the basic auth block? but the bearer auth is shared across all apps, like the basic auth. i mean, it's generated by this recipe. Yes, one basic auth secret and one bearer secret that is used for every metric that requires it. But of course, we could extend this if needed. We could add a label like `prometheus.io/authsecret=default` to specify using another secret.

❤️ 1

fauno added 1 commit 2026-06-16 23:05:25 +00:00

feat: bearer auth support 64cb07a4a2

fauno referenced this pull request from coop-cloud/garage

2026-06-16 23:05:41 +00:00

feat: support scrape discovery #19

fauno commented

2026-06-17 00:02:13 +00:00

@dannygroenewegen please check coop-cloud/garage#19 in combination with bearer auth.

where would i find indication that the labels are getting discovered?

i noticed no coopcloud container is on the docker_gwbridge network, so i added alloy to the proxy and changed the network filter on config.alloy.tmpl. this way i could curl metrics from garage on the alloy container, but i still don't see them appearing on grafana (other metrics such as alloy_build_info are appearing, so sending metrics works)

@dannygroenewegen please check https://git.coopcloud.tech/coop-cloud/garage/pulls/19 in combination with bearer auth. where would i find indication that the labels are getting discovered? i noticed no coopcloud container is on the `docker_gwbridge` network, so i added alloy to the `proxy` and changed the network filter on `config.alloy.tmpl`. this way i could `curl` metrics from garage on the alloy container, but i still don't see them appearing on grafana (other metrics such as `alloy_build_info` are appearing, so sending metrics works)

fauno added 1 commit 2026-06-17 00:20:44 +00:00

feat: enable alloy web ui fa76179987

fauno added 4 commits 2026-06-17 01:02:02 +00:00

feat: live debugging alloy 1970061ff8

fix: needed labels come from docker swarm d085c66d68

fix: send container metrics directly to prometheus 03227f1907

fix: filter by proxy network 23acf56637

fauno commented

2026-06-17 01:09:06 +00:00

got it to work by scraping and relabeling from dockerswarm! the only issue is that swarm doesn't expose the domain names, so jobs are labeled by service names (instance="git.sutty.abyaya.la" to job="git_sutty_abyaya_la_app" on the screenshot)

<img width="278" alt="image.png" src="attachments/a1d6cfc8-6b8c-4dd6-9c97-3b3e5d61858f"> got it to work by scraping and relabeling from dockerswarm! the only issue is that swarm doesn't expose the domain names, so jobs are labeled by service names (`instance="git.sutty.abyaya.la"` to `job="git_sutty_abyaya_la_app"` on the screenshot)

image.png

7.6 KiB

fauno commented

2026-06-17 01:45:04 +00:00

so far it's working for bearer and noauth. @dannygroenewegen let me know if this is what you intended and we can merge back to alloy

fauno referenced this pull request

2026-06-17 02:16:17 +00:00

Promtail is EOL, replace with Alloy? #20

dannygroenewegen commented

2026-06-17 09:26:45 +00:00

so far it's working for bearer and noauth. @dannygroenewegen let me know if this is what you intended and we can merge back to alloy

Yes, great that you got it working! Some thoughts:

Should there be a seperate secret for bearer auth? It guess it works to reuse one secret for basicauth and bearer since they both accept a similar value. But bearer_token = "{{ secret "basic_auth" }}" reads a bit confusing. The basic_auth is also used for the loki and prometheus write url. Should we add something like a scrapemetric secret? Which can then be used in the scrape blocks for basicauth and bearer?
I don't understand the changes in 03227f1907. Doesn't this result in scraping every container and resulting in errors? Many won't have any metrics endpoint.
The live debugging could be useful. Should we put the 0.0.0.0:12345 listen address and the Traefik labels for that in a separate compose.alloydebug.yml? That would prevent unnecessarily exposing the alloy web ui publicly.

Anyway, I think it's ok to merge this into alloy so there's only one branch to work with. Once I get to testing, I could also address the above points.

> so far it's working for bearer and noauth. @dannygroenewegen let me know if this is what you intended and we can merge back to alloy Yes, great that you got it working! Some thoughts: - Should there be a seperate secret for bearer auth? It guess it works to reuse one secret for basicauth and bearer since they both accept a similar value. But `bearer_token = "{{ secret "basic_auth" }}"` reads a bit confusing. The basic_auth is also used for the loki and prometheus write url. Should we add something like a `scrapemetric` secret? Which can then be used in the scrape blocks for basicauth and bearer? - I don't understand the changes in 03227f1907. Doesn't this result in scraping every container and resulting in errors? Many won't have any metrics endpoint. - The live debugging could be useful. Should we put the `0.0.0.0:12345` listen address and the Traefik labels for that in a separate compose.alloydebug.yml? That would prevent unnecessarily exposing the alloy web ui publicly. Anyway, I think it's ok to merge this into alloy so there's only one branch to work with. Once I get to testing, I could also address the above points.

fauno commented

2026-06-17 12:46:29 +00:00

I don't understand the changes in 03227f1907. Doesn't this result in scraping every container and resulting in errors? Many won't have any metrics endpoint.

it's to collect every metric produced, it doesn't go through relabeling and i didn't want to lose it

The live debugging could be useful. Should we put the 0.0.0.0:12345 listen address and the Traefik labels for that in a separate compose.alloydebug.yml? That would prevent unnecessarily exposing the alloy web ui publicly.

it's protected by basicauth like prometheus, but it could be moved to save on resources, alloy uses a lot of ram for a small vps!

> I don't understand the changes in 03227f1907. Doesn't this result in scraping every container and resulting in errors? Many won't have any metrics endpoint. it's to collect every metric produced, it doesn't go through relabeling and i didn't want to lose it > The live debugging could be useful. Should we put the 0.0.0.0:12345 listen address and the Traefik labels for that in a separate compose.alloydebug.yml? That would prevent unnecessarily exposing the alloy web ui publicly. it's protected by basicauth like prometheus, but it could be moved to save on resources, alloy uses a lot of ram for a small vps!

fauno changed title from ~~WIP: improve alloy config and match with main branch~~ to improve alloy config and match with main branch

2026-06-17 12:52:14 +00:00

fauno approved these changes 2026-06-17 12:52:44 +00:00

fauno merged commit 1d9eb10004 into alloy

2026-06-17 12:53:09 +00:00

fauno deleted branch alloy-updates

2026-06-17 12:53:09 +00:00

fauno referenced this issue from a commit

2026-06-17 12:53:11 +00:00

improve alloy config and match with main branch (#24)

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: coop-cloud/monitoring-ng#24