Split image tags, relabel journald logs #30

2026-09-16T15:44:10Z

simon commented

2026-09-16 15:44:10 +00:00

I wanted to step up our monitoring game by adding more helpful dashboards and noticed some shortcomings of the labels we currently use for logs and metrics. Did so with the help of Claude via Browser chat, so #llm_enabled

was still able to build this beautiful dashboard (with claude assistance..)

but it relays on regex based label replacements of this kind:

count by (image_name, image_tag) (
  group by (image_name, image_tag, container_label_com_docker_stack_namespace) (
    label_replace(
      label_replace(container_last_seen, "image_name", "$1", "image", "([^@]+):[^:@]+.*"),
      "image_tag", "$1", "image", "[^@]+:([^:@]+).*"
    )
  )
)

the changes introduced with prometheus.relabel "container_meta" address this issue and should make working with image tags and names more straightforward.

the second change affects the journald kernel logs to e.g. distinguish source, priority etc.:

for this I also dropped the NetworkDB stats as they produce a lot of log volume while not being very helpful at all, especially on single node deployments.

I'm currently also working on a way to add labels for version (of the recipe) and the actual recipe via abra - along with a method to set custom labels via .env

I wanted to step up our monitoring game by adding more helpful dashboards and noticed some shortcomings of the labels we currently use for logs and metrics. Did so with the help of Claude via Browser chat, so #llm_enabled was still able to build this beautiful dashboard (with claude assistance..) ![grafik.png](/attachments/d08a78fa-b83b-41b9-8068-4445f2f8c6b8) but it relays on regex based label replacements of this kind: ``` count by (image_name, image_tag) ( group by (image_name, image_tag, container_label_com_docker_stack_namespace) ( label_replace( label_replace(container_last_seen, "image_name", "$1", "image", "([^@]+):[^:@]+.*"), "image_tag", "$1", "image", "[^@]+:([^:@]+).*" ) ) ) ``` the changes introduced with `prometheus.relabel "container_meta"` address this issue and should make working with image tags and names more straightforward. the second change affects the journald kernel logs to e.g. distinguish source, priority etc.: ![grafik.png](/attachments/c8ff1bf7-728b-43f2-bf88-5095d9029c61) for this I also dropped the NetworkDB stats as they produce a lot of log volume while not being very helpful at all, especially on single node deployments. I'm currently also working on a way to add labels for `version` (of the recipe) and the actual `recipe` via abra - along with a method to set custom labels via .env

grafik.png

200 KiB

grafik.png

244 KiB

simon added 3 commits 2026-09-16 15:44:11 +00:00

split image tag cffb387580

relabel journald logs and drop networkDB stats e22d6bae83

Merge remote-tracking branch 'origin/main' into split-image-tags 33e767737a

simon requested review from monitoring-ng-maintainers 2026-09-16 15:44:55 +00:00

ammaratef45 approved these changes

2026-09-17 06:04:38 +00:00

simon added 1 commit 2026-09-17 13:36:45 +00:00