make disk usage per container metric optional #29

2026-09-08T08:37:18Z

simon commented

2026-09-08 08:37:18 +00:00

Following the discussion from the matrix channel, I made the disk usage metrics optional, as it currently causes ~35% of cpu usage on our system.

Old cadvisor-config from this repo had a - "--housekeeping_interval=120s" option enabled which seems currently not configurable for alloy - maybe this allows us reenabling disk metrics at a later point again.

dropped from 35% to 5% when removing disk from
enabled_metrics = ["cpu", "cpuLoad", "diskIO", "memory", "network"] for cadvisor.

go profiling:

File: alloy
Build ID: de7ba8978f6753c25c13f6886ab3b896d3b05d3f
Type: cpu
Time: Sep 7, 2026 at 12:53pm (CEST)
Duration: 120s, Total samples = 37.23s (31.02%)
Showing nodes accounting for 32.20s, 86.49% of 37.23s total
Dropped 790 nodes (cum <= 0.19s)
      flat  flat%   sum%        cum   cum%
    28.72s 77.14% 77.14%     28.72s 77.14%  internal/runtime/syscall/linux.Syscall6
     0.47s  1.26% 78.40%      0.48s  1.29%  internal/filepathlite.(*lazybuf).append (inline)
     0.30s  0.81% 79.21%      0.94s  2.52%  internal/filepathlite.Clean
     0.26s   0.7% 79.91%      0.26s   0.7%  runtime.nextFreeFast (inline)
     0.23s  0.62% 80.53%      0.23s  0.62%  runtime.futex
     0.21s  0.56% 81.09%      0.21s  0.56%  runtime.memclrNoHeapPointers
     0.21s  0.56% 81.65%      0.21s  0.56%  runtime.memmove
     0.19s  0.51% 82.16%      6.19s 16.63%  os.(*File).readdir
     0.17s  0.46% 82.62%     31.24s 83.91%  path/filepath.walk
     0.12s  0.32% 82.94%      0.20s  0.54%  runtime.exitsyscall
     0.08s  0.21% 83.16%      0.24s  0.64%  runtime.scanObject
     0.08s  0.21% 83.37%      0.24s  0.64%  runtime.sweepone
     0.08s  0.21% 83.59%      0.29s  0.78%  slices.pdqsortOrdered[go.shape.string]
     0.07s  0.19% 83.78%      0.27s  0.73%  runtime.makeslicecopy
     0.06s  0.16% 83.94%     28.48s 76.50%  syscall.RawSyscall6
     0.05s  0.13% 84.07%      0.27s  0.73%  github.com/google/cadvisor/fs.GetDirUsage.func1
     0.05s  0.13% 84.21%     19.99s 53.69%  os.lstatNolog
     0.05s  0.13% 84.34%      1.26s  3.38%  runtime.mallocgc
     0.05s  0.13% 84.47%      0.43s  1.15%  runtime.mallocgcSmallNoscan
     0.04s  0.11% 84.58%      0.46s  1.24%  runtime.newobject
     0.04s  0.11% 84.69%      0.19s  0.51%  runtime.selectgo
     0.04s  0.11% 84.80%      1.18s  3.17%  runtime.systemstack

Following the discussion from the matrix channel, I made the disk usage metrics optional, as it currently causes ~35% of cpu usage on our system. [Old cadvisor-config](https://git.coopcloud.tech/coop-cloud/monitoring-ng/src/commit/02b01e5c236e963ccdcfc422c3c099c6bbf41e8e/compose.yml) from this repo had a `- "--housekeeping_interval=120s"` option enabled which seems currently not configurable for alloy - maybe this allows us reenabling disk metrics at a later point again. dropped from 35% to 5% when removing `disk` from ` enabled_metrics = ["cpu", "cpuLoad", "diskIO", "memory", "network"]` for cadvisor. go profiling: ``` File: alloy Build ID: de7ba8978f6753c25c13f6886ab3b896d3b05d3f Type: cpu Time: Sep 7, 2026 at 12:53pm (CEST) Duration: 120s, Total samples = 37.23s (31.02%) Showing nodes accounting for 32.20s, 86.49% of 37.23s total Dropped 790 nodes (cum <= 0.19s) flat flat% sum% cum cum% 28.72s 77.14% 77.14% 28.72s 77.14% internal/runtime/syscall/linux.Syscall6 0.47s 1.26% 78.40% 0.48s 1.29% internal/filepathlite.(*lazybuf).append (inline) 0.30s 0.81% 79.21% 0.94s 2.52% internal/filepathlite.Clean 0.26s 0.7% 79.91% 0.26s 0.7% runtime.nextFreeFast (inline) 0.23s 0.62% 80.53% 0.23s 0.62% runtime.futex 0.21s 0.56% 81.09% 0.21s 0.56% runtime.memclrNoHeapPointers 0.21s 0.56% 81.65% 0.21s 0.56% runtime.memmove 0.19s 0.51% 82.16% 6.19s 16.63% os.(*File).readdir 0.17s 0.46% 82.62% 31.24s 83.91% path/filepath.walk 0.12s 0.32% 82.94% 0.20s 0.54% runtime.exitsyscall 0.08s 0.21% 83.16% 0.24s 0.64% runtime.scanObject 0.08s 0.21% 83.37% 0.24s 0.64% runtime.sweepone 0.08s 0.21% 83.59% 0.29s 0.78% slices.pdqsortOrdered[go.shape.string] 0.07s 0.19% 83.78% 0.27s 0.73% runtime.makeslicecopy 0.06s 0.16% 83.94% 28.48s 76.50% syscall.RawSyscall6 0.05s 0.13% 84.07% 0.27s 0.73% github.com/google/cadvisor/fs.GetDirUsage.func1 0.05s 0.13% 84.21% 19.99s 53.69% os.lstatNolog 0.05s 0.13% 84.34% 1.26s 3.38% runtime.mallocgc 0.05s 0.13% 84.47% 0.43s 1.15% runtime.mallocgcSmallNoscan 0.04s 0.11% 84.58% 0.46s 1.24% runtime.newobject 0.04s 0.11% 84.69% 0.19s 0.51% runtime.selectgo 0.04s 0.11% 84.80% 1.18s 3.17% runtime.systemstack ```

simon added 1 commit 2026-09-08 08:37:20 +00:00

make disk usage per container metric optional 3ea485bfc7

simon requested review from monitoring-ng-maintainers 2026-09-08 08:37:28 +00:00

dannygroenewegen commented

2026-09-08 12:01:13 +00:00

Looks good, but have you considered making the whole enabled_metrics value configurable with an env? So, setting the full ["cpu", "cpuLoad", "diskIO", "memory", "network"] as an env value, with a default fallback if unset.

This method of CADVISOR_DISK_USAGE=1 quickly becomes complicated in the config file if we want to add the option to enable/disable more categories.

Looks good, but have you considered making the whole `enabled_metrics` value configurable with an env? So, setting the full `["cpu", "cpuLoad", "diskIO", "memory", "network"]` as an env value, with a default fallback if unset. This method of `CADVISOR_DISK_USAGE=1` quickly becomes complicated in the config file if we want to add the option to enable/disable more categories.

simon commented

2026-09-08 13:22:57 +00:00

I have, but since there was no demand to make any other metric configurable in the past years It seemed easier to just add a flag for this one. But mainly because I wanted to avoid potentials issues with quoting and correct escaping of quotes when passing the full array as env.

I'm fine with either, though

I have, but since there was no demand to make any other metric configurable in the past years It seemed easier to just add a flag for this one. But mainly because I wanted to avoid potentials issues with quoting and correct escaping of quotes when passing the full array as env. I'm fine with either, though

👍 1

dannygroenewegen approved these changes

2026-09-08 13:25:01 +00:00

simon merged commit f48aa64193 into main

2026-09-08 14:25:50 +00:00

simon deleted branch optional-disk-metric

2026-09-08 14:25:55 +00:00

Sign in to join this conversation.

2 Participants

Notifications

Due Date

No due date set.

Dependencies

No dependencies set.

Reference: coop-cloud/monitoring-ng#29