toolshed/abra
4
4
Fork 22
Code Issues 76 Pull Requests 9 Packages Releases 45 Activity

out of order logs? #875

New Issue
Open
opened 2026-06-29 13:23:36 +00:00 by fauno · 5 comments
fauno commented 2026-06-29 13:23:36 +00:00
Owner
Copy Link
Copy Source

while running abra app logs for a monitoring-ng recipe, i found that logs lines seem to be out of order. i can see lines from 2026-06-28, then 2026-06-29, then 2026-06-28 again. i thought the two containers may have different timezones, but running date on them show they are sharing the same time. docker service logs -f doesn't seem to have this problem.

while running `abra app logs` for a monitoring-ng recipe, i found that logs lines seem to be out of order. i can see lines from 2026-06-28, then 2026-06-29, then 2026-06-28 again. i thought the two containers may have different timezones, but running `date` on them show they are sharing the same time. `docker service logs -f` doesn't seem to have this problem.
fauno commented 2026-06-29 13:25:04 +00:00
Author
Owner
Copy Link
Copy Source

not sure where the characters at the start of each line are coming from

+2026-06-28T23:23:26.925627488Z time=2026-06-28T23:23:26.925Z level=ERROR source=write_handler.go:230 msg="Out of order sample from remote write" component=web err="out of order sample" series="{__name__=\"up\", instance=\"10.0.1.5:3901\", job=\"prometheus.scrape.default\"}" timestamp=1782688872180
*2026-06-28T23:23:32.080351867Z time=2026-06-28T23:23:32.080Z level=ERROR source=write_handler.go:230 msg="Out of order sample from remote write" component=web err="out of order sample" series="{__name__=\"up\", instance=\"10.0.1.5:443\", job=\"prometheus.scrape.default\"}" timestamp=1782688878548
+2026-06-28T23:24:07.565748741Z time=2026-06-28T23:24:07.565Z level=ERROR source=write_handler.go:230 msg="Out of order sample from remote write" component=web err="out of order sample" series="{__name__=\"up\", instance=\"10.0.2.2:3901\", job=\"prometheus.scrape.default\"}" timestamp=1782688913003
)2026-06-28T23:24:22.730180956Z time=2026-06-28T23:24:22.729Z level=ERROR source=write_handler.go:230 msg="Out of order sample from remote write" component=web err="out of order sample" series="{__name__=\"up\", instance=\"10.0.1.5:80\", job=\"prometheus.scrape.default\"}" timestamp=1782688925846
)2026-06-28T23:42:09.414066062Z time=2026-06-28T23:42:09.392Z level=ERROR source=write_handler.go:230 msg="Out of order sample from remote write" component=web err="out of order sample" series="{__name__=\"up\", instance=\"10.0.2.2:80\", job=\"prometheus.scrape.default\"}" timestamp=1782689995354
�2026-06-29T01:00:03.964805615Z time=2026-06-29T01:00:03.952Z level=INFO source=compact.go:590 msg="write block started" component=tsdb mint=1782684000602 maxt=1782691200000 ulid=01KW8E7YEB10H5Z9AVG0VPYW6R
�2026-06-29T01:00:04.461016619Z time=2026-06-29T01:00:04.460Z level=INFO source=compact.go:627 msg="write block completed" component=tsdb mint=1782684000602 maxt=1782691200000 ulid=01KW8E7YEB10H5Z9AVG0VPYW6R duration=554.69921ms ooo=false
�2026-06-29T01:00:04.465857462Z time=2026-06-29T01:00:04.465Z level=INFO source=db.go:2077 msg="Deleting obsolete block" component=tsdb block=01KVPV6JGY6725101XFD30J2WT
�2026-06-29T01:00:04.466102605Z time=2026-06-29T01:00:04.466Z level=INFO source=head.go:1508 msg="Head GC started" component=tsdb caller=truncateMemory
�2026-06-29T01:00:04.595545506Z time=2026-06-29T01:00:04.595Z level=INFO source=head.go:1512 msg="Head GC completed" component=tsdb caller=truncateMemory duration=129.217203ms
�2026-06-29T03:00:04.146793697Z time=2026-06-29T03:00:04.131Z level=INFO source=compact.go:590 msg="write block started" component=tsdb mint=1782691200602 maxt=1782698400000 ulid=01KW8N3NW4KH93CCESP53KSTG7
�2026-06-29T03:00:04.656181629Z time=2026-06-29T03:00:04.655Z level=INFO source=compact.go:627 msg="write block completed" component=tsdb mint=1782691200602 maxt=1782698400000 ulid=01KW8N3NW4KH93CCESP53KSTG7 duration=562.883823ms ooo=false
�2026-06-29T03:00:04.664594682Z time=2026-06-29T03:00:04.658Z level=INFO source=head.go:1508 msg="Head GC started" component=tsdb caller=truncateMemory
�2026-06-29T03:00:04.684713199Z time=2026-06-29T03:00:04.679Z level=INFO source=head.go:1512 msg="Head GC completed" component=tsdb caller=truncateMemory duration=21.386971ms
�2026-06-29T03:00:04.684746793Z time=2026-06-29T03:00:04.680Z level=INFO source=checkpoint.go:109 msg="Creating checkpoint" component=tsdb from_segment=361 to_segment=362 mint=1782698400000
�2026-06-29T03:00:04.896624873Z time=2026-06-29T03:00:04.896Z level=INFO source=head.go:1473 msg="WAL checkpoint complete" component=tsdb first=361 last=362 duration=215.81151ms
�2026-06-29T05:00:04.147826661Z time=2026-06-29T05:00:04.141Z level=INFO source=compact.go:590 msg="write block started" component=tsdb mint=1782698400602 maxt=1782705600000 ulid=01KW8VZD4X0TWZF4MTX6RN1VN4
�2026-06-29T05:00:04.549142960Z time=2026-06-29T05:00:04.548Z level=INFO source=compact.go:627 msg="write block completed" component=tsdb mint=1782698400602 maxt=1782705600000 ulid=01KW8VZD4X0TWZF4MTX6RN1VN4 duration=429.277508ms ooo=false
�2026-06-29T05:00:04.551789475Z time=2026-06-29T05:00:04.551Z level=INFO source=head.go:1508 msg="Head GC started" component=tsdb caller=truncateMemory
�2026-06-29T05:00:04.575877685Z time=2026-06-29T05:00:04.575Z level=INFO source=head.go:1512 msg="Head GC completed" component=tsdb caller=truncateMemory duration=23.483139ms
C2026-06-29T05:00:05.240236565Z time=2026-06-29T05:00:05.240Z level=INFO source=compact.go:562 msg="compact blocks" component=tsdb count=3 mint=1782671632763 maxt=1782691200000 ulid=01KW8VZDK2WPQC7PS0FBSC44CG sources="[01KW82EKY86TVKE79R6PHZYAHS 01KW87C8NH7YS8M5AVA476KP51 01KW8E7YEB10H5Z9AVG0VPYW6R]" duration=662.031259ms
�2026-06-29T05:00:05.249124835Z time=2026-06-29T05:00:05.248Z level=INFO source=db.go:2077 msg="Deleting obsolete block" component=tsdb block=01KW82EKY86TVKE79R6PHZYAHS
�2026-06-29T05:00:05.252470498Z time=2026-06-29T05:00:05.252Z level=INFO source=db.go:2077 msg="Deleting obsolete block" component=tsdb block=01KW8E7YEB10H5Z9AVG0VPYW6R
�2026-06-29T05:00:05.255973408Z time=2026-06-29T05:00:05.255Z level=INFO source=db.go:2077 msg="Deleting obsolete block" component=tsdb block=01KW87C8NH7YS8M5AVA476KP51
�2026-06-29T07:00:04.044546120Z time=2026-06-29T07:00:04.037Z level=INFO source=compact.go:590 msg="write block started" component=tsdb mint=1782705600602 maxt=1782712800000 ulid=01KW92V49S95SHFPQ88WX6EJRW
�2026-06-29T07:00:04.430469988Z time=2026-06-29T07:00:04.430Z level=INFO source=compact.go:627 msg="write block completed" component=tsdb mint=1782705600602 maxt=1782712800000 ulid=01KW92V49S95SHFPQ88WX6EJRW duration=407.035577ms ooo=false
�2026-06-29T07:00:04.434369004Z time=2026-06-29T07:00:04.434Z level=INFO source=db.go:2077 msg="Deleting obsolete block" component=tsdb block=01KVQFSMFQ28AKWPFSMK1MAARY
�2026-06-29T07:00:04.434575192Z time=2026-06-29T07:00:04.434Z level=INFO source=head.go:1508 msg="Head GC started" component=tsdb caller=truncateMemory
�2026-06-29T07:00:04.455969954Z time=2026-06-29T07:00:04.455Z level=INFO source=head.go:1512 msg="Head GC completed" component=tsdb caller=truncateMemory duration=21.289564ms
�2026-06-29T07:00:04.456650178Z time=2026-06-29T07:00:04.456Z level=INFO source=checkpoint.go:109 msg="Creating checkpoint" component=tsdb from_segment=363 to_segment=364 mint=1782712800000
�2026-06-29T07:00:04.862895844Z time=2026-06-29T07:00:04.862Z level=INFO source=head.go:1473 msg="WAL checkpoint complete" component=tsdb first=363 last=364 duration=406.394797ms
�2026-06-29T09:00:04.103985656Z time=2026-06-29T09:00:04.096Z level=INFO source=compact.go:590 msg="write block started" component=tsdb mint=1782712800602 maxt=1782720000000 ulid=01KW99PVKG455STX71ENCYBGTG
�2026-06-29T09:00:04.513412897Z time=2026-06-29T09:00:04.513Z level=INFO source=compact.go:627 msg="write block completed" component=tsdb mint=1782712800602 maxt=1782720000000 ulid=01KW99PVKG455STX71ENCYBGTG duration=441.170476ms ooo=false
�2026-06-29T09:00:04.519210414Z time=2026-06-29T09:00:04.518Z level=INFO source=head.go:1508 msg="Head GC started" component=tsdb caller=truncateMemory
�2026-06-29T09:00:04.538122251Z time=2026-06-29T09:00:04.537Z level=INFO source=head.go:1512 msg="Head GC completed" component=tsdb caller=truncateMemory duration=18.854369ms
�2026-06-29T11:00:04.016799140Z time=2026-06-29T11:00:03.997Z level=INFO source=compact.go:590 msg="write block started" component=tsdb mint=1782720000602 maxt=1782727200000 ulid=01KW9GJJRA172TM4EDDRKWQTCR
�2026-06-29T11:00:04.508509292Z time=2026-06-29T11:00:04.508Z level=INFO source=compact.go:627 msg="write block completed" component=tsdb mint=1782720000602 maxt=1782727200000 ulid=01KW9GJJRA172TM4EDDRKWQTCR duration=534.886827ms ooo=false
�2026-06-29T11:00:04.522162816Z time=2026-06-29T11:00:04.521Z level=INFO source=head.go:1508 msg="Head GC started" component=tsdb caller=truncateMemory
�2026-06-29T11:00:04.538547475Z time=2026-06-29T11:00:04.538Z level=INFO source=head.go:1512 msg="Head GC completed" component=tsdb caller=truncateMemory duration=16.523631ms
�2026-06-29T11:00:04.539847368Z time=2026-06-29T11:00:04.539Z level=INFO source=checkpoint.go:109 msg="Creating checkpoint" component=tsdb from_segment=365 to_segment=366 mint=1782727200000
�2026-06-29T11:00:04.918643457Z time=2026-06-29T11:00:04.918Z level=INFO source=head.go:1473 msg="WAL checkpoint complete" component=tsdb first=365 last=366 duration=378.88175ms
B2026-06-29T11:00:05.655559528Z time=2026-06-29T11:00:05.655Z level=INFO source=compact.go:562 msg="compact blocks" component=tsdb count=3 mint=1782691200602 maxt=1782712800000 ulid=01KW9GJKNQZMTA9V3N9Q69XBY7 sources="[01KW8N3NW4KH93CCESP53KSTG7 01KW8VZD4X0TWZF4MTX6RN1VN4 01KW92V49S95SHFPQ88WX6EJRW]" duration=736.27373ms
�2026-06-29T11:00:05.661352499Z time=2026-06-29T11:00:05.661Z level=INFO source=db.go:2077 msg="Deleting obsolete block" component=tsdb block=01KW8N3NW4KH93CCESP53KSTG7
�2026-06-29T11:00:05.665339524Z time=2026-06-29T11:00:05.665Z level=INFO source=db.go:2077 msg="Deleting obsolete block" component=tsdb block=01KW92V49S95SHFPQ88WX6EJRW
�2026-06-29T11:00:05.669270681Z time=2026-06-29T11:00:05.669Z level=INFO source=db.go:2077 msg="Deleting obsolete block" component=tsdb block=01KW8VZD4X0TWZF4MTX6RN1VN4
�2026-06-29T13:00:03.245358823Z time=2026-06-29T13:00:03.169Z level=INFO source=compact.go:590 msg="write block started" component=tsdb mint=1782727200602 maxt=1782734400000 ulid=01KW9QE94RYA1TSY77BMG8JHYK
�2026-06-29T13:00:05.587809290Z time=2026-06-29T13:00:05.587Z level=INFO source=compact.go:627 msg="write block completed" component=tsdb mint=1782727200602 maxt=1782734400000 ulid=01KW9QE94RYA1TSY77BMG8JHYK duration=2.492913445s ooo=false
�2026-06-29T13:00:05.609152706Z time=2026-06-29T13:00:05.609Z level=INFO source=db.go:2077 msg="Deleting obsolete block" component=tsdb block=01KVR4CYN9E4G2CT9JXBW8DZSD
�2026-06-29T13:00:05.609345912Z time=2026-06-29T13:00:05.609Z level=INFO source=head.go:1508 msg="Head GC started" component=tsdb caller=truncateMemory
�2026-06-29T13:00:05.668362336Z time=2026-06-29T13:00:05.667Z level=INFO source=head.go:1512 msg="Head GC completed" component=tsdb caller=truncateMemory duration=58.650746ms
�2026-06-28T20:03:48.384109678Z time=2026-06-28T20:03:48.382Z level=ERROR source=file.go:233 msg="Error adding file watch" component="discovery manager scrape" discovery=file config=default path=/prometheus/scrape_configs/ err="no such file or directory"
�2026-06-28T20:08:48.414983034Z time=2026-06-28T20:08:48.394Z level=ERROR source=file.go:233 msg="Error adding file watch" component="discovery manager scrape" discovery=file config=default path=/prometheus/scrape_configs/ err="no such file or directory"
�2026-06-28T20:13:48.414114972Z time=2026-06-28T20:13:48.399Z level=ERROR source=file.go:233 msg="Error adding file watch" component="discovery manager scrape" discovery=file config=default path=/prometheus/scrape_configs/ err="no such file or directory"
�2026-06-28T20:18:48.408910491Z time=2026-06-28T20:18:48.399Z level=ERROR source=file.go:233 msg="Error adding file watch" component="discovery manager scrape" discovery=file config=default path=/prometheus/scrape_configs/ err="no such file or directory"
�2026-06-28T20:23:48.392161652Z time=2026-06-28T20:23:48.388Z level=ERROR source=file.go:233 msg="Error adding file watch" component="discovery manager scrape" discovery=file config=default path=/prometheus/scrape_configs/ err="no such file or directory"
�2026-06-28T20:28:48.396142784Z time=2026-06-28T20:28:48.389Z level=ERROR source=file.go:233 msg="Error adding file watch" component="discovery manager scrape" discovery=file config=default path=/prometheus/scrape_configs/ err="no such file or directory"
�2026-06-28T20:33:48.382218002Z time=2026-06-28T20:33:48.381Z level=ERROR source=file.go:233 msg="Error adding file watch" component="discovery manager scrape" discovery=file config=default path=/prometheus/scrape_configs/ err="no such file or directory"
�2026-06-28T20:38:48.461231580Z time=2026-06-28T20:38:48.446Z level=ERROR source=file.go:233 msg="Error adding file watch" component="discovery manager scrape" discovery=file config=default path=/prometheus/scrape_configs/ err="no such file or directory"
�2026-06-28T20:43:48.413667928Z time=2026-06-28T20:43:48.398Z level=ERROR source=file.go:233 msg="Error adding file watch" component="discovery manager scrape" discovery=file config=default path=/prometheus/scrape_configs/ err="no such file or directory"
�2026-06-28T20:48:48.388197715Z time=2026-06-28T20:48:48.382Z level=ERROR source=file.go:233 msg="Error adding file watch" component="discovery manager scrape" discovery=file config=default path=/prometheus/scrape_configs/ err="no such file or directory"
�2026-06-28T20:53:48.409402177Z time=2026-06-28T20:53:48.393Z level=ERROR source=file.go:233 msg="Error adding file watch" component="discovery manager scrape" discovery=file config=default path=/prometheus/scrape_configs/ err="no such file or directory"
�2026-06-28T20:58:48.388051210Z time=2026-06-28T20:58:48.383Z level=ERROR source=file.go:233 msg="Error adding file watch" component="discovery manager scrape" discovery=file config=default path=/prometheus/scrape_configs/ err="no such file or directory"
�2026-06-28T21:03:48.401932496Z time=2026-06-28T21:03:48.392Z level=ERROR source=file.go:233 msg="Error adding file watch" component="discovery manager scrape" discovery=file config=default path=/prometheus/scrape_configs/ err="no such file or directory"
�2026-06-28T21:08:48.393321232Z time=2026-06-28T21:08:48.384Z level=ERROR source=file.go:233 msg="Error adding file watch" component="discovery manager scrape" discovery=file config=default path=/prometheus/scrape_configs/ err="no such file or directory"
�2026-06-28T21:13:48.428444144Z time=2026-06-28T21:13:48.426Z level=ERROR source=file.go:233 msg="Error adding file watch" component="discovery manager scrape" discovery=file config=default path=/prometheus/scrape_configs/ err="no such file or directory"
�2026-06-28T21:18:48.448649967Z time=2026-06-28T21:18:48.422Z level=ERROR source=file.go:233 msg="Error adding file watch" component="discovery manager scrape" discovery=file config=default path=/prometheus/scrape_configs/ err="no such file or directory"
�2026-06-28T21:23:48.436133110Z time=2026-06-28T21:23:48.421Z level=ERROR source=file.go:233 msg="Error adding file watch" component="discovery manager scrape" discovery=file config=default path=/prometheus/scrape_configs/ err="no such file or directory"
�2026-06-28T21:28:48.432378903Z time=2026-06-28T21:28:48.423Z level=ERROR source=file.go:233 msg="Error adding file watch" component="discovery manager scrape" discovery=file config=default path=/prometheus/scrape_configs/ err="no such file or directory"
�2026-06-28T21:33:48.382403255Z time=2026-06-28T21:33:48.382Z level=ERROR source=file.go:233 msg="Error adding file watch" component="discovery manager scrape" discovery=file config=default path=/prometheus/scrape_configs/ err="no such file or directory"
�2026-06-28T21:33:59.659162222Z time=2026-06-28T21:33:59.645Z level=INFO source=compact.go:590 msg="write block started" component=tsdb mint=1782671632763 maxt=1782676800000 ulid=01KW82EKY86TVKE79R6PHZYAHS
�2026-06-28T21:34:00.086336385Z time=2026-06-28T21:34:00.086Z level=INFO source=compact.go:627 msg="write block completed" component=tsdb mint=1782671632763 maxt=1782676800000 ulid=01KW82EKY86TVKE79R6PHZYAHS duration=461.240696ms ooo=false
�2026-06-28T21:34:00.100639181Z time=2026-06-28T21:34:00.100Z level=INFO source=db.go:2077 msg="Deleting obsolete block" component=tsdb block=01KVMXD0GM6VCR6V55YVE07Q62
�2026-06-28T21:34:00.102984516Z time=2026-06-28T21:34:00.102Z level=INFO source=db.go:2077 msg="Deleting obsolete block" component=tsdb block=01KVNJ038B6625YWFXQJ80DNMB
�2026-06-28T21:34:00.104674354Z time=2026-06-28T21:34:00.104Z level=INFO source=db.go:2077 msg="Deleting obsolete block" component=tsdb block=01KVP6K9QWQ4N30G0B0666DDGE
�2026-06-28T21:34:00.106695167Z time=2026-06-28T21:34:00.106Z level=INFO source=db.go:2077 msg="Deleting obsolete block" component=tsdb block=01KVM8SVB1HKWCG0A8EZYSNJKX
�2026-06-28T21:34:00.106819702Z time=2026-06-28T21:34:00.106Z level=INFO source=head.go:1508 msg="Head GC started" component=tsdb caller=truncateMemory
�2026-06-28T21:34:00.122926772Z time=2026-06-28T21:34:00.122Z level=INFO source=head.go:1512 msg="Head GC completed" component=tsdb caller=truncateMemory duration=16.132468ms
�2026-06-28T21:35:06.121609733Z time=2026-06-28T21:35:06.108Z level=WARN source=main.go:1221 msg="Received an OS signal, exiting gracefully..." signal=terminated
�2026-06-28T21:35:06.125683138Z time=2026-06-28T21:35:06.121Z level=INFO source=main.go:1246 msg="Stopping scrape discovery manager..."
�2026-06-28T21:35:06.133123146Z time=2026-06-28T21:35:06.122Z level=INFO source=main.go:1260 msg="Stopping notify discovery manager..."
�2026-06-28T21:35:06.133140539Z time=2026-06-28T21:35:06.124Z level=INFO source=main.go:1275 msg="Stopping rule manager manager..."
�2026-06-28T21:35:06.137368545Z time=2026-06-28T21:35:06.130Z level=INFO source=main.go:1242 msg="Scrape discovery manager stopped"
�2026-06-28T21:35:06.174844513Z time=2026-06-28T21:35:06.171Z level=INFO source=manager.go:223 msg="Stopping rule manager..." component="rule manager"
�2026-06-28T21:35:06.175292028Z time=2026-06-28T21:35:06.174Z level=INFO source=manager.go:239 msg="Rule manager stopped" component="rule manager"
}2026-06-28T21:35:06.175300003Z time=2026-06-28T21:35:06.174Z level=INFO source=main.go:1299 msg="Stopping scrape manager..."
~2026-06-28T21:35:06.177044534Z time=2026-06-28T21:35:06.176Z level=INFO source=main.go:1314 msg="Stopping tracing manager..."
x2026-06-28T21:35:06.177068700Z time=2026-06-28T21:35:06.177Z level=INFO source=main.go:1399 msg="Stopping reloaders..."
v2026-06-28T21:35:06.177144132Z time=2026-06-28T21:35:06.177Z level=INFO source=main.go:1488 msg="Stopping storage..."
y2026-06-28T21:35:06.180542804Z time=2026-06-28T21:35:06.179Z level=INFO source=main.go:1291 msg="Scrape manager stopped"
w2026-06-28T21:35:06.183467021Z time=2026-06-28T21:35:06.182Z level=INFO source=main.go:1271 msg="Rule manager stopped"
z2026-06-28T21:35:06.183611754Z time=2026-06-28T21:35:06.182Z level=INFO source=main.go:1310 msg="Tracing manager stopped"
t2026-06-28T21:35:06.183758131Z time=2026-06-28T21:35:06.182Z level=INFO source=main.go:1391 msg="Reloaders stopped"
�2026-06-28T21:35:06.184905996Z time=2026-06-28T21:35:06.184Z level=INFO source=main.go:1256 msg="Notify discovery manager stopped"
z2026-06-28T21:35:06.262732424Z time=2026-06-28T21:35:06.262Z level=INFO source=main.go:1566 msg="Stopping web handler..."
2026-06-28T21:35:06.262907414Z time=2026-06-28T21:35:06.262Z level=INFO source=main.go:1589 msg="Stopping notifier manager..."
�2026-06-28T21:35:06.262917413Z time=2026-06-28T21:35:06.262Z level=INFO source=manager.go:321 msg="Stopping notification manager..." component=notifier
{2026-06-28T21:35:06.262926009Z time=2026-06-28T21:35:06.262Z level=INFO source=main.go:1585 msg="Notifier manager stopped"
o2026-06-28T21:35:06.262930658Z time=2026-06-28T21:35:06.262Z level=INFO source=main.go:1484 msg="TSDB stopped"
v2026-06-28T21:35:06.266192401Z time=2026-06-28T21:35:06.266Z level=INFO source=main.go:1562 msg="Web handler stopped"
u2026-06-28T21:35:06.266210536Z time=2026-06-28T21:35:06.266Z level=INFO source=main.go:1600 msg="See you next time!"
2026-06-28T21:35:19.834198190Z ts=2026-06-28T21:35:19.830116559Z level=info msg="Starting scraped metadata watcher" component_path=/ component_id=prometheus.remote_write.prometheus subcomponent=rw remote_name=6a1c19 url=https://prometheus.monitoring.sutty.abyaya.la/api/v1/write
�2026-06-28T21:35:19.834203660Z ts=2026-06-28T21:35:19.830132389Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=d64d192b9721197cf6171a96bcef64c2 node_id=prometheus.remote_write.prometheus duration=111.714836ms
2026-06-28T21:35:19.834207377Z ts=2026-06-28T21:35:19.830228991Z level=info msg="Replaying WAL" component_path=/ component_id=prometheus.remote_write.prometheus subcomponent=rw remote_name=6a1c19 url=https://prometheus.monitoring.sutty.abyaya.la/api/v1/write queue=6a1c19
�2026-06-28T21:35:19.834211244Z ts=2026-06-28T21:35:19.830840364Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=d64d192b9721197cf6171a96bcef64c2 node_id=prometheus.scrape.containers_bearerauth duration=690.282µs
�2026-06-28T21:35:19.834735974Z ts=2026-06-28T21:35:19.831193892Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=d64d192b9721197cf6171a96bcef64c2 node_id=prometheus.scrape.containers_basicauth duration=339.962µs
�2026-06-28T21:35:19.834740262Z ts=2026-06-28T21:35:19.83153739Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=d64d192b9721197cf6171a96bcef64c2 node_id=prometheus.scrape.containers duration=332.658µs
�2026-06-28T21:35:19.835021304Z ts=2026-06-28T21:35:19.831876239Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=d64d192b9721197cf6171a96bcef64c2 node_id=prometheus.scrape.default duration=328.851µs
�2026-06-28T21:35:19.835025592Z ts=2026-06-28T21:35:19.831891247Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=d64d192b9721197cf6171a96bcef64c2 node_id=otel duration=4.699µs
�2026-06-28T21:35:19.835364631Z ts=2026-06-28T21:35:19.831896307Z level=info msg="finished complete graph evaluation" controller_path=/ controller_id="" trace_id=d64d192b9721197cf6171a96bcef64c2 duration=267.272241ms
i2026-06-28T21:35:19.835371244Z ts=2026-06-28T21:35:19.832381222Z level=info msg="{^_^} Alloy is running"
�2026-06-28T21:35:19.835375591Z ts=2026-06-28T21:35:19.83259177Z level=info msg="scheduling loaded components and services" controller_id=""
�2026-06-28T21:35:19.835380050Z ts=2026-06-28T21:35:19.834519486Z level=info msg="starting cluster node" service=cluster peers_count=0 peers="" advertise_addr=127.0.0.1:12345 minimum_cluster_size=0 minimum_size_wait_timeout=0s
�2026-06-28T21:35:19.843265337Z ts=2026-06-28T21:35:19.842392439Z level=error msg="failed to register collector with remote server" service=remotecfg id=ea827bbe-2a03-4225-9b9e-eba251d34885 name="" err="noop client"
�2026-06-28T21:35:19.844810332Z ts=2026-06-28T21:35:19.8447085Z level=info msg="peers changed" service=cluster peers_count=1 min_cluster_size=0 peers=monitoring.sutty.abyaya.la
�2026-06-28T21:35:19.847650750Z ts=2026-06-28T21:35:19.847579757Z level=info msg="now listening for http traffic" service=http addr=0.0.0.0:12345
2026-06-28T21:35:30.049221645Z ts=2026-06-28T21:35:30.048980449Z level=info msg="Done replaying WAL" component_path=/ component_id=prometheus.remote_write.prometheus subcomponent=rw remote_name=6a1c19 url=https://prometheus.monitoring.sutty.abyaya.la/api/v1/write duration=10.218799439s
�2026-06-28T23:31:45.727419343Z 2026/06/28 23:31:45 Unsolicited response received on idle HTTP channel starting with "SSH-2.0-Go\r\n"; err=<nil>
�2026-06-28T23:33:19.641970740Z ts=2026-06-28T23:33:19.569330345Z level=info msg="reporting Alloy stats" date=2026-06-28T23:33:19.565Z
p2026-06-28T23:33:19.755934216Z ts=2026-06-28T23:33:19.75574587Z level=info msg="usage report sent with success"
�2026-06-28T23:35:19.867041211Z ts=2026-06-28T23:35:19.866144019Z level=info msg="series GC completed" component_path=/ component_id=prometheus.remote_write.prometheus subcomponent=wal duration=26.692449ms
�2026-06-28T23:35:19.890642870Z ts=2026-06-28T23:35:19.875822449Z level=info msg="Creating checkpoint" component_path=/ component_id=prometheus.remote_write.prometheus subcomponent=wal from_segment=412 to_segment=414 mint=1782689418000
�2026-06-28T23:35:20.205042292Z ts=2026-06-28T23:35:20.204820415Z level=info msg="WAL checkpoint complete" component_path=/ component_id=prometheus.remote_write.prometheus subcomponent=wal first=412 last=414 duration=365.369986ms
�2026-06-29T00:43:45.748321056Z 2026/06/29 00:43:45 Unsolicited response received on idle HTTP channel starting with "SSH-2.0-Go\r\n"; err=<nil>
�2026-06-29T00:55:45.721176022Z 2026/06/29 00:55:45 Unsolicited response received on idle HTTP channel starting with "SSH-2.0-Go\r\n"; err=<nil>
�2026-06-29T01:35:20.300758116Z ts=2026-06-29T01:35:20.236109084Z level=info msg="series GC completed" component_path=/ component_id=prometheus.remote_write.prometheus subcomponent=wal duration=15.979159ms
�2026-06-29T02:23:45.722635562Z 2026/06/29 02:23:45 Unsolicited response received on idle HTTP channel starting with "SSH-2.0-Go\r\n"; err=<nil>
�2026-06-29T03:33:19.567587580Z ts=2026-06-29T03:33:19.564707086Z level=info msg="reporting Alloy stats" date=2026-06-29T03:33:19.564Z
q2026-06-29T03:33:19.646068268Z ts=2026-06-29T03:33:19.645646762Z level=info msg="usage report sent with success"
�2026-06-29T03:35:20.311780888Z ts=2026-06-29T03:35:20.311448661Z level=info msg="series GC completed" component_path=/ component_id=prometheus.remote_write.prometheus subcomponent=wal duration=4.938859ms
�2026-06-29T03:35:20.312890670Z ts=2026-06-29T03:35:20.312495886Z level=info msg="Creating checkpoint" component_path=/ component_id=prometheus.remote_write.prometheus subcomponent=wal from_segment=415 to_segment=416 mint=1782703818000
�2026-06-29T03:35:20.483627116Z ts=2026-06-29T03:35:20.483514212Z level=info msg="WAL checkpoint complete" component_path=/ component_id=prometheus.remote_write.prometheus subcomponent=wal first=415 last=416 duration=177.021914ms
�2026-06-29T04:27:45.721473586Z 2026/06/29 04:27:45 Unsolicited response received on idle HTTP channel starting with "SSH-2.0-Go\r\n"; err=<nil>
�2026-06-29T04:31:45.720218112Z 2026/06/29 04:31:45 Unsolicited response received on idle HTTP channel starting with "SSH-2.0-Go\r\n"; err=<nil>
�2026-06-29T04:35:45.719972929Z 2026/06/29 04:35:45 Unsolicited response received on idle HTTP channel starting with "SSH-2.0-Go\r\n"; err=<nil>
�2026-06-29T04:37:45.720197929Z 2026/06/29 04:37:45 Unsolicited response received on idle HTTP channel starting with "SSH-2.0-Go\r\n"; err=<nil>
�2026-06-29T05:35:20.574848440Z ts=2026-06-29T05:35:20.512906369Z level=info msg="series GC completed" component_path=/ component_id=prometheus.remote_write.prometheus subcomponent=wal duration=15.764211ms
�2026-06-29T07:33:19.658854263Z ts=2026-06-29T07:33:19.56551342Z level=info msg="reporting Alloy stats" date=2026-06-29T07:33:19.565Z
q2026-06-29T07:33:19.801925506Z ts=2026-06-29T07:33:19.801796783Z level=info msg="usage report sent with success"
�2026-06-29T07:35:20.591786813Z ts=2026-06-29T07:35:20.591488791Z level=info msg="series GC completed" component_path=/ component_id=prometheus.remote_write.prometheus subcomponent=wal duration=3.435512ms
�2026-06-29T07:35:20.628231608Z ts=2026-06-29T07:35:20.626559764Z level=info msg="Creating checkpoint" component_path=/ component_id=prometheus.remote_write.prometheus subcomponent=wal from_segment=417 to_segment=418 mint=1782718218000
�2026-06-29T07:35:20.760911497Z ts=2026-06-29T07:35:20.760795939Z level=info msg="WAL checkpoint complete" component_path=/ component_id=prometheus.remote_write.prometheus subcomponent=wal first=417 last=418 duration=181.868246ms
�2026-06-29T09:35:20.832057458Z ts=2026-06-29T09:35:20.78928268Z level=info msg="series GC completed" component_path=/ component_id=prometheus.remote_write.prometheus subcomponent=wal duration=3.055744ms
�2026-06-29T11:13:45.743723566Z 2026/06/29 11:13:45 Unsolicited response received on idle HTTP channel starting with "SSH-2.0-Go\r\n"; err=<nil>
�2026-06-29T11:33:19.605560110Z ts=2026-06-29T11:33:19.565353189Z level=info msg="reporting Alloy stats" date=2026-06-29T11:33:19.565Z
q2026-06-29T11:33:19.697074665Z ts=2026-06-29T11:33:19.696885838Z level=info msg="usage report sent with success"
�2026-06-29T11:35:20.850061748Z ts=2026-06-29T11:35:20.849858725Z level=info msg="series GC completed" component_path=/ component_id=prometheus.remote_write.prometheus subcomponent=wal duration=2.255816ms
�2026-06-29T11:35:20.871394256Z ts=2026-06-29T11:35:20.870349585Z level=info msg="Creating checkpoint" component_path=/ component_id=prometheus.remote_write.prometheus subcomponent=wal from_segment=419 to_segment=420 mint=1782732618000
�2026-06-29T11:35:20.987716127Z ts=2026-06-29T11:35:20.987570121Z level=info msg="WAL checkpoint complete" component_path=/ component_id=prometheus.remote_write.prometheus subcomponent=wal first=419 last=420 duration=139.966071ms
�2026-06-29T12:15:45.721565443Z 2026/06/29 12:15:45 Unsolicited response received on idle HTTP channel starting with "SSH-2.0-Go\r\n"; err=<nil>
�2026-06-29T12:25:45.723211317Z 2026/06/29 12:25:45 Unsolicited response received on idle HTTP channel starting with "SSH-2.0-Go\r\n"; err=<nil>
2026-06-28T19:33:47.571431433Z ts=2026-06-28T19:33:47.52090193Z level=info msg="Starting scraped metadata watcher" component_path=/ component_id=prometheus.remote_write.prometheus subcomponent=rw remote_name=6a1c19 url=https://prometheus.monitoring.sutty.abyaya.la/api/v1/write
�2026-06-28T19:33:47.576076516Z ts=2026-06-28T19:33:47.520915916Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=78c717cab8190f7ee0319e8bd9eb8d43 node_id=prometheus.remote_write.prometheus duration=156.659128ms
�2026-06-28T19:33:47.576086515Z ts=2026-06-28T19:33:47.521053084Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=78c717cab8190f7ee0319e8bd9eb8d43 node_id=discovery.dockerswarm.swarm duration=124.404µs
�2026-06-28T19:33:47.576093558Z ts=2026-06-28T19:33:47.521219538Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=78c717cab8190f7ee0319e8bd9eb8d43 node_id=discovery.relabel.metrics duration=161.454µs
�2026-06-28T19:33:47.576402291Z ts=2026-06-28T19:33:47.521254244Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=78c717cab8190f7ee0319e8bd9eb8d43 node_id=discovery.relabel.metrics_bearerauth duration=30.558µs
2026-06-28T19:33:47.576406790Z ts=2026-06-28T19:33:47.521477014Z level=info msg="Replaying WAL" component_path=/ component_id=prometheus.remote_write.prometheus subcomponent=rw remote_name=6a1c19 url=https://prometheus.monitoring.sutty.abyaya.la/api/v1/write queue=6a1c19
�2026-06-28T19:33:47.576410997Z ts=2026-06-28T19:33:47.543453096Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=78c717cab8190f7ee0319e8bd9eb8d43 node_id=prometheus.scrape.containers_bearerauth duration=22.192621ms
�2026-06-28T19:33:47.576414634Z ts=2026-06-28T19:33:47.543685124Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=78c717cab8190f7ee0319e8bd9eb8d43 node_id=discovery.relabel.metrics_basicauth duration=211.8µs
�2026-06-28T19:33:47.576418121Z ts=2026-06-28T19:33:47.545470953Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=78c717cab8190f7ee0319e8bd9eb8d43 node_id=prometheus.scrape.containers_basicauth duration=1.77573ms
�2026-06-28T19:33:47.576421707Z ts=2026-06-28T19:33:47.545689044Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=78c717cab8190f7ee0319e8bd9eb8d43 node_id=discovery.relabel.metrics_noauth duration=210.066µs
�2026-06-28T19:33:47.576425184Z ts=2026-06-28T19:33:47.550879567Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=78c717cab8190f7ee0319e8bd9eb8d43 node_id=prometheus.scrape.containers duration=5.182978ms
�2026-06-28T19:33:47.576428290Z ts=2026-06-28T19:33:47.552913825Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=78c717cab8190f7ee0319e8bd9eb8d43 node_id=discovery.docker.containers duration=2.026445ms
�2026-06-28T19:33:47.576434922Z ts=2026-06-28T19:33:47.552980251Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=78c717cab8190f7ee0319e8bd9eb8d43 node_id=prometheus.exporter.self.alloy duration=58.23µs
�2026-06-28T19:33:47.578710285Z ts=2026-06-28T19:33:47.557620816Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=78c717cab8190f7ee0319e8bd9eb8d43 node_id=prometheus.scrape.default duration=4.634102ms
�2026-06-28T19:33:47.579166957Z ts=2026-06-28T19:33:47.557653428Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=78c717cab8190f7ee0319e8bd9eb8d43 node_id=tracing duration=18.745µs
�2026-06-28T19:33:47.579192365Z ts=2026-06-28T19:33:47.55830159Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=78c717cab8190f7ee0319e8bd9eb8d43 node_id=logging duration=644.084µs
�2026-06-28T19:33:47.579197284Z ts=2026-06-28T19:33:47.558308282Z level=info msg="finished complete graph evaluation" controller_path=/ controller_id="" trace_id=78c717cab8190f7ee0319e8bd9eb8d43 duration=851.67724ms
i2026-06-28T19:33:47.579201702Z ts=2026-06-28T19:33:47.558658553Z level=info msg="{^_^} Alloy is running"
�2026-06-28T19:33:47.579206011Z ts=2026-06-28T19:33:47.565591113Z level=info msg="scheduling loaded components and services" controller_id=""
�2026-06-28T19:33:47.579209658Z ts=2026-06-28T19:33:47.570800212Z level=error msg="failed to register collector with remote server" service=remotecfg id=ea827bbe-2a03-4225-9b9e-eba251d34885 name="" err="noop client"
�2026-06-28T19:33:47.586887503Z ts=2026-06-28T19:33:47.57526729Z level=info msg="starting cluster node" service=cluster peers_count=0 peers="" advertise_addr=127.0.0.1:12345 minimum_cluster_size=0 minimum_size_wait_timeout=0s
�2026-06-28T19:33:47.586903304Z ts=2026-06-28T19:33:47.575539414Z level=info msg="peers changed" service=cluster peers_count=1 min_cluster_size=0 peers=monitoring.sutty.abyaya.la
�2026-06-28T19:33:47.586908423Z ts=2026-06-28T19:33:47.577443595Z level=info msg="now listening for http traffic" service=http addr=0.0.0.0:12345
2026-06-28T19:33:52.863260941Z ts=2026-06-28T19:33:52.86281064Z level=info msg="Done replaying WAL" component_path=/ component_id=prometheus.remote_write.prometheus subcomponent=rw remote_name=6a1c19 url=https://prometheus.monitoring.sutty.abyaya.la/api/v1/write duration=5.341352371s
�2026-06-28T20:21:45.725010719Z 2026/06/28 20:21:45 Unsolicited response received on idle HTTP channel starting with "SSH-2.0-Go\r\n"; err=<nil>
�2026-06-28T21:33:47.660605538Z ts=2026-06-28T21:33:47.598901959Z level=info msg="series GC completed" component_path=/ component_id=prometheus.remote_write.prometheus subcomponent=wal duration=11.618661ms
�2026-06-28T21:33:47.660647046Z ts=2026-06-28T21:33:47.654919349Z level=info msg="Creating checkpoint" component_path=/ component_id=prometheus.remote_write.prometheus subcomponent=wal from_segment=409 to_segment=411 mint=1782682121000
�2026-06-28T21:33:47.863031660Z ts=2026-06-28T21:33:47.862833166Z level=info msg="WAL checkpoint complete" component_path=/ component_id=prometheus.remote_write.prometheus subcomponent=wal first=409 last=411 duration=282.373382ms
22026-06-28T21:35:06.544999956Z interrupt received
�2026-06-28T21:35:06.579240599Z ts=2026-06-28T21:35:06.577395778Z level=info msg="node exited without error" controller_id="" node=ui
�2026-06-28T21:35:06.579671222Z ts=2026-06-28T21:35:06.57918862Z level=info msg="node exited without error" controller_id="" node=otel
�2026-06-28T21:35:06.580590356Z ts=2026-06-28T21:35:06.580351636Z level=info msg="node exited without error" controller_id="" node=labelstore
�2026-06-28T21:35:06.581124794Z ts=2026-06-28T21:35:06.581041487Z level=info msg="node exited without error" controller_id="" node=discovery.docker.linux
�2026-06-28T21:35:06.596877264Z ts=2026-06-28T21:35:06.590996017Z level=info msg="node exited without error" controller_id="" node=prometheus.scrape.containers
�2026-06-28T21:35:06.597244537Z ts=2026-06-28T21:35:06.596866384Z level=info msg="scrape manager stopped" component_path=/ component_id=prometheus.scrape.containers
�2026-06-28T21:35:06.598288276Z ts=2026-06-28T21:35:06.597688755Z level=info msg="node exited without error" controller_id="" node=prometheus.scrape.containers_basicauth
�2026-06-28T21:35:06.602489182Z ts=2026-06-28T21:35:06.598600155Z level=info msg="scrape manager stopped" component_path=/ component_id=prometheus.scrape.containers_basicauth
�2026-06-28T21:35:06.602504881Z ts=2026-06-28T21:35:06.598690645Z level=info msg="node exited without error" controller_id="" node=prometheus.scrape.containers_bearerauth
�2026-06-28T21:35:06.602509741Z ts=2026-06-28T21:35:06.601060767Z level=info msg="scrape manager stopped" component_path=/ component_id=prometheus.scrape.containers_bearerauth
�2026-06-28T21:35:06.602513920Z ts=2026-06-28T21:35:06.601436286Z level=info msg="node exited without error" controller_id="" node=prometheus.scrape.default
�2026-06-28T21:35:06.610976456Z ts=2026-06-28T21:35:06.610884261Z level=info msg="node exited without error" controller_id="" node=cluster
2026-06-28T21:35:06.618033681Z ts=2026-06-28T21:35:06.617059753Z level=info msg="Stopping remote storage..." component_path=/ component_id=prometheus.remote_write.prometheus subcomponent=rw remote_name=6a1c19 url=https://prometheus.monitoring.sutty.abyaya.la/api/v1/write
�2026-06-28T21:35:06.619691338Z ts=2026-06-28T21:35:06.617944652Z level=info msg="http server closed" service=http addr=[::]:12345 err="http: Server closed"
�2026-06-28T21:35:06.621640145Z ts=2026-06-28T21:35:06.621536439Z level=info msg="scrape manager stopped" component_path=/ component_id=prometheus.scrape.default
�2026-06-28T21:35:06.624162363Z ts=2026-06-28T21:35:06.621895727Z level=info msg="http server closed" service=http addr=memory err="http: Server closed"
�2026-06-28T21:35:06.624177762Z ts=2026-06-28T21:35:06.623679022Z level=info msg="node exited without error" controller_id="" node=http
�2026-06-28T21:35:06.625010233Z ts=2026-06-28T21:35:06.624712471Z level=error msg="failed to unregister collector with remote server" service=remotecfg id=ea827bbe-2a03-4225-9b9e-eba251d34885 err="noop client"
�2026-06-28T21:35:06.625343181Z ts=2026-06-28T21:35:06.624810306Z level=error msg="failed to unregister collector during service shutdown" service=remotecfg err="noop client"
�2026-06-28T21:35:06.625971125Z ts=2026-06-28T21:35:06.624893803Z level=info msg="node exited without error" controller_id="" node=remotecfg
�2026-06-28T21:35:06.626178948Z ts=2026-06-28T21:35:06.625889672Z level=info msg="node exited without error" controller_id="" node=livedebugging
not sure where the characters at the start of each line are coming from ``` +2026-06-28T23:23:26.925627488Z time=2026-06-28T23:23:26.925Z level=ERROR source=write_handler.go:230 msg="Out of order sample from remote write" component=web err="out of order sample" series="{__name__=\"up\", instance=\"10.0.1.5:3901\", job=\"prometheus.scrape.default\"}" timestamp=1782688872180 *2026-06-28T23:23:32.080351867Z time=2026-06-28T23:23:32.080Z level=ERROR source=write_handler.go:230 msg="Out of order sample from remote write" component=web err="out of order sample" series="{__name__=\"up\", instance=\"10.0.1.5:443\", job=\"prometheus.scrape.default\"}" timestamp=1782688878548 +2026-06-28T23:24:07.565748741Z time=2026-06-28T23:24:07.565Z level=ERROR source=write_handler.go:230 msg="Out of order sample from remote write" component=web err="out of order sample" series="{__name__=\"up\", instance=\"10.0.2.2:3901\", job=\"prometheus.scrape.default\"}" timestamp=1782688913003 )2026-06-28T23:24:22.730180956Z time=2026-06-28T23:24:22.729Z level=ERROR source=write_handler.go:230 msg="Out of order sample from remote write" component=web err="out of order sample" series="{__name__=\"up\", instance=\"10.0.1.5:80\", job=\"prometheus.scrape.default\"}" timestamp=1782688925846 )2026-06-28T23:42:09.414066062Z time=2026-06-28T23:42:09.392Z level=ERROR source=write_handler.go:230 msg="Out of order sample from remote write" component=web err="out of order sample" series="{__name__=\"up\", instance=\"10.0.2.2:80\", job=\"prometheus.scrape.default\"}" timestamp=1782689995354 �2026-06-29T01:00:03.964805615Z time=2026-06-29T01:00:03.952Z level=INFO source=compact.go:590 msg="write block started" component=tsdb mint=1782684000602 maxt=1782691200000 ulid=01KW8E7YEB10H5Z9AVG0VPYW6R �2026-06-29T01:00:04.461016619Z time=2026-06-29T01:00:04.460Z level=INFO source=compact.go:627 msg="write block completed" component=tsdb mint=1782684000602 maxt=1782691200000 ulid=01KW8E7YEB10H5Z9AVG0VPYW6R duration=554.69921ms ooo=false �2026-06-29T01:00:04.465857462Z time=2026-06-29T01:00:04.465Z level=INFO source=db.go:2077 msg="Deleting obsolete block" component=tsdb block=01KVPV6JGY6725101XFD30J2WT �2026-06-29T01:00:04.466102605Z time=2026-06-29T01:00:04.466Z level=INFO source=head.go:1508 msg="Head GC started" component=tsdb caller=truncateMemory �2026-06-29T01:00:04.595545506Z time=2026-06-29T01:00:04.595Z level=INFO source=head.go:1512 msg="Head GC completed" component=tsdb caller=truncateMemory duration=129.217203ms �2026-06-29T03:00:04.146793697Z time=2026-06-29T03:00:04.131Z level=INFO source=compact.go:590 msg="write block started" component=tsdb mint=1782691200602 maxt=1782698400000 ulid=01KW8N3NW4KH93CCESP53KSTG7 �2026-06-29T03:00:04.656181629Z time=2026-06-29T03:00:04.655Z level=INFO source=compact.go:627 msg="write block completed" component=tsdb mint=1782691200602 maxt=1782698400000 ulid=01KW8N3NW4KH93CCESP53KSTG7 duration=562.883823ms ooo=false �2026-06-29T03:00:04.664594682Z time=2026-06-29T03:00:04.658Z level=INFO source=head.go:1508 msg="Head GC started" component=tsdb caller=truncateMemory �2026-06-29T03:00:04.684713199Z time=2026-06-29T03:00:04.679Z level=INFO source=head.go:1512 msg="Head GC completed" component=tsdb caller=truncateMemory duration=21.386971ms �2026-06-29T03:00:04.684746793Z time=2026-06-29T03:00:04.680Z level=INFO source=checkpoint.go:109 msg="Creating checkpoint" component=tsdb from_segment=361 to_segment=362 mint=1782698400000 �2026-06-29T03:00:04.896624873Z time=2026-06-29T03:00:04.896Z level=INFO source=head.go:1473 msg="WAL checkpoint complete" component=tsdb first=361 last=362 duration=215.81151ms �2026-06-29T05:00:04.147826661Z time=2026-06-29T05:00:04.141Z level=INFO source=compact.go:590 msg="write block started" component=tsdb mint=1782698400602 maxt=1782705600000 ulid=01KW8VZD4X0TWZF4MTX6RN1VN4 �2026-06-29T05:00:04.549142960Z time=2026-06-29T05:00:04.548Z level=INFO source=compact.go:627 msg="write block completed" component=tsdb mint=1782698400602 maxt=1782705600000 ulid=01KW8VZD4X0TWZF4MTX6RN1VN4 duration=429.277508ms ooo=false �2026-06-29T05:00:04.551789475Z time=2026-06-29T05:00:04.551Z level=INFO source=head.go:1508 msg="Head GC started" component=tsdb caller=truncateMemory �2026-06-29T05:00:04.575877685Z time=2026-06-29T05:00:04.575Z level=INFO source=head.go:1512 msg="Head GC completed" component=tsdb caller=truncateMemory duration=23.483139ms C2026-06-29T05:00:05.240236565Z time=2026-06-29T05:00:05.240Z level=INFO source=compact.go:562 msg="compact blocks" component=tsdb count=3 mint=1782671632763 maxt=1782691200000 ulid=01KW8VZDK2WPQC7PS0FBSC44CG sources="[01KW82EKY86TVKE79R6PHZYAHS 01KW87C8NH7YS8M5AVA476KP51 01KW8E7YEB10H5Z9AVG0VPYW6R]" duration=662.031259ms �2026-06-29T05:00:05.249124835Z time=2026-06-29T05:00:05.248Z level=INFO source=db.go:2077 msg="Deleting obsolete block" component=tsdb block=01KW82EKY86TVKE79R6PHZYAHS �2026-06-29T05:00:05.252470498Z time=2026-06-29T05:00:05.252Z level=INFO source=db.go:2077 msg="Deleting obsolete block" component=tsdb block=01KW8E7YEB10H5Z9AVG0VPYW6R �2026-06-29T05:00:05.255973408Z time=2026-06-29T05:00:05.255Z level=INFO source=db.go:2077 msg="Deleting obsolete block" component=tsdb block=01KW87C8NH7YS8M5AVA476KP51 �2026-06-29T07:00:04.044546120Z time=2026-06-29T07:00:04.037Z level=INFO source=compact.go:590 msg="write block started" component=tsdb mint=1782705600602 maxt=1782712800000 ulid=01KW92V49S95SHFPQ88WX6EJRW �2026-06-29T07:00:04.430469988Z time=2026-06-29T07:00:04.430Z level=INFO source=compact.go:627 msg="write block completed" component=tsdb mint=1782705600602 maxt=1782712800000 ulid=01KW92V49S95SHFPQ88WX6EJRW duration=407.035577ms ooo=false �2026-06-29T07:00:04.434369004Z time=2026-06-29T07:00:04.434Z level=INFO source=db.go:2077 msg="Deleting obsolete block" component=tsdb block=01KVQFSMFQ28AKWPFSMK1MAARY �2026-06-29T07:00:04.434575192Z time=2026-06-29T07:00:04.434Z level=INFO source=head.go:1508 msg="Head GC started" component=tsdb caller=truncateMemory �2026-06-29T07:00:04.455969954Z time=2026-06-29T07:00:04.455Z level=INFO source=head.go:1512 msg="Head GC completed" component=tsdb caller=truncateMemory duration=21.289564ms �2026-06-29T07:00:04.456650178Z time=2026-06-29T07:00:04.456Z level=INFO source=checkpoint.go:109 msg="Creating checkpoint" component=tsdb from_segment=363 to_segment=364 mint=1782712800000 �2026-06-29T07:00:04.862895844Z time=2026-06-29T07:00:04.862Z level=INFO source=head.go:1473 msg="WAL checkpoint complete" component=tsdb first=363 last=364 duration=406.394797ms �2026-06-29T09:00:04.103985656Z time=2026-06-29T09:00:04.096Z level=INFO source=compact.go:590 msg="write block started" component=tsdb mint=1782712800602 maxt=1782720000000 ulid=01KW99PVKG455STX71ENCYBGTG �2026-06-29T09:00:04.513412897Z time=2026-06-29T09:00:04.513Z level=INFO source=compact.go:627 msg="write block completed" component=tsdb mint=1782712800602 maxt=1782720000000 ulid=01KW99PVKG455STX71ENCYBGTG duration=441.170476ms ooo=false �2026-06-29T09:00:04.519210414Z time=2026-06-29T09:00:04.518Z level=INFO source=head.go:1508 msg="Head GC started" component=tsdb caller=truncateMemory �2026-06-29T09:00:04.538122251Z time=2026-06-29T09:00:04.537Z level=INFO source=head.go:1512 msg="Head GC completed" component=tsdb caller=truncateMemory duration=18.854369ms �2026-06-29T11:00:04.016799140Z time=2026-06-29T11:00:03.997Z level=INFO source=compact.go:590 msg="write block started" component=tsdb mint=1782720000602 maxt=1782727200000 ulid=01KW9GJJRA172TM4EDDRKWQTCR �2026-06-29T11:00:04.508509292Z time=2026-06-29T11:00:04.508Z level=INFO source=compact.go:627 msg="write block completed" component=tsdb mint=1782720000602 maxt=1782727200000 ulid=01KW9GJJRA172TM4EDDRKWQTCR duration=534.886827ms ooo=false �2026-06-29T11:00:04.522162816Z time=2026-06-29T11:00:04.521Z level=INFO source=head.go:1508 msg="Head GC started" component=tsdb caller=truncateMemory �2026-06-29T11:00:04.538547475Z time=2026-06-29T11:00:04.538Z level=INFO source=head.go:1512 msg="Head GC completed" component=tsdb caller=truncateMemory duration=16.523631ms �2026-06-29T11:00:04.539847368Z time=2026-06-29T11:00:04.539Z level=INFO source=checkpoint.go:109 msg="Creating checkpoint" component=tsdb from_segment=365 to_segment=366 mint=1782727200000 �2026-06-29T11:00:04.918643457Z time=2026-06-29T11:00:04.918Z level=INFO source=head.go:1473 msg="WAL checkpoint complete" component=tsdb first=365 last=366 duration=378.88175ms B2026-06-29T11:00:05.655559528Z time=2026-06-29T11:00:05.655Z level=INFO source=compact.go:562 msg="compact blocks" component=tsdb count=3 mint=1782691200602 maxt=1782712800000 ulid=01KW9GJKNQZMTA9V3N9Q69XBY7 sources="[01KW8N3NW4KH93CCESP53KSTG7 01KW8VZD4X0TWZF4MTX6RN1VN4 01KW92V49S95SHFPQ88WX6EJRW]" duration=736.27373ms �2026-06-29T11:00:05.661352499Z time=2026-06-29T11:00:05.661Z level=INFO source=db.go:2077 msg="Deleting obsolete block" component=tsdb block=01KW8N3NW4KH93CCESP53KSTG7 �2026-06-29T11:00:05.665339524Z time=2026-06-29T11:00:05.665Z level=INFO source=db.go:2077 msg="Deleting obsolete block" component=tsdb block=01KW92V49S95SHFPQ88WX6EJRW �2026-06-29T11:00:05.669270681Z time=2026-06-29T11:00:05.669Z level=INFO source=db.go:2077 msg="Deleting obsolete block" component=tsdb block=01KW8VZD4X0TWZF4MTX6RN1VN4 �2026-06-29T13:00:03.245358823Z time=2026-06-29T13:00:03.169Z level=INFO source=compact.go:590 msg="write block started" component=tsdb mint=1782727200602 maxt=1782734400000 ulid=01KW9QE94RYA1TSY77BMG8JHYK �2026-06-29T13:00:05.587809290Z time=2026-06-29T13:00:05.587Z level=INFO source=compact.go:627 msg="write block completed" component=tsdb mint=1782727200602 maxt=1782734400000 ulid=01KW9QE94RYA1TSY77BMG8JHYK duration=2.492913445s ooo=false �2026-06-29T13:00:05.609152706Z time=2026-06-29T13:00:05.609Z level=INFO source=db.go:2077 msg="Deleting obsolete block" component=tsdb block=01KVR4CYN9E4G2CT9JXBW8DZSD �2026-06-29T13:00:05.609345912Z time=2026-06-29T13:00:05.609Z level=INFO source=head.go:1508 msg="Head GC started" component=tsdb caller=truncateMemory �2026-06-29T13:00:05.668362336Z time=2026-06-29T13:00:05.667Z level=INFO source=head.go:1512 msg="Head GC completed" component=tsdb caller=truncateMemory duration=58.650746ms �2026-06-28T20:03:48.384109678Z time=2026-06-28T20:03:48.382Z level=ERROR source=file.go:233 msg="Error adding file watch" component="discovery manager scrape" discovery=file config=default path=/prometheus/scrape_configs/ err="no such file or directory" �2026-06-28T20:08:48.414983034Z time=2026-06-28T20:08:48.394Z level=ERROR source=file.go:233 msg="Error adding file watch" component="discovery manager scrape" discovery=file config=default path=/prometheus/scrape_configs/ err="no such file or directory" �2026-06-28T20:13:48.414114972Z time=2026-06-28T20:13:48.399Z level=ERROR source=file.go:233 msg="Error adding file watch" component="discovery manager scrape" discovery=file config=default path=/prometheus/scrape_configs/ err="no such file or directory" �2026-06-28T20:18:48.408910491Z time=2026-06-28T20:18:48.399Z level=ERROR source=file.go:233 msg="Error adding file watch" component="discovery manager scrape" discovery=file config=default path=/prometheus/scrape_configs/ err="no such file or directory" �2026-06-28T20:23:48.392161652Z time=2026-06-28T20:23:48.388Z level=ERROR source=file.go:233 msg="Error adding file watch" component="discovery manager scrape" discovery=file config=default path=/prometheus/scrape_configs/ err="no such file or directory" �2026-06-28T20:28:48.396142784Z time=2026-06-28T20:28:48.389Z level=ERROR source=file.go:233 msg="Error adding file watch" component="discovery manager scrape" discovery=file config=default path=/prometheus/scrape_configs/ err="no such file or directory" �2026-06-28T20:33:48.382218002Z time=2026-06-28T20:33:48.381Z level=ERROR source=file.go:233 msg="Error adding file watch" component="discovery manager scrape" discovery=file config=default path=/prometheus/scrape_configs/ err="no such file or directory" �2026-06-28T20:38:48.461231580Z time=2026-06-28T20:38:48.446Z level=ERROR source=file.go:233 msg="Error adding file watch" component="discovery manager scrape" discovery=file config=default path=/prometheus/scrape_configs/ err="no such file or directory" �2026-06-28T20:43:48.413667928Z time=2026-06-28T20:43:48.398Z level=ERROR source=file.go:233 msg="Error adding file watch" component="discovery manager scrape" discovery=file config=default path=/prometheus/scrape_configs/ err="no such file or directory" �2026-06-28T20:48:48.388197715Z time=2026-06-28T20:48:48.382Z level=ERROR source=file.go:233 msg="Error adding file watch" component="discovery manager scrape" discovery=file config=default path=/prometheus/scrape_configs/ err="no such file or directory" �2026-06-28T20:53:48.409402177Z time=2026-06-28T20:53:48.393Z level=ERROR source=file.go:233 msg="Error adding file watch" component="discovery manager scrape" discovery=file config=default path=/prometheus/scrape_configs/ err="no such file or directory" �2026-06-28T20:58:48.388051210Z time=2026-06-28T20:58:48.383Z level=ERROR source=file.go:233 msg="Error adding file watch" component="discovery manager scrape" discovery=file config=default path=/prometheus/scrape_configs/ err="no such file or directory" �2026-06-28T21:03:48.401932496Z time=2026-06-28T21:03:48.392Z level=ERROR source=file.go:233 msg="Error adding file watch" component="discovery manager scrape" discovery=file config=default path=/prometheus/scrape_configs/ err="no such file or directory" �2026-06-28T21:08:48.393321232Z time=2026-06-28T21:08:48.384Z level=ERROR source=file.go:233 msg="Error adding file watch" component="discovery manager scrape" discovery=file config=default path=/prometheus/scrape_configs/ err="no such file or directory" �2026-06-28T21:13:48.428444144Z time=2026-06-28T21:13:48.426Z level=ERROR source=file.go:233 msg="Error adding file watch" component="discovery manager scrape" discovery=file config=default path=/prometheus/scrape_configs/ err="no such file or directory" �2026-06-28T21:18:48.448649967Z time=2026-06-28T21:18:48.422Z level=ERROR source=file.go:233 msg="Error adding file watch" component="discovery manager scrape" discovery=file config=default path=/prometheus/scrape_configs/ err="no such file or directory" �2026-06-28T21:23:48.436133110Z time=2026-06-28T21:23:48.421Z level=ERROR source=file.go:233 msg="Error adding file watch" component="discovery manager scrape" discovery=file config=default path=/prometheus/scrape_configs/ err="no such file or directory" �2026-06-28T21:28:48.432378903Z time=2026-06-28T21:28:48.423Z level=ERROR source=file.go:233 msg="Error adding file watch" component="discovery manager scrape" discovery=file config=default path=/prometheus/scrape_configs/ err="no such file or directory" �2026-06-28T21:33:48.382403255Z time=2026-06-28T21:33:48.382Z level=ERROR source=file.go:233 msg="Error adding file watch" component="discovery manager scrape" discovery=file config=default path=/prometheus/scrape_configs/ err="no such file or directory" �2026-06-28T21:33:59.659162222Z time=2026-06-28T21:33:59.645Z level=INFO source=compact.go:590 msg="write block started" component=tsdb mint=1782671632763 maxt=1782676800000 ulid=01KW82EKY86TVKE79R6PHZYAHS �2026-06-28T21:34:00.086336385Z time=2026-06-28T21:34:00.086Z level=INFO source=compact.go:627 msg="write block completed" component=tsdb mint=1782671632763 maxt=1782676800000 ulid=01KW82EKY86TVKE79R6PHZYAHS duration=461.240696ms ooo=false �2026-06-28T21:34:00.100639181Z time=2026-06-28T21:34:00.100Z level=INFO source=db.go:2077 msg="Deleting obsolete block" component=tsdb block=01KVMXD0GM6VCR6V55YVE07Q62 �2026-06-28T21:34:00.102984516Z time=2026-06-28T21:34:00.102Z level=INFO source=db.go:2077 msg="Deleting obsolete block" component=tsdb block=01KVNJ038B6625YWFXQJ80DNMB �2026-06-28T21:34:00.104674354Z time=2026-06-28T21:34:00.104Z level=INFO source=db.go:2077 msg="Deleting obsolete block" component=tsdb block=01KVP6K9QWQ4N30G0B0666DDGE �2026-06-28T21:34:00.106695167Z time=2026-06-28T21:34:00.106Z level=INFO source=db.go:2077 msg="Deleting obsolete block" component=tsdb block=01KVM8SVB1HKWCG0A8EZYSNJKX �2026-06-28T21:34:00.106819702Z time=2026-06-28T21:34:00.106Z level=INFO source=head.go:1508 msg="Head GC started" component=tsdb caller=truncateMemory �2026-06-28T21:34:00.122926772Z time=2026-06-28T21:34:00.122Z level=INFO source=head.go:1512 msg="Head GC completed" component=tsdb caller=truncateMemory duration=16.132468ms �2026-06-28T21:35:06.121609733Z time=2026-06-28T21:35:06.108Z level=WARN source=main.go:1221 msg="Received an OS signal, exiting gracefully..." signal=terminated �2026-06-28T21:35:06.125683138Z time=2026-06-28T21:35:06.121Z level=INFO source=main.go:1246 msg="Stopping scrape discovery manager..." �2026-06-28T21:35:06.133123146Z time=2026-06-28T21:35:06.122Z level=INFO source=main.go:1260 msg="Stopping notify discovery manager..." �2026-06-28T21:35:06.133140539Z time=2026-06-28T21:35:06.124Z level=INFO source=main.go:1275 msg="Stopping rule manager manager..." �2026-06-28T21:35:06.137368545Z time=2026-06-28T21:35:06.130Z level=INFO source=main.go:1242 msg="Scrape discovery manager stopped" �2026-06-28T21:35:06.174844513Z time=2026-06-28T21:35:06.171Z level=INFO source=manager.go:223 msg="Stopping rule manager..." component="rule manager" �2026-06-28T21:35:06.175292028Z time=2026-06-28T21:35:06.174Z level=INFO source=manager.go:239 msg="Rule manager stopped" component="rule manager" }2026-06-28T21:35:06.175300003Z time=2026-06-28T21:35:06.174Z level=INFO source=main.go:1299 msg="Stopping scrape manager..." ~2026-06-28T21:35:06.177044534Z time=2026-06-28T21:35:06.176Z level=INFO source=main.go:1314 msg="Stopping tracing manager..." x2026-06-28T21:35:06.177068700Z time=2026-06-28T21:35:06.177Z level=INFO source=main.go:1399 msg="Stopping reloaders..." v2026-06-28T21:35:06.177144132Z time=2026-06-28T21:35:06.177Z level=INFO source=main.go:1488 msg="Stopping storage..." y2026-06-28T21:35:06.180542804Z time=2026-06-28T21:35:06.179Z level=INFO source=main.go:1291 msg="Scrape manager stopped" w2026-06-28T21:35:06.183467021Z time=2026-06-28T21:35:06.182Z level=INFO source=main.go:1271 msg="Rule manager stopped" z2026-06-28T21:35:06.183611754Z time=2026-06-28T21:35:06.182Z level=INFO source=main.go:1310 msg="Tracing manager stopped" t2026-06-28T21:35:06.183758131Z time=2026-06-28T21:35:06.182Z level=INFO source=main.go:1391 msg="Reloaders stopped" �2026-06-28T21:35:06.184905996Z time=2026-06-28T21:35:06.184Z level=INFO source=main.go:1256 msg="Notify discovery manager stopped" z2026-06-28T21:35:06.262732424Z time=2026-06-28T21:35:06.262Z level=INFO source=main.go:1566 msg="Stopping web handler..." 2026-06-28T21:35:06.262907414Z time=2026-06-28T21:35:06.262Z level=INFO source=main.go:1589 msg="Stopping notifier manager..." �2026-06-28T21:35:06.262917413Z time=2026-06-28T21:35:06.262Z level=INFO source=manager.go:321 msg="Stopping notification manager..." component=notifier {2026-06-28T21:35:06.262926009Z time=2026-06-28T21:35:06.262Z level=INFO source=main.go:1585 msg="Notifier manager stopped" o2026-06-28T21:35:06.262930658Z time=2026-06-28T21:35:06.262Z level=INFO source=main.go:1484 msg="TSDB stopped" v2026-06-28T21:35:06.266192401Z time=2026-06-28T21:35:06.266Z level=INFO source=main.go:1562 msg="Web handler stopped" u2026-06-28T21:35:06.266210536Z time=2026-06-28T21:35:06.266Z level=INFO source=main.go:1600 msg="See you next time!" 2026-06-28T21:35:19.834198190Z ts=2026-06-28T21:35:19.830116559Z level=info msg="Starting scraped metadata watcher" component_path=/ component_id=prometheus.remote_write.prometheus subcomponent=rw remote_name=6a1c19 url=https://prometheus.monitoring.sutty.abyaya.la/api/v1/write �2026-06-28T21:35:19.834203660Z ts=2026-06-28T21:35:19.830132389Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=d64d192b9721197cf6171a96bcef64c2 node_id=prometheus.remote_write.prometheus duration=111.714836ms 2026-06-28T21:35:19.834207377Z ts=2026-06-28T21:35:19.830228991Z level=info msg="Replaying WAL" component_path=/ component_id=prometheus.remote_write.prometheus subcomponent=rw remote_name=6a1c19 url=https://prometheus.monitoring.sutty.abyaya.la/api/v1/write queue=6a1c19 �2026-06-28T21:35:19.834211244Z ts=2026-06-28T21:35:19.830840364Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=d64d192b9721197cf6171a96bcef64c2 node_id=prometheus.scrape.containers_bearerauth duration=690.282µs �2026-06-28T21:35:19.834735974Z ts=2026-06-28T21:35:19.831193892Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=d64d192b9721197cf6171a96bcef64c2 node_id=prometheus.scrape.containers_basicauth duration=339.962µs �2026-06-28T21:35:19.834740262Z ts=2026-06-28T21:35:19.83153739Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=d64d192b9721197cf6171a96bcef64c2 node_id=prometheus.scrape.containers duration=332.658µs �2026-06-28T21:35:19.835021304Z ts=2026-06-28T21:35:19.831876239Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=d64d192b9721197cf6171a96bcef64c2 node_id=prometheus.scrape.default duration=328.851µs �2026-06-28T21:35:19.835025592Z ts=2026-06-28T21:35:19.831891247Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=d64d192b9721197cf6171a96bcef64c2 node_id=otel duration=4.699µs �2026-06-28T21:35:19.835364631Z ts=2026-06-28T21:35:19.831896307Z level=info msg="finished complete graph evaluation" controller_path=/ controller_id="" trace_id=d64d192b9721197cf6171a96bcef64c2 duration=267.272241ms i2026-06-28T21:35:19.835371244Z ts=2026-06-28T21:35:19.832381222Z level=info msg="{^_^} Alloy is running" �2026-06-28T21:35:19.835375591Z ts=2026-06-28T21:35:19.83259177Z level=info msg="scheduling loaded components and services" controller_id="" �2026-06-28T21:35:19.835380050Z ts=2026-06-28T21:35:19.834519486Z level=info msg="starting cluster node" service=cluster peers_count=0 peers="" advertise_addr=127.0.0.1:12345 minimum_cluster_size=0 minimum_size_wait_timeout=0s �2026-06-28T21:35:19.843265337Z ts=2026-06-28T21:35:19.842392439Z level=error msg="failed to register collector with remote server" service=remotecfg id=ea827bbe-2a03-4225-9b9e-eba251d34885 name="" err="noop client" �2026-06-28T21:35:19.844810332Z ts=2026-06-28T21:35:19.8447085Z level=info msg="peers changed" service=cluster peers_count=1 min_cluster_size=0 peers=monitoring.sutty.abyaya.la �2026-06-28T21:35:19.847650750Z ts=2026-06-28T21:35:19.847579757Z level=info msg="now listening for http traffic" service=http addr=0.0.0.0:12345 2026-06-28T21:35:30.049221645Z ts=2026-06-28T21:35:30.048980449Z level=info msg="Done replaying WAL" component_path=/ component_id=prometheus.remote_write.prometheus subcomponent=rw remote_name=6a1c19 url=https://prometheus.monitoring.sutty.abyaya.la/api/v1/write duration=10.218799439s �2026-06-28T23:31:45.727419343Z 2026/06/28 23:31:45 Unsolicited response received on idle HTTP channel starting with "SSH-2.0-Go\r\n"; err=<nil> �2026-06-28T23:33:19.641970740Z ts=2026-06-28T23:33:19.569330345Z level=info msg="reporting Alloy stats" date=2026-06-28T23:33:19.565Z p2026-06-28T23:33:19.755934216Z ts=2026-06-28T23:33:19.75574587Z level=info msg="usage report sent with success" �2026-06-28T23:35:19.867041211Z ts=2026-06-28T23:35:19.866144019Z level=info msg="series GC completed" component_path=/ component_id=prometheus.remote_write.prometheus subcomponent=wal duration=26.692449ms �2026-06-28T23:35:19.890642870Z ts=2026-06-28T23:35:19.875822449Z level=info msg="Creating checkpoint" component_path=/ component_id=prometheus.remote_write.prometheus subcomponent=wal from_segment=412 to_segment=414 mint=1782689418000 �2026-06-28T23:35:20.205042292Z ts=2026-06-28T23:35:20.204820415Z level=info msg="WAL checkpoint complete" component_path=/ component_id=prometheus.remote_write.prometheus subcomponent=wal first=412 last=414 duration=365.369986ms �2026-06-29T00:43:45.748321056Z 2026/06/29 00:43:45 Unsolicited response received on idle HTTP channel starting with "SSH-2.0-Go\r\n"; err=<nil> �2026-06-29T00:55:45.721176022Z 2026/06/29 00:55:45 Unsolicited response received on idle HTTP channel starting with "SSH-2.0-Go\r\n"; err=<nil> �2026-06-29T01:35:20.300758116Z ts=2026-06-29T01:35:20.236109084Z level=info msg="series GC completed" component_path=/ component_id=prometheus.remote_write.prometheus subcomponent=wal duration=15.979159ms �2026-06-29T02:23:45.722635562Z 2026/06/29 02:23:45 Unsolicited response received on idle HTTP channel starting with "SSH-2.0-Go\r\n"; err=<nil> �2026-06-29T03:33:19.567587580Z ts=2026-06-29T03:33:19.564707086Z level=info msg="reporting Alloy stats" date=2026-06-29T03:33:19.564Z q2026-06-29T03:33:19.646068268Z ts=2026-06-29T03:33:19.645646762Z level=info msg="usage report sent with success" �2026-06-29T03:35:20.311780888Z ts=2026-06-29T03:35:20.311448661Z level=info msg="series GC completed" component_path=/ component_id=prometheus.remote_write.prometheus subcomponent=wal duration=4.938859ms �2026-06-29T03:35:20.312890670Z ts=2026-06-29T03:35:20.312495886Z level=info msg="Creating checkpoint" component_path=/ component_id=prometheus.remote_write.prometheus subcomponent=wal from_segment=415 to_segment=416 mint=1782703818000 �2026-06-29T03:35:20.483627116Z ts=2026-06-29T03:35:20.483514212Z level=info msg="WAL checkpoint complete" component_path=/ component_id=prometheus.remote_write.prometheus subcomponent=wal first=415 last=416 duration=177.021914ms �2026-06-29T04:27:45.721473586Z 2026/06/29 04:27:45 Unsolicited response received on idle HTTP channel starting with "SSH-2.0-Go\r\n"; err=<nil> �2026-06-29T04:31:45.720218112Z 2026/06/29 04:31:45 Unsolicited response received on idle HTTP channel starting with "SSH-2.0-Go\r\n"; err=<nil> �2026-06-29T04:35:45.719972929Z 2026/06/29 04:35:45 Unsolicited response received on idle HTTP channel starting with "SSH-2.0-Go\r\n"; err=<nil> �2026-06-29T04:37:45.720197929Z 2026/06/29 04:37:45 Unsolicited response received on idle HTTP channel starting with "SSH-2.0-Go\r\n"; err=<nil> �2026-06-29T05:35:20.574848440Z ts=2026-06-29T05:35:20.512906369Z level=info msg="series GC completed" component_path=/ component_id=prometheus.remote_write.prometheus subcomponent=wal duration=15.764211ms �2026-06-29T07:33:19.658854263Z ts=2026-06-29T07:33:19.56551342Z level=info msg="reporting Alloy stats" date=2026-06-29T07:33:19.565Z q2026-06-29T07:33:19.801925506Z ts=2026-06-29T07:33:19.801796783Z level=info msg="usage report sent with success" �2026-06-29T07:35:20.591786813Z ts=2026-06-29T07:35:20.591488791Z level=info msg="series GC completed" component_path=/ component_id=prometheus.remote_write.prometheus subcomponent=wal duration=3.435512ms �2026-06-29T07:35:20.628231608Z ts=2026-06-29T07:35:20.626559764Z level=info msg="Creating checkpoint" component_path=/ component_id=prometheus.remote_write.prometheus subcomponent=wal from_segment=417 to_segment=418 mint=1782718218000 �2026-06-29T07:35:20.760911497Z ts=2026-06-29T07:35:20.760795939Z level=info msg="WAL checkpoint complete" component_path=/ component_id=prometheus.remote_write.prometheus subcomponent=wal first=417 last=418 duration=181.868246ms �2026-06-29T09:35:20.832057458Z ts=2026-06-29T09:35:20.78928268Z level=info msg="series GC completed" component_path=/ component_id=prometheus.remote_write.prometheus subcomponent=wal duration=3.055744ms �2026-06-29T11:13:45.743723566Z 2026/06/29 11:13:45 Unsolicited response received on idle HTTP channel starting with "SSH-2.0-Go\r\n"; err=<nil> �2026-06-29T11:33:19.605560110Z ts=2026-06-29T11:33:19.565353189Z level=info msg="reporting Alloy stats" date=2026-06-29T11:33:19.565Z q2026-06-29T11:33:19.697074665Z ts=2026-06-29T11:33:19.696885838Z level=info msg="usage report sent with success" �2026-06-29T11:35:20.850061748Z ts=2026-06-29T11:35:20.849858725Z level=info msg="series GC completed" component_path=/ component_id=prometheus.remote_write.prometheus subcomponent=wal duration=2.255816ms �2026-06-29T11:35:20.871394256Z ts=2026-06-29T11:35:20.870349585Z level=info msg="Creating checkpoint" component_path=/ component_id=prometheus.remote_write.prometheus subcomponent=wal from_segment=419 to_segment=420 mint=1782732618000 �2026-06-29T11:35:20.987716127Z ts=2026-06-29T11:35:20.987570121Z level=info msg="WAL checkpoint complete" component_path=/ component_id=prometheus.remote_write.prometheus subcomponent=wal first=419 last=420 duration=139.966071ms �2026-06-29T12:15:45.721565443Z 2026/06/29 12:15:45 Unsolicited response received on idle HTTP channel starting with "SSH-2.0-Go\r\n"; err=<nil> �2026-06-29T12:25:45.723211317Z 2026/06/29 12:25:45 Unsolicited response received on idle HTTP channel starting with "SSH-2.0-Go\r\n"; err=<nil> 2026-06-28T19:33:47.571431433Z ts=2026-06-28T19:33:47.52090193Z level=info msg="Starting scraped metadata watcher" component_path=/ component_id=prometheus.remote_write.prometheus subcomponent=rw remote_name=6a1c19 url=https://prometheus.monitoring.sutty.abyaya.la/api/v1/write �2026-06-28T19:33:47.576076516Z ts=2026-06-28T19:33:47.520915916Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=78c717cab8190f7ee0319e8bd9eb8d43 node_id=prometheus.remote_write.prometheus duration=156.659128ms �2026-06-28T19:33:47.576086515Z ts=2026-06-28T19:33:47.521053084Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=78c717cab8190f7ee0319e8bd9eb8d43 node_id=discovery.dockerswarm.swarm duration=124.404µs �2026-06-28T19:33:47.576093558Z ts=2026-06-28T19:33:47.521219538Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=78c717cab8190f7ee0319e8bd9eb8d43 node_id=discovery.relabel.metrics duration=161.454µs �2026-06-28T19:33:47.576402291Z ts=2026-06-28T19:33:47.521254244Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=78c717cab8190f7ee0319e8bd9eb8d43 node_id=discovery.relabel.metrics_bearerauth duration=30.558µs 2026-06-28T19:33:47.576406790Z ts=2026-06-28T19:33:47.521477014Z level=info msg="Replaying WAL" component_path=/ component_id=prometheus.remote_write.prometheus subcomponent=rw remote_name=6a1c19 url=https://prometheus.monitoring.sutty.abyaya.la/api/v1/write queue=6a1c19 �2026-06-28T19:33:47.576410997Z ts=2026-06-28T19:33:47.543453096Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=78c717cab8190f7ee0319e8bd9eb8d43 node_id=prometheus.scrape.containers_bearerauth duration=22.192621ms �2026-06-28T19:33:47.576414634Z ts=2026-06-28T19:33:47.543685124Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=78c717cab8190f7ee0319e8bd9eb8d43 node_id=discovery.relabel.metrics_basicauth duration=211.8µs �2026-06-28T19:33:47.576418121Z ts=2026-06-28T19:33:47.545470953Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=78c717cab8190f7ee0319e8bd9eb8d43 node_id=prometheus.scrape.containers_basicauth duration=1.77573ms �2026-06-28T19:33:47.576421707Z ts=2026-06-28T19:33:47.545689044Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=78c717cab8190f7ee0319e8bd9eb8d43 node_id=discovery.relabel.metrics_noauth duration=210.066µs �2026-06-28T19:33:47.576425184Z ts=2026-06-28T19:33:47.550879567Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=78c717cab8190f7ee0319e8bd9eb8d43 node_id=prometheus.scrape.containers duration=5.182978ms �2026-06-28T19:33:47.576428290Z ts=2026-06-28T19:33:47.552913825Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=78c717cab8190f7ee0319e8bd9eb8d43 node_id=discovery.docker.containers duration=2.026445ms �2026-06-28T19:33:47.576434922Z ts=2026-06-28T19:33:47.552980251Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=78c717cab8190f7ee0319e8bd9eb8d43 node_id=prometheus.exporter.self.alloy duration=58.23µs �2026-06-28T19:33:47.578710285Z ts=2026-06-28T19:33:47.557620816Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=78c717cab8190f7ee0319e8bd9eb8d43 node_id=prometheus.scrape.default duration=4.634102ms �2026-06-28T19:33:47.579166957Z ts=2026-06-28T19:33:47.557653428Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=78c717cab8190f7ee0319e8bd9eb8d43 node_id=tracing duration=18.745µs �2026-06-28T19:33:47.579192365Z ts=2026-06-28T19:33:47.55830159Z level=info msg="finished node evaluation" controller_path=/ controller_id="" trace_id=78c717cab8190f7ee0319e8bd9eb8d43 node_id=logging duration=644.084µs �2026-06-28T19:33:47.579197284Z ts=2026-06-28T19:33:47.558308282Z level=info msg="finished complete graph evaluation" controller_path=/ controller_id="" trace_id=78c717cab8190f7ee0319e8bd9eb8d43 duration=851.67724ms i2026-06-28T19:33:47.579201702Z ts=2026-06-28T19:33:47.558658553Z level=info msg="{^_^} Alloy is running" �2026-06-28T19:33:47.579206011Z ts=2026-06-28T19:33:47.565591113Z level=info msg="scheduling loaded components and services" controller_id="" �2026-06-28T19:33:47.579209658Z ts=2026-06-28T19:33:47.570800212Z level=error msg="failed to register collector with remote server" service=remotecfg id=ea827bbe-2a03-4225-9b9e-eba251d34885 name="" err="noop client" �2026-06-28T19:33:47.586887503Z ts=2026-06-28T19:33:47.57526729Z level=info msg="starting cluster node" service=cluster peers_count=0 peers="" advertise_addr=127.0.0.1:12345 minimum_cluster_size=0 minimum_size_wait_timeout=0s �2026-06-28T19:33:47.586903304Z ts=2026-06-28T19:33:47.575539414Z level=info msg="peers changed" service=cluster peers_count=1 min_cluster_size=0 peers=monitoring.sutty.abyaya.la �2026-06-28T19:33:47.586908423Z ts=2026-06-28T19:33:47.577443595Z level=info msg="now listening for http traffic" service=http addr=0.0.0.0:12345 2026-06-28T19:33:52.863260941Z ts=2026-06-28T19:33:52.86281064Z level=info msg="Done replaying WAL" component_path=/ component_id=prometheus.remote_write.prometheus subcomponent=rw remote_name=6a1c19 url=https://prometheus.monitoring.sutty.abyaya.la/api/v1/write duration=5.341352371s �2026-06-28T20:21:45.725010719Z 2026/06/28 20:21:45 Unsolicited response received on idle HTTP channel starting with "SSH-2.0-Go\r\n"; err=<nil> �2026-06-28T21:33:47.660605538Z ts=2026-06-28T21:33:47.598901959Z level=info msg="series GC completed" component_path=/ component_id=prometheus.remote_write.prometheus subcomponent=wal duration=11.618661ms �2026-06-28T21:33:47.660647046Z ts=2026-06-28T21:33:47.654919349Z level=info msg="Creating checkpoint" component_path=/ component_id=prometheus.remote_write.prometheus subcomponent=wal from_segment=409 to_segment=411 mint=1782682121000 �2026-06-28T21:33:47.863031660Z ts=2026-06-28T21:33:47.862833166Z level=info msg="WAL checkpoint complete" component_path=/ component_id=prometheus.remote_write.prometheus subcomponent=wal first=409 last=411 duration=282.373382ms 22026-06-28T21:35:06.544999956Z interrupt received �2026-06-28T21:35:06.579240599Z ts=2026-06-28T21:35:06.577395778Z level=info msg="node exited without error" controller_id="" node=ui �2026-06-28T21:35:06.579671222Z ts=2026-06-28T21:35:06.57918862Z level=info msg="node exited without error" controller_id="" node=otel �2026-06-28T21:35:06.580590356Z ts=2026-06-28T21:35:06.580351636Z level=info msg="node exited without error" controller_id="" node=labelstore �2026-06-28T21:35:06.581124794Z ts=2026-06-28T21:35:06.581041487Z level=info msg="node exited without error" controller_id="" node=discovery.docker.linux �2026-06-28T21:35:06.596877264Z ts=2026-06-28T21:35:06.590996017Z level=info msg="node exited without error" controller_id="" node=prometheus.scrape.containers �2026-06-28T21:35:06.597244537Z ts=2026-06-28T21:35:06.596866384Z level=info msg="scrape manager stopped" component_path=/ component_id=prometheus.scrape.containers �2026-06-28T21:35:06.598288276Z ts=2026-06-28T21:35:06.597688755Z level=info msg="node exited without error" controller_id="" node=prometheus.scrape.containers_basicauth �2026-06-28T21:35:06.602489182Z ts=2026-06-28T21:35:06.598600155Z level=info msg="scrape manager stopped" component_path=/ component_id=prometheus.scrape.containers_basicauth �2026-06-28T21:35:06.602504881Z ts=2026-06-28T21:35:06.598690645Z level=info msg="node exited without error" controller_id="" node=prometheus.scrape.containers_bearerauth �2026-06-28T21:35:06.602509741Z ts=2026-06-28T21:35:06.601060767Z level=info msg="scrape manager stopped" component_path=/ component_id=prometheus.scrape.containers_bearerauth �2026-06-28T21:35:06.602513920Z ts=2026-06-28T21:35:06.601436286Z level=info msg="node exited without error" controller_id="" node=prometheus.scrape.default �2026-06-28T21:35:06.610976456Z ts=2026-06-28T21:35:06.610884261Z level=info msg="node exited without error" controller_id="" node=cluster 2026-06-28T21:35:06.618033681Z ts=2026-06-28T21:35:06.617059753Z level=info msg="Stopping remote storage..." component_path=/ component_id=prometheus.remote_write.prometheus subcomponent=rw remote_name=6a1c19 url=https://prometheus.monitoring.sutty.abyaya.la/api/v1/write �2026-06-28T21:35:06.619691338Z ts=2026-06-28T21:35:06.617944652Z level=info msg="http server closed" service=http addr=[::]:12345 err="http: Server closed" �2026-06-28T21:35:06.621640145Z ts=2026-06-28T21:35:06.621536439Z level=info msg="scrape manager stopped" component_path=/ component_id=prometheus.scrape.default �2026-06-28T21:35:06.624162363Z ts=2026-06-28T21:35:06.621895727Z level=info msg="http server closed" service=http addr=memory err="http: Server closed" �2026-06-28T21:35:06.624177762Z ts=2026-06-28T21:35:06.623679022Z level=info msg="node exited without error" controller_id="" node=http �2026-06-28T21:35:06.625010233Z ts=2026-06-28T21:35:06.624712471Z level=error msg="failed to unregister collector with remote server" service=remotecfg id=ea827bbe-2a03-4225-9b9e-eba251d34885 err="noop client" �2026-06-28T21:35:06.625343181Z ts=2026-06-28T21:35:06.624810306Z level=error msg="failed to unregister collector during service shutdown" service=remotecfg err="noop client" �2026-06-28T21:35:06.625971125Z ts=2026-06-28T21:35:06.624893803Z level=info msg="node exited without error" controller_id="" node=remotecfg �2026-06-28T21:35:06.626178948Z ts=2026-06-28T21:35:06.625889672Z level=info msg="node exited without error" controller_id="" node=livedebugging ```
decentral1se commented 2026-06-29 14:21:22 +00:00
Owner
Copy Link
Copy Source

The �2026-06-28T21:35:06.626178948Z ts=2026-06-28T21:35:06.625889672Z is usually coming from the fact your docker daemon is hooked into systemd? However, I think we also configure this by default in abra:

abra/pkg/logs/logs.go Lines 62 to 71 in f3943d25d1
logs, err := cl.ServiceLogs(context.Background(), serviceID, containerTypes.LogsOptions{
ShowStderr: true,
ShowStdout: !opts.StdErr,
Since: opts.Since,
Until: "",
Timestamps: true,
Follow: true,
Tail: tail,
Details: false,
})

Timestamps: true could be the issue? I can't remember where they're coming from either 🙃

As far as I remember, the abra code just calls "show me your logs" in parallel and they all throw their logs to stdout at the same time. So everything can arrive in a weird order. I thought that was also the docker logs behaviour.

The `�2026-06-28T21:35:06.626178948Z ts=2026-06-28T21:35:06.625889672Z` is usually coming from the fact your docker daemon is hooked into `systemd`? However, I think we also configure this by default in `abra`: https://git.coopcloud.tech/toolshed/abra/src/commit/f3943d25d19a832e2eee269c849394dda87a7312/pkg/logs/logs.go#L62-L71 `Timestamps: true` could be the issue? I can't remember where they're coming from either 🙃 As far as I remember, the `abra` code just calls "show me your logs" in parallel and they all throw their logs to stdout at the same time. So everything can arrive in a weird order. I thought that was also the `docker logs` behaviour.
fauno commented 2026-06-29 14:36:38 +00:00
Author
Owner
Copy Link
Copy Source

this isn't a systemd server, but yeah docker is configured to send logs to syslog. these are two different services so i had to run two docker service logs. the issue is that they are out of order within the same service, you can see alloy starting and then stopping, when the order was stop and then start.

# docker service ls | grep monitoring
49onser933ly   monitoring_sutty_abyaya_la_app          replicated   1/1        grafana/alloy:v1.17.0                                     *:514->514/tcp
ttv51pf0ciwp   monitoring_sutty_abyaya_la_prometheus   replicated   1/1        prom/prometheus:v3.12.0
this isn't a systemd server, but yeah docker is configured to send logs to syslog. these are two different services so i had to run two `docker service logs`. the issue is that they are out of order within the same service, you can see alloy starting and then stopping, when the order was stop and then start. ``` # docker service ls | grep monitoring 49onser933ly monitoring_sutty_abyaya_la_app replicated 1/1 grafana/alloy:v1.17.0 *:514->514/tcp ttv51pf0ciwp monitoring_sutty_abyaya_la_prometheus replicated 1/1 prom/prometheus:v3.12.0 ```
decentral1se commented 2026-06-29 14:40:11 +00:00
Owner
Copy Link
Copy Source

Ah, I see. So, we need to sort the output by timestamp on a per-service level 🤔 I guess a dip back into the source of how docker service logs works might be warranted. I had tried also wanted to have a kind of "mini-ui" for being able to flick between logs of each service but I never got around to it... thanks for the report!

Ah, I see. So, we need to sort the output by timestamp on a per-service level 🤔 I guess a dip back into the source of how `docker service logs` works might be warranted. I had tried also wanted to have a kind of "mini-ui" for being able to flick between logs of each service but I never got around to it... thanks for the report!
decentral1se added the bug label 2026-06-29 14:40:17 +00:00
decentral1se added this to the Abra "next" project 2026-06-29 14:40:20 +00:00
fauno commented 2026-06-29 14:47:44 +00:00
Author
Owner
Copy Link
Copy Source

new logs arrive correctly, with the occasional message from another dimension (!):

2▮26↑▮6↑29T14:33:▮8↓52▮58▮792Z ┌⎺±±␊⎼=␌⎺┼├␊│├ ┤⎽␊⎼I␍=▮ ⎺⎼±I␍=▮ ┤┼▒└␊= ├=2▮26↑▮6↑29T14:33:▮8↓518775▮▮4Z ┌␊┴␊┌=␋┼°⎺ └⎽±="R␊─┤␊⎽├ C⎺└⎻┌␊├␊␍
" └␊├␤⎺␍=GET ⎻▒├␤=/ ⎽├▒├┤⎽=3▮2 ⎼␊└⎺├␊_▒␍␍⎼=1▮↓13↓12↓1 ├␋└␊_└⎽=47 ␍┤⎼▒├␋⎺┼=47↓966649└⎽ ⎽␋≥␊=29 ⎼␊°␊⎼␊⎼= ␤▒┼␍┌␊⎼=/ ⎽├▒├┤⎽_⎽⎺┤⎼␌␊=⎽␊⎼┴␊⎼
^C┌▒⎼±␊↑┤⎻┌⎺▒␍⎽ ├⎼▒␊°␋┐ >⎼␊⎽␊├
new logs arrive correctly, with the occasional message from another dimension (!): ``` 2▮26↑▮6↑29T14:33:▮8↓52▮58▮792Z ┌⎺±±␊⎼=␌⎺┼├␊│├ ┤⎽␊⎼I␍=▮ ⎺⎼±I␍=▮ ┤┼▒└␊= ├=2▮26↑▮6↑29T14:33:▮8↓518775▮▮4Z ┌␊┴␊┌=␋┼°⎺ └⎽±="R␊─┤␊⎽├ C⎺└⎻┌␊├␊␍ " └␊├␤⎺␍=GET ⎻▒├␤=/ ⎽├▒├┤⎽=3▮2 ⎼␊└⎺├␊_▒␍␍⎼=1▮↓13↓12↓1 ├␋└␊_└⎽=47 ␍┤⎼▒├␋⎺┼=47↓966649└⎽ ⎽␋≥␊=29 ⎼␊°␊⎼␊⎼= ␤▒┼␍┌␊⎼=/ ⎽├▒├┤⎽_⎽⎺┤⎼␌␊=⎽␊⎼┴␊⎼ ^C┌▒⎼±␊↑┤⎻┌⎺▒␍⎽ ├⎼▒␊°␋┐ >⎼␊⎽␊├ ```
😆 1
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
Labels
Clear labels
bug
Something is not working
 build
go build related issues
 ci/cd
Building things with CI/CD
 critical fix
https://docs.coopcloud.tech/federation/resolutions/passed/010/
 design
UI/UX
 documentation
Documenting all the things
 duplicate
This issue or pull request already exists
 easy-first-issue
Something for new people to get stuck into. We hope it's easy!
 enhancement
New feature
 help wanted
Need some help
 i10n
Everything to do with localisation
 i18n
Everything to do with internationalisation
 installer
Everything to do with the install script.
 invalid
Something is wrong
 question
More information is needed
 release
Release management
 release-candidate
Related to the new release candidate
 security
Security related
 tech-debt
test
Unit/integration testing
 wontfix
This won't be fixed
No Label bug
Milestone
No items
No Milestone
Projects
Clear projects
No project Abra "next"
Assignees
Clear assignees
3wordchant aadil (Aadil Ayub) abra-bot (Abra Bot) ammaratef45 amras (Sarma) Apfelwurm BornDeleuze Brooke carla cas (Cassowary) coopcloud cyrnel decentral1se (decentral1se) dede fauno (fauno) iexos jade (Jade Ambrose) jjsfunhouse jmakdah2 joe-irving kawaiipunk (KawaiiPunk) knoflook kolaente lambdabundesverband linnealovespie moosemower moritz notplants oxaliq (sorrel) p4u1 pharaohgraphy (Andrew 🐦‍🔥❤️‍🔥✴️) renovate-bot (Comrade Renovate Bot) ripclap simon sixsmith (Sixsmith) stevensting trav val (val (he/him)) yksflip
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: toolshed/abra#875
Delete Branch "%!s()"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?