coop-cloud/kimai
2
0
Fork 2
Code Issues 2 Pull Requests 2 Releases Wiki Activity

Arreglo/healthcheck kimai #14

Merged
moritz merged 2 commits from escuela-comun/kimai:arreglo/healthcheck-kimai into main 2026-10-08 09:06:40 +00:00
Conversation 2 Commits 2 Files Changed 1
+2 -2
ChasquiLabo commented 2026-10-07 22:40:59 +00:00
Contributor
Copy Link
Copy Source

During a fresh deployment of Kimai 2.68.0, Abra reported the application as successfully deployed, but the Kimai web interface returned HTTP 500.

There were two related problems:

  1. The current healthcheck does not use curl --fail, so HTTP 500 responses can still result in a successful healthcheck.
  2. The initial database migration took longer than the current 3-minute start_period, causing the application to be restarted while migrations were still running.

That left the database partially migrated. Subsequent starts failed with errors such as existing tables, missing columns and a missing kimai2_sessions table.

After increasing the startup grace period and making the healthcheck fail on HTTP errors, a fresh deployment completed successfully and the web interface worked normally.

The debugging, analysis and preparation of this patch were done collaboratively with ChatGPT (LLM), using the deployment logs and the recipe configuration as the basis for the changes.

During a fresh deployment of Kimai 2.68.0, Abra reported the application as successfully deployed, but the Kimai web interface returned HTTP 500. There were two related problems: 1. The current healthcheck does not use `curl --fail`, so HTTP 500 responses can still result in a successful healthcheck. 2. The initial database migration took longer than the current 3-minute `start_period`, causing the application to be restarted while migrations were still running. That left the database partially migrated. Subsequent starts failed with errors such as existing tables, missing columns and a missing `kimai2_sessions` table. After increasing the startup grace period and making the healthcheck fail on HTTP errors, a fresh deployment completed successfully and the web interface worked normally. The debugging, analysis and preparation of this patch were done collaboratively with ChatGPT (LLM), using the deployment logs and the recipe configuration as the basis for the changes.
ChasquiLabo added 2 commits 2026-10-07 22:41:00 +00:00
fix: fail healthcheck on HTTP errors 400e447d61
fix: allow time for initial database migrations
continuous-integration/drone/pr Build is failing
Details
d5c78d3549
moritz merged commit 6f8c08fbb6 into main 2026-10-08 09:06:40 +00:00
compose.yml
@@ -42,3 +42,3 @@
entrypoint: /docker-entrypoint.sh
healthcheck:
test: curl -s -o /dev/null http://localhost:8001 || exit 1
test: curl -fsS -o /dev/null http://localhost:8001 || exit 1
fauno commented 2026-10-08 13:44:46 +00:00
Owner
Copy Link
Copy Source

@moritz just a comment on this! if curl is going to exit > 0 on failure, the || exit 1 is redundant, unless the healthcheck expects only 1 instead of > 0. which makes me think we'll need to review every healthcheck, for redundancy and missing -f flag

@moritz just a comment on this! if `curl` is going to exit > 0 on failure, the `|| exit 1` is redundant, unless the healthcheck expects only 1 instead of > 0. which makes me think we'll need to review every healthcheck, for redundancy and missing `-f` flag
moritz commented 2026-10-08 19:33:21 +00:00
Owner
Copy Link
Copy Source

I'm not sure what the healthcheck expects but I think || exit 1 is redundant, as most recipes don't have it. But some do. Maybe it's just copy pasta. I would favor a default healthcheck for all recipies, if possible.

Here is just an overview of all curl healthchecks:

./owncast/compose.yml:      test: ["CMD", "curl", "-f", "http://localhost:8080/api/status"]
./xwiki/compose.yml:    #   test: ["CMD", "curl", "-f", "http://localhost"]
./lauti/compose.yml:      test: ["CMD", "curl", "-f", "http://localhost:3333"]
./dokuwiki/compose.yml:      test: ["CMD", "curl", "http://localhost:80"]
./zammad/compose.yml:      test: ["CMD", "curl", "-f", "http://localhost:8080"]
./zammad/compose.yml:      test: ["CMD", "curl", "-f", "http://localhost:3000"]
./drutopia/compose.yml:    #   test: ["CMD", "curl", "-f", "http://localhost"]
./botamusique/compose.yml:    #   test: ["CMD", "curl", "-f", "http://localhost"]
./minio/compose.yml:      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
./kanboard/compose.yml:      test: ["CMD", "curl", "-f", "http://localhost"]
./wordpress-bedrock/compose.yml:    #   test: ["CMD", "curl", "-f", "http://localhost"]
./custom-html/compose.yml:        #  test: ["CMD", "curl", "-f", "http://localhost"]
./wordpress/compose.yml:      test: ["CMD", "curl", "-f", "http://localhost"]
./dashboard-recipe/compose.yml:      test: ["CMD", "curl", "-f", "http://localhost"]
./monica/compose.yml:        #   test: ["CMD", "curl", "-f", "http://localhost:8080"]
./4get/compose.yml:      test: [ "CMD", "curl", "-f", "http://localhost" ]
./onlyoffice/compose.yml:      test: ["CMD", "curl", "-f", "http://localhost/welcome"]
./farmos/compose.yml:    #   test: ["CMD", "curl", "-f", "http://localhost"]
./radicale/compose.yml:      test: ["CMD", "curl", "-f", "http://localhost:5232"]
./minecraft/compose.yml:    #   test: ["CMD", "curl", "-f", "http://localhost"]
./fab-manager/compose.yml:      test: ["CMD", "curl", "-f", "http://localhost:3000"]
./fab-manager/compose.yml:      test: ["CMD-SHELL", "curl --silent --fail localhost:9200/_cluster/health || exit 1"]
./singlelink/compose.yml:    #   test: ["CMD", "curl", "-f", "http://localhost"]
./seafile/compose.yml:      test: ["CMD", "curl", "-f", "http://localhost"]
./maubot/compose.yml:      #test: ["CMD", "curl", "-f", "http://localhost"]
./jellyfin/compose.yml:    #   test: ["CMD", "curl", "-f", "http://localhost:8096"]
./agora/compose.yml:      #   test: ["CMD", "curl", "-f", "http://localhost"]
./etherpad/compose.yml:          #      test: ["CMD", "curl", "-f", "http://localhost:9001"]
./abra-test-recipe/compose.yml:      test: ["CMD", "curl", "-f", "http://localhost"]
./minetest/compose.yml:    #   test: ["CMD", "curl", "-f", "http://localhost"]
./loomio/compose.yml:      test: ["CMD", "curl", "-f", "http://localhost:3000"]
./kimai/compose.yml:     test: curl -fsS -o /dev/null http://localhost:8001 || exit 1
./bonfire/compose.yml:    #   test: ["CMD", "curl", "-f", "http://localhost"]
./laplace/compose.yml:    #   test: ["CMD", "curl", "-f", "http://localhost"]
./synapse-admin/compose.yml:      test: ["CMD", "curl", "-f", "http://localhost"]
./keycloak/compose.yml:    #   test: ["CMD", "curl", "-f", "http://localhost:8080"]
./restic-rest-server/compose.yml:    #   test: ["CMD", "curl", "-f", "http://localhost:8000"]
./firefly-iii/compose.yml:      test: ["CMD", "curl", "http://localhost:8080"]
./compy/compose.yml:    #   test: ["CMD", "curl", "-f", "http://localhost"]
./voila/compose.yml:    #   test: ["CMD", "curl", "-f", "http://localhost"]
./rustdesk-server/compose.yml:    #   test: ["CMD", "curl", "-f", "http://localhost"]
./mobilizon/compose.yml:    #   test: ["CMD", "curl", "-f", "http://localhost"]
./oasis/compose.yml:    #   test: ["CMD", "curl", "-f", "http://localhost"]
./forgejo/compose.yml:      test: ["CMD", "curl", "-f", "http://localhost:3000/api/healthz"]
./cal/compose.yml:    #   test: ["CMD", "curl", "-f", "http://localhost"]
./nextcloud/compose.yml:      test: ["CMD-SHELL", "curl -fsS http://localhost/status.php | grep -q '\"installed\":true'"]
./cobalt/compose.yml:#      test: ["CMD", "curl", "-f", "http://localhost:9000"]
./adapt_authoring/compose.yml:    #  test: ["CMD", "curl", "-f", "http://localhost:5000"]
./gitea/compose.yml:      test: ["CMD", "curl", "-f", "http://localhost:3000/api/healthz"]
./swarmpit/compose.yml:      test: ["CMD", "curl", "-f", "http://localhost:8080"]
./vaultwarden/compose.yml:      test: curl -f http://localhost/alive || exit 1
./jupyter-lab/compose.voila.yml:    #   test: ["CMD", "curl", "-f", "http://localhost"]
./jupyter-lab/compose.yml:    #   test: ["CMD", "curl", "-f", "http://localhost"]
./mumble/compose.yml:    #  test: ["CMD", "curl", "-f", "http://localhost"]
./matrix-synapse/compose.yml:      test: curl -f http://${STACK_NAME}_app:8008/health || exit 1
./matrix-synapse/compose.yml:      test: ["CMD", "curl", "-f", "http://localhost:8008/health"]
./matrix-synapse/compose.admin.yml:      test: ["CMD", "curl", "-f", "http://localhost"]
./pretix/compose.yml:      test: ["CMD", "curl", "-f", "http://localhost/healthcheck"]
./peertube/compose.yml:      test: curl -f http://localhost:9000/v1/api/ping || exit 1
./notea/compose.yml:    #   test: ["CMD", "curl", "-f", "http://localhost"]
./icecast/compose.yml:    #   test: ["CMD", "curl", "-f", "http://localhost:8000"]
./projectsend/compose.yml:      test: ["CMD", "curl", "-f", "http://localhost"]
./custom-php/compose.yml:    #   test: ["CMD", "curl", "-f", "http://localhost"]
./roundcube/compose.yml:      test: ["CMD", "curl", "-f", "http://localhost/"]
./limesurvey/compose.yml:      test: ["CMD", "curl", "-f", "http://localhost:8080"]
./foodsoft/compose.yml:      test: ["CMD", "curl", "-f", "http://localhost:3000"]
./cryptpad/compose.yml:    #   test: ["CMD", "curl", "-f", "http://localhost"]
./screensy/compose.yml:    #   test: ["CMD", "curl", "-f", "http://localhost"]
./osticket/compose.yml:#      test: ["CMD", "curl", "-f", "http://localhost"]
./redmine/compose.yml:      test: ["CMD", "curl", "-f", "http://localhost:3000"]
./karrot/compose.yml:      test: ["CMD", "curl", "-f", "http://localhost/"]
./karrot/compose.yml:      test: ["CMD", "curl", "-f", "http://localhost:8000/api/settings/"]
./fluffychat/compose.yml:      test: ["CMD", "curl", "-f", "http://localhost"]
./uptime-kuma/compose.yml:      test: "curl -L localhost:3001"
./keycloak-collective-portal/compose.yml:      test: curl --fail 0.0.0.0:8000/healthz || exit 1
I'm not sure what the healthcheck expects but I think `|| exit 1` is redundant, as most recipes don't have it. But some do. Maybe it's just copy pasta. I would favor a default healthcheck for all recipies, if possible. Here is just an overview of all curl healthchecks: ``` ./owncast/compose.yml: test: ["CMD", "curl", "-f", "http://localhost:8080/api/status"] ./xwiki/compose.yml: # test: ["CMD", "curl", "-f", "http://localhost"] ./lauti/compose.yml: test: ["CMD", "curl", "-f", "http://localhost:3333"] ./dokuwiki/compose.yml: test: ["CMD", "curl", "http://localhost:80"] ./zammad/compose.yml: test: ["CMD", "curl", "-f", "http://localhost:8080"] ./zammad/compose.yml: test: ["CMD", "curl", "-f", "http://localhost:3000"] ./drutopia/compose.yml: # test: ["CMD", "curl", "-f", "http://localhost"] ./botamusique/compose.yml: # test: ["CMD", "curl", "-f", "http://localhost"] ./minio/compose.yml: test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"] ./kanboard/compose.yml: test: ["CMD", "curl", "-f", "http://localhost"] ./wordpress-bedrock/compose.yml: # test: ["CMD", "curl", "-f", "http://localhost"] ./custom-html/compose.yml: # test: ["CMD", "curl", "-f", "http://localhost"] ./wordpress/compose.yml: test: ["CMD", "curl", "-f", "http://localhost"] ./dashboard-recipe/compose.yml: test: ["CMD", "curl", "-f", "http://localhost"] ./monica/compose.yml: # test: ["CMD", "curl", "-f", "http://localhost:8080"] ./4get/compose.yml: test: [ "CMD", "curl", "-f", "http://localhost" ] ./onlyoffice/compose.yml: test: ["CMD", "curl", "-f", "http://localhost/welcome"] ./farmos/compose.yml: # test: ["CMD", "curl", "-f", "http://localhost"] ./radicale/compose.yml: test: ["CMD", "curl", "-f", "http://localhost:5232"] ./minecraft/compose.yml: # test: ["CMD", "curl", "-f", "http://localhost"] ./fab-manager/compose.yml: test: ["CMD", "curl", "-f", "http://localhost:3000"] ./fab-manager/compose.yml: test: ["CMD-SHELL", "curl --silent --fail localhost:9200/_cluster/health || exit 1"] ./singlelink/compose.yml: # test: ["CMD", "curl", "-f", "http://localhost"] ./seafile/compose.yml: test: ["CMD", "curl", "-f", "http://localhost"] ./maubot/compose.yml: #test: ["CMD", "curl", "-f", "http://localhost"] ./jellyfin/compose.yml: # test: ["CMD", "curl", "-f", "http://localhost:8096"] ./agora/compose.yml: # test: ["CMD", "curl", "-f", "http://localhost"] ./etherpad/compose.yml: # test: ["CMD", "curl", "-f", "http://localhost:9001"] ./abra-test-recipe/compose.yml: test: ["CMD", "curl", "-f", "http://localhost"] ./minetest/compose.yml: # test: ["CMD", "curl", "-f", "http://localhost"] ./loomio/compose.yml: test: ["CMD", "curl", "-f", "http://localhost:3000"] ./kimai/compose.yml: test: curl -fsS -o /dev/null http://localhost:8001 || exit 1 ./bonfire/compose.yml: # test: ["CMD", "curl", "-f", "http://localhost"] ./laplace/compose.yml: # test: ["CMD", "curl", "-f", "http://localhost"] ./synapse-admin/compose.yml: test: ["CMD", "curl", "-f", "http://localhost"] ./keycloak/compose.yml: # test: ["CMD", "curl", "-f", "http://localhost:8080"] ./restic-rest-server/compose.yml: # test: ["CMD", "curl", "-f", "http://localhost:8000"] ./firefly-iii/compose.yml: test: ["CMD", "curl", "http://localhost:8080"] ./compy/compose.yml: # test: ["CMD", "curl", "-f", "http://localhost"] ./voila/compose.yml: # test: ["CMD", "curl", "-f", "http://localhost"] ./rustdesk-server/compose.yml: # test: ["CMD", "curl", "-f", "http://localhost"] ./mobilizon/compose.yml: # test: ["CMD", "curl", "-f", "http://localhost"] ./oasis/compose.yml: # test: ["CMD", "curl", "-f", "http://localhost"] ./forgejo/compose.yml: test: ["CMD", "curl", "-f", "http://localhost:3000/api/healthz"] ./cal/compose.yml: # test: ["CMD", "curl", "-f", "http://localhost"] ./nextcloud/compose.yml: test: ["CMD-SHELL", "curl -fsS http://localhost/status.php | grep -q '\"installed\":true'"] ./cobalt/compose.yml:# test: ["CMD", "curl", "-f", "http://localhost:9000"] ./adapt_authoring/compose.yml: # test: ["CMD", "curl", "-f", "http://localhost:5000"] ./gitea/compose.yml: test: ["CMD", "curl", "-f", "http://localhost:3000/api/healthz"] ./swarmpit/compose.yml: test: ["CMD", "curl", "-f", "http://localhost:8080"] ./vaultwarden/compose.yml: test: curl -f http://localhost/alive || exit 1 ./jupyter-lab/compose.voila.yml: # test: ["CMD", "curl", "-f", "http://localhost"] ./jupyter-lab/compose.yml: # test: ["CMD", "curl", "-f", "http://localhost"] ./mumble/compose.yml: # test: ["CMD", "curl", "-f", "http://localhost"] ./matrix-synapse/compose.yml: test: curl -f http://${STACK_NAME}_app:8008/health || exit 1 ./matrix-synapse/compose.yml: test: ["CMD", "curl", "-f", "http://localhost:8008/health"] ./matrix-synapse/compose.admin.yml: test: ["CMD", "curl", "-f", "http://localhost"] ./pretix/compose.yml: test: ["CMD", "curl", "-f", "http://localhost/healthcheck"] ./peertube/compose.yml: test: curl -f http://localhost:9000/v1/api/ping || exit 1 ./notea/compose.yml: # test: ["CMD", "curl", "-f", "http://localhost"] ./icecast/compose.yml: # test: ["CMD", "curl", "-f", "http://localhost:8000"] ./projectsend/compose.yml: test: ["CMD", "curl", "-f", "http://localhost"] ./custom-php/compose.yml: # test: ["CMD", "curl", "-f", "http://localhost"] ./roundcube/compose.yml: test: ["CMD", "curl", "-f", "http://localhost/"] ./limesurvey/compose.yml: test: ["CMD", "curl", "-f", "http://localhost:8080"] ./foodsoft/compose.yml: test: ["CMD", "curl", "-f", "http://localhost:3000"] ./cryptpad/compose.yml: # test: ["CMD", "curl", "-f", "http://localhost"] ./screensy/compose.yml: # test: ["CMD", "curl", "-f", "http://localhost"] ./osticket/compose.yml:# test: ["CMD", "curl", "-f", "http://localhost"] ./redmine/compose.yml: test: ["CMD", "curl", "-f", "http://localhost:3000"] ./karrot/compose.yml: test: ["CMD", "curl", "-f", "http://localhost/"] ./karrot/compose.yml: test: ["CMD", "curl", "-f", "http://localhost:8000/api/settings/"] ./fluffychat/compose.yml: test: ["CMD", "curl", "-f", "http://localhost"] ./uptime-kuma/compose.yml: test: "curl -L localhost:3001" ./keycloak-collective-portal/compose.yml: test: curl --fail 0.0.0.0:8000/healthz || exit 1 ```
compose.yml
@@ -45,2 +44,3 @@
test: curl -fsS -o /dev/null http://localhost:8001 || exit 1
interval: 30s
start_period: 3m
start_period: 20m
fauno commented 2026-10-08 13:45:07 +00:00
Owner
Copy Link
Copy Source

does this mean it may take 20m for the whole recipe to deploy?

does this mean it may take 20m for the whole recipe to deploy?
ChasquiLabo commented 2026-10-08 13:47:16 +00:00
Author
Contributor
Copy Link
Copy Source

no, it doesn't. For me the first migration took more than 10m, thats way the big 20m.

no, it doesn't. For me the first migration took more than 10m, thats way the big 20m.
moritz commented 2026-10-08 19:27:07 +00:00
Owner
Copy Link
Copy Source

The healthcheck is just waiting 20m before it starts to restart the container. This value must be high enough, that even on low power systems the healthcheck isn't breaking anything. The only disadvantage is, that if the container is really stuck it takes 20m until it restarts.

The healthcheck is just waiting 20m before it starts to restart the container. This value must be high enough, that even on low power systems the healthcheck isn't breaking anything. The only disadvantage is, that if the container is really stuck it takes 20m until it restarts.
👍 1
ChasquiLabo commented 2026-10-09 00:26:08 +00:00
Author
Contributor
Copy Link
Copy Source

20 min goes flying. With all the time coopcloud saves, this shouldn´t be call a disadvantege <3

20 min goes flying. With all the time coopcloud saves, this shouldn´t be call a disadvantege <3
Sign in to join this conversation.
Reviewers
No Reviewers
fauno
Labels
Clear labels
bug
Something is not working
 duplicate
This issue or pull request already exists
 enhancement
New feature
 help wanted
Need some help
 invalid
Something is wrong
 question
More information is needed
 wontfix
This won't be fixed
No labels
Milestone
No items
No Milestone
Assignees
Clear assignees
3wordchant aadil (Aadil Ayub) abra-bot (Abra Bot) ammaratef45 amras (Sarma) Apfelwurm appletalk arjan basebuilder BornDeleuze (Born Deleuze) Brooke carla cas (Cassowary) codegod100 coopcloud cyrnel dannygroenewegen (Danny Groenewegen) decentral1se (d1) dede devydave fauno (fauno) flancian Frando iexos jade (Jade Ambrose) javielico (Javielico) jjsfunhouse jmakdah2 (Jackie Makdah) joe-irving (Joe Irving) kawaiipunk (KawaiiPunk) knoflook kolaente lambdabundesverband linnealovespie (April) marlon (marlon) mayel mirsal moosemower moritz nicksellen (Nick Sellen) notplants oxaliq (sorrel) p4u1 pau pharaohgraphy (Andrew 🐦‍🔥❤️‍🔥✴️) PhiNatalie renovate-bot (Comrade Renovate Bot) ripclap rix rscmbbng sef (sef) simon sixsmith (Sixsmith) stevensting tobias trav (Trav Fryer) val (val (he/him)) vaznasty virtualboys wolcen (Chris Thompson) wykwit xynosis yksflip
No Assignees
3 Participants
Notifications
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: coop-cloud/kimai#14