toolshed/abra-bash
Archived
5
0
Fork 3
Code Pull Requests Releases Activity

Make ensure_stack_deployed more reliable #177

Merged
decentral1se merged 1 commits from improved-stack-deploy-guarantees into main 2021-06-27 19:03:48 +00:00
Conversation 2 Commits 1 Files Changed 4 +161 -67
decentral1se commented 2021-06-09 08:32:58 +00:00
Owner
Copy Link

Closes https://git.autonomic.zone/coop-cloud/abra/issues/165.

Cases to be solved from my testing:

  • Sucessful deployment
  • Failed update (we see the healthcheck fail on the new container)
  • Container refuses to start because MaxRetryAttempts has happened (when no existing container is up - this is typically when you're trying to deploy the app for the first time)
  • Handling missing healthchecks gracefully

Overall goals are: make it work and keep it fast.

All of these will rely on healthchecks on each container, so, we need to implement them in all our apps and make them standard from now on, unfortunately...but I am getting better at understanding how they work! I have hope.

Seems like a default new deploy: ... configuration is emerging:

deploy:
  update_config:
    failure_action: rollback
    order: start-first
  rollback_config:
    order: start-first
  restart_policy:
    max_attempts: 3

Where you ensure that the service won't be in a restart loop forever.

Closes https://git.autonomic.zone/coop-cloud/abra/issues/165. Cases to be solved from my testing: - [x] Sucessful deployment - [x] Failed update (we see the healthcheck fail on the new container) - [ ] Container refuses to start because MaxRetryAttempts has happened (when no existing container is up - this is typically when you're trying to deploy the app for the first time) - [x] Handling missing healthchecks gracefully Overall goals are: *make it work and keep it fast*. All of these will rely on healthchecks on each container, so, we need to implement them in all our apps and make them standard from now on, unfortunately...but I am getting better at understanding how they work! I have hope. Seems like a default new `deploy: ...` configuration is emerging: ``` deploy: update_config: failure_action: rollback order: start-first rollback_config: order: start-first restart_policy: max_attempts: 3 ``` Where you ensure that the service won't be in a restart loop forever.
decentral1se added 1 commit 2021-06-09 08:32:59 +00:00
WIP WORLD
Some checks failed
continuous-integration/drone/pr Build is failing
Details
f151018f10
decentral1se force-pushed improved-stack-deploy-guarantees from f151018f10 to c7f838178c 2021-06-10 22:44:09 +00:00 Compare
decentral1se commented 2021-06-15 20:15:43 +00:00
Author
Owner
Copy Link

It seems a core element of making this run fast is configuring the healthcheck:

The default settings are really default. A redis container is up in like 3 seconds but the first healthcheck won't run until 30 seconds. So by tailoring each of these values to some better default, we can get speedy deployments and abra doesn't have to manage it. It pushes the complexity to swarm itself which is ideal.

Also:

https://engineering.issuu.com/2018/09/12/confident-deployments

It seems a core element of making this run fast is configuring the healthcheck: - https://docs.docker.com/engine/reference/builder/#healthcheck - https://docs.docker.com/compose/compose-file/compose-file-v3/#healthcheck The default settings are really default. A redis container is up in like 3 seconds but the first healthcheck won't run until 30 seconds. So by tailoring each of these values to some better default, we can get speedy deployments and `abra` doesn't have to manage it. It pushes the complexity to swarm itself which is ideal. Also: > https://engineering.issuu.com/2018/09/12/confident-deployments
❤️ 1
decentral1se force-pushed improved-stack-deploy-guarantees from c7f838178c to 4ed9f119d8 2021-06-15 21:53:16 +00:00 Compare
decentral1se force-pushed improved-stack-deploy-guarantees from 4ed9f119d8 to 7322105fd9 2021-06-17 08:32:42 +00:00 Compare
decentral1se force-pushed improved-stack-deploy-guarantees from 7322105fd9 to 4593451dab 2021-06-19 19:06:45 +00:00 Compare
decentral1se force-pushed improved-stack-deploy-guarantees from 4593451dab to 62999a1732 2021-06-27 19:01:30 +00:00 Compare
decentral1se commented 2021-06-27 19:01:37 +00:00
Author
Owner
Copy Link

OK, I'm gonna merge this as is now, one edge case is still not covered. Example output:

INFO: works_r5_should_also_work is healthy!
INFO: works_r4_disabled_health_check has no healthcheck configured, cannot guarantee this service comes up successfully...
INFO: Deploying: 2/5 (timeout: 1/60)
INFO: works_r3_no_health_check has no healthcheck configured, cannot guarantee this service comes up successfully...
INFO: Deploying: 3/5 (timeout: 2/60)
INFO: Deploying: 3/5 (timeout: 3/60)
INFO: Deploying: 3/5 (timeout: 4/60)
INFO: works_r1_should_work is healthy!
INFO: Deploying: 4/5 (timeout: 5/60)
WARNING: Healthcheck for new instance of works_r2_broken_health_check is failing (exit code: 127)
WARNING: /bin/sh: foobar: not found
ERROR: healthcheck for works_r2_broken_health_check is failing, this deployment did not succeed :(
OK, I'm gonna merge this as is now, one edge case is still not covered. Example output: ``` INFO: works_r5_should_also_work is healthy! INFO: works_r4_disabled_health_check has no healthcheck configured, cannot guarantee this service comes up successfully... INFO: Deploying: 2/5 (timeout: 1/60) INFO: works_r3_no_health_check has no healthcheck configured, cannot guarantee this service comes up successfully... INFO: Deploying: 3/5 (timeout: 2/60) INFO: Deploying: 3/5 (timeout: 3/60) INFO: Deploying: 3/5 (timeout: 4/60) INFO: works_r1_should_work is healthy! INFO: Deploying: 4/5 (timeout: 5/60) WARNING: Healthcheck for new instance of works_r2_broken_health_check is failing (exit code: 127) WARNING: /bin/sh: foobar: not found ERROR: healthcheck for works_r2_broken_health_check is failing, this deployment did not succeed :( ```
decentral1se force-pushed improved-stack-deploy-guarantees from 62999a1732 to dccfff0c87 2021-06-27 19:02:01 +00:00 Compare
decentral1se force-pushed improved-stack-deploy-guarantees from dccfff0c87 to 93714a593b 2021-06-27 19:03:29 +00:00 Compare
decentral1se changed title from WIP: make ensure_stack_deployed reliable to Make ensure_stack_deployed more reliable 2021-06-27 19:03:42 +00:00
decentral1se merged commit 0ab2b3a652 into main 2021-06-27 19:03:48 +00:00
decentral1se deleted branch improved-stack-deploy-guarantees 2021-06-27 19:10:12 +00:00
This repo is archived. You cannot comment on pull requests.
Reviewers
No Reviewers
Labels
Clear labels
breaking-change
Anything that breaks the existing API
bug
Something is not working
CI/CD
Anything to do with testing and infra
design
Anything to with UX of this tool
documentation
Documentation
duplicate
This issue or pull request already exists
enhancement
New feature
help wanted
Need some help
invalid
Something is wrong
plugin
Related to the plugin architecture
question
More information is needed
secrets
Everything to do with docker secrets
shell-completion
To do with shell completion
versioning
All the things to do with versioning apps
wontfix
This won't be fixed
No Label
Milestone
No items
No Milestone
Assignees
Clear assignees
3wordchant aadil abra-bot ammaratef45 Apfelwurm carla cas coopcloud cyrnel decentral1se fauno iexos jmakdah2 joe-irving kawaiipunk knoflook kolaente lambdabundesverband moritz p4u1 pharaohgraphy renovate-bot ripclap simon sixsmith trav val yksflip
No Assignees
1 Participants
Due Date
No due date set.
Dependencies

No dependencies set.

Reference: toolshed/abra-bash#177
No description provided.
Delete Branch "improved-stack-deploy-guarantees"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?