Bump to the next version

Add label generation checking
Closes https://git.autonomic.zone/coop-cloud/abra/issues/186.
2021-07-06 12:34:15 +02:00 · 2021-07-06 12:27:06 +02:00 · 2021-07-06 11:51:03 +02:00 · 2021-07-06 11:50:09 +02:00 · 2021-07-06 11:18:50 +02:00 · 2021-07-06 11:12:49 +02:00
21 changed files with 1054 additions and 2795 deletions
--- a/.drone.yml
+++ b/.drone.yml
@ -3,47 +3,86 @@ kind: pipeline
 name: linters
 steps:
  - name: run shellcheck
-    image: koalaman/shellcheck-alpine:v0.7.1
+    image: koalaman/shellcheck-alpine
    commands:
      - shellcheck abra
      - shellcheck bin/*.sh
+      - shellcheck deploy/install.abra.coopcloud.tech/installer

  - name: run flake8
-    image: alpine/flake8:3.9.0
+    image: alpine/flake8
    commands:
-      - flake8 --max-line-length 100 bin/app-json.py
+      - flake8 --max-line-length 100 bin/*.py

  - name: run unit tests
    image: decentral1se/docker-dind-bats-kcov
    commands:
      - bats tests

-  - name: collect code coverage
-    failure: ignore # until we fix this
-    image: decentral1se/docker-dind-bats-kcov
+  - name: test installation script
+    image: debian:buster
    commands:
-      - kcov . bats tests || true
+      - apt update && apt install -yqq sudo lsb-release
+      - deploy/install.abra.coopcloud.tech/installer --no-prompt
+      - ~/.local/bin/abra version

-  - name: send code coverage report to codecov
-    failure: ignore # until we fix this
-    image: plugins/codecov
+  - name: publish image
+    image: plugins/docker
    settings:
+      auto_tag: true
+      username: thecoopcloud
+      password:
+        from_secret: thecoopcloud_password
+      repo: thecoopcloud/abra
+      tags: latest
+    depends_on:
+      - run shellcheck
+      - run flake8
+      - run unit tests
+      - test installation script
+    when:
+      event:
+        exclude:
+          - pull_request
+
+  - name: trigger downstream builds
+    image: plugins/downstream
+    settings:
+      server: https://drone.autonomic.zone
      token:
-        from_secret: codecov_token
-        required: true
+        from_secret: decentral1se_token
+      fork: true
+      repositories:
+        - coop-cloud/drone-abra
+    depends_on:
+      - run shellcheck
+      - run flake8
+      - run unit tests
+      - test installation script
+      - publish image
+    when:
+      event:
+        exclude:
+          - pull_request

-  - name: notify rocket chat
-    image: plugins/slack
+  - name: notify on failure
+    image: plugins/matrix
    settings:
-      webhook:
-        from_secret: rc_builds_url
-      username: comradebritney
-      channel: "internal.builds"
-      template: "{{repo.owner}}/{{repo.name}} build failed: {{build.link}}"
+      homeserver: https://matrix.autonomic.zone
+      roomid: "IFazIpLtxiScqbHqoa:autonomic.zone"
+      userid: "@autono-bot:autonomic.zone"
+      accesstoken:
+        from_secret: autono_bot_access_token
+    depends_on:
+      - run shellcheck
+      - run flake8
+      - run unit tests
+      - test installation script
+      - publish image
+      - trigger downstream builds
    when:
      status:
        - failure
-
 trigger:
  branch:
    - main
--- a/.gitignore
+++ b/.gitignore
@ -1,2 +1,5 @@
+*.json
+*.pyc
 /.venv
+__pycache__
 coverage/
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@ -9,6 +9,42 @@

 # abra x.x.x (UNRELEASED)

+# abra 10.0.0 (2021-07-06)
+
+- Add `--bump` to `deploy` command to allow packagers to make minor package related releases ([#173](https://git.autonomic.zone/coop-cloud/abra/issues/173))
+- Drop `--skip-version-check`/`--no-domain-poll`/`--no-state-poll` in favour of `--fast` ([#169](https://git.autonomic.zone/coop-cloud/abra/issues/169))
+- Move `abra` image under the new `thecoopcloud/...` namespace ([#1](https://git.autonomic.zone/coop-cloud/auto-apps-json/issues/1))
+- Add a `--output` flag to the `app-json.py` app generator for the CI environment ([#2](https://git.autonomic.zone/coop-cloud/auto-apps-json/issues/2))
+- Support logging in as new `thecoopcloud` Docker account via `skopeo` when generating new `apps.json` ([7482362af1](https://git.autonomic.zone/coop-cloud/abra/commit/7482362af1d01cc02828abd45b1222fa643d1f80))
+- App deployment checks are somewhat more reliable (see [#193](https://git.autonomic.zone/coop-cloud/abra/issues/193) for remaining work) ([#165](https://git.autonomic.zone/coop-cloud/abra/issues/165))
+- Skip generation of commented out secrets and correctly fail deploy when secret generation fails ([#133](https://git.autonomic.zone/coop-cloud/abra/issues/133))
+- Fix logging for chaos deploys and recipe selection logic ([#185](https://git.autonomic.zone/coop-cloud/abra/issues/185))
+- Improve reliability of selecting when to download a new `apps.json` ([#170](https://git.autonomic.zone/coop-cloud/abra/issues/170))
+- Implement `pwgen`/`pwqgen` native fallback for password generation ([#167](https://git.autonomic.zone/coop-cloud/abra/issues/167) / [#197](https://git.autonomic.zone/coop-cloud/abra/issues/197))
+- `abra` installer script will now try to install system requirements ([#196](https://git.autonomic.zone/coop-cloud/abra/issues/196))
+- Use latest [v4.9.6](https://github.com/mikefarah/yq/releases/tag/v4.9.6) install of `yq` for vendoring (**upgrade HOWTO:** `rm -rf ~/.abra/vendor/*`)
+- Support overriding `$ARGS` from `abra.sh` custom commands and error out correctly when these commands fail ([#1](https://github.com/Coop-Cloud/peertube/issues/1))
+- Add `abra <app> restart <service>` to support restarting individual services ([#200](https://git.autonomic.zone/coop-cloud/abra/issues/200))
+- Output changes when asking to commit during release logic ([3f0a907b5](https://git.autonomic.zone/coop-cloud/rocketchat/commit/3f0a907b57acf960d6befab3d943982d956e2474))
+- Add validation for generated output when making new release labels ([#186](https://git.autonomic.zone/coop-cloud/abra/issues/186))
+
+# abra 9.0.0 (2021-06-10)
+
+- Add Docker image for `abra` ([64d578cf91](https://git.autonomic.zone/coop-cloud/abra/commit/64d578cf914bd2bad378ea4ef375747d10b33191))
+- Support unattended mode for recipe releasing ([3759bcd641](https://git.autonomic.zone/coop-cloud/abra/commit/3759bcd641cf60611c13927e83425e773d2bb629))
+- Add Renovate bot configuraiton script ([9fadc430a7](https://git.autonomic.zone/coop-cloud/abra/commit/9fadc430a7bb2d554c0ee26c0f9b6c51dc5b0475))
+- Add release automation via [drone-abra](https://git.autonomic.zone/coop-cloud/drone-abra) ([#56](https://git.autonomic.zone/coop-cloud/organising/issues/56))
+- Move `apps.json` generation to [auto-apps-json](https://git.autonomic.zone/coop-cloud/auto-apps-json) ([#125](https://git.autonomic.zone/coop-cloud/abra/issues/125))
+- Add Github mirroring script ([4ef433312d](https://git.autonomic.zone/coop-cloud/abra/commit/4ef433312dd0b0ace91b3c285f82f3973093d92d))
+- Add `--chaos` flag to deploy (always choose latest Git commit) ([#178](https://git.autonomic.zone/coop-cloud/abra/issues/178))
+
+# abra 8.0.1 (2021-05-31)
+
+- Fix help for `... app ... volume ls` ([efad71c470](https://git.autonomic.zone/coop-cloud/abra/commits/branch/main))
+- Only output secrets warnings once ([#143](https://git.autonomic.zone/coop-cloud/abra/issues/143))
+- Migrate `abra` installation script to `coopcloud.tech` domain ([#150](https://git.autonomic.zone/coop-cloud/abra/issues/150))
+- Add `--no-state-poll` to avoid success/failure forecasting on deployment ([#165](https://git.autonomic.zone/coop-cloud/abra/issues/165))
+
 # abra 8.0.0 (2021-05-30)

 - Fix secret length generation ([f537417](https://git.autonomic.zone/coop-cloud/abra/commit/1b85bf3d37280e9632c315d759c0f2d09c039fef))
--- a/33
+++ b/33
@ -0,0 +1,33 @@
+FROM alpine:latest
+
+RUN apk add --upgrade --no-cache \
+  bash \
+  curl \
+  git \
+  grep \
+  openssh-client \
+  py3-requests \
+  skopeo \
+  util-linux
+
+RUN mkdir -p ~./local/bin
+RUN mkdir -p ~/.abra/apps
+RUN mkdir -p ~/.abra/vendor
+RUN mkdir -p ~/.ssh/
+
+RUN ssh-keyscan -p 2222 git.autonomic.zone > ~/.ssh/known_hosts
+
+RUN curl -L https://github.com/stedolan/jq/releases/download/jq-1.6/jq-linux64 --output ~/.abra/vendor/jq
+RUN chmod +x ~/.abra/vendor/jq
+
+RUN curl -L https://github.com/mikefarah/yq/releases/download/v4.9.3/yq_linux_amd64 --output ~/.abra/vendor/yq
+RUN chmod +x ~/.abra/vendor/yq
+
+# Note(decentral1se): it is fine to always use the development branch because
+# our Drone CI docker auto-tagger will publish official release tags and
+# otherwise give us the latest abra on the latest tag
+RUN curl https://install.abra.coopcloud.tech | bash -s -- --dev
+
+COPY bin/* /root/.local/bin/
+
+ENTRYPOINT ["/root/.local/bin/abra"]
--- a/README.md
+++ b/README.md
@ -1,11 +1,10 @@
 # abra

 [![Build Status](https://drone.autonomic.zone/api/badges/coop-cloud/abra/status.svg)](https://drone.autonomic.zone/coop-cloud/abra)
-[![codecov](https://codecov.io/gh/Autonomic-Cooperative/abra/branch/main/graph/badge.svg?token=aX3I5NMRsj)](undefined)

 > https://coopcloud.tech

-The cooperative cloud utility belt 🎩🐇
+The Co-op Cloud utility belt 🎩🐇

 `abra` is a command-line tool for managing your own [Co-op Cloud](https://coopcloud.tech). It can provision new servers, create applications, deploy them, run backup and restore operations and a whole lot of other things. It is the go-to tool for day-to-day operations when managing a Co-op Cloud instance.

@ -26,29 +25,40 @@ See [CHANGELOG.md](./CHANGELOG.md).

 > [docs.coopcloud.tech](https://docs.coopcloud.tech)

-## Install
+## Requirements

-Requirements:
-
- `pwqgen` (optional)
- `pwgen` (optional)
 - `curl`
 - `docker`
 - `bash` >= 4

+## Install
+
 Install the latest stable release:

 ```sh
-curl https://install.abra.autonomic.zone | bash
+curl https://install.abra.coopcloud.tech | bash
 ```

-or the bleeding-edge development version:
+The source for this script is [here](./deploy/install.abra.coopcloud.tech/installer).
+
+You can pass options to the script like so (e.g. install the bleeding edge development version):

 ```sh
-curl https://install.abra.autonomic.zone | bash -s -- --dev
+curl https://install.abra.coopcloud.tech | bash -s -- --dev
 ```

-The source for this script is [here](./deploy/install.abra.autonomic.zone/installer).
+Other options available are as follows:
+
+- **--no-prompt**: non-interactive installation
+- **--no-deps**: do not attempt to install [requirements](#requirements)
+
+## Container
+
+An [image](https://hub.docker.com/r/thecoopcloud/abra) is also provided.
+
+```
+docker run thecoopcloud/abra app ls
+```

 ## Update

@ -61,41 +71,22 @@ To update the development version, run `abra upgrade --dev`.

 It's written in Bash version 4 or greater!

-Install it via `curl https://install.abra.autonomic.zone | bash -s -- --dev`, then you can hack on the source in `~/.abra/src`.
+Install it via `curl https://install.abra.coopcloud.tech | bash -s -- --dev`, then you can hack on the source in `~/.abra/src`.

 The command-line interface is generated via [docopt](http://docopt.org/). If you add arguments then you need to run `make docopt` ro regenerate the parser.

 Please remember to update the [CHANGELOG](./CHANGELOG.md) when you make a change.

-## Generating a new apps.json
-
-You'll need to install the following requirements:
-
- [requests](https://docs.python-requests.org/en/master/) (`apt install python3-requests` / `pip install requests`)
- [skopeo](https://github.com/containers/skopeo) (check [the install docs](https://github.com/containers/skopeo/blob/master/install.md))
- [jq](https://stedolan.github.io/jq/tutorial/) (`sudo apt-get install jq` or see [the install docs](https://stedolan.github.io/jq/download/))
- [yq](https://mikefarah.gitbook.io/yq/) (see [the install docs](https://mikefarah.gitbook.io/yq/#install))
-
-Then run `./bin/app-json.py` ([source](./bin/app-json.py)) and it will spit out the JSON file into [deploy/apps.coopcloud.tech/apps.json](./deploy/apps.coopcloud.tech/apps.json).
-
 ## Releasing

 ### `abra`

-> [install.abra.autonomic.zone](https://install.abra.autonomic.zone)
+> [install.abra.coopcloud.tech](https://install.abra.coopcloud.tech)

 - Change the `x.x.x` header in [CHANGELOG.md](./CHANGELOG.md) to reflect new version and mark date
 - Update the version in [abra](./abra)
- Update the version in [deploy/install.abra.autonomic.zone/installer](./deploy/install.abra.autonomic.zone/installer)
+- Update the version in [deploy/install.abra.coopcloud.tech/installer](./deploy/install.abra.coopcloud.tech/installer)
 - `git commit` the above changes and then tag it with `git tag <your-new-version>`
 - `git push` and `git push --tags`
 - Deploy a new installer script `make release-installer`
 - Tell the world (CoTech forum, Matrix public channel, Autonomic mastodon, etc.)
-
-### apps.coopcloud.tech
-
-> [apps.coopcloud.tech](https://apps.coopcloud.tech)
-
-```bash
-$ make release-apps
-```
--- a/569
+++ b/569
@ -1,9 +1,11 @@
 #!/usr/bin/env bash

+# shellcheck disable=SC2154
+
 GIT_URL="https://git.autonomic.zone/coop-cloud/"
 ABRA_APPS_URL="https://apps.coopcloud.tech"
 ABRA_DIR="${ABRA_DIR:-$HOME/.abra}"
-ABRA_VERSION="8.0.0"
+ABRA_VERSION="10.0.0"
 ABRA_BACKUP_DIR="${ABRA_BACKUP_DIR:-$ABRA_DIR/backups}"
 ABRA_VENDOR_DIR="$ABRA_DIR/vendor"
 ABRA_APPS_JSON="${ABRA_DIR}/apps.json"
@ -13,13 +15,20 @@ ABRA_APPS_JSON="${ABRA_DIR}/apps.json"
 #######################################

 DOC="
-The cooperative cloud utility belt 🎩🐇
+The Co-op Cloud utility belt 🎩🐇
+
+    ____                           ____ _                 _
+   / ___|___         ___  _ __    / ___| | ___  _   _  __| |
+  | |   / _ \ _____ / _ \| '_ \  | |   | |/ _ \| | | |/ _' |
+  | |__| (_) |_____| (_) | |_) | | |___| | (_) | |_| | (_| |
+   \____\___/       \___/| .__/   \____|_|\___/ \__,_|\__,_|
+                         |_|

 Usage:
  abra [options] app (list|ls) [--status] [--server=<server>] [--type=<type>]
  abra [options] app new [--server=<server>] [--domain=<domain>] [--app-name=<app_name>] [--pass] [--secrets] <type>
  abra [options] app <app> backup (<service>|--all)
-  abra [options] app <app> deploy [--update] [--force] [--fast] [--skip-version-check] [--no-domain-poll] [<version>]
+  abra [options] app <app> deploy [--update] [--force] [--fast] [--chaos] [<version>]
  abra [options] app <app> check
  abra [options] app <app> version
  abra [options] app <app> config
@ -31,6 +40,7 @@ Usage:
  abra [options] app <app> restore <service> [<backup file>]
  abra [options] app <app> run [--no-tty] [--user=<user>] <service> <args>...
  abra [options] app <app> rollback [<version>]
+  abra [options] app <app> restart <service>
  abra [options] app <app> secret generate (<secret> <version>|--all) [<cmd>] [--pass]
  abra [options] app <app> secret insert <secret> <version> <data> [--pass]
  abra [options] app <app> secret (rm|delete) (<secret>|--all) [--pass]
@ -40,7 +50,7 @@ Usage:
  abra [options] app <app> <command> [<args>...]
  abra [options] recipe ls
  abra [options] recipe create <recipe>
-  abra [options] recipe <recipe> release [--force]
+  abra [options] recipe <recipe> release [--force] [--bump]
  abra [options] recipe <recipe> versions
  abra [options] server add <host> [<user>] [<port>]
  abra [options] server new <provider> -- <args>
@ -167,23 +177,23 @@ eval "var_$1+=($value)"; else eval "var_$1=$value"; fi; return 0; fi; done
 return 1; }; stdout() { printf -- "cat <<'EOM'\n%s\nEOM\n" "$1"; }; stderr() {
 printf -- "cat <<'EOM' >&2\n%s\nEOM\n" "$1"; }; error() {
 [[ -n $1 ]] && stderr "$1"; stderr "$usage"; _return 1; }; _return() {
-printf -- "exit %d\n" "$1"; exit "$1"; }; set -e; trimmed_doc=${DOC:1:2474}
-usage=${DOC:40:1865}; digest=c7702
-shorts=(-h -U -C -s -v -b -d -n -e '' '' '' '' '' '' '' '' '' '' '' '' '' '' '' '' '')
-longs=(--help --skip-update --skip-check --stack --verbose --branch --debug --no-prompt --env --status --server --type --domain --app-name --pass --secrets --all --update --force --fast --skip-version-check --no-domain-poll --volumes --no-tty --user --dev)
-argcounts=(0 0 0 1 0 1 0 0 1 0 1 1 1 1 0 0 0 0 0 0 0 0 0 0 1 0); node_0(){
-switch __help 0; }; node_1(){ switch __skip_update 1; }; node_2(){
-switch __skip_check 2; }; node_3(){ value __stack 3; }; node_4(){
-switch __verbose 4; }; node_5(){ value __branch 5; }; node_6(){ switch __debug 6
-}; node_7(){ switch __no_prompt 7; }; node_8(){ value __env 8; }; node_9(){
+printf -- "exit %d\n" "$1"; exit "$1"; }; set -e; trimmed_doc=${DOC:1:2824}
+usage=${DOC:368:1887}; digest=6ab46
+shorts=(-n -b -v -U -e -h -d -C -s '' '' '' '' '' '' '' '' '' '' '' '' '' '' '' '' '')
+longs=(--no-prompt --branch --verbose --skip-update --env --help --debug --skip-check --stack --status --server --type --domain --app-name --pass --secrets --all --update --force --fast --chaos --volumes --no-tty --user --bump --dev)
+argcounts=(0 1 0 0 1 0 0 0 1 0 1 1 1 1 0 0 0 0 0 0 0 0 0 1 0 0); node_0(){
+switch __no_prompt 0; }; node_1(){ value __branch 1; }; node_2(){
+switch __verbose 2; }; node_3(){ switch __skip_update 3; }; node_4(){
+value __env 4; }; node_5(){ switch __help 5; }; node_6(){ switch __debug 6; }
+node_7(){ switch __skip_check 7; }; node_8(){ value __stack 8; }; node_9(){
 switch __status 9; }; node_10(){ value __server 10; }; node_11(){
 value __type 11; }; node_12(){ value __domain 12; }; node_13(){
 value __app_name 13; }; node_14(){ switch __pass 14; }; node_15(){
 switch __secrets 15; }; node_16(){ switch __all 16; }; node_17(){
 switch __update 17; }; node_18(){ switch __force 18; }; node_19(){
-switch __fast 19; }; node_20(){ switch __skip_version_check 20; }; node_21(){
-switch __no_domain_poll 21; }; node_22(){ switch __volumes 22; }; node_23(){
-switch __no_tty 23; }; node_24(){ value __user 24; }; node_25(){ switch __dev 25
+switch __fast 19; }; node_20(){ switch __chaos 20; }; node_21(){
+switch __volumes 21; }; node_22(){ switch __no_tty 22; }; node_23(){
+value __user 23; }; node_24(){ switch __bump 24; }; node_25(){ switch __dev 25
 }; node_26(){ value _type_ a; }; node_27(){ value _app_ a; }; node_28(){
 value _service_ a; }; node_29(){ value _version_ a; }; node_30(){ value _src_ a
 }; node_31(){ value _dst_ a; }; node_32(){ value _backup_file_ a; }; node_33(){
@ -198,97 +208,97 @@ _command deploy; }; node_51(){ _command check; }; node_52(){ _command version; }
 node_53(){ _command config; }; node_54(){ _command cp; }; node_55(){
 _command logs; }; node_56(){ _command ps; }; node_57(){ _command restore; }
 node_58(){ _command rm; }; node_59(){ _command delete; }; node_60(){
-_command run; }; node_61(){ _command rollback; }; node_62(){ _command secret; }
-node_63(){ _command generate; }; node_64(){ _command insert; }; node_65(){
-_command undeploy; }; node_66(){ _command volume; }; node_67(){ _command recipe
-}; node_68(){ _command create; }; node_69(){ _command release; }; node_70(){
-_command versions; }; node_71(){ _command server; }; node_72(){ _command add; }
-node_73(){ _command __ --; }; node_74(){ _command init; }; node_75(){
-_command apps; }; node_76(){ _command upgrade; }; node_77(){ _command doctor; }
-node_78(){ _command help; }; node_79(){ optional 0 1 2 3 4 5 6 7 8; }
-node_80(){ optional 79; }; node_81(){ either 46 47; }; node_82(){ required 81; }
-node_83(){ optional 9; }; node_84(){ optional 10; }; node_85(){ optional 11; }
-node_86(){ required 80 45 82 83 84 85; }; node_87(){ optional 12; }; node_88(){
-optional 13; }; node_89(){ optional 14; }; node_90(){ optional 15; }; node_91(){
-required 80 45 48 84 87 88 89 90 26; }; node_92(){ either 28 16; }; node_93(){
-required 92; }; node_94(){ required 80 45 27 49 93; }; node_95(){ optional 17; }
-node_96(){ optional 18; }; node_97(){ optional 19; }; node_98(){ optional 20; }
-node_99(){ optional 21; }; node_100(){ optional 29; }; node_101(){
-required 80 45 27 50 95 96 97 98 99 100; }; node_102(){ required 80 45 27 51; }
-node_103(){ required 80 45 27 52; }; node_104(){ required 80 45 27 53; }
-node_105(){ required 80 45 27 54 30 31; }; node_106(){ optional 28; }
-node_107(){ required 80 45 27 55 106; }; node_108(){ required 80 45 27 56; }
-node_109(){ required 80 45 27 57 93; }; node_110(){ either 58 59; }; node_111(){
-required 110; }; node_112(){ optional 22; }; node_113(){
-required 80 45 27 111 112 90; }; node_114(){ optional 32; }; node_115(){
-required 80 45 27 57 28 114; }; node_116(){ optional 23; }; node_117(){
-optional 24; }; node_118(){ oneormore 33; }; node_119(){
-required 80 45 27 60 116 117 28 118; }; node_120(){ required 80 45 27 61 100; }
-node_121(){ required 34 29; }; node_122(){ either 121 16; }; node_123(){
-required 122; }; node_124(){ optional 35; }; node_125(){
-required 80 45 27 62 63 123 124 89; }; node_126(){
-required 80 45 27 62 64 34 29 36 89; }; node_127(){ either 34 16; }; node_128(){
-required 127; }; node_129(){ required 80 45 27 62 111 128 89; }; node_130(){
-required 80 45 27 65; }; node_131(){ required 80 45 27 66 47; }; node_132(){
-either 37 16; }; node_133(){ required 132; }; node_134(){
-required 80 45 27 66 111 133; }; node_135(){ optional 118; }; node_136(){
-required 80 45 27 38 135; }; node_137(){ required 80 67 47; }; node_138(){
-required 80 67 68 39; }; node_139(){ required 80 67 39 69 96; }; node_140(){
-required 80 67 39 70; }; node_141(){ optional 41; }; node_142(){ optional 42; }
-node_143(){ required 80 71 72 40 141 142; }; node_144(){
-required 80 71 48 43 73 33; }; node_145(){ required 80 71 82; }; node_146(){
-required 80 71 40 58; }; node_147(){ required 80 71 40 74; }; node_148(){
-required 80 71 40 75 83; }; node_149(){ optional 25; }; node_150(){
-required 80 76 149; }; node_151(){ required 80 52; }; node_152(){ required 80 77
-}; node_153(){ oneormore 44; }; node_154(){ optional 153; }; node_155(){
-required 80 78 154; }; node_156(){ required 80; }; node_157(){
-either 86 91 94 101 102 103 104 105 107 108 109 113 115 119 120 125 126 129 130 131 134 136 137 138 139 140 143 144 145 146 147 148 150 151 152 155 156
-}; node_158(){ required 157; }; cat <<<' docopt_exit() {
-[[ -n $1 ]] && printf "%s\n" "$1" >&2; printf "%s\n" "${DOC:40:1865}" >&2
-exit 1; }'; unset var___help var___skip_update var___skip_check var___stack \
-var___verbose var___branch var___debug var___no_prompt var___env var___status \
+_command run; }; node_61(){ _command rollback; }; node_62(){ _command restart; }
+node_63(){ _command secret; }; node_64(){ _command generate; }; node_65(){
+_command insert; }; node_66(){ _command undeploy; }; node_67(){ _command volume
+}; node_68(){ _command recipe; }; node_69(){ _command create; }; node_70(){
+_command release; }; node_71(){ _command versions; }; node_72(){ _command server
+}; node_73(){ _command add; }; node_74(){ _command __ --; }; node_75(){
+_command init; }; node_76(){ _command apps; }; node_77(){ _command upgrade; }
+node_78(){ _command doctor; }; node_79(){ _command help; }; node_80(){
+optional 0 1 2 3 4 5 6 7 8; }; node_81(){ optional 80; }; node_82(){
+either 46 47; }; node_83(){ required 82; }; node_84(){ optional 9; }; node_85(){
+optional 10; }; node_86(){ optional 11; }; node_87(){ required 81 45 83 84 85 86
+}; node_88(){ optional 12; }; node_89(){ optional 13; }; node_90(){ optional 14
+}; node_91(){ optional 15; }; node_92(){ required 81 45 48 85 88 89 90 91 26; }
+node_93(){ either 28 16; }; node_94(){ required 93; }; node_95(){
+required 81 45 27 49 94; }; node_96(){ optional 17; }; node_97(){ optional 18; }
+node_98(){ optional 19; }; node_99(){ optional 20; }; node_100(){ optional 29; }
+node_101(){ required 81 45 27 50 96 97 98 99 100; }; node_102(){
+required 81 45 27 51; }; node_103(){ required 81 45 27 52; }; node_104(){
+required 81 45 27 53; }; node_105(){ required 81 45 27 54 30 31; }; node_106(){
+optional 28; }; node_107(){ required 81 45 27 55 106; }; node_108(){
+required 81 45 27 56; }; node_109(){ required 81 45 27 57 94; }; node_110(){
+either 58 59; }; node_111(){ required 110; }; node_112(){ optional 21; }
+node_113(){ required 81 45 27 111 112 91; }; node_114(){ optional 32; }
+node_115(){ required 81 45 27 57 28 114; }; node_116(){ optional 22; }
+node_117(){ optional 23; }; node_118(){ oneormore 33; }; node_119(){
+required 81 45 27 60 116 117 28 118; }; node_120(){ required 81 45 27 61 100; }
+node_121(){ required 81 45 27 62 28; }; node_122(){ required 34 29; }
+node_123(){ either 122 16; }; node_124(){ required 123; }; node_125(){
+optional 35; }; node_126(){ required 81 45 27 63 64 124 125 90; }; node_127(){
+required 81 45 27 63 65 34 29 36 90; }; node_128(){ either 34 16; }; node_129(){
+required 128; }; node_130(){ required 81 45 27 63 111 129 90; }; node_131(){
+required 81 45 27 66; }; node_132(){ required 81 45 27 67 47; }; node_133(){
+either 37 16; }; node_134(){ required 133; }; node_135(){
+required 81 45 27 67 111 134; }; node_136(){ optional 118; }; node_137(){
+required 81 45 27 38 136; }; node_138(){ required 81 68 47; }; node_139(){
+required 81 68 69 39; }; node_140(){ optional 24; }; node_141(){
+required 81 68 39 70 97 140; }; node_142(){ required 81 68 39 71; }; node_143(){
+optional 41; }; node_144(){ optional 42; }; node_145(){
+required 81 72 73 40 143 144; }; node_146(){ required 81 72 48 43 74 33; }
+node_147(){ required 81 72 83; }; node_148(){ required 81 72 40 58; }
+node_149(){ required 81 72 40 75; }; node_150(){ required 81 72 40 76 84; }
+node_151(){ optional 25; }; node_152(){ required 81 77 151; }; node_153(){
+required 81 52; }; node_154(){ required 81 78; }; node_155(){ oneormore 44; }
+node_156(){ optional 155; }; node_157(){ required 81 79 156; }; node_158(){
+required 81; }; node_159(){
+either 87 92 95 101 102 103 104 105 107 108 109 113 115 119 120 121 126 127 130 131 132 135 137 138 139 141 142 145 146 147 148 149 150 152 153 154 157 158
+}; node_160(){ required 159; }; cat <<<' docopt_exit() {
+[[ -n $1 ]] && printf "%s\n" "$1" >&2; printf "%s\n" "${DOC:368:1887}" >&2
+exit 1; }'; unset var___no_prompt var___branch var___verbose var___skip_update \
+var___env var___help var___debug var___skip_check var___stack var___status \
 var___server var___type var___domain var___app_name var___pass var___secrets \
-var___all var___update var___force var___fast var___skip_version_check \
-var___no_domain_poll var___volumes var___no_tty var___user var___dev \
-var__type_ var__app_ var__service_ var__version_ var__src_ var__dst_ \
-var__backup_file_ var__args_ var__secret_ var__cmd_ var__data_ var__volume_ \
-var__command_ var__recipe_ var__host_ var__user_ var__port_ var__provider_ \
-var__subcommands_ var_app var_list var_ls var_new var_backup var_deploy \
-var_check var_version var_config var_cp var_logs var_ps var_restore var_rm \
-var_delete var_run var_rollback var_secret var_generate var_insert \
-var_undeploy var_volume var_recipe var_create var_release var_versions \
-var_server var_add var___ var_init var_apps var_upgrade var_doctor var_help
-parse 158 "$@"; local prefix=${DOCOPT_PREFIX:-''}; unset "${prefix}__help" \
-"${prefix}__skip_update" "${prefix}__skip_check" "${prefix}__stack" \
-"${prefix}__verbose" "${prefix}__branch" "${prefix}__debug" \
-"${prefix}__no_prompt" "${prefix}__env" "${prefix}__status" \
-"${prefix}__server" "${prefix}__type" "${prefix}__domain" \
-"${prefix}__app_name" "${prefix}__pass" "${prefix}__secrets" "${prefix}__all" \
-"${prefix}__update" "${prefix}__force" "${prefix}__fast" \
-"${prefix}__skip_version_check" "${prefix}__no_domain_poll" \
-"${prefix}__volumes" "${prefix}__no_tty" "${prefix}__user" "${prefix}__dev" \
-"${prefix}_type_" "${prefix}_app_" "${prefix}_service_" "${prefix}_version_" \
-"${prefix}_src_" "${prefix}_dst_" "${prefix}_backup_file_" "${prefix}_args_" \
+var___all var___update var___force var___fast var___chaos var___volumes \
+var___no_tty var___user var___bump var___dev var__type_ var__app_ \
+var__service_ var__version_ var__src_ var__dst_ var__backup_file_ var__args_ \
+var__secret_ var__cmd_ var__data_ var__volume_ var__command_ var__recipe_ \
+var__host_ var__user_ var__port_ var__provider_ var__subcommands_ var_app \
+var_list var_ls var_new var_backup var_deploy var_check var_version var_config \
+var_cp var_logs var_ps var_restore var_rm var_delete var_run var_rollback \
+var_restart var_secret var_generate var_insert var_undeploy var_volume \
+var_recipe var_create var_release var_versions var_server var_add var___ \
+var_init var_apps var_upgrade var_doctor var_help; parse 160 "$@"
+local prefix=${DOCOPT_PREFIX:-''}; unset "${prefix}__no_prompt" \
+"${prefix}__branch" "${prefix}__verbose" "${prefix}__skip_update" \
+"${prefix}__env" "${prefix}__help" "${prefix}__debug" "${prefix}__skip_check" \
+"${prefix}__stack" "${prefix}__status" "${prefix}__server" "${prefix}__type" \
+"${prefix}__domain" "${prefix}__app_name" "${prefix}__pass" \
+"${prefix}__secrets" "${prefix}__all" "${prefix}__update" "${prefix}__force" \
+"${prefix}__fast" "${prefix}__chaos" "${prefix}__volumes" "${prefix}__no_tty" \
+"${prefix}__user" "${prefix}__bump" "${prefix}__dev" "${prefix}_type_" \
+"${prefix}_app_" "${prefix}_service_" "${prefix}_version_" "${prefix}_src_" \
+"${prefix}_dst_" "${prefix}_backup_file_" "${prefix}_args_" \
 "${prefix}_secret_" "${prefix}_cmd_" "${prefix}_data_" "${prefix}_volume_" \
 "${prefix}_command_" "${prefix}_recipe_" "${prefix}_host_" "${prefix}_user_" \
 "${prefix}_port_" "${prefix}_provider_" "${prefix}_subcommands_" \
 "${prefix}app" "${prefix}list" "${prefix}ls" "${prefix}new" "${prefix}backup" \
 "${prefix}deploy" "${prefix}check" "${prefix}version" "${prefix}config" \
 "${prefix}cp" "${prefix}logs" "${prefix}ps" "${prefix}restore" "${prefix}rm" \
-"${prefix}delete" "${prefix}run" "${prefix}rollback" "${prefix}secret" \
-"${prefix}generate" "${prefix}insert" "${prefix}undeploy" "${prefix}volume" \
-"${prefix}recipe" "${prefix}create" "${prefix}release" "${prefix}versions" \
-"${prefix}server" "${prefix}add" "${prefix}__" "${prefix}init" "${prefix}apps" \
-"${prefix}upgrade" "${prefix}doctor" "${prefix}help"
-eval "${prefix}"'__help=${var___help:-false}'
+"${prefix}delete" "${prefix}run" "${prefix}rollback" "${prefix}restart" \
+"${prefix}secret" "${prefix}generate" "${prefix}insert" "${prefix}undeploy" \
+"${prefix}volume" "${prefix}recipe" "${prefix}create" "${prefix}release" \
+"${prefix}versions" "${prefix}server" "${prefix}add" "${prefix}__" \
+"${prefix}init" "${prefix}apps" "${prefix}upgrade" "${prefix}doctor" \
+"${prefix}help"; eval "${prefix}"'__no_prompt=${var___no_prompt:-false}'
+eval "${prefix}"'__branch=${var___branch:-}'
+eval "${prefix}"'__verbose=${var___verbose:-false}'
 eval "${prefix}"'__skip_update=${var___skip_update:-false}'
+eval "${prefix}"'__env=${var___env:-}'
+eval "${prefix}"'__help=${var___help:-false}'
+eval "${prefix}"'__debug=${var___debug:-false}'
 eval "${prefix}"'__skip_check=${var___skip_check:-false}'
 eval "${prefix}"'__stack=${var___stack:-}'
-eval "${prefix}"'__verbose=${var___verbose:-false}'
-eval "${prefix}"'__branch=${var___branch:-}'
-eval "${prefix}"'__debug=${var___debug:-false}'
-eval "${prefix}"'__no_prompt=${var___no_prompt:-false}'
-eval "${prefix}"'__env=${var___env:-}'
 eval "${prefix}"'__status=${var___status:-false}'
 eval "${prefix}"'__server=${var___server:-}'
 eval "${prefix}"'__type=${var___type:-}'
@ -300,11 +310,11 @@ eval "${prefix}"'__all=${var___all:-false}'
 eval "${prefix}"'__update=${var___update:-false}'
 eval "${prefix}"'__force=${var___force:-false}'
 eval "${prefix}"'__fast=${var___fast:-false}'
-eval "${prefix}"'__skip_version_check=${var___skip_version_check:-false}'
-eval "${prefix}"'__no_domain_poll=${var___no_domain_poll:-false}'
+eval "${prefix}"'__chaos=${var___chaos:-false}'
 eval "${prefix}"'__volumes=${var___volumes:-false}'
 eval "${prefix}"'__no_tty=${var___no_tty:-false}'
 eval "${prefix}"'__user=${var___user:-}'
+eval "${prefix}"'__bump=${var___bump:-false}'
 eval "${prefix}"'__dev=${var___dev:-false}'
 eval "${prefix}"'_type_=${var__type_:-}'; eval "${prefix}"'_app_=${var__app_:-}'
 eval "${prefix}"'_service_=${var__service_:-}'
@ -339,6 +349,7 @@ eval "${prefix}"'rm=${var_rm:-false}'
 eval "${prefix}"'delete=${var_delete:-false}'
 eval "${prefix}"'run=${var_run:-false}'
 eval "${prefix}"'rollback=${var_rollback:-false}'
+eval "${prefix}"'restart=${var_restart:-false}'
 eval "${prefix}"'secret=${var_secret:-false}'
 eval "${prefix}"'generate=${var_generate:-false}'
 eval "${prefix}"'insert=${var_insert:-false}'
@ -356,27 +367,28 @@ eval "${prefix}"'upgrade=${var_upgrade:-false}'
 eval "${prefix}"'doctor=${var_doctor:-false}'
 eval "${prefix}"'help=${var_help:-false}'; local docopt_i=1
 [[ $BASH_VERSION =~ ^4.3 ]] && docopt_i=2; for ((;docopt_i>0;docopt_i--)); do
-declare -p "${prefix}__help" "${prefix}__skip_update" "${prefix}__skip_check" \
-"${prefix}__stack" "${prefix}__verbose" "${prefix}__branch" "${prefix}__debug" \
-"${prefix}__no_prompt" "${prefix}__env" "${prefix}__status" \
+declare -p "${prefix}__no_prompt" "${prefix}__branch" "${prefix}__verbose" \
+"${prefix}__skip_update" "${prefix}__env" "${prefix}__help" "${prefix}__debug" \
+"${prefix}__skip_check" "${prefix}__stack" "${prefix}__status" \
 "${prefix}__server" "${prefix}__type" "${prefix}__domain" \
 "${prefix}__app_name" "${prefix}__pass" "${prefix}__secrets" "${prefix}__all" \
-"${prefix}__update" "${prefix}__force" "${prefix}__fast" \
-"${prefix}__skip_version_check" "${prefix}__no_domain_poll" \
-"${prefix}__volumes" "${prefix}__no_tty" "${prefix}__user" "${prefix}__dev" \
-"${prefix}_type_" "${prefix}_app_" "${prefix}_service_" "${prefix}_version_" \
-"${prefix}_src_" "${prefix}_dst_" "${prefix}_backup_file_" "${prefix}_args_" \
-"${prefix}_secret_" "${prefix}_cmd_" "${prefix}_data_" "${prefix}_volume_" \
-"${prefix}_command_" "${prefix}_recipe_" "${prefix}_host_" "${prefix}_user_" \
-"${prefix}_port_" "${prefix}_provider_" "${prefix}_subcommands_" \
-"${prefix}app" "${prefix}list" "${prefix}ls" "${prefix}new" "${prefix}backup" \
-"${prefix}deploy" "${prefix}check" "${prefix}version" "${prefix}config" \
-"${prefix}cp" "${prefix}logs" "${prefix}ps" "${prefix}restore" "${prefix}rm" \
-"${prefix}delete" "${prefix}run" "${prefix}rollback" "${prefix}secret" \
-"${prefix}generate" "${prefix}insert" "${prefix}undeploy" "${prefix}volume" \
-"${prefix}recipe" "${prefix}create" "${prefix}release" "${prefix}versions" \
-"${prefix}server" "${prefix}add" "${prefix}__" "${prefix}init" "${prefix}apps" \
-"${prefix}upgrade" "${prefix}doctor" "${prefix}help"; done; }
+"${prefix}__update" "${prefix}__force" "${prefix}__fast" "${prefix}__chaos" \
+"${prefix}__volumes" "${prefix}__no_tty" "${prefix}__user" "${prefix}__bump" \
+"${prefix}__dev" "${prefix}_type_" "${prefix}_app_" "${prefix}_service_" \
+"${prefix}_version_" "${prefix}_src_" "${prefix}_dst_" \
+"${prefix}_backup_file_" "${prefix}_args_" "${prefix}_secret_" \
+"${prefix}_cmd_" "${prefix}_data_" "${prefix}_volume_" "${prefix}_command_" \
+"${prefix}_recipe_" "${prefix}_host_" "${prefix}_user_" "${prefix}_port_" \
+"${prefix}_provider_" "${prefix}_subcommands_" "${prefix}app" "${prefix}list" \
+"${prefix}ls" "${prefix}new" "${prefix}backup" "${prefix}deploy" \
+"${prefix}check" "${prefix}version" "${prefix}config" "${prefix}cp" \
+"${prefix}logs" "${prefix}ps" "${prefix}restore" "${prefix}rm" \
+"${prefix}delete" "${prefix}run" "${prefix}rollback" "${prefix}restart" \
+"${prefix}secret" "${prefix}generate" "${prefix}insert" "${prefix}undeploy" \
+"${prefix}volume" "${prefix}recipe" "${prefix}create" "${prefix}release" \
+"${prefix}versions" "${prefix}server" "${prefix}add" "${prefix}__" \
+"${prefix}init" "${prefix}apps" "${prefix}upgrade" "${prefix}doctor" \
+"${prefix}help"; done; }
 # docopt parser above, complete command for generating this parser is `docopt.sh abra`

 PROGRAM_NAME=$(basename "$0")
@ -488,13 +500,7 @@ require_docker_version() {
  done
 }

-require_valid_json() {
-  require_jq
-  $JQ "$1" > /dev/null || error "Invalid JSON '$1'"
-}
-
 require_valid_yaml() {
-  require_yq
  $YQ e "$1" > /dev/null || error "Invalid YAML '$1'"
 }

@ -509,8 +515,8 @@ require_apps_json() {

  if [ -f "$ABRA_APPS_JSON" ]; then
    modified=$(curl --silent --head "$ABRA_APPS_URL" | \
-             awk '/^Last-Modified/{print $0}' | \
-             sed 's/^Last-Modified: //')
+             awk '/^last-modified/{print tolower($0)}' | \
+             sed 's/^last-modified: //I')
    remote_ctime=$(date --date="$modified" +%s)
    local_ctime=$(stat -c %Z "$ABRA_APPS_JSON")

@ -600,7 +606,11 @@ require_app_version() {
  if [ -z "$VERSION" ]; then
    warning "No version specified, dangerously using latest git 😨"
  else
-    git checkout -q "$VERSION" || error "Can't find version $VERSION"
+    if [ "$abra___chaos" = "false" ]; then
+      git checkout -q "$VERSION" || error "Can't find version $VERSION"
+    else
+      warning "Chaos deploy specified, dangerously using latest git 😨"
+    fi
  fi
 }

@ -642,13 +652,31 @@ require_jq() {
 }

 require_yq() {
-  vendor_binary "https://github.com/mikefarah/yq" "v4.6.1" "yq_linux_amd64"
+  vendor_binary "https://github.com/mikefarah/yq" "v4.9.6" "yq_linux_amd64"
 }

 checkout_main_or_master() {
  git checkout main > /dev/null 2>&1 || git checkout master > /dev/null 2>&1
 }

+pwgen_native() {
+  if type pwgen > /dev/null 2>&1; then
+    pwgen -s "$length" 1
+    return
+  fi
+
+  tr -dc 'a-zA-Z0-9' < /dev/urandom | head -c "$1"
+}
+
+pwqgen_native() {
+  if type pwqgen > /dev/null 2>&1; then
+    pwqgen
+    return
+  fi
+
+  shuf -n 3 /usr/share/dict/words | tr -dc 'a-zA-Z0-9' | tr -d '\n'
+}
+
 # FIXME 3wc: update or remove
 if [ -z "$ABRA_ENV" ] && [ -f .env ] && type direnv > /dev/null 2>&1 && ! direnv status | grep -q 'Found RC allowed true'; then
  error "direnv is blocked, run direnv allow"
@ -682,7 +710,11 @@ get_recipe_version_latest() {
    info "No versions found"
  else
    VERSION="${RECIPE_VERSIONS[-1]}"
-    info "Chose version $VERSION"
+    if [ "$abra___chaos" = "true" ]; then
+      info "Not choosing a version and instead deploying from latest commit"
+    else
+      info "Chose version $VERSION"
+    fi
  fi
 }

@ -709,6 +741,7 @@ output_version_summary() {

  CONSENT_TO_UPDATE=$abra___update
  FORCE_DEPLOY=$abra___force
+  CHAOS_DEPLOY=$abra___chaos

  local -a IS_AN_UPDATE="false"
  local -a UNABLE_TO_DETECT="false"
@ -750,7 +783,11 @@ output_version_summary() {
          if [ "$live_version" != "$service_tag" ] || [ "$live_digest" != "$service_digest" ]; then
            IS_AN_UPDATE="true"
          fi
-          echo "      to be deployed: $(tput setaf 1)$service_tag ($service_digest)$(tput sgr0)"
+          if [ "$abra___chaos" = "true" ]; then
+            echo "      to be deployed: $(tput setaf 1)$service_tag ($service_digest) (+ latest git)$(tput sgr0)"
+          else
+            echo "      to be deployed: $(tput setaf 1)$service_tag ($service_digest)$(tput sgr0)"
+          fi
        fi
      else
        if [[ $UNDEPLOYED_STATE == "true" ]]; then
@ -776,54 +813,64 @@ output_version_summary() {
  else
    if [[ $UNABLE_TO_DETECT == "false" ]] && \
       [[ $UNDEPLOYED_STATE == "false" ]] && \
-       [[ $FORCE_DEPLOY == "false" ]]; then
+       [[ $FORCE_DEPLOY == "false" ]] && \
+       [[ $CHAOS_DEPLOY = "false" ]]; then
      success "Nothing to deploy, you're on latest (use --force to re-deploy anyway)"
      exit 0
    fi
  fi
 }

-# Note(decentral1se): inspired by https://github.com/vitalets/docker-stack-wait-deploy
 ensure_stack_deployed() {
-  STACK_NAME=$1
+  local -a HEALTHY # mapping
+  local -a MISSING # mapping

-  info "Waiting for deployment to succeed"
+  TIMEOUT=60
+  idx=0

-  while true; do
-    all_services_done=1
-    has_errors=0
+  IFS=' ' read -r -a SERVICES <<< "$(docker stack services "${STACK_NAME}" --format "{{.ID}}" | tr '\n' ' ')"
+  debug "Considering the following service IDs: ${SERVICES[*]} for ${STACK_NAME} deployment"

-    service_ids=$(docker stack services -q "$STACK_NAME")
+  while [ ! $(( ${#HEALTHY[@]} + ${#MISSING[@]} )) -eq ${#SERVICES[@]} ]; do
+    for service in $(docker ps -f "name=$STACK_NAME" -q); do
+      debug "Polling $service for deployment status"

-    for service_id in $service_ids; do
-      # see: https://github.com/moby/moby/issues/28012
-      service_state=$(docker service inspect --format "{{if .UpdateStatus}}{{.UpdateStatus.State}}{{else}}created{{end}}" "$service_id")
+      healthcheck=$(docker inspect --format "{{ json .State }}" "$service" | jq "try(.Health.Status // \"missing\")")
+      name=$(docker inspect --format '{{ index .Config.Labels "com.docker.swarm.service.name" }}' "$service")

-      debug "$service_id has state: $service_state"
+      if [[ ${MISSING[*]} =~ ${name} ]] || [[ ${HEALTHY[*]} =~ ${name} ]]; then
+        debug "$name already marked as missing healthcheck / healthy status"
+        continue
+      fi

-      case "$service_state" in
-        created|completed)
-          ;;
-        paused|rollback_completed)
-          has_errors=1
-          ;;
-        *)
-          all_services_done=0
-          ;;
-      esac
+      if [[ "$healthcheck" == "\"missing\"" ]] && [[ ! "${MISSING[*]}" =~ $name ]]; then
+        MISSING+=("$name")
+        debug "No healthcheck configured for $name"
+        continue
+      fi
+
+      if [[ "$healthcheck" == "\"healthy\"" ]] && [[ ! "${HEALTHY[*]}" =~ $name ]]; then
+        HEALTHY+=("$name")
+        debug "Marking $name with healthy status"
+        continue
+      fi
+
+      if [[ "$healthcheck" == \""unhealthy"\" ]]; then
+        logs=$(docker inspect --format "{{ json .State.Health.Log }}" "$service")
+        exitcode="$(echo "$logs" | $JQ '.[-1] | .ExitCode')"
+        warning "Healthcheck for new instance of $name is failing (exit code: $exitcode)"
+        warning "$(echo "$logs" | $JQ -r '.[-1] | .Output')"
+        error "healthcheck for $name is failing, this deployment did not succeed :("
+      fi
    done

-    if [ "$all_services_done" == "1" ]; then
-      if [ "$has_errors" == "1" ]; then
-        debug "Deployment appears to have failed"
-        break
-      else
-        debug "Deployment appears to have suceeded"
-        break
-      fi
-    else
-      sleep 1
+    idx=$(("$idx" + 1))
+    if [[ $idx -eq "$TIMEOUT" ]]; then
+      error "Waiting for healthy status timed out, this deployment did not succeed :("
    fi
+
+    sleep 1
+    debug "Deploying: $(( ${#HEALTHY[@]} + ${#MISSING[@]} ))/${#SERVICES[@]} (timeout: $idx/$TIMEOUT)"
  done
 }

@ -852,14 +899,14 @@ get_servers() {

 get_app_secrets() {
  # FIXME 3wc: requires bash 4, use for loop instead
-  mapfile -t PASSWORDS < <(grep "SECRET.*VERSION.*" "$ENV_FILE")
+  mapfile -t PASSWORDS < <(grep "^SECRET.*VERSION.*" "$ENV_FILE")
 }

 load_instance() {
  APP="$abra__app_"

  # load all files matching "$APP.env" into ENV_FILES array
-  mapfile -t ENV_FILES < <(find -L "$ABRA_DIR" -name "$APP.env")
+  mapfile -t ENV_FILES < <(find -L "$ABRA_DIR/servers/" -name "$APP.env")
  # FIXME 3wc: requires bash 4, use for loop instead

  case "${#ENV_FILES[@]}" in
@ -1298,16 +1345,15 @@ _abra_backup_mysql() {

 ###### .. app deploy
 help_app_deploy (){
-  echo "abra [options] app <app> deploy [--update] [--force] [--fast] [--skip-version-check] [--no-domain-poll]
+  echo "abra [options] app <app> deploy [--update] [--force] [--fast]

 Deploy app <app> to the configured server.

 OPTIONS
  --update                Consent to deploying an updated app version
  --force                 Force a deployment regardless of state
-  --skip-version-check    Don't try and detect deployed version
-  --no-domain-poll        Don't wait for the configured domain to come up
-  --fast                  Alias for --skip-version-check --no-domain-poll
+  --fast                  Run deployment without various safety checks
+  --chaos                 Deploy straight from latest Git version (potentially chaotic!)

 POWERED BY
  docker stack deploy -c compose.yml <app>"
@ -1319,9 +1365,11 @@ sub_app_deploy (){
  if [ "$abra___fast" = "true" ]; then
    SKIP_VERSION_CHECK=true
    NO_DOMAIN_POLL=true
+    NO_STATE_POLL=true
  else
-    SKIP_VERSION_CHECK=$abra___skip_version_check
-    NO_DOMAIN_POLL=$abra___no_domain_poll
+    SKIP_VERSION_CHECK=false
+    NO_DOMAIN_POLL=false
+    NO_STATE_POLL=false
  fi

  if [ ! "$abra__version_" = "dev" ]; then
@ -1330,7 +1378,7 @@ sub_app_deploy (){
    if [ -n "$abra__version_" ]; then
      VERSION="$abra__version_"
      if ! printf '%s\0' "${RECIPE_VERSIONS[@]}" | grep -Fqxz -- "$VERSION"; then
-        error "'$version' doesn't appear to be a valid version of $TYPE"
+        error "'$VERSION' doesn't appear to be a valid version of $TYPE"
      fi
      info "Chose version $VERSION"
    else
@ -1376,7 +1424,9 @@ sub_app_deploy (){
        success "Something happened! Hope it was good 🙏"
        exit 0
      fi
-      ensure_stack_deployed "$STACK_NAME"
+      if [[ $NO_STATE_POLL == "false" ]]; then
+        ensure_stack_deployed "$STACK_NAME"
+      fi
      if [ -n "$DOMAIN" ]; then
        if [[ $NO_DOMAIN_POLL == "false" ]]; then
          ensure_domain_deployed "https://${DOMAIN}"
@ -1584,6 +1634,9 @@ sub_app_secret_insert() {
  # shellcheck disable=SC2059
  printf "$PW" | docker secret create "${STACK_NAME}_${SECRET}_${VERSION}" - > /dev/null

+  # shellcheck disable=SC2181
+  if [[ $? != 0 ]]; then exit 1; fi
+
  if [ "$STORE_WITH_PASS" == "true" ] && type pass > /dev/null 2>&1; then
    echo "$PW" | pass insert "hosts/$DOCKER_CONTEXT/${STACK_NAME}/${SECRET}" -m > /dev/null
    success "pass: hosts/$DOCKER_CONTEXT/${STACK_NAME}/${SECRET}"
@ -1658,6 +1711,15 @@ sub_app_secret_generate(){
  local secret="$abra__secret_"
  local version="$abra__version_"
  local length="$abra__length_"
+  local msg_already_outputted=${msg_already_outputted:-"false"}
+
+  if [ "$msg_already_outputted" == "false" ]; then
+    warning "These generated secrets are now stored as encrypted data on your server"
+    warning "Please take a moment to make sure you have saved a copy of the passwords"
+    warning "Abra is not able to show the password values in plain text again"
+    warning "See https://docs.cloud.autonomic.zone/secrets/ for more on secrets"
+    msg_already_outputted="true"
+  fi

  if [ "$abra___all" == "true" ]; then
    # Note(decentral1se): we need to reset the flag here to avoid the infinite
@ -1669,11 +1731,9 @@ sub_app_secret_generate(){
  fi

  if [[ -n "$length" ]]; then
-    require_binary pwgen
-    abra__cmd_="pwgen -s $length 1"
+    abra__cmd_="pwgen_native $length"
  else
-    require_binary pwqgen
-    abra__cmd_=pwqgen
+    abra__cmd_=pwqgen_native
  fi

  PWGEN=${abra__cmd_}
@ -1683,7 +1743,7 @@ sub_app_secret_generate(){
    error "Required arguments missing"
  fi

-  PW=$($PWGEN|tr -d "\n")
+  PW=$($PWGEN)

  success "Password: $PW"

@ -1692,11 +1752,6 @@ sub_app_secret_generate(){
  abra__data_="$PW"

  sub_app_secret_insert
-
-  warning "These generated secrets are now stored as encrypted data on your server"
-  warning "Please take a moment to make sure you have saved a copy of the passwords"
-  warning "Abra is not able to show the password values in plain text again"
-  warning "See https://docs.docker.com/engine/swarm/secrets/ for more on secrets"
 }

 ###### .. app volume
@ -1715,10 +1770,6 @@ help_app_volume_list (){

 Show all volumes associated with <app>.

-OPTIONS
-  <secret>  Generate a single secret
-  --all     Auto-generate all secrets
-
 POWERED BY
  docker volume ls"
 }
@ -1793,10 +1844,12 @@ sub_app_run(){
    RUN_USER="-u $abra___user"
  fi

+  local -a ARGS="${ARGS:-""}"
+
  if [ "$abra___no_tty" = "true" ]; then
-    ARGS="-i"
+    ARGS+=" -i"
  else
-    ARGS="-it"
+    ARGS+=" -it"
  fi

  CONTAINER=$(docker container ls --format "table {{.ID}},{{.Names}}" \
@ -1808,11 +1861,14 @@ sub_app_run(){
  fi

  debug "Using container ID ${CONTAINER}"
+  # shellcheck disable=SC2145
+  debug "Running command: docker exec $RUN_USER $ARGS $CONTAINER $@"
+  exit

-  # 3wc: we want the "splitting" that shellcheck warns us about, so that -u and
-  # $RUN_USER aren't treated as a single argument:
  # shellcheck disable=SC2086
-  docker exec $RUN_USER $ARGS "$CONTAINER" "$@"
+  if ! docker exec $RUN_USER $ARGS "$CONTAINER" "$@"; then
+    exit 1
+  fi

  return
 }
@ -1856,6 +1912,31 @@ sub_app_rollback(){
  sub_app_deploy
 }

+###### .. app restart
+help_app_restart (){
+  echo "abra [options] app <app> restart <service>
+
+Restart a service.
+
+EXAMPLES
+  abra app peertube restart app
+
+POWERED BY
+  docker service scale foo=bar"
+}
+
+sub_app_restart(){
+  SERVICE="${abra__service_}"
+
+  debug "Scaling ${STACK_NAME}_${SERVICE} to 0 running containers..."
+  docker service scale "${STACK_NAME}_${SERVICE}=0"
+
+  debug "Scaling ${STACK_NAME}_${SERVICE} back to 1 running container..."
+  docker service scale "${STACK_NAME}_${SERVICE}=1"
+
+  success "${STACK_NAME}_${SERVICE} was succesfully restarted!"
+}
+
 ###### .. app logs
 help_app_logs (){
  echo "abra [options] app <app> logs [<service>]
@ -2042,7 +2123,7 @@ sub_recipe_versions() {

 ###### .. recipe <recipe> release
 help_recipe_release() {
-  echo "abra [options] recipe <recipe> release
+  echo "abra [options] recipe <recipe> release [--force] [--bump]

 (For recipe maintainers)

@ -2055,6 +2136,7 @@ any of the images in <recipe>.
 OPTIONS
  --force   Over-write existing tag; use this if you have a git tag for the
            recipe version already, to make sure labels are in sync.
+  --bump    Make an n+1 release (packager specific changes to recipe)

 POWERED BY
  skopeo inspect docker://image:tag
@ -2069,8 +2151,13 @@ sub_recipe_release() {

  recipe="$abra__recipe_"
  force="$abra___force"
+  bump="$abra___bump"
  recipe_dir="$ABRA_DIR/apps/$recipe"

+  if [ "$bump" = "true" ] && [ "$force" = "true" ]; then
+    error "--bump and --force don't play nicely together"
+  fi
+
  cd "$recipe_dir" || error "Can't find recipe dir '$recipe_dir'"

  get_recipe_versions "$recipe"
@ -2083,22 +2170,29 @@ sub_recipe_release() {
      latest_version_message=$(git show -s --format=%s)
    fi
    info "Latest available version: '$latest_version'"
-    info "Latest verion message: '$latest_version_message'"
+    info "Latest version message: '$latest_version_message'"
  else
    latest_version=""
    latest_version_message="Initial tagged release"
    info "No previous releases found"
+    if [ "$bump" = "true" ]; then
+      error "--bump can't do its work when there are no existing release versions"
+    fi
  fi

  current_tag=$(git tag --points-at HEAD)
-  if [ "$force" = "false" ] && [ -n "$current_tag" ]; then
-    error "$recipe is already on $current_tag, no release needed"
+  if [ "$force" = "false" ] && [ -n "$current_tag" ] && [ "$bump" = "false" ]; then
+    success "$recipe is already on $current_tag, no release needed"
  fi

  if [ "$(git rev-parse --abbrev-ref --symbolic-full-name HEAD)" = "HEAD" ]; then
    warning "It looks like $recipe_dir is in 'detached HEAD' state"
-    read -rp "Check out main/master branch first? [Y/n] "
-    if [ "${choice,,}" != "n" ]; then
+    if [ "$abra___no_prompt" = "false" ]; then
+      read -rp "Check out main/master branch first? [Y/n] "
+      if [ "${choice,,}" != "n" ]; then
+        checkout_main_or_master
+      fi
+    else
      checkout_main_or_master
    fi
  fi
@ -2110,7 +2204,12 @@ sub_recipe_release() {
  new_version="false"

  for compose_file in "${compose_files[@]}"; do
+    if [ "$bump" = "true" ]; then
+      continue  # skip trying to upgrade labels for --bump logic
+    fi
+
    mapfile -t services < <($YQ e -N '.services | keys | .[]' "$compose_file" | sort -u)
+
    if [ "$compose_file" = "compose.yml" ] && ! printf '%s\0' "${services[@]}" | grep -Fqxz -- "app"; then
      # shellcheck disable=SC2016
      warning 'No `app` service found; which one should we use for the version number?'
@ -2151,7 +2250,7 @@ sub_recipe_release() {
        fi
        info "Fetching $service_image metadata from Docker Hub"
        service_data=$(skopeo inspect "docker://$service_image")
-        service_digest=$(echo "$service_data" | jq -r '.Digest' | cut -d':' -f2 | cut -c-8)
+        service_digest=$(echo "$service_data" | $JQ -r '.Digest' | cut -d':' -f2 | cut -c-8)

        label="coop-cloud.\${STACK_NAME}.$service.version=${service_tag}-${service_digest}"

@ -2184,24 +2283,57 @@ sub_recipe_release() {
    error "Hmm, something went wrong generating a new version number.."
  fi

+  local -a expected_format="^coop-cloud.[a-z]*.[a-z]*.[a-z]*=v\\d*.\\d*.\\d*-\\S{8}"
+  if [[ ! "$new_version" =~ $expected_format ]]; then
+    error "'$new_version' does not match the expected label format, bailing out..."
+  fi
+
  success "All compose files updated; new version is $new_version"

-  read -rp "Commit your changes to git? [y/N]? " choice
+  if [ "$abra___no_prompt" = "false" ] && [ "$bump" = "false" ]; then
+    git --no-pager diff
+    read -rp "Commit your changes to git? [y/N]? " choice

-  if [ "${choice,,}" != "y" ]; then
-    return
+    if [ "${choice,,}" != "y" ]; then
+      return
+    fi
  fi

-  git commit -avem "Version $new_version; sync labels" || exit
-
-  read -rp "Tag this as \`$new_version\`? [y/N]? " choice
-
-  if [ "${choice,,}" != "y" ]; then
-    return
+  if [ "$abra___no_prompt" = "false" ] && [ "$bump" = "false" ]; then
+    git commit -avem "Version $new_version; sync labels" || exit
+  else
+    git commit -am "Version $new_version; sync labels" || true
  fi

-  test "$force" = "true" && git tag -d "$new_version"
-  git tag -aem "$latest_version_message" "$new_version"
+  if [ "$abra___no_prompt" = "false" ]; then
+    read -rp "Tag this as \`$new_version\`? [y/N]? " choice
+
+    if [ "${choice,,}" != "y" ]; then
+      return
+    fi
+  fi
+
+  if [ "$force" = "true" ]; then
+    git tag -d "$new_version" || true
+    git push origin --delete "$new_version" || true
+    debug "Deleted local tag and remote tag if present"
+  fi
+
+  if [ "$abra___no_prompt" = "false" ]; then
+    git tag -aem "$latest_version_message" "$new_version"
+  else
+    git tag -am "$latest_version_message" "$new_version" || true
+  fi
+
+  if [ "$abra___no_prompt" = "false" ]; then
+    read -rp "Git push this new tag? [y/N]? " choice
+
+    if [ "${choice,,}" = "y" ]; then
+      git push && git push --tags
+    fi
+  else
+    git push && git push --tags
+  fi
 }

 #######################################
@ -2398,9 +2530,9 @@ OPTIONS

 sub_upgrade() {
  if [[ "$abra___dev" == "true" ]]; then
-    curl https://install.abra.autonomic.zone | bash -s -- --dev
+    curl https://install.abra.coopcloud.tech | bash -s -- --dev --no-deps
  else
-    curl https://install.abra.autonomic.zone | bash
+    curl https://install.abra.coopcloud.tech | bash --no-deps
  fi
 }

@ -2483,19 +2615,6 @@ sub_network() {
 abra() {
  require_bash_4

-  # TODO (3wc): we either need to do this, or add 'shellcheck disable' all over
-  # the place to handle the dynamically-defined vars
-  declare abra___stack abra___env abra__command_ abra__args_ \
-    abra__secret_ abra__version_ abra__data_ abra___user abra__host_ \
-    abra__type_ abra__port_ abra__user_ abra__service_ abra__src_ abra__dst_ \
-    abra___server abra___domain abra___pass abra___secrets abra___status  \
-    abra___no_tty abra___app_name abra__subcommands_ abra___skip_update \
-    abra___skip_check abra__backup_file_ abra___verbose abra___debug \
-    abra___help abra___branch abra___volumes abra__provider_ abra___type \
-    abra___dev abra___update abra___no_prompt abra___force \
-    abra___skip_version_check abra__recipe_ abra___no_domain_poll \
-    abra___fast abra__volume_
-
  if ! type tput > /dev/null 2>&1; then
    tput() {
      echo -n
@ -2559,7 +2678,7 @@ abra() {
  # Use abra__command_ in case `command` is provided (i.e. `volume` or `stack`)
  CMD="sub_${abra__command_}"
  if type "$CMD" > /dev/null 2>&1; then
-    # shellcheck disable=SC2086
+    # shellcheck disable=SC2086,SC2048
    "$CMD" ${abra__args_[*]}
  else
    docopt_exit
--- a/bin/abralib.py
+++ b/bin/abralib.py
@ -0,0 +1,108 @@
+"""Shared utilities for bin/*.py scripts."""
+
+from logging import DEBUG, basicConfig, getLogger
+from os import chdir, mkdir
+from os.path import exists, expanduser
+from pathlib import Path
+from shlex import split
+from subprocess import check_output
+from sys import exit
+
+from requests import get
+
+HOME_PATH = expanduser("~/")
+CLONES_PATH = Path(f"{HOME_PATH}/.abra/apps").absolute()
+REPOS_TO_SKIP = (
+    "abra",
+    "abra-apps",
+    "abra-gandi",
+    "abra-hetzner",
+    "auto-apps-json",
+    "auto-mirror",
+    "backup-bot",
+    "coopcloud.tech",
+    "coturn",
+    "docker-cp-deploy",
+    "docker-dind-bats-kcov",
+    "docs.coopcloud.tech",
+    "example",
+    "gardening",
+    "organising",
+    "pyabra",
+    "radicle-seed-node",
+    "stack-ssh-deploy",
+    "swarm-cronjob",
+)
+YQ_PATH = Path(f"{HOME_PATH}/.abra/vendor/yq")
+JQ_PATH = Path(f"{HOME_PATH}/.abra/vendor/jq")
+
+log = getLogger(__name__)
+basicConfig()
+log.setLevel(DEBUG)
+
+
+def _run_cmd(cmd, shell=False, **kwargs):
+    """Run a shell command."""
+    args = [split(cmd)]
+
+    if shell:
+        args = [cmd]
+        kwargs = {"shell": shell}
+
+    try:
+        return check_output(*args, **kwargs).decode("utf-8").strip()
+    except Exception as exception:
+        log.error(f"Failed to run {cmd}, saw {str(exception)}")
+        exit(1)
+
+
+def get_repos_json():
+    """ Retrieve repo list from Gitea """
+
+    url = "https://git.autonomic.zone/api/v1/orgs/coop-cloud/repos"
+
+    log.info(f"Retrieving {url}")
+
+    repos = []
+    response = True
+    page = 1
+
+    try:
+        while response:
+            log.info(f"Trying to fetch page {page}")
+            response = get(url + f"?page={page}", timeout=10).json()
+            repos.extend(response)
+            page += 1
+
+        return repos
+    except Exception as exception:
+        log.error(f"Failed to retrieve {url}, saw {str(exception)}")
+        exit(1)
+
+
+def clone_all_apps(repos_json, ssh=False):
+    """Clone all Co-op Cloud apps to ~/.abra/apps."""
+    if not exists(CLONES_PATH):
+        mkdir(CLONES_PATH)
+
+    if ssh:
+        repos = [[p["name"], p["ssh_url"]] for p in repos_json]
+    else:
+        repos = [[p["name"], p["clone_url"]] for p in repos_json]
+
+    for name, url in repos:
+        if name in REPOS_TO_SKIP:
+            continue
+
+        if not exists(f"{CLONES_PATH}/{name}"):
+            log.info(f"Retrieving {url}")
+            _run_cmd(f"git clone {url} {CLONES_PATH}/{name}")
+
+            chdir(f"{CLONES_PATH}/{name}")
+            if not int(_run_cmd("git branch --list | wc -l", shell=True)):
+                log.info(f"Guessing main branch is HEAD for {name}")
+                _run_cmd("git checkout main")
+        else:
+            log.info(f"Updating {name}")
+            chdir(f"{CLONES_PATH}/{name}")
+            _run_cmd("git fetch -a")
--- a/bin/app-json.py
+++ b/bin/app-json.py
@ -6,84 +6,44 @@
 # ~/.abra/apps), and format it as JSON so that it can be hosted here:
 # https://apps.coopcloud.tech

+import argparse
 from json import dump
-from logging import DEBUG, basicConfig, getLogger
-from os import chdir, listdir, mkdir
-from os.path import basename, exists, expanduser
+from os import chdir, environ, getcwd, listdir
+from os.path import basename
 from pathlib import Path
 from re import findall, search
-from shlex import split
-from subprocess import DEVNULL, check_output
-from sys import exit
+from subprocess import DEVNULL

 from requests import get

-HOME_PATH = expanduser("~/")
-CLONES_PATH = Path(f"{HOME_PATH}/.abra/apps").absolute()
-YQ_PATH = Path(f"{HOME_PATH}/.abra/vendor/yq")
-SCRIPT_PATH = Path(__file__).absolute().parent
-REPOS_TO_SKIP = (
-    "abra",
-    "abra-apps",
-    "abra-gandi",
-    "abra-hetzner",
-    "backup-bot",
-    "coopcloud.tech",
-    "coturn",
-    "docker-cp-deploy",
-    "docker-dind-bats-kcov",
-    "docs.coopcloud.tech",
-    "example",
-    "gardening",
-    "organising",
-    "pyabra",
-    "radicle-seed-node",
-    "stack-ssh-deploy",
-    "swarm-cronjob",
+from abralib import (
+    CLONES_PATH,
+    JQ_PATH,
+    REPOS_TO_SKIP,
+    YQ_PATH,
+    _run_cmd,
+    clone_all_apps,
+    get_repos_json,
+    log,
 )

-log = getLogger(__name__)
-basicConfig()
-log.setLevel(DEBUG)
+parser = argparse.ArgumentParser(description="Generate a new apps.json")
+parser.add_argument("--output", type=Path, default=f"{getcwd()}/apps.json")


-def _run_cmd(cmd, shell=False, **kwargs):
-    """Run a shell command."""
-    args = [split(cmd)]
+def skopeo_login():
+    """Log into the docker registry to avoid rate limits."""
+    user = environ.get("SKOPEO_USER")
+    password = environ.get("SKOPEO_PASSWORD")
+    registry = environ.get("SKOPEO_REGISTRY", "docker.io")

-    if shell:
-        args = [cmd]
-        kwargs = {"shell": shell}
+    if not user or not password:
+        log.info("Failed to log in via Skopeo due to missing env vars")
+        return

-    try:
-        return check_output(*args, **kwargs).decode("utf-8").strip()
-    except Exception as exception:
-        log.error(f"Failed to run {cmd}, saw {str(exception)}")
-        exit(1)
-
-
-def get_repos_json():
-    """ Retrieve repo list from Gitea """
-
-    url = "https://git.autonomic.zone/api/v1/orgs/coop-cloud/repos"
-
-    log.info(f"Retrieving {url}")
-
-    repos = []
-    response = True
-    page = 1
-
-    try:
-        while response:
-            log.info(f"Trying to fetch page {page}")
-            response = get(url + f"?page={page}", timeout=10).json()
-            repos.extend(response)
-            page += 1
-
-        return repos
-    except Exception as exception:
-        log.error(f"Failed to retrieve {url}, saw {str(exception)}")
-        exit(1)
+    login_cmd = f"skopeo login {registry} -u {user} -p {password}"
+    output = _run_cmd(login_cmd, shell=True)
+    log.info(f"Skopeo login attempt: {output}")


 def get_published_apps_json():
@ -99,31 +59,6 @@ def get_published_apps_json():
        return {}


-def clone_all_apps(repos_json):
-    """Clone all Co-op Cloud apps to ~/.abra/apps."""
-    if not exists(CLONES_PATH):
-        mkdir(CLONES_PATH)
-
-    repos = [[p["name"], p["ssh_url"]] for p in repos_json]
-
-    for name, url in repos:
-        if name in REPOS_TO_SKIP:
-            continue
-
-        if not exists(f"{CLONES_PATH}/{name}"):
-            log.info(f"Retrieving {url}")
-            _run_cmd(f"git clone {url} {CLONES_PATH}/{name}")
-
-            chdir(f"{CLONES_PATH}/{name}")
-            if not int(_run_cmd("git branch --list | wc -l", shell=True)):
-                log.info(f"Guessing main branch is HEAD for {name}")
-                _run_cmd("git checkout main")
-        else:
-            log.info(f"Updating {name}")
-            chdir(f"{CLONES_PATH}/{name}")
-            _run_cmd("git fetch -a")
-
-
 def generate_apps_json(repos_json):
    """Generate the abra-apps.json application versions file."""
    apps_json = {}
@ -257,7 +192,7 @@ def get_app_versions(app_path, cached_apps_json):
                if image in ("null", "---"):
                    continue

-                images_cmd = f"skopeo inspect docker://{image} | jq '.Digest'"
+                images_cmd = f"skopeo inspect docker://{image} | {JQ_PATH} '.Digest'"
                output = _run_cmd(images_cmd, shell=True)

                service_version_info = {
@ -280,11 +215,14 @@ def get_app_versions(app_path, cached_apps_json):

 def main():
    """Run the script."""
+    args = parser.parse_args()
+
+    skopeo_login()
+
    repos_json = get_repos_json()
    clone_all_apps(repos_json)

-    target = f"{SCRIPT_PATH}/../deploy/apps.coopcloud.tech/apps.json"
-    with open(target, "w", encoding="utf-8") as handle:
+    with open(args.output, "w", encoding="utf-8") as handle:
        dump(
            generate_apps_json(repos_json),
            handle,
@ -293,7 +231,7 @@ def main():
            sort_keys=True,
        )

-    log.info(f"Successfully generated {target}")
+    log.info(f"Successfully generated {args.output}")


 main()
--- a/bin/clone-all-apps.py
+++ b/bin/clone-all-apps.py
@ -0,0 +1,16 @@
+#!/usr/bin/env python3
+
+# Usage: ./clone-all-apps.py
+#
+# Clone all available apps into ~/.abra/apps using ssh:// URLs
+
+from abralib import clone_all_apps, get_repos_json
+
+
+def main():
+    """Run the script."""
+    repos_json = get_repos_json()
+    clone_all_apps(repos_json, ssh=True)
+
+
+main()
--- a/bin/github-sync.py
+++ b/bin/github-sync.py
@ -0,0 +1,47 @@
+#!/usr/bin/env python3
+
+# Usage: ./github-sync.py
+#
+# Mirror repositories to Github (Fuck M$, get it straight)
+
+from os import chdir, environ, listdir
+
+from abralib import CLONES_PATH, _run_cmd, clone_all_apps, get_repos_json, log
+
+REPOS_TO_SKIP = (
+    "backup-bot",
+    "docker-dind-bats-kcov",
+    "docs.coopcloud.tech",
+    "pyabra",
+    "radicle-seed-node",
+    "swarm-cronjob",
+)
+
+
+def main():
+    """Run the script."""
+    repos_json = get_repos_json()
+    clone_all_apps(repos_json)
+
+    for app in listdir(CLONES_PATH):
+        if app in REPOS_TO_SKIP:
+            log.info(f"Skipping {app}")
+            continue
+
+        app_path = f"{CLONES_PATH}/{app}"
+        chdir(app_path)
+
+        log.info(f"Mirroring {app}...")
+
+        token = environ.get("GITHUB_ACCESS_TOKEN")
+        remote = f"https://coopcloudbot:{token}@github.com/Coop-Cloud/{app}.git"
+
+        _run_cmd(
+            f"git remote add github {remote} || true",
+            shell=True,
+        )
+
+        _run_cmd("git push github --all")
+
+
+main()
--- a/bin/renovate-ls-apps.py
+++ b/bin/renovate-ls-apps.py
@ -0,0 +1,20 @@
+#!/usr/bin/env python3
+
+# Usage: ./renovate-ls-apps.py
+#
+# Output list of apps for Renovate bot configuration
+
+from abralib import REPOS_TO_SKIP, get_repos_json
+
+
+def main():
+    """Run the script."""
+    repos = [p["full_name"] for p in get_repos_json()]
+    repos.sort()
+    for repo in repos:
+        if repo.split("/")[-1] in REPOS_TO_SKIP:
+            continue
+        print(f'"{repo}",')
+
+
+main()
--- a/deploy/apps.coopcloud.tech/apps.json
+++ b/deploy/apps.coopcloud.tech/apps.json
--- a/deploy/apps.coopcloud.tech/compose.yml
+++ b/deploy/apps.coopcloud.tech/compose.yml
@ -1,41 +0,0 @@
---
-version: "3.8"
-
-services:
-  app:
-    image: "nginx:stable"
-    configs:
-      - source: abra_conf
-        target: /etc/nginx/conf.d/abra.conf
-      - source: abra_apps_json
-        target: /var/www/abra-apps/apps.json
-    volumes:
-      - "public:/var/www/abra-apps"
-    networks:
-      - proxy
-    deploy:
-      update_config:
-        failure_action: rollback
-        order: start-first
-      labels:
-        - "traefik.enable=true"
-        - "traefik.http.services.abra-apps.loadbalancer.server.port=80"
-        - "traefik.http.routers.abra-apps.rule=Host(`apps.coopcloud.tech`, `abra-apps.cloud.autonomic.zone`)"
-        - "traefik.http.routers.abra-apps.entrypoints=web-secure"
-        - "traefik.http.routers.abra-apps.tls.certresolver=production"
-        - "traefik.http.routers.abra-apps.middlewares=abra-apps-redirect"
-        - "traefik.http.middlewares.abra-apps-redirect.headers.SSLForceHost=true"
-        - "traefik.http.middlewares.abra-apps-redirect.headers.SSLHost=apps.coopcloud.tech"
-
-configs:
-  abra_apps_json:
-    file: apps.json
-  abra_conf:
-    file: nginx.conf
-
-networks:
-  proxy:
-    external: true
-
-volumes:
-  public:
--- a/deploy/apps.coopcloud.tech/nginx.conf
+++ b/deploy/apps.coopcloud.tech/nginx.conf
@ -1,10 +0,0 @@
-server {
-  listen         80 default_server;
-  server_name    apps.coopcloud.tech;
-
-  location / {
-    root /var/www/abra-apps;
-    add_header Content-Type application/json;
-    index apps.json;
-  }
-}
--- a/deploy/install.abra.autonomic.zone/installer
+++ b/deploy/install.abra.autonomic.zone/installer
@ -1,35 +0,0 @@
-#!/bin/bash
-
-ABRA_VERSION="8.0.0"
-GIT_URL="https://git.autonomic.zone/coop-cloud/abra"
-ABRA_SRC="$GIT_URL/raw/tag/$ABRA_VERSION/abra"
-ABRA_DIR="${ABRA_DIR:-$HOME/.abra/}"
-
-function install_abra_release {
-  mkdir -p "$HOME/.local/bin"
-  curl "$ABRA_SRC" > "$HOME/.local/bin/abra"
-  chmod +x "$HOME/.local/bin/abra"
-  echo "abra installed to $HOME/.local/bin/abra"
-}
-
-function install_abra_dev {
-  mkdir -p "$ABRA_DIR/"
-  if [[ ! -d "$ABRA_DIR/src" ]]; then
-    git clone "$GIT_URL" "$ABRA_DIR/src"
-  fi
-  (cd "$ABRA_DIR/src" && git pull origin main && cd - || exit)
-  mkdir -p "$HOME/.local/bin"
-  ln -sf "$ABRA_DIR/src/abra" "$HOME/.local/bin/abra"
-  echo "abra installed to $HOME/.local/bin/abra (development bleeding edge)"
-}
-
-function run_installation {
-  if [ "$1" = "--dev" ]; then
-    install_abra_dev
-  else
-    install_abra_release
-  fi
-}
-
-run_installation "$@"
-exit 0
--- a/deploy/install.abra.coopcloud.tech/compose.yml
+++ b/deploy/install.abra.coopcloud.tech/compose.yml
@ -20,7 +20,7 @@ services:
      labels:
        - "traefik.enable=true"
        - "traefik.http.services.abra-installer.loadbalancer.server.port=80"
-        - "traefik.http.routers.abra-installer.rule=Host(`install.abra.autonomic.zone`)"
+        - "traefik.http.routers.abra-installer.rule=Host(`install.abra.autonomic.zone`,`install.abra.coopcloud.tech`)"
        - "traefik.http.routers.abra-installer.entrypoints=web-secure"
        - "traefik.http.routers.abra-installer.tls.certresolver=production"

--- a/deploy/install.abra.coopcloud.tech/installer
+++ b/deploy/install.abra.coopcloud.tech/installer
@ -0,0 +1,221 @@
+#!/usr/bin/env bash
+
+# shellcheck disable=SC2154,SC2034
+
+ABRA_VERSION="10.0.0"
+GIT_URL="https://git.autonomic.zone/coop-cloud/abra"
+ABRA_SRC="$GIT_URL/raw/tag/$ABRA_VERSION/abra"
+ABRA_DIR="${ABRA_DIR:-$HOME/.abra}"
+
+DOC="
+abra command-line installer script
+
+Usage:
+  installer [options]
+
+Options:
+  -h, --help            Show this message and exit
+  -d, --dev             Install bleeding edge development version
+  -n, --no-prompt       Don't prompt for input and run non-interactively
+  -p, --no-deps         Don't attempt to install system dependencies
+"
+
+# docopt parser below, refresh this parser with `docopt.sh installer`
+# shellcheck disable=2016,1075
+docopt() { parse() { if ${DOCOPT_DOC_CHECK:-true}; then local doc_hash
+if doc_hash=$(printf "%s" "$DOC" | (sha256sum 2>/dev/null || shasum -a 256)); then
+if [[ ${doc_hash:0:5} != "$digest" ]]; then
+stderr "The current usage doc (${doc_hash:0:5}) does not match \
+what the parser was generated with (${digest})
+Run \`docopt.sh\` to refresh the parser."; _return 70; fi; fi; fi
+local root_idx=$1; shift; argv=("$@"); parsed_params=(); parsed_values=()
+left=(); testdepth=0; local arg; while [[ ${#argv[@]} -gt 0 ]]; do
+if [[ ${argv[0]} = "--" ]]; then for arg in "${argv[@]}"; do
+parsed_params+=('a'); parsed_values+=("$arg"); done; break
+elif [[ ${argv[0]} = --* ]]; then parse_long
+elif [[ ${argv[0]} = -* && ${argv[0]} != "-" ]]; then parse_shorts
+elif ${DOCOPT_OPTIONS_FIRST:-false}; then for arg in "${argv[@]}"; do
+parsed_params+=('a'); parsed_values+=("$arg"); done; break; else
+parsed_params+=('a'); parsed_values+=("${argv[0]}"); argv=("${argv[@]:1}"); fi
+done; local idx; if ${DOCOPT_ADD_HELP:-true}; then
+for idx in "${parsed_params[@]}"; do [[ $idx = 'a' ]] && continue
+if [[ ${shorts[$idx]} = "-h" || ${longs[$idx]} = "--help" ]]; then
+stdout "$trimmed_doc"; _return 0; fi; done; fi
+if [[ ${DOCOPT_PROGRAM_VERSION:-false} != 'false' ]]; then
+for idx in "${parsed_params[@]}"; do [[ $idx = 'a' ]] && continue
+if [[ ${longs[$idx]} = "--version" ]]; then stdout "$DOCOPT_PROGRAM_VERSION"
+_return 0; fi; done; fi; local i=0; while [[ $i -lt ${#parsed_params[@]} ]]; do
+left+=("$i"); ((i++)) || true; done
+if ! required "$root_idx" || [ ${#left[@]} -gt 0 ]; then error; fi; return 0; }
+parse_shorts() { local token=${argv[0]}; local value; argv=("${argv[@]:1}")
+[[ $token = -* && $token != --* ]] || _return 88; local remaining=${token#-}
+while [[ -n $remaining ]]; do local short="-${remaining:0:1}"
+remaining="${remaining:1}"; local i=0; local similar=(); local match=false
+for o in "${shorts[@]}"; do if [[ $o = "$short" ]]; then similar+=("$short")
+[[ $match = false ]] && match=$i; fi; ((i++)) || true; done
+if [[ ${#similar[@]} -gt 1 ]]; then
+error "${short} is specified ambiguously ${#similar[@]} times"
+elif [[ ${#similar[@]} -lt 1 ]]; then match=${#shorts[@]}; value=true
+shorts+=("$short"); longs+=(''); argcounts+=(0); else value=false
+if [[ ${argcounts[$match]} -ne 0 ]]; then if [[ $remaining = '' ]]; then
+if [[ ${#argv[@]} -eq 0 || ${argv[0]} = '--' ]]; then
+error "${short} requires argument"; fi; value=${argv[0]}; argv=("${argv[@]:1}")
+else value=$remaining; remaining=''; fi; fi; if [[ $value = false ]]; then
+value=true; fi; fi; parsed_params+=("$match"); parsed_values+=("$value"); done
+}; parse_long() { local token=${argv[0]}; local long=${token%%=*}
+local value=${token#*=}; local argcount; argv=("${argv[@]:1}")
+[[ $token = --* ]] || _return 88; if [[ $token = *=* ]]; then eq='='; else eq=''
+value=false; fi; local i=0; local similar=(); local match=false
+for o in "${longs[@]}"; do if [[ $o = "$long" ]]; then similar+=("$long")
+[[ $match = false ]] && match=$i; fi; ((i++)) || true; done
+if [[ $match = false ]]; then i=0; for o in "${longs[@]}"; do
+if [[ $o = $long* ]]; then similar+=("$long"); [[ $match = false ]] && match=$i
+fi; ((i++)) || true; done; fi; if [[ ${#similar[@]} -gt 1 ]]; then
+error "${long} is not a unique prefix: ${similar[*]}?"
+elif [[ ${#similar[@]} -lt 1 ]]; then
+[[ $eq = '=' ]] && argcount=1 || argcount=0; match=${#shorts[@]}
+[[ $argcount -eq 0 ]] && value=true; shorts+=(''); longs+=("$long")
+argcounts+=("$argcount"); else if [[ ${argcounts[$match]} -eq 0 ]]; then
+if [[ $value != false ]]; then
+error "${longs[$match]} must not have an argument"; fi
+elif [[ $value = false ]]; then
+if [[ ${#argv[@]} -eq 0 || ${argv[0]} = '--' ]]; then
+error "${long} requires argument"; fi; value=${argv[0]}; argv=("${argv[@]:1}")
+fi; if [[ $value = false ]]; then value=true; fi; fi; parsed_params+=("$match")
+parsed_values+=("$value"); }; required() { local initial_left=("${left[@]}")
+local node_idx; ((testdepth++)) || true; for node_idx in "$@"; do
+if ! "node_$node_idx"; then left=("${initial_left[@]}"); ((testdepth--)) || true
+return 1; fi; done; if [[ $((--testdepth)) -eq 0 ]]; then
+left=("${initial_left[@]}"); for node_idx in "$@"; do "node_$node_idx"; done; fi
+return 0; }; optional() { local node_idx; for node_idx in "$@"; do
+"node_$node_idx"; done; return 0; }; switch() { local i
+for i in "${!left[@]}"; do local l=${left[$i]}
+if [[ ${parsed_params[$l]} = "$2" ]]; then
+left=("${left[@]:0:$i}" "${left[@]:((i+1))}")
+[[ $testdepth -gt 0 ]] && return 0; if [[ $3 = true ]]; then
+eval "((var_$1++))" || true; else eval "var_$1=true"; fi; return 0; fi; done
+return 1; }; stdout() { printf -- "cat <<'EOM'\n%s\nEOM\n" "$1"; }; stderr() {
+printf -- "cat <<'EOM' >&2\n%s\nEOM\n" "$1"; }; error() {
+[[ -n $1 ]] && stderr "$1"; stderr "$usage"; _return 1; }; _return() {
+printf -- "exit %d\n" "$1"; exit "$1"; }; set -e; trimmed_doc=${DOC:1:333}
+usage=${DOC:37:28}; digest=36916; shorts=(-h -d -n -p)
+longs=(--help --dev --no-prompt --no-deps); argcounts=(0 0 0 0); node_0(){
+switch __help 0; }; node_1(){ switch __dev 1; }; node_2(){ switch __no_prompt 2
+}; node_3(){ switch __no_deps 3; }; node_4(){ optional 0 1 2 3; }; node_5(){
+optional 4; }; node_6(){ required 5; }; node_7(){ required 6; }
+cat <<<' docopt_exit() { [[ -n $1 ]] && printf "%s\n" "$1" >&2
+printf "%s\n" "${DOC:37:28}" >&2; exit 1; }'; unset var___help var___dev \
+var___no_prompt var___no_deps; parse 7 "$@"; local prefix=${DOCOPT_PREFIX:-''}
+unset "${prefix}__help" "${prefix}__dev" "${prefix}__no_prompt" \
+"${prefix}__no_deps"; eval "${prefix}"'__help=${var___help:-false}'
+eval "${prefix}"'__dev=${var___dev:-false}'
+eval "${prefix}"'__no_prompt=${var___no_prompt:-false}'
+eval "${prefix}"'__no_deps=${var___no_deps:-false}'; local docopt_i=1
+[[ $BASH_VERSION =~ ^4.3 ]] && docopt_i=2; for ((;docopt_i>0;docopt_i--)); do
+declare -p "${prefix}__help" "${prefix}__dev" "${prefix}__no_prompt" \
+"${prefix}__no_deps"; done; }
+# docopt parser above, complete command for generating this parser is `docopt.sh installer`
+
+function prompt_confirm {
+  if [ "$no_prompt" == "true" ]; then
+    return
+  fi
+
+  read -rp "Continue? [y/N]? " choice
+
+  case "$choice" in
+    y|Y ) return ;;
+    * ) exit;;
+  esac
+}
+
+function show_banner {
+  echo ""
+  echo "   ____                           ____ _                 _ "
+  echo "  / ___|___         ___  _ __    / ___| | ___  _   _  __| |"
+  echo " | |   / _ \ _____ / _ \| '_ \  | |   | |/ _ \| | | |/ _' |"
+  echo " | |__| (_) |_____| (_) | |_) | | |___| | (_) | |_| | (_| |"
+  echo "  \____\___/       \___/| .__/   \____|_|\___/ \__,_|\__,_|"
+  echo "                        |_|"
+  echo ""
+}
+
+function install_docker {
+  sudo apt-get remove docker docker-engine docker.io containerd runc
+  sudo apt-get install -yq \
+    apt-transport-https \
+    ca-certificates \
+    gnupg \
+    lsb-release
+  curl -fsSL https://download.docker.com/linux/debian/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
+  echo \
+  "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian \
+  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
+  sudo apt-get update
+  sudo apt-get install -yq docker-ce docker-ce-cli containerd.io
+}
+
+function install_requirements {
+  if [ -f "/etc/debian_version" ]; then
+    echo "Detected Debian based distribution, attempting to install system requirements..."
+
+    sudo apt update && sudo apt install -y \
+      passwdqc \
+      pwgen
+
+    echo "Install Docker (https://docs.docker.com/engine/install/debian/)?"
+    prompt_confirm
+    install_docker
+  else
+    echo "Sorry, we only support Debian based distributions at the moment"
+    echo "You'll have to install the requirements manually for your distribution"
+    echo "See https://git.autonomic.zone/coop-cloud/abra#requirements for more"
+  fi
+}
+
+function install_abra_release {
+  mkdir -p "$HOME/.local/bin"
+  curl "$ABRA_SRC" > "$HOME/.local/bin/abra"
+  chmod +x "$HOME/.local/bin/abra"
+  echo "abra installed to $HOME/.local/bin/abra"
+}
+
+function install_abra_dev {
+  mkdir -p "$ABRA_DIR/"
+  if [[ ! -d "$ABRA_DIR/src" ]]; then
+    git clone "$GIT_URL" "$ABRA_DIR/src"
+  fi
+  (cd "$ABRA_DIR/src" && git pull origin main && cd - || exit)
+  mkdir -p "$HOME/.local/bin"
+  ln -sf "$ABRA_DIR/src/abra" "$HOME/.local/bin/abra"
+  echo "abra installed to $HOME/.local/bin/abra (development bleeding edge)"
+}
+
+function run_installation {
+  show_banner
+
+  DOCOPT_PREFIX=installer_
+  DOCOPT_ADD_HELP=false
+  eval "$(docopt "$@")"
+
+  dev="$installer___dev"
+  no_prompt="$installer___no_prompt"
+  no_deps="$installer___no_deps"
+
+  if [ "$no_deps" == "false" ]; then
+    install_requirements
+  fi
+
+  if ! type curl > /dev/null 2>&1; then
+    error "'curl' program is not installed, cannot continue..."
+  fi
+
+  if [ "$dev" == "true" ]; then
+    install_abra_dev
+  else
+    install_abra_release
+  fi
+}
+
+run_installation "$@"
+exit 0
--- a/deploy/install.abra.coopcloud.tech/nginx.conf
+++ b/deploy/install.abra.coopcloud.tech/nginx.conf
@ -1,6 +1,6 @@
 server {
  listen         80 default_server;
-  server_name    install.abra.autonomic.zone;
+  server_name    install.abra.autonomic.zone install.abra.coopcloud.tech;

  location / {
    root /var/www/abra-installer;
--- a/45
+++ b/45
@ -1,4 +1,4 @@
-.PHONY: test shellcheck docopt kcov codecov release-installer
+.PHONY: test shellcheck docopt release-installer build push deploy-docopt symlink

 test:
 	@sudo DOCKER_CONTEXT=default docker run \
@ -7,7 +7,7 @@ test:
 		-d \
 		--name=abra-test-dind \
 		-e DOCKER_TLS_CERTDIR="" \
-		decentral1se/docker-dind-bats-kcov
+		decentral1se/docker-dind-bats-kcov \
 	@DOCKER_CONTEXT=default sudo docker exec \
 		-it \
 		abra-test-dind \
@ -21,8 +21,9 @@ shellcheck:
 		--rm \
 		-v $$(pwd):/workdir \
 		koalaman/shellcheck-alpine \
-		shellcheck /workdir/abra && \
-		shellcheck /workdir/bin/*.sh
+		sh -c "shellcheck /workdir/abra && \
+		       shellcheck /workdir/bin/*.sh && \
+		       shellcheck /workdir/deploy/install.abra.coopcloud.tech/installer"

 docopt:
 	@if [ ! -d ".venv" ]; then \
@ -32,27 +33,27 @@ docopt:
 	fi
 	.venv/bin/docopt.sh abra

-kcov:
-	@docker run \
-		-it \
-		--rm \
-		-v $$(pwd):/workdir \
-		kcov/kcov:latest \
-		sh -c "kcov /workdir/coverage /workdir/abra || true"
-
-codecov: SHELL:=/bin/bash
-codecov:
-	@bash <(curl -s https://codecov.io/bash) \
-		-s coverage -t $$(pass show hosts/swarm.autonomic.zone/drone/codecov/token)
+deploy-docopt:
+	@if [ ! -d ".venv" ]; then \
+		python3 -m venv .venv && \
+		.venv/bin/pip install -U pip setuptools wheel && \
+		.venv/bin/pip install docopt-sh; \
+	fi
+	.venv/bin/docopt.sh deploy/install.abra.coopcloud.tech/installer

 release-installer:
 	@DOCKER_CONTEXT=swarm.autonomic.zone \
 		docker stack rm abra-installer-script && \
-		cd deploy/install.abra.autonomic.zone && \
+		cd deploy/install.abra.coopcloud.tech && \
 		DOCKER_CONTEXT=swarm.autonomic.zone docker stack deploy -c compose.yml abra-installer-script

-release-apps:
-	@DOCKER_CONTEXT=swarm.autonomic.zone \
-		docker stack rm abra-apps-json && \
-		cd deploy/apps.coopcloud.tech && \
-		DOCKER_CONTEXT=swarm.autonomic.zone docker stack deploy -c compose.yml abra-apps-json
+build:
+	@docker build -t thecoopcloud/abra .
+
+push: build
+	@docker push thecoopcloud/abra
+
+symlink:
+	@mkdir -p ~/.abra/servers/ && \
+	ln -srf tests/default ~/.abra/servers && \
+	ln -srf tests/apps/* ~/.abra/apps
--- a/tests/apps/works/compose.yml
+++ b/tests/apps/works/compose.yml
@ -0,0 +1,84 @@
+---
+
+# The goal of this compose file is to have a testing ground for understanding
+# what cases we need to handle to get stable deployments. For that, we need to
+# work with healthchecks and deploy configurations quite closely. If you run
+# the `make symlink` target then this will be loaded into a "fake" app on your
+# local machine which you can deploy with `abra`.
+
+version: "3.8"
+services:
+  r1_should_work:
+    image: redis:alpine
+    deploy:
+      update_config:
+        failure_action: rollback
+        order: start-first
+      rollback_config:
+        order: start-first
+      restart_policy:
+        max_attempts: 1
+    healthcheck:
+      test: redis-cli ping
+      interval: 2s
+      retries: 3
+      start_period: 1s
+      timeout: 3s
+
+  r2_broken_health_check:
+    image: redis:alpine
+    deploy:
+      update_config:
+        failure_action: rollback
+        order: start-first
+      rollback_config:
+        order: start-first
+      restart_policy:
+        max_attempts: 3
+    healthcheck:
+      test: foobar
+      interval: 2s
+      retries: 3
+      start_period: 1s
+      timeout: 3s
+
+  r3_no_health_check:
+    image: redis:alpine
+    deploy:
+      update_config:
+        failure_action: rollback
+        order: start-first
+      rollback_config:
+        order: start-first
+      restart_policy:
+        max_attempts: 3
+
+  r4_disabled_health_check:
+    image: redis:alpine
+    deploy:
+      update_config:
+        failure_action: rollback
+        order: start-first
+      rollback_config:
+        order: start-first
+      restart_policy:
+        max_attempts: 3
+    healthcheck:
+      disable: true
+
+  r5_should_also_work:
+    image: redis:alpine
+    deploy:
+      update_config:
+        failure_action: rollback
+        order: start-first
+      rollback_config:
+        order: start-first
+      restart_policy:
+        max_attempts: 1
+    healthcheck:
+      test: redis-cli ping
+      interval: 2s
+      retries: 3
+      start_period: 1s
+      timeout: 3s
--- a/tests/default/works.env
+++ b/tests/default/works.env
@ -0,0 +1 @@
+TYPE=works
Author	SHA1	Message	Date
decentral1se	0581559783	Bump to the next version	2021-07-06 12:34:15 +02:00
decentral1se	cdd196346a	Add label generation checking Closes https://git.autonomic.zone/coop-cloud/abra/issues/186.	2021-07-06 12:27:06 +02:00
decentral1se	78287fec37	Add change log entry	2021-07-06 11:51:03 +02:00
decentral1se	4b820457de	Output diff before committing changes See https://git.autonomic.zone/coop-cloud/abra/issues/174.	2021-07-06 11:50:09 +02:00
decentral1se	8519cb8661	Appease shellcheck and revert to original quote handling	2021-07-06 11:18:50 +02:00
decentral1se	3c30d3621b	Support restarting a service Closes https://git.autonomic.zone/coop-cloud/abra/issues/200.	2021-07-06 11:12:49 +02:00
decentral1se	76a0badc5a	Add new change log entries	2021-07-06 10:53:31 +02:00
decentral1se	b2e0a95a11	Override ARGS and fail correctly See https://github.com/Coop-Cloud/peertube/issues/1.	2021-07-06 10:53:08 +02:00
decentral1se	ace854e1d7	Don't describe the what here	2021-07-06 10:40:09 +02:00
decentral1se	bd7688f9e7	Do not install deps on CLI upgrade [ci skip]	2021-07-06 00:13:22 +02:00
decentral1se	d7a4c2cebe	This has to be already installed [ci skip]	2021-07-06 00:12:02 +02:00
decentral1se	d9a0922b2c	Add upgrade note [ci skip]	2021-07-06 00:09:25 +02:00
decentral1se	32a86e0317	Add change log entry	2021-07-06 00:06:39 +02:00
decentral1se	b1c5391a91	Use latest yq	2021-07-06 00:06:07 +02:00
decentral1se	b813f6b90e	Drop additional check This forces the `require yq` to only happen at the start of the top-level functions which makes more sense and is easier to manage. Closes https://git.autonomic.zone/coop-cloud/abra/issues/183.	2021-07-06 00:04:19 +02:00
decentral1se	73de76fc04	Remove unused function	2021-07-06 00:04:14 +02:00
decentral1se	5c5cbbf20f	Add ASCII radness [ci skip]	2021-07-06 00:01:32 +02:00
decentral1se	19498d9494	Fix typo [ci skip]	2021-07-05 23:58:39 +02:00
decentral1se	6f6a9ab413	Update change log entry	2021-07-05 23:56:37 +02:00
decentral1se	aa81d26d08	Use pwgen/pwqgen if installed Closes https://git.autonomic.zone/coop-cloud/abra/issues/197.	2021-07-05 23:55:23 +02:00
decentral1se	cc4efe69bf	Merge pull request 'Install requirements via install script' (#198 ) from requirements-install-script into main Reviewed-on: https://git.autonomic.zone/coop-cloud/abra/pulls/198	2021-07-05 23:49:26 +02:00
decentral1se	6c7b53f585	Install requirements via install script Closes https://git.autonomic.zone/coop-cloud/abra/issues/196.	2021-07-05 23:48:03 +02:00
decentral1se	32bf28e7a9	Follow our usual convention of writing it now	2021-07-05 23:41:45 +02:00
decentral1se	624815e5b1	Use global ignore and avoid this hack	2021-07-05 23:35:48 +02:00
decentral1se	e9fb9e56ad	Drop pwgen/pwqgen requirements Closes https://git.autonomic.zone/coop-cloud/abra/issues/167.	2021-07-04 23:06:20 +02:00
decentral1se	283eb21e29	Add log entry	2021-07-04 22:42:44 +02:00
decentral1se	92f49d56dd	Use case insensitive awk/sed Closes https://git.autonomic.zone/coop-cloud/abra/issues/170.	2021-07-04 22:41:48 +02:00
decentral1se	d9ff48b55b	Add change log entry	2021-07-04 22:14:42 +02:00
decentral1se	3d8ce3492e	Reflect chaos deploy when selecting recipe version Closes https://git.autonomic.zone/coop-cloud/abra/issues/185.	2021-07-04 22:13:57 +02:00
decentral1se	07696760b7	Output all elements when debugging	2021-07-04 21:59:47 +02:00
decentral1se	43b4a01f8a	Make logging reflect reality + do more debugging See https://git.autonomic.zone/coop-cloud/abra/issues/193.	2021-07-04 21:51:55 +02:00
decentral1se	bb3b324e07	Add change log entry	2021-07-04 21:45:02 +02:00
decentral1se	eb9d1b883b	Ignore this warning for now	2021-07-04 21:44:13 +02:00
3wordchant	6f6140ced2	Merge pull request 'Don't generate commented out secrets. Throw an error when can't put the secret in docker' (#195 ) from knoflook/abra:main into main Reviewed-on: https://git.autonomic.zone/coop-cloud/abra/pulls/195	2021-07-04 21:18:13 +02:00
knoflook	cb225908d0	Don't generate commented out secrets. Throw an error when can't put the secret in docker	2021-07-03 19:43:42 +02:00
decentral1se	f2892bad6f	Fix that sentence [ci skip]	2021-06-27 21:14:19 +02:00
decentral1se	480b1453ec	Add change log entry [ci skip]	2021-06-27 21:13:21 +02:00
decentral1se	0ab2b3a652	Merge pull request 'Make ensure_stack_deployed more reliable' (#177 ) from improved-stack-deploy-guarantees into main Reviewed-on: https://git.autonomic.zone/coop-cloud/abra/pulls/177	2021-06-27 21:03:48 +02:00
decentral1se	93714a593b	`ensure_stack_deployed` is now somewhat more reliable Closes https://git.autonomic.zone/coop-cloud/abra/issues/165.	2021-06-27 21:03:24 +02:00
decentral1se	57f3f96bbc	Use new name	2021-06-17 21:48:55 +02:00
decentral1se	e1959506c7	Add change log entry [ci skip]	2021-06-17 21:42:33 +02:00
decentral1se	7482362af1	Support logging in via Skopeo See https://git.autonomic.zone/coop-cloud/auto-apps-json/issues/1.	2021-06-17 21:40:58 +02:00
decentral1se	e8510c8aeb	Add change log and --output for app-json.py Closes https://git.autonomic.zone/coop-cloud/auto-apps-json/issues/2.	2021-06-17 21:25:29 +02:00
decentral1se	4042e10985	Use new image	2021-06-17 21:00:43 +02:00
decentral1se	f7cd0eb54c	Use new name	2021-06-17 16:34:03 +02:00
decentral1se	a571b839a8	Use proper jq path Closes https://git.autonomic.zone/coop-cloud/abra/issues/184.	2021-06-17 09:34:36 +02:00
decentral1se	fae13d9af8	Define our own repos to skip here for mirroring	2021-06-17 07:54:54 +02:00
decentral1se	9c9f7225e7	Use new user/org for mirroring	2021-06-17 07:43:00 +02:00
3wc	352cc0939b	Fix typo in missing version error message	2021-06-13 20:48:20 +02:00
decentral1se	2ca7884bbe	Fix bump releases Closes https://git.autonomic.zone/coop-cloud/abra/issues/180.	2021-06-11 00:36:40 +02:00
decentral1se	fa54705f79	Merge pull request 'Prefer --fast for skipping all checks' (#175 ) from prefer-fast-option into main Reviewed-on: https://git.autonomic.zone/coop-cloud/abra/pulls/175	2021-06-11 00:29:20 +02:00
decentral1se	8d802c78aa	Prefer --fast for skipping all checks Closes https://git.autonomic.zone/coop-cloud/abra/issues/169	2021-06-11 00:27:50 +02:00
decentral1se	1c022fb616	Merge pull request 'Add --bump release logic' (#176 ) from add-bump-logic into main Reviewed-on: https://git.autonomic.zone/coop-cloud/abra/pulls/176	2021-06-10 12:19:35 +02:00
decentral1se	0655c03434	Add --bump release logic Closes https://git.autonomic.zone/coop-cloud/abra/issues/173.	2021-06-10 12:18:27 +02:00
decentral1se	f6bdf596f5	Bump to next version	2021-06-10 11:43:00 +02:00
decentral1se	6c6e6808c9	Merge pull request 'Add --chaos flag' (#179 ) from chaos-deploy-flag into main Reviewed-on: https://git.autonomic.zone/coop-cloud/abra/pulls/179	2021-06-10 11:41:26 +02:00
decentral1se	a3ffd7f239	Add --chaos flag Closes https://git.autonomic.zone/coop-cloud/abra/issues/178.	2021-06-10 11:40:25 +02:00
decentral1se	a019417fd2	Avoid PR publishing	2021-06-08 12:27:50 +02:00
decentral1se	743600b94e	Use access token for pushing	2021-06-05 22:54:16 +02:00
decentral1se	f4c2da894b	Add change log entry [ci skip]	2021-06-05 22:43:50 +02:00
decentral1se	4ef433312d	Add mirroring script	2021-06-05 22:41:50 +02:00
decentral1se	9f69532dca	Use vendored JQ	2021-06-05 08:57:35 +02:00
decentral1se	3a97358f30	Vendor later versions of jq/yq	2021-06-05 08:51:10 +02:00
decentral1se	1e19805757	Drop trailing slash	2021-06-05 08:51:02 +02:00
decentral1se	389ad9d049	Drop non-existant flag	2021-06-05 08:41:14 +02:00
decentral1se	93ffc633f3	Prepare more things for this image	2021-06-05 08:39:13 +02:00
decentral1se	b61c9410a0	Drop the apps.json in the cwd	2021-06-05 08:26:43 +02:00
decentral1se	bbab900ebc	Move apps.json generation stuff out of abra See https://git.autonomic.zone/coop-cloud/abra/issues/125.	2021-06-05 08:22:01 +02:00
decentral1se	36d4dbc5cf	Ignore that new repo	2021-06-05 08:15:11 +02:00
decentral1se	a4ade1463f	Copy over app-json script	2021-06-05 08:05:40 +02:00
decentral1se	20af4666c6	Remove and ignore pycache folders	2021-06-05 08:05:29 +02:00
decentral1se	d15b031f33	Ignore those pyc files	2021-06-05 08:04:58 +02:00
decentral1se	a7f0bbde62	Add openssh machinery	2021-06-05 08:03:42 +02:00
decentral1se	76d5a1026a	Support HTTPS/SSH cloning	2021-06-05 07:58:24 +02:00
decentral1se	7b0fb50e7f	Abstract common functions into a library	2021-06-05 07:55:05 +02:00
decentral1se	f92364af80	Run downstream builds See https://git.autonomic.zone/coop-cloud/abra/issues/171.	2021-06-05 07:25:41 +02:00
decentral1se	ca2a3c8b58	Add notify failures	2021-06-03 23:31:31 +02:00
decentral1se	a5c5526948	Add log entry [ci skip]	2021-06-03 23:15:08 +02:00
decentral1se	d16eb0e309	Drop force and keep going on non-interctive git stuff	2021-06-03 22:12:47 +02:00
decentral1se	3cff8aaada	Better grep and apps folder	2021-06-03 21:52:51 +02:00
decentral1se	4ff4c83154	Use dev all the time	2021-06-03 21:04:58 +02:00
decentral1se	f953743a7c	Let the plugin do tagging for us as well	2021-06-03 21:01:39 +02:00
decentral1se	e84062e67c	Use success instead (helpful for automation)	2021-06-03 20:58:38 +02:00
decentral1se	e573794367	Skip those repos too	2021-06-03 11:45:30 +02:00
decentral1se	87f9c16db4	Add log entry [ci skip]	2021-06-03 11:41:47 +02:00
decentral1se	9fadc430a7	Add renovate script	2021-06-03 11:40:55 +02:00
decentral1se	53cec2469b	Handle forcing re-upload	2021-06-03 10:34:19 +02:00
decentral1se	a1de7f10cb	Don't edit git stuff when running non-interactively	2021-06-03 10:17:09 +02:00
decentral1se	ece968478d	Add log entry	2021-06-03 10:07:23 +02:00
decentral1se	3759bcd641	Support unattended mode for recipe releasing	2021-06-03 10:06:40 +02:00
decentral1se	0ff08b5d34	Add missing dep and make special place in docs	2021-06-03 09:58:28 +02:00
decentral1se	8b541623ad	Add change log entry [ci skip]	2021-06-03 09:55:35 +02:00
decentral1se	f24259dbfc	Sort on lines [ci skip]	2021-06-03 09:54:32 +02:00
decentral1se	40259f5e97	Add git also	2021-06-03 09:53:56 +02:00
decentral1se	fd471eb3f1	Install dependencies	2021-06-03 09:52:14 +02:00
decentral1se	a4633f06bd	Add note about container [ci skip]	2021-06-03 09:47:46 +02:00
decentral1se	0d6031fef9	Also depend on tests [ci skip]	2021-06-03 09:44:47 +02:00
decentral1se	64d578cf91	Add docker image publishing	2021-06-03 09:43:44 +02:00
decentral1se	e216fe290b	Actually use that image as it is required	2021-06-03 09:11:50 +02:00
decentral1se	207278af75	Use same language	2021-06-03 09:09:44 +02:00
decentral1se	ff309182ea	Drop kcov/codecov for now, use upstream bats	2021-06-03 09:07:55 +02:00
decentral1se	542cf793d2	Remove app which is gone away now	2021-06-01 00:08:21 +02:00
decentral1se	ad1fe2b8d7	Bump new patch release	2021-05-31 23:59:38 +02:00
decentral1se	0771d58b69	Fix typo and add help fix commit	2021-05-31 23:58:25 +02:00
decentral1se	b24cdce354	Add --no-state-poll See https://git.autonomic.zone/coop-cloud/abra/issues/165.	2021-05-31 23:23:14 +02:00
decentral1se	499cc46583	Migrate `abra` installer to coopcloud.tech domain Closes https://git.autonomic.zone/coop-cloud/abra/issues/150.	2021-05-31 21:10:51 +02:00
decentral1se	0af0ea096f	Add change log entry	2021-05-31 21:01:21 +02:00
decentral1se	925df196fc	Only ouput secrets warning once Closes https://git.autonomic.zone/coop-cloud/abra/issues/143.	2021-05-31 21:00:40 +02:00
3wc	efad71c470	Fix help for `.. app .. volume ls`	2021-05-31 12:16:27 +02:00