Front-line that deprecation notice

Add deprecation notice
See coop-cloud/organising#107.
2021-08-02 11:32:52 +02:00 · 2021-08-02 11:29:33 +02:00 · 2021-07-30 19:33:29 +02:00 · 2021-07-30 17:01:07 +00:00 · 2021-07-30 16:48:29 +00:00 · 2021-07-14 10:16:05 +02:00
25 changed files with 4348 additions and 281 deletions
--- a/.drone.yml
+++ b/.drone.yml
@ -3,11 +3,86 @@ kind: pipeline
 name: linters
 steps:
  - name: run shellcheck
+    image: koalaman/shellcheck-alpine
+    commands:
+      - shellcheck abra
+      - shellcheck bin/*.sh
+      - shellcheck deploy/install.abra.coopcloud.tech/installer
+
+  - name: run flake8
+    image: alpine/flake8
+    commands:
+      - flake8 --max-line-length 100 bin/*.py
+
+  - name: run unit tests
+    image: decentral1se/docker-dind-bats-kcov
+    commands:
+      - bats tests
+
+  - name: test installation script
    image: debian:buster
    commands:
-      - apt update
-      - apt install -y shellcheck
-      - shellcheck abra installer
+      - apt update && apt install -yqq sudo lsb-release
+      - deploy/install.abra.coopcloud.tech/installer --no-prompt
+      - ~/.local/bin/abra version
+
+  - name: publish image
+    image: plugins/docker
+    settings:
+      auto_tag: true
+      username: thecoopcloud
+      password:
+        from_secret: thecoopcloud_password
+      repo: thecoopcloud/abra
+      tags: latest
+    depends_on:
+      - run shellcheck
+      - run flake8
+      - run unit tests
+      - test installation script
+    when:
+      event:
+        exclude:
+          - pull_request
+
+  - name: trigger downstream builds
+    image: plugins/downstream
+    settings:
+      server: https://drone.autonomic.zone
+      token:
+        from_secret: decentral1se_token
+      fork: true
+      repositories:
+        - coop-cloud/drone-abra
+    depends_on:
+      - run shellcheck
+      - run flake8
+      - run unit tests
+      - test installation script
+      - publish image
+    when:
+      event:
+        exclude:
+          - pull_request
+
+  - name: notify on failure
+    image: plugins/matrix
+    settings:
+      homeserver: https://matrix.autonomic.zone
+      roomid: "IFazIpLtxiScqbHqoa:autonomic.zone"
+      userid: "@autono-bot:autonomic.zone"
+      accesstoken:
+        from_secret: autono_bot_access_token
+    depends_on:
+      - run shellcheck
+      - run flake8
+      - run unit tests
+      - test installation script
+      - publish image
+      - trigger downstream builds
+    when:
+      status:
+        - failure
 trigger:
  branch:
    - main
--- a/.gitignore
+++ b/.gitignore
@ -0,0 +1,5 @@
+*.json
+*.pyc
+/.venv
+__pycache__
+coverage/
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@ -1,3 +1,187 @@
+> 🔥 🔥 🔥 Please note, while we are still in
+> [public alpha](https://docs.cloud.autonomic.zone/roadmap/), the `abra` release
+> versioning scheme is not following [semver](https://semver.org/) conventions
+> because we are still in the exploratory phases of building this tool. Please
+> read the changes before upgrading your `abra` installation as there are
+> **most likely** breaking changes coming each release. Sorry for any
+> inconvenience caused, we're working hard to make this tool stable. Semver
+> will be respected when we reach public beta. 🔥 🔥 🔥
+
+# abra x.x.x (UNRELEASED)
+
+# abra 10.0.1 (2021-07-30)
+
+- New `recipe .. lint` command ([#202](https://git.autonomic.zone/coop-cloud/abra/issues/202))
+- `abra-capsul` plugin ([e85bcc4](https://git.autonomic.zone/coop-cloud/abra/commit/e85bcc4))
+- Fix `run <service> <cmd>` invocations ([#204](https://git.autonomic.zone/coop-cloud/abra/issues/204))
+- Point installer at our [half-migrated new Gitea instance](https://git.coopcloud.tech) ([#207](https://git.coopcloud.tech/coop-cloud/abra/pulls/207) thanks @nicksellen!)
+
+# abra 10.0.0 (2021-07-06)
+
+- Add `--bump` to `deploy` command to allow packagers to make minor package related releases ([#173](https://git.autonomic.zone/coop-cloud/abra/issues/173))
+- Drop `--skip-version-check`/`--no-domain-poll`/`--no-state-poll` in favour of `--fast` ([#169](https://git.autonomic.zone/coop-cloud/abra/issues/169))
+- Move `abra` image under the new `thecoopcloud/...` namespace ([#1](https://git.autonomic.zone/coop-cloud/auto-apps-json/issues/1))
+- Add a `--output` flag to the `app-json.py` app generator for the CI environment ([#2](https://git.autonomic.zone/coop-cloud/auto-apps-json/issues/2))
+- Support logging in as new `thecoopcloud` Docker account via `skopeo` when generating new `apps.json` ([`7482362`](https://git.autonomic.zone/coop-cloud/abra/commit/7482362af1d01cc02828abd45b1222fa643d1f80))
+- App deployment checks are somewhat more reliable (see [#193](https://git.autonomic.zone/coop-cloud/abra/issues/193) for remaining work) ([#165](https://git.autonomic.zone/coop-cloud/abra/issues/165))
+- Skip generation of commented out secrets and correctly fail deploy when secret generation fails ([#133](https://git.autonomic.zone/coop-cloud/abra/issues/133))
+- Fix logging for chaos deploys and recipe selection logic ([#185](https://git.autonomic.zone/coop-cloud/abra/issues/185))
+- Improve reliability of selecting when to download a new `apps.json` ([#170](https://git.autonomic.zone/coop-cloud/abra/issues/170))
+- Implement `pwgen`/`pwqgen` native fallback for password generation ([#167](https://git.autonomic.zone/coop-cloud/abra/issues/167) / [#197](https://git.autonomic.zone/coop-cloud/abra/issues/197))
+- `abra` installer script will now try to install system requirements ([#196](https://git.autonomic.zone/coop-cloud/abra/issues/196))
+- Use latest [v4.9.6](https://github.com/mikefarah/yq/releases/tag/v4.9.6) install of `yq` for vendoring (**upgrade HOWTO:** `rm -rf ~/.abra/vendor/*`)
+- Support overriding `$ARGS` from `abra.sh` custom commands and error out correctly when these commands fail ([#1](https://github.com/Coop-Cloud/peertube/issues/1))
+- Add `abra <app> restart <service>` to support restarting individual services ([#200](https://git.autonomic.zone/coop-cloud/abra/issues/200))
+- Output diff of proposed changes when asking to commit during release logic ([`4b82045`](https://git.autonomic.zone/coop-cloud/abra/commit/4b820457defe1511208b6caa8b9feb9603ffc8be))
+- Add validation for generated output when making new release labels ([#186](https://git.autonomic.zone/coop-cloud/abra/issues/186))
+
+# abra 9.0.0 (2021-06-10)
+
+- Add Docker image for `abra` ([`64d578cf`](https://git.autonomic.zone/coop-cloud/abra/commit/64d578cf914bd2bad378ea4ef375747d10b33191))
+- Support unattended mode for recipe releasing ([`3759bcd6`](https://git.autonomic.zone/coop-cloud/abra/commit/3759bcd641cf60611c13927e83425e773d2bb629))
+- Add Renovate bot configuraiton script ([`9fadc430`](https://git.autonomic.zone/coop-cloud/abra/commit/9fadc430a7bb2d554c0ee26c0f9b6c51dc5b0475))
+- Add release automation via [drone-abra](https://git.autonomic.zone/coop-cloud/drone-abra) ([#56](https://git.autonomic.zone/coop-cloud/organising/issues/56))
+- Move `apps.json` generation to [auto-apps-json](https://git.autonomic.zone/coop-cloud/auto-apps-json) ([#125](https://git.autonomic.zone/coop-cloud/abra/issues/125))
+- Add Github mirroring script ([`4ef43331`](https://git.autonomic.zone/coop-cloud/abra/commit/4ef433312dd0b0ace91b3c285f82f3973093d92d))
+- Add `--chaos` flag to deploy (always choose latest Git commit) ([#178](https://git.autonomic.zone/coop-cloud/abra/issues/178))
+
+# abra 8.0.1 (2021-05-31)
+
+- Fix help for `... app ... volume ls` ([`efad71c4`](https://git.autonomic.zone/coop-cloud/abra/commit/efad71c470d6d65f7e4bfe39c5f68ff1097f80a2))
+- Only output secrets warnings once ([#143](https://git.autonomic.zone/coop-cloud/abra/issues/143))
+- Migrate `abra` installation script to `coopcloud.tech` domain ([#150](https://git.autonomic.zone/coop-cloud/abra/issues/150))
+- Add `--no-state-poll` to avoid success/failure forecasting on deployment ([#165](https://git.autonomic.zone/coop-cloud/abra/issues/165))
+
+# abra 8.0.0 (2021-05-30)
+
+- Fix secret length generation ([`f537417`](https://git.autonomic.zone/coop-cloud/abra/commit/1b85bf3d37280e9632c315d759c0f2d09c039fef))
+- Fix checking out new apps ([#164](https://git.autonomic.zone/coop-cloud/abra/issues/164)
+- Give up if YAML is invalid ([#154](https://git.autonomic.zone/coop-cloud/abra/issues/154))
+- Switch from wget to cURL ([`fc0caaa`](https://git.autonomic.zone/coop-cloud/abra/commit/fc0caaa))
+- Add Bash completion for `recipe ..` ([`8c93d1a`](https://git.autonomic.zone/coop-cloud/abra/commit/8c93d1a))
+- Tweak README parsing in `app-json.py` ([`b14219b`](https://git.autonomic.zone/coop-cloud/abra/commit/b14219b))
+- Add fallback names to `app.json` ([#157](https://git.autonomic.zone/coop-cloud/abra/issues/157))
+- Remove duplicate message ([#155](https://git.autonomic.zone/coop-cloud/abra/issues/155))
+- Add `deploy --fast` ([`a7f7c96`](https://git.autonomic.zone/coop-cloud/abra/commit/a7f7c96))
+- Add `app .. volume` commands, fix volume deletion with `app .. delete --volumes` ([#161](https://git.autonomic.zone/coop-cloud/abra/issues/161))
+
+# abra 0.7.4 (2021-05-10)
+
+- Sort `apps.json` when publishing ([`39a7fc0`](https://git.autonomic.zone/coop-cloud/abra/commit/39a7fc04fb5df1a6d78b84f51838530ab3eb76db))
+- Fix publishing of rating for new apps ([`0e28af9`](https://git.autonomic.zone/coop-cloud/abra/commit/0e28af9eb1af6c6da705b4614ddd173c60576629))
+- Detect compose filenames in `n+1` release generation ([`ffc569e`](https://git.autonomic.zone/coop-cloud/abra/commit/ffc569e275df7ca784a4db1a3331e17975fd8c87))
+- Fix secret generation when specifying length ([`3a353f4`](https://git.autonomic.zone/coop-cloud/abra/commit/3a353f4062baccde2c9f175b03afb2db6d462ae4))
+
+# abra 0.7.3 (2021-04-28)
+
+- Only check for pw(q)gen if we're actually trying to use them ([#147](https://git.autonomic.zone/coop-cloud/abra/issues/147))
+- Use apps.coopcloud.tech for app data hosting & download ([`75bd599`](https://git.autonomic.zone/coop-cloud/abra/commit/75bd599))
+- Choose latest commit messages for new tags ([#144](https://git.autonomic.zone/coop-cloud/abra/issues/144))
+- Handle recipes without an `app` service in `recipe .. release` ([#151](https://git.autonomic.zone/coop-cloud/abra/issues/151))
+
+# abra 0.7.2 (2021-04-07)
+
+- Fix installation script development installs (again! Thanks Bash!) ([`4747d9b7`](https://git.autonomic.zone/coop-cloud/abra/commit/4747d9b7fb5fba914f210b6570bfe2db0b53da23))
+
+# abra 0.7.1 (2021-04-07)
+
+- Fix installation script development installs ([`8f2fadb3c`](https://git.autonomic.zone/coop-cloud/abra/commit/8f2fadb3c43c5915520f5ea531ea3815c2ba8531))
+
+# abra 0.7.0 (2021-04-07)
+
+- Add `--force` to the `deploy` command to allow overriding deployment logic ([#105](https://git.autonomic.zone/coop-cloud/abra/issues/105))
+- Handle undeployed apps in version summaries when deploying ([#104](https://git.autonomic.zone/coop-cloud/abra/issues/104))
+- Add `--force` to `undeploy` command ([`e5e98d5`](https://git.autonomic.zone/coop-cloud/abra/commit/e5e98d5))
+- Rename "app type" back to "stack" in the deployment overview ([`54b6acc`](https://git.autonomic.zone/coop-cloud/abra/commit/54b6acc))
+- Show context connection details on `abra server ls` ([#110](https://git.autonomic.zone/coop-cloud/abra/issues/110))
+- Allow to debug the SSH connection details on swarm init ([#109](https://git.autonomic.zone/coop-cloud/abra/issues/109))
+- Show correct status for apps deployed on servers with missing context ([#99](https://git.autonomic.zone/coop-cloud/abra/issues/99))
+- Search for subcommands in descending order of how many components there are ([#108](https://git.autonomic.zone/coop-cloud/abra/issues/108))
+- Add specific app version checking command (`abra app <app> version`) ([#108](https://git.autonomic.zone/coop-cloud/abra/issues/108))
+- Add docker version check (guestimating < v19 is a bad idea) ([#15](https://git.autonomic.zone/coop-cloud/abra/issues/15))
+- Fix git branch handling when not passing `-b <branch>` ([#122](https://git.autonomic.zone/coop-cloud/abra/issues/122))
+- Add work-around to correctly git clone non-master default branch app repositories ([#122](https://git.autonomic.zone/coop-cloud/abra/issues/122))
+- Replace `--force` (except for the `deploy` command) with a global `--no-prompt` for avoiding interactive questions ([#118](https://git.autonomic.zone/coop-cloud/abra/issues/118))
+- Use [docker-stack-wait-deploy](https://github.com/vitalets/docker-stack-wait-deploy) inspired logic to deploy apps ([#116](https://git.autonomic.zone/coop-cloud/abra/issues/116))
+- Add a domain polling check when deploying apps ([#113](https://git.autonomic.zone/coop-cloud/abra/issues/113))
+- Recognise when apps are already undeployed with `abra app <app> undeploy` ([#123](https://git.autonomic.zone/coop-cloud/abra/issues/123))
+- Add `abra doctor` command to help diagnose setup issues ([#119](https://git.autonomic.zone/coop-cloud/abra/issues/119))
+- Add apps version and feature catalogue generation script ([#121](https://git.autonomic.zone/coop-cloud/abra/issues/121))
+- New `--skip-version-check` option to `deploy` ([`df4e504`](https://git.autonomic.zone/coop-cloud/abra/commit/df4e504))
+- Look up local available version from compose files instead of `abra.sh` ([#131](https://git.autonomic.zone/coop-cloud/abra/issues/131))
+- Improve domain polling logging and allow to skip the check altogether with `--no-domain-poll` ([#140](https://git.autonomic.zone/coop-cloud/abra/issues/140), [#141](https://git.autonomic.zone/coop-cloud/abra/issues/141))
+- Support `ABRA_DIR` in the installer script ([`4e94a424e`](https://git.autonomic.zone/coop-cloud/abra/commit/4e94a424e94a42))
+- Support [abra-hetzner](https://git.autonomic.zone/coop-cloud/abra-hetzner) plugin ([#88](https://git.autonomic.zone/coop-cloud/abra/issues/88))
+
+# abra 0.6.0 (2021-03-17)
+
+- Show version and digest of app if labelled ([`98e674b8e`](https://git.autonomic.zone/coop-cloud/abra/commit/98e674b8e83458a83dcbf331e8e34c7188559c4a))
+- Implement basic version checking on deployment ([#82](https://git.autonomic.zone/coop-cloud/abra/issues/82))
+- New `app-catalogue.sh` script to auto-generate app list for documentation ([`f163d4b0f`](https://git.autonomic.zone/coop-cloud/abra/commit/f163d4b0fa920232e9d995a22d20fe78b174b3a9))
+- Support app service rollbacks with `abra <app> rollback <service>` ([#76](https://git.autonomic.zone/coop-cloud/abra/issues/76))
+- Detect when latest version is deployed and perform a no-op ([#87](https://git.autonomic.zone/coop-cloud/abra/issues/87))
+- Allow cloning of app repos with different main branches using `-b, --branch=<branch>` ([#80](https://git.autonomic.zone/coop-cloud/abra/issues/80))
+- Protect against lengthy app names which gives Docker trouble later on ([#83](https://git.autonomic.zone/coop-cloud/abra/issues/83))
+- Support removal of secrets and volumes when `rm`'ing apps ([#44](https://git.autonomic.zone/coop-cloud/abra/issues/44))
+- Always choose the default IPv4 address with `abra server <host> init` ([#91](https://git.autonomic.zone/coop-cloud/abra/issues/91))
+- Add `--type=<type>` filtering option to `abra <app> ls` ([`0828189`](https://git.autonomic.zone/coop-cloud/abra/commit/0828189))
+- Check for bash 4+ ([#96](https://git.autonomic.zone/coop-cloud/abra/commit/0828189))
+- Add `--dev` option to installer using `git clone` ([`88d2a75`](https://git.autonomic.zone/coop-cloud/abra/commit/88d2a75))
+- Support `--dev` on the `abra upgrade` command also ([`bcc15ec`](https://git.autonomic.zone/coop-cloud/abra/commit/bcc15ec))
+- Vendor [yq](https://github.com/mikefarah/yq/releases) automatically ([`3b59adf`](https://git.autonomic.zone/coop-cloud/abra/commit/3b59adf))
+- Extend version handling logic to support all underlying services ([#90](https://git.autonomic.zone/coop-cloud/abra/issues/90))
+- Fix development installation script symlink issue ([#98](https://git.autonomic.zone/coop-cloud/abra/issues/98))
+- Add `app-version.sh` script to help packagers version apps ([`28618bd`](https://git.autonomic.zone/coop-cloud/abra/commit/28618bd))
+- Add git digest to `abra version` output ([`8b41416`](https://git.autonomic.zone/coop-cloud/abra/commit/8b41416))
+
+# abra 0.5.0 (2021-03-01)
+
+- `secret auto` merged into `secret generate` and `app new --auto` is now `app new --secrets` ([#64](https://git.autonomic.zone/coop-cloud/abra/pulls/64))
+- Avoid outputting length during secret generation when not in use ([#67](https://git.autonomic.zone/coop-cloud/abra/issues/67))
+- Support graceful failure when missing secret generation commands ([`44d3ac3`](https://git.autonomic.zone/coop-cloud/abra/commit/44d3ac3a1cb86edc9b9e91eea1a00e70eae14965))
+- Fix secret detection when using new `.env` file format in apps ([`5532452`](https://git.autonomic.zone/coop-cloud/abra/commit/55324524ca77141666ffe6cc41b62cc71cf89ace))
+- Support choosing an `$EDITOR` when editing configs ([`29cc392`](https://git.autonomic.zone/coop-cloud/abra/commit/29cc392dff3e93e48e0e2edd3ce11b405c66a95a))
+- "server" shell completion fixed ([`8839bd4`](https://git.autonomic.zone/coop-cloud/abra/commit/8839bd45951d00dccf4ef81ece445bcc49e13ee6))
+- Drop `multilogs` command ([#56](https://git.autonomic.zone/coop-cloud/abra/pulls/56))
+- Remove `server use` command ([#51](https://git.autonomic.zone/coop-cloud/abra/issues/51))
+- `new <app>` becomes `new <type>` ([#48](https://git.autonomic.zone/coop-cloud/abra/issues/48))
+- `check` is run on `deploy` now and configurable ([`77ba565`](https://git.autonomic.zone/coop-cloud/abra/commit/77ba5652b2fe15820f5edfa0f642636f7b8eae7e))
+- App configurations are always updated now ([#42](https://git.autonomic.zone/coop-cloud/abra/issues/42))
+- We use docker format `.env` files (no "export" syntax) from now now ([#55](https://git.autonomic.zone/coop-cloud/abra/pulls/55))
+- Rename `<domain>` option to `<app>` and `APP` variable to `TYPE`, see ([#47](https://git.autonomic.zone/coop-cloud/abra/issues/47))
+- Use Docker-in-Docker (dind), and `dind-bats-kcov` Docker image, for `make test` ([`1600b62`](https://git.autonomic.zone/coop-cloud/abra/commit/1600b6277fbbffc4c6de1e4ba799c7bbe72ec6a0))
+- Add built-in documentation using `abra help <subcommand>...`, see ([#50](https://git.autonomic.zone/coop-cloud/abra/issues/50))
+- `version` subcommand ([e6b24fe](https://git.autonomic.zone/coop-cloud/abra/commit/e6b24fe))
+- Use `# length=x` comments to generate passwords with `pwgen` and drop `KEY`/`PASSWORD` logic ([#68](https://git.autonomic.zone/coop-cloud/abra/issues/68))
+- Global `--skip-update|-U` / `--skip-check|-C` options to make things quicker ([`37e8b00`](https://git.autonomic.zone/coop-cloud/abra/commit/37e8b00))
+- `app backup` and `app restore` commands; requires per-app definition ([#70](https://git.autonomic.zone/coop-cloud/abra/issues/70))
+- Rename per-type `abra-commands.sh` to `abra.sh`, and include config versions as type-level instead of app-level config ([#43](https://git.autonomic.zone/coop-cloud/abra/issues/43))
+- Show per-subcommand help by adding `-h/--help` to a command line ([#38](https://git.autonomic.zone/coop-cloud/abra/issues/78))
+
+# abra 0.4.1 (2020-12-24)
+
+- Bug-fixes on `app ls --status` & custom commands
+- Add `app ls --server=...` and alias
+
+# abra 0.4.0 (2020-12-24)
+
+- New command-line interface based on docopt
+- `~/.abra` directory instead of expecting local `.env` files
+- Integration tests & code coverage
+
+# abra 0.3.1 (2020-09-27)
+
+- Fix installer version
+
+# abra 0.3.0 (2020-09-27)
+
+- Add multilogs stack logs implementation ([#8](https://git.autonomic.zone/compose-stacks/abra/issues/8))
+- Add beginnings of "monorepo" functionality
+
+# abra 0.2.0 (2020-09-24)
+
+- Prepare for swarm install script using script.d ([#12](https://git.autonomic.zone/compose-stacks/planning/issues/12))
+
 # abra 0.1.2 (2020-09-22)

 - Add upgrade command ([#10](https://git.autonomic.zone/autonomic-cooperative/abra/issues/10))
--- a/33
+++ b/33
@ -0,0 +1,33 @@
+FROM alpine:latest
+
+RUN apk add --upgrade --no-cache \
+  bash \
+  curl \
+  git \
+  grep \
+  openssh-client \
+  py3-requests \
+  skopeo \
+  util-linux
+
+RUN mkdir -p ~./local/bin
+RUN mkdir -p ~/.abra/apps
+RUN mkdir -p ~/.abra/vendor
+RUN mkdir -p ~/.ssh/
+
+RUN ssh-keyscan -p 2222 git.autonomic.zone > ~/.ssh/known_hosts
+
+RUN curl -L https://github.com/stedolan/jq/releases/download/jq-1.6/jq-linux64 --output ~/.abra/vendor/jq
+RUN chmod +x ~/.abra/vendor/jq
+
+RUN curl -L https://github.com/mikefarah/yq/releases/download/v4.9.3/yq_linux_amd64 --output ~/.abra/vendor/yq
+RUN chmod +x ~/.abra/vendor/yq
+
+# Note(decentral1se): it is fine to always use the development branch because
+# our Drone CI docker auto-tagger will publish official release tags and
+# otherwise give us the latest abra on the latest tag
+RUN curl https://install.abra.coopcloud.tech | bash -s -- --dev
+
+COPY bin/* /root/.local/bin/
+
+ENTRYPOINT ["/root/.local/bin/abra"]
--- a/14
+++ b/14
@ -1,14 +0,0 @@
-default: install
-
-dev_install:
-	ln -sf $(PWD)/abra ~/.local/bin
-
-install:
-	install abra /usr/bin/abra
-
-get_yq:
-	wget https://github.com/mikefarah/yq/releases/download/3.3.2/yq_linux_amd64 && \
-		chmod +x yq_linux_amd64 && \
-		mv yq_linux_amd64 yq
-
-.PHONY: dev_install install get_yq
--- a/README.md
+++ b/README.md
@ -1,31 +1,100 @@
 # abra

-[![Build Status](https://drone.autonomic.zone/api/badges/autonomic-cooperative/abra/status.svg)](https://drone.autonomic.zone/autonomic-cooperative/abra)
+## 🔥 🔥 🔥 D E P R E C A T E D 🔥 🔥 🔥

-Docker stack magic 🎩🐇
+[`abra`](https://git.coopcloud.tech/coop-cloud/abra) served us well but we're porting it to [Golang](https://golang.org) over in [`go-abra`](https://git.coopcloud.tech/coop-cloud/go-abra). To learn more about the reasons for that, see [this blog post](https://coopcloud.tech/blog/this-month-in-coop-cloud-july/). This means this repository and tool are officially deprecated as of August 1rst 2021. We will still provide bug security fixes but no new features will be developed in `abra`. Feel free to go on using it and reporting issues against this issue tracker. Thanks for all the good times Bash.

-## Install
+## 🔥 🔥 🔥 D E P R E C A T E D 🔥 🔥 🔥

-```sh
-curl -fsSL https://install.abra.autonomic.zone | bash
-```
+---

-Specific releases are available via the project [release page](https://git.autonomic.zone/autonomic-cooperative/abra/releases).
+[![Build Status](https://drone.autonomic.zone/api/badges/coop-cloud/abra/status.svg)](https://drone.autonomic.zone/coop-cloud/abra)

-## Changes
+> https://coopcloud.tech
+
+The Co-op Cloud utility belt 🎩🐇
+
+`abra` is a command-line tool for managing your own [Co-op Cloud](https://coopcloud.tech). It can provision new servers, create applications, deploy them, run backup and restore operations and a whole lot of other things. It is the go-to tool for day-to-day operations when managing a Co-op Cloud instance.
+
+## Change log
+
+> 🔥 🔥 🔥 Please note, while we are still in [public
+> alpha](https://docs.coopcloud.tech/roadmap/), the `abra` release
+> versioning scheme is not following [semver](https://semver.org/) conventions
+> because we are still in the exploratory phases of building this tool. Please
+> read the changes before upgrading your `abra` installation as there are
+> **most likely** breaking changes coming each release. Sorry for any
+> inconvenience caused, we're working hard to make this tool stable. Semver
+> will be respected when we reach public beta. 🔥 🔥 🔥

 See [CHANGELOG.md](./CHANGELOG.md).

-## Hacking
+## Documentation
+
+> [docs.coopcloud.tech](https://docs.coopcloud.tech)
+
+## Requirements
+
+- `curl`
+- `docker`
+- `bash` >= 4
+
+## Install
+
+Install the latest stable release:

 ```sh
-git clone ssh://git@git.autonomic.zone:2222/autonomic-cooperative/abra.git
-cd abra
-make dev_install
+curl https://install.abra.coopcloud.tech | bash
 ```

-See [autonomic-cooperative/installer-scripts](https://git.autonomic.zone/autonomic-cooperative/installer-scripts) for the installer script deployment. To make a release, just add an entry to [CHANGELOG.md](./CHANGELOG.md) and the [installer](./installer) (following [semver](https://semver.org/) please) and then `git tag x.x.x && git push origin main --tags`. If you want the [installer-scripts](https://git.autonomic.zone/autonomic-cooperative/installer-scripts) deployment to pick that up, you'll need to change the version number in the [Makefile](https://git.autonomic.zone/autonomic-cooperative/installer-scripts/src/branch/main/Makefile) and run `make` in that repository and push the changes.
+The source for this script is [here](./deploy/install.abra.coopcloud.tech/installer).

-## Examples
+You can pass options to the script like so (e.g. install the bleeding edge development version):

- `abra run mariadb mysqldump gitea -p'GdIbMeS09SURRktBnm3jcTufsL5z0MPd' | gzip > ../git.autonomic.zone_mariadb_`date +%F`.sql.gz`
+```sh
+curl https://install.abra.coopcloud.tech | bash -s -- --dev
+```
+
+Other options available are as follows:
+
+- **--no-prompt**: non-interactive installation
+- **--no-deps**: do not attempt to install [requirements](#requirements)
+
+## Container
+
+An [image](https://hub.docker.com/r/thecoopcloud/abra) is also provided.
+
+```
+docker run thecoopcloud/abra app ls
+```
+
+## Update
+
+Run `abra upgrade` to automatically download and install the latest release
+version.
+
+To update the development version, run `abra upgrade --dev`.
+
+## Hack
+
+It's written in Bash version 4 or greater!
+
+Install it via `curl https://install.abra.coopcloud.tech | bash -s -- --dev`, then you can hack on the source in `~/.abra/src`.
+
+The command-line interface is generated via [docopt](http://docopt.org/). If you add arguments then you need to run `make docopt` ro regenerate the parser.
+
+Please remember to update the [CHANGELOG](./CHANGELOG.md) when you make a change.
+
+## Releasing
+
+### `abra`
+
+> [install.abra.coopcloud.tech](https://install.abra.coopcloud.tech)
+
+- Change the `x.x.x` header in [CHANGELOG.md](./CHANGELOG.md) to reflect new version and mark date
+- Update the version in [abra](./abra)
+- Update the version in [deploy/install.abra.coopcloud.tech/installer](./deploy/install.abra.coopcloud.tech/installer)
+- `git commit` the above changes and then tag it with `git tag <your-new-version>`
+- `git push` and `git push --tags`
+- Deploy a new installer script `make release-installer`
+- Tell the world (CoTech forum, Matrix public channel, Autonomic mastodon, etc.)
--- a/2977
+++ b/2977
--- a/bin/abralib.py
+++ b/bin/abralib.py
@ -0,0 +1,112 @@
+"""Shared utilities for bin/*.py scripts."""
+
+from logging import DEBUG, basicConfig, getLogger
+from os import chdir, mkdir
+from os.path import exists, expanduser
+from pathlib import Path
+from shlex import split
+from subprocess import check_output
+from sys import exit
+
+from requests import get
+
+HOME_PATH = expanduser("~/")
+CLONES_PATH = Path(f"{HOME_PATH}/.abra/apps").absolute()
+REPOS_TO_SKIP = (
+    "abra",
+    "abra-apps",
+    "abra-capsul",
+    "abra-gandi",
+    "abra-hetzner",
+    "apps",
+    "auto-apps-json",
+    "auto-mirror",
+    "backup-bot",
+    "coopcloud.tech",
+    "coturn",
+    "docker-cp-deploy",
+    "docker-dind-bats-kcov",
+    "docs.coopcloud.tech",
+    "example",
+    "gardening",
+    "go-abra",
+    "organising",
+    "pyabra",
+    "radicle-seed-node",
+    "stack-ssh-deploy",
+    "swarm-cronjob",
+    "tyop",
+)
+YQ_PATH = Path(f"{HOME_PATH}/.abra/vendor/yq")
+JQ_PATH = Path(f"{HOME_PATH}/.abra/vendor/jq")
+
+log = getLogger(__name__)
+basicConfig()
+log.setLevel(DEBUG)
+
+
+def _run_cmd(cmd, shell=False, **kwargs):
+    """Run a shell command."""
+    args = [split(cmd)]
+
+    if shell:
+        args = [cmd]
+        kwargs = {"shell": shell}
+
+    try:
+        return check_output(*args, **kwargs).decode("utf-8").strip()
+    except Exception as exception:
+        log.error(f"Failed to run {cmd}, saw {str(exception)}")
+        exit(1)
+
+
+def get_repos_json():
+    """ Retrieve repo list from Gitea """
+
+    url = "https://git.autonomic.zone/api/v1/orgs/coop-cloud/repos"
+
+    log.info(f"Retrieving {url}")
+
+    repos = []
+    response = True
+    page = 1
+
+    try:
+        while response:
+            log.info(f"Trying to fetch page {page}")
+            response = get(url + f"?page={page}", timeout=10).json()
+            repos.extend(response)
+            page += 1
+
+        return repos
+    except Exception as exception:
+        log.error(f"Failed to retrieve {url}, saw {str(exception)}")
+        exit(1)
+
+
+def clone_all_apps(repos_json, ssh=False):
+    """Clone all Co-op Cloud apps to ~/.abra/apps."""
+    if not exists(CLONES_PATH):
+        mkdir(CLONES_PATH)
+
+    if ssh:
+        repos = [[p["name"], p["ssh_url"]] for p in repos_json]
+    else:
+        repos = [[p["name"], p["clone_url"]] for p in repos_json]
+
+    for name, url in repos:
+        if name in REPOS_TO_SKIP:
+            continue
+
+        if not exists(f"{CLONES_PATH}/{name}"):
+            log.info(f"Retrieving {url}")
+            _run_cmd(f"git clone {url} {CLONES_PATH}/{name}")
+
+            chdir(f"{CLONES_PATH}/{name}")
+            if not int(_run_cmd("git branch --list | wc -l", shell=True)):
+                log.info(f"Guessing main branch is HEAD for {name}")
+                _run_cmd("git checkout main")
+        else:
+            log.info(f"Updating {name}")
+            chdir(f"{CLONES_PATH}/{name}")
+            _run_cmd("git fetch -a")
--- a/bin/app-catalogue.sh
+++ b/bin/app-catalogue.sh
@ -0,0 +1,103 @@
+#!/bin/bash
+
+# shellcheck disable=SC2119
+
+# Usage: ./app-catalogue.sh
+#
+# Gather metadata from Co-op Cloud apps in $ABRA_DIR/apps (default
+# ~/.abra/apps), and format it as a Markdown table for this page:
+# https://docs.cloud.autonomic.zone/apps/
+
+stack_dir="${ABRA_DIR:-$HOME/.abra}/apps/"
+
+cd "$stack_dir" || exit
+
+# load all README files into ENV_FILES array
+mapfile -t readmes < <(find -L . -name "README.md")
+# FIXME 3wc: requires bash 4, use for loop instead
+
+base_url="https://git.autonomic.zone/coop-cloud"
+
+cat_apps=()
+cat_development=()
+cat_utilities=()
+cat_graveyard=()
+
+get_var() {
+	echo "$1" | grep "$2" | sed 's/^[^:]*: //'
+}
+
+# shellcheck disable=SC2120
+trim() {
+	# accept input as argument or from STDIN, see here:
+	# https://zwbetz.com/passing-input-to-a-bash-function-via-arguments-or-stdin/
+	# shellcheck disable=SC2155
+	local input="$([[ -p /dev/stdin ]] && cat - || echo "$@")"
+	[[ -z "$input" ]] && return 1
+	echo "$input" | tr -d ' '
+}
+
+# shellcheck disable=SC2120
+prettify() {
+	# as above
+	# shellcheck disable=SC2155
+	local input="$([[ -p /dev/stdin ]] && cat - || echo "$@")"
+	[[ -z "$input" ]] && return 1
+
+	echo "$input" | sed -e 's/Yes/✅/' -e 's/No/❌/' -e 's/N\/A/⛔/'
+}
+
+for readme in "${readmes[@]}"; do
+	type="$(basename "${readme%README.md}")"
+	if [ "$type" = "example" ]; then
+		continue
+	fi
+	title="$(grep '^# ' "$type/README.md" | sed 's/^# //' )"
+	# find section between 'metadata' and 'endmetadata' comments
+	metadata="$(awk '/-- metadata --/,/-- endmetadata --/' "$type/README.md")"
+	status="$(get_var "$metadata" "Status")"
+	category="$(get_var "$metadata" "Category" | cut -d',' -f2 | trim)"
+
+	if [ -z "$category" ]; then
+		echo "ERROR: missing category for $type"
+		continue
+	fi
+
+	image="$(get_var "$metadata" "Image" | cut -d',' -f2 | trim)"
+	healthcheck="$(get_var "$metadata" "Healthcheck" | prettify)"
+	backups="$(get_var "$metadata" "Backups" | prettify)"
+	email="$(get_var "$metadata" "Email" | prettify)"
+	tests="$(get_var "$metadata" "Tests" | prettify)"
+	sso="$(get_var "$metadata" "SSO" | prettify)"
+
+	row="| [$title]($base_url/$type) | $status | $image | $healthcheck | $backups | $email | $tests | $sso |"
+
+	category_lower="$(echo "$category" | tr '[:upper:]' '[:lower:]')"
+	eval "cat_$category_lower+=( '$row' )"
+done
+
+headers="
+| **Name** | **Status** | **Image** | **Healtcheck** | **Backups** | **Email** | **CI** | **Single-Sign-On** |
+| --- | --- | --- | --- | --- | --- | --- | --- |"
+
+echo "## Applications"
+echo "$headers"
+printf '%s\n' "${cat_apps[@]}" | sort
+
+echo
+
+echo "## Developer tools"
+echo "$headers"
+printf '%s\n' "${cat_development[@]}" | sort
+
+echo
+
+echo "## Utilities"
+echo "$headers"
+printf '%s\n' "${cat_utilities[@]}" | sort
+
+echo
+
+echo "## Graveyard"
+echo "$headers"
+printf '%s\n' "${cat_graveyard[@]}" | sort
--- a/bin/app-json.py
+++ b/bin/app-json.py
@ -0,0 +1,237 @@
+#!/usr/bin/env python3
+
+# Usage: ./app-json.py
+#
+# Gather metadata from Co-op Cloud apps in $ABRA_DIR/apps (default
+# ~/.abra/apps), and format it as JSON so that it can be hosted here:
+# https://apps.coopcloud.tech
+
+import argparse
+from json import dump
+from os import chdir, environ, getcwd, listdir
+from os.path import basename
+from pathlib import Path
+from re import findall, search
+from subprocess import DEVNULL
+
+from requests import get
+
+from abralib import (
+    CLONES_PATH,
+    JQ_PATH,
+    REPOS_TO_SKIP,
+    YQ_PATH,
+    _run_cmd,
+    clone_all_apps,
+    get_repos_json,
+    log,
+)
+
+parser = argparse.ArgumentParser(description="Generate a new apps.json")
+parser.add_argument("--output", type=Path, default=f"{getcwd()}/apps.json")
+
+
+def skopeo_login():
+    """Log into the docker registry to avoid rate limits."""
+    user = environ.get("SKOPEO_USER")
+    password = environ.get("SKOPEO_PASSWORD")
+    registry = environ.get("SKOPEO_REGISTRY", "docker.io")
+
+    if not user or not password:
+        log.info("Failed to log in via Skopeo due to missing env vars")
+        return
+
+    login_cmd = f"skopeo login {registry} -u {user} -p {password}"
+    output = _run_cmd(login_cmd, shell=True)
+    log.info(f"Skopeo login attempt: {output}")
+
+
+def get_published_apps_json():
+    """Retrieve already published apps json."""
+    url = "https://apps.coopcloud.tech"
+
+    log.info(f"Retrieving {url}")
+
+    try:
+        return get(url, timeout=5).json()
+    except Exception as exception:
+        log.error(f"Failed to retrieve {url}, saw {str(exception)}")
+        return {}
+
+
+def generate_apps_json(repos_json):
+    """Generate the abra-apps.json application versions file."""
+    apps_json = {}
+    cached_apps_json = get_published_apps_json()
+
+    for app in listdir(CLONES_PATH):
+        if app in REPOS_TO_SKIP:
+            log.info(f"Skipping {app}")
+            continue
+
+        repo_details = next(filter(lambda x: x["name"] == app, repos_json), {})
+
+        app_path = f"{CLONES_PATH}/{app}"
+        chdir(app_path)
+
+        metadata = get_app_metadata(app_path)
+
+        name = metadata.pop("name", app)
+
+        log.info(f"Processing {app}")
+        apps_json[app] = {
+            "name": name,
+            "category": metadata.get("category", ""),
+            "repository": repo_details.get("clone_url", ""),
+            "default_branch": repo_details.get("default_branch", ""),
+            "description": repo_details.get("description", ""),
+            "website": repo_details.get("website", ""),
+            "features": metadata,
+            "versions": get_app_versions(app_path, cached_apps_json),
+            "icon": repo_details.get("avatar_url", ""),
+        }
+
+    return apps_json
+
+
+def get_app_metadata(app_path):
+    """Parse metadata from app repo README files."""
+    metadata = {}
+
+    chdir(app_path)
+
+    try:
+        with open(f"{app_path}/README.md", "r") as handle:
+            log.info(f"{app_path}/README.md")
+            contents = handle.read()
+    except Exception:
+        log.info(f"No {app_path}/README.md discovered, moving on")
+        return {}
+
+    try:
+        for match in findall(r"\*\*.*", contents):
+            title = search(r"(?<=\*\*).*(?=\*\*)", match).group().lower()
+
+            if title == "image":
+                value = {
+                    "image": search(r"(?<=`).*(?=`)", match).group(),
+                    "url": search(r"(?<=\().*(?=\))", match).group(),
+                    "rating": match.split(",")[1].strip(),
+                    "source": match.split(",")[-1].replace("*", "").strip(),
+                }
+            elif title == "status":
+                value = {"❶💚": 1, "❷💛": 2, "❸🍎": 3, "❹💣": 4, "?": 5, "": 5}[
+                    match.split(":")[-1].replace("*", "").strip()
+                ]
+            else:
+                value = match.split(":")[-1].replace("*", "").strip()
+
+            metadata[title] = value
+        metadata["name"] = findall(r"^# (.*)", contents)[0]
+    except (IndexError, AttributeError):
+        log.info(f"Can't parse {app_path}/README.md")
+        return {}
+    finally:
+        _run_cmd("git checkout HEAD")
+
+    log.info(f"Parsed {metadata}")
+
+    return metadata
+
+
+def get_app_versions(app_path, cached_apps_json):
+    versions = {}
+
+    chdir(app_path)
+
+    tags = _run_cmd("git tag --list").split()
+
+    if not tags:
+        log.info("No tags discovered, moving on")
+        return {}
+
+    initial_branch = _run_cmd("git rev-parse --abbrev-ref HEAD")
+
+    app_name = basename(app_path)
+
+    try:
+        existing_tags = cached_apps_json[app_name]["versions"].keys()
+    except KeyError:
+        existing_tags = []
+
+    for tag in tags:
+        _run_cmd(f"git checkout {tag}", stderr=DEVNULL)
+
+        services_cmd = f"{YQ_PATH} e '.services | keys | .[]' compose*.yml"
+        services = _run_cmd(services_cmd, shell=True).split()
+
+        parsed_services = []
+        service_versions = {}
+        for service in services:
+            if service in ("null", "---"):
+                continue
+
+            if (
+                tag in existing_tags
+                and service in cached_apps_json[app_name]["versions"][tag]
+            ):
+                log.info(f"Skipping {tag} because we've already processed it")
+                existing_versions = cached_apps_json[app_name]["versions"][tag][service]
+                service_versions[service] = existing_versions
+                _run_cmd(f"git checkout {initial_branch}")
+                continue
+
+            if service in parsed_services:
+                log.info(f"Skipped {service}, we've already parsed it locally")
+                continue
+
+            services_cmd = f"{YQ_PATH} e '.services.{service}.image' compose*.yml"
+            images = _run_cmd(services_cmd, shell=True).split()
+
+            for image in images:
+                if image in ("null", "---"):
+                    continue
+
+                images_cmd = f"skopeo inspect docker://{image} | {JQ_PATH} '.Digest'"
+                output = _run_cmd(images_cmd, shell=True)
+
+                service_version_info = {
+                    "image": image.split(":")[0],
+                    "tag": image.split(":")[-1],
+                    "digest": output.split(":")[-1][:8],
+                }
+
+                log.info(f"Parsed {service_version_info}")
+                service_versions[service] = service_version_info
+
+            parsed_services.append(service)
+
+        versions[tag] = service_versions
+
+    _run_cmd(f"git checkout {initial_branch}")
+
+    return versions
+
+
+def main():
+    """Run the script."""
+    args = parser.parse_args()
+
+    skopeo_login()
+
+    repos_json = get_repos_json()
+    clone_all_apps(repos_json)
+
+    with open(args.output, "w", encoding="utf-8") as handle:
+        dump(
+            generate_apps_json(repos_json),
+            handle,
+            ensure_ascii=False,
+            indent=4,
+            sort_keys=True,
+        )
+
+    log.info(f"Successfully generated {args.output}")
+
+
+main()
--- a/bin/clone-all-apps.py
+++ b/bin/clone-all-apps.py
@ -0,0 +1,16 @@
+#!/usr/bin/env python3
+
+# Usage: ./clone-all-apps.py
+#
+# Clone all available apps into ~/.abra/apps using ssh:// URLs
+
+from abralib import clone_all_apps, get_repos_json
+
+
+def main():
+    """Run the script."""
+    repos_json = get_repos_json()
+    clone_all_apps(repos_json, ssh=True)
+
+
+main()
--- a/bin/github-sync.py
+++ b/bin/github-sync.py
@ -0,0 +1,49 @@
+#!/usr/bin/env python3
+
+# Usage: ./github-sync.py
+#
+# Mirror repositories to Github (Fuck M$, get it straight)
+
+from os import chdir, environ, listdir
+
+from abralib import CLONES_PATH, _run_cmd, clone_all_apps, get_repos_json, log
+
+REPOS_TO_SKIP = (
+    "apps",
+    "backup-bot",
+    "docker-dind-bats-kcov",
+    "docs.coopcloud.tech",
+    "pyabra",
+    "radicle-seed-node",
+    "swarm-cronjob",
+    "tyop",
+)
+
+
+def main():
+    """Run the script."""
+    repos_json = get_repos_json()
+    clone_all_apps(repos_json)
+
+    for app in listdir(CLONES_PATH):
+        if app in REPOS_TO_SKIP:
+            log.info(f"Skipping {app}")
+            continue
+
+        app_path = f"{CLONES_PATH}/{app}"
+        chdir(app_path)
+
+        log.info(f"Mirroring {app}...")
+
+        token = environ.get("GITHUB_ACCESS_TOKEN")
+        remote = f"https://coopcloudbot:{token}@github.com/Coop-Cloud/{app}.git"
+
+        _run_cmd(
+            f"git remote add github {remote} || true",
+            shell=True,
+        )
+
+        _run_cmd("git push github --all")
+
+
+main()
--- a/bin/renovate-ls-apps.py
+++ b/bin/renovate-ls-apps.py
@ -0,0 +1,20 @@
+#!/usr/bin/env python3
+
+# Usage: ./renovate-ls-apps.py
+#
+# Output list of apps for Renovate bot configuration
+
+from abralib import REPOS_TO_SKIP, get_repos_json
+
+
+def main():
+    """Run the script."""
+    repos = [p["full_name"] for p in get_repos_json()]
+    repos.sort()
+    for repo in repos:
+        if repo.split("/")[-1] in REPOS_TO_SKIP:
+            continue
+        print(f'"{repo}",')
+
+
+main()
--- a/completion/_abra
+++ b/completion/_abra
@ -0,0 +1,52 @@
+#compdef abra
+
+_abra () {
+	local context state line curcontext="$curcontext" ret=1
+	_arguments -n : \
+		{-h,--help}'[Help message]' \
+		'1:commands:(app server)' \
+		'*::arguments:->arguments' \
+		&& ret=0
+
+	case $state in
+		(arguments)
+			curcontext="${curcontext%:*:*}:abra-arguments-$words[1]:"
+			case $words[1] in
+				(app)
+					_arguments \
+						'1: :_abra_apps' \
+					&& ret=0
+				;;
+				(server)
+					_arguments \
+						'1:servers:_abra_servers' \
+					&& ret=0
+				;;
+			esac
+		;;
+	esac
+
+	return ret
+}
+
+_abra_servers() {
+	_path_files -/W $HOME/.abra/servers
+}
+
+_abra_apps()
+{
+	local newapps apps=($HOME/.abra/servers/*/*.env)
+	typeset -a apps
+	newapps=()
+	for app in $apps; do
+		newapps+=($(_abra_basename "${app}"))
+	done
+	_describe -t apps 'app' newapps
+}
+
+_abra_basename()
+{
+    printf -- "${1##*/}"
+}
+
+_abra "$@"
--- a/completion/abra.bash
+++ b/completion/abra.bash
@ -0,0 +1,140 @@
+#!/usr/bin/env bash
+
+_abra_basename()
+{
+    echo "${1##*/}"
+}
+
+_abra_servers()
+{
+    # FIXME 3wc: copied from abra/get_servers()
+    shopt -s nullglob dotglob
+    local SERVERS=(~/.abra/servers/*)
+    shopt -u nullglob dotglob
+
+    for SERVER in "${SERVERS[@]}"; do
+        _abra_basename "${SERVER}"
+    done
+}
+
+_abra_complete_servers()
+{
+    mapfile -t COMPREPLY < <(compgen -W "$(_abra_servers)" -- "$1")
+}
+
+_abra_apps()
+{
+    shopt -s nullglob dotglob
+    local APPS=(~/.abra/servers/*/*.env)
+    shopt -u nullglob dotglob
+
+    for APP in "${APPS[@]}"; do
+        _abra_basename "${APP%.env}"
+    done
+}
+
+_abra_complete_apps()
+{
+    mapfile -t COMPREPLY < <(compgen -W "$(_abra_apps)" -- "$1")
+}
+
+_abra_recipes()
+{
+    shopt -s nullglob dotglob
+    local RECIPES=(~/.abra/apps/*)
+    shopt -u nullglob dotglob
+
+    for RECIPE in "${RECIPES[@]}"; do
+        _abra_basename "${RECIPE%.env}"
+    done
+}
+
+_abra_complete_recipes()
+{
+    mapfile -t COMPREPLY < <(compgen -W "$(_abra_recipes)" -- "$1")
+}
+
+_abra_complete()
+{
+    compopt +o default +o nospace
+    COMPREPLY=()
+
+    local -r cmds='
+        app
+        server
+        recipe
+    '
+    local -r short_opts='-e      -h     -s      -v'
+    local -r long_opts='--env --help --stack --version'
+
+    # Scan through the command line and find the abra command
+    # (if present), as well as its expected position.
+    local cmd
+    local cmd_index=1 # Expected index of the command token.
+    local i
+    for (( i = 1; i < ${#COMP_WORDS[@]}; i++ )); do
+        local word="${COMP_WORDS[i]}"
+        case "$word" in
+            -*)
+                ((cmd_index++))
+                ;;
+            *)
+                cmd="$word"
+                break
+                ;;
+        esac
+    done
+
+    local cur="${COMP_WORDS[COMP_CWORD]}"
+
+    if (( COMP_CWORD < cmd_index )); then
+        # Offer option completions.
+        case "$cur" in
+            --*)
+                mapfile -t COMPREPLY < <(compgen -W "$long_opts" -- "$cur")
+                ;;
+            -*)
+                mapfile -t COMPREPLY < <(compgen -W "$short_opts" -- "$cur")
+                ;;
+            *)
+                # Skip completion; we should never get here.
+                ;;
+        esac
+    elif (( COMP_CWORD == cmd_index )); then
+        # Offer command name completions.
+        mapfile -t COMPREPLY < <(compgen -W "$cmds" -- "$cur")
+    else
+        # Offer command argument completions.
+        case "$cmd" in
+            server)
+                # Offer exactly one server name completion.
+                if (( COMP_CWORD == cmd_index + 1 )); then
+                    _abra_complete_servers "$cur"
+                fi
+                ;;
+            app)
+                # Offer exactly one app completion.
+                if (( COMP_CWORD == cmd_index + 1 )); then
+                    _abra_complete_apps "$cur"
+                fi
+                ;;
+            recipe)
+                # Offer exactly one app completion.
+                if (( COMP_CWORD == cmd_index + 1 )); then
+                    _abra_complete_recipes "$cur"
+                fi
+                ;;
+            #help)
+            #    # Offer exactly one command name completion.
+            #    if (( COMP_CWORD == cmd_index + 1 )); then
+            #        COMPREPLY=($(compgen -W "$cmds" -- "$cur"))
+            #    fi
+            #    ;;
+            *)
+                # Unknown command or unknowable argument.
+                ;;
+        esac
+    fi
+}
+
+complete -o default -F _abra_complete abra
--- a/deploy/install.abra.coopcloud.tech/compose.yml
+++ b/deploy/install.abra.coopcloud.tech/compose.yml
@ -0,0 +1,38 @@
+---
+version: "3.8"
+
+services:
+  app:
+    image: "nginx:stable"
+    configs:
+      - source: abra_conf
+        target: /etc/nginx/conf.d/abra.conf
+      - source: abra_installer
+        target: /var/www/abra-installer/installer
+    volumes:
+      - "public:/var/www/abra-installer"
+    networks:
+      - proxy
+    deploy:
+      update_config:
+        failure_action: rollback
+        order: start-first
+      labels:
+        - "traefik.enable=true"
+        - "traefik.http.services.abra-installer.loadbalancer.server.port=80"
+        - "traefik.http.routers.abra-installer.rule=Host(`install.abra.autonomic.zone`,`install.abra.coopcloud.tech`)"
+        - "traefik.http.routers.abra-installer.entrypoints=web-secure"
+        - "traefik.http.routers.abra-installer.tls.certresolver=production"
+
+configs:
+  abra_installer:
+    file: installer
+  abra_conf:
+    file: nginx.conf
+
+networks:
+  proxy:
+    external: true
+
+volumes:
+  public:
--- a/deploy/install.abra.coopcloud.tech/installer
+++ b/deploy/install.abra.coopcloud.tech/installer
@ -0,0 +1,217 @@
+#!/usr/bin/env bash
+
+# shellcheck disable=SC2154,SC2034
+
+ABRA_VERSION="10.0.1"
+GIT_URL="https://git.coopcloud.tech/coop-cloud/abra"
+ABRA_SRC="$GIT_URL/raw/tag/$ABRA_VERSION/abra"
+ABRA_DIR="${ABRA_DIR:-$HOME/.abra}"
+
+DOC="
+abra command-line installer script
+
+Usage:
+  installer [options]
+
+Options:
+  -h, --help            Show this message and exit
+  -d, --dev             Install bleeding edge development version
+  -n, --no-prompt       Don't prompt for input and run non-interactively
+  -p, --no-deps         Don't attempt to install system dependencies
+"
+
+# docopt parser below, refresh this parser with `docopt.sh installer`
+# shellcheck disable=2016,1075
+docopt() { parse() { if ${DOCOPT_DOC_CHECK:-true}; then local doc_hash
+if doc_hash=$(printf "%s" "$DOC" | (sha256sum 2>/dev/null || shasum -a 256)); then
+if [[ ${doc_hash:0:5} != "$digest" ]]; then
+stderr "The current usage doc (${doc_hash:0:5}) does not match \
+what the parser was generated with (${digest})
+Run \`docopt.sh\` to refresh the parser."; _return 70; fi; fi; fi
+local root_idx=$1; shift; argv=("$@"); parsed_params=(); parsed_values=()
+left=(); testdepth=0; local arg; while [[ ${#argv[@]} -gt 0 ]]; do
+if [[ ${argv[0]} = "--" ]]; then for arg in "${argv[@]}"; do
+parsed_params+=('a'); parsed_values+=("$arg"); done; break
+elif [[ ${argv[0]} = --* ]]; then parse_long
+elif [[ ${argv[0]} = -* && ${argv[0]} != "-" ]]; then parse_shorts
+elif ${DOCOPT_OPTIONS_FIRST:-false}; then for arg in "${argv[@]}"; do
+parsed_params+=('a'); parsed_values+=("$arg"); done; break; else
+parsed_params+=('a'); parsed_values+=("${argv[0]}"); argv=("${argv[@]:1}"); fi
+done; local idx; if ${DOCOPT_ADD_HELP:-true}; then
+for idx in "${parsed_params[@]}"; do [[ $idx = 'a' ]] && continue
+if [[ ${shorts[$idx]} = "-h" || ${longs[$idx]} = "--help" ]]; then
+stdout "$trimmed_doc"; _return 0; fi; done; fi
+if [[ ${DOCOPT_PROGRAM_VERSION:-false} != 'false' ]]; then
+for idx in "${parsed_params[@]}"; do [[ $idx = 'a' ]] && continue
+if [[ ${longs[$idx]} = "--version" ]]; then stdout "$DOCOPT_PROGRAM_VERSION"
+_return 0; fi; done; fi; local i=0; while [[ $i -lt ${#parsed_params[@]} ]]; do
+left+=("$i"); ((i++)) || true; done
+if ! required "$root_idx" || [ ${#left[@]} -gt 0 ]; then error; fi; return 0; }
+parse_shorts() { local token=${argv[0]}; local value; argv=("${argv[@]:1}")
+[[ $token = -* && $token != --* ]] || _return 88; local remaining=${token#-}
+while [[ -n $remaining ]]; do local short="-${remaining:0:1}"
+remaining="${remaining:1}"; local i=0; local similar=(); local match=false
+for o in "${shorts[@]}"; do if [[ $o = "$short" ]]; then similar+=("$short")
+[[ $match = false ]] && match=$i; fi; ((i++)) || true; done
+if [[ ${#similar[@]} -gt 1 ]]; then
+error "${short} is specified ambiguously ${#similar[@]} times"
+elif [[ ${#similar[@]} -lt 1 ]]; then match=${#shorts[@]}; value=true
+shorts+=("$short"); longs+=(''); argcounts+=(0); else value=false
+if [[ ${argcounts[$match]} -ne 0 ]]; then if [[ $remaining = '' ]]; then
+if [[ ${#argv[@]} -eq 0 || ${argv[0]} = '--' ]]; then
+error "${short} requires argument"; fi; value=${argv[0]}; argv=("${argv[@]:1}")
+else value=$remaining; remaining=''; fi; fi; if [[ $value = false ]]; then
+value=true; fi; fi; parsed_params+=("$match"); parsed_values+=("$value"); done
+}; parse_long() { local token=${argv[0]}; local long=${token%%=*}
+local value=${token#*=}; local argcount; argv=("${argv[@]:1}")
+[[ $token = --* ]] || _return 88; if [[ $token = *=* ]]; then eq='='; else eq=''
+value=false; fi; local i=0; local similar=(); local match=false
+for o in "${longs[@]}"; do if [[ $o = "$long" ]]; then similar+=("$long")
+[[ $match = false ]] && match=$i; fi; ((i++)) || true; done
+if [[ $match = false ]]; then i=0; for o in "${longs[@]}"; do
+if [[ $o = $long* ]]; then similar+=("$long"); [[ $match = false ]] && match=$i
+fi; ((i++)) || true; done; fi; if [[ ${#similar[@]} -gt 1 ]]; then
+error "${long} is not a unique prefix: ${similar[*]}?"
+elif [[ ${#similar[@]} -lt 1 ]]; then
+[[ $eq = '=' ]] && argcount=1 || argcount=0; match=${#shorts[@]}
+[[ $argcount -eq 0 ]] && value=true; shorts+=(''); longs+=("$long")
+argcounts+=("$argcount"); else if [[ ${argcounts[$match]} -eq 0 ]]; then
+if [[ $value != false ]]; then
+error "${longs[$match]} must not have an argument"; fi
+elif [[ $value = false ]]; then
+if [[ ${#argv[@]} -eq 0 || ${argv[0]} = '--' ]]; then
+error "${long} requires argument"; fi; value=${argv[0]}; argv=("${argv[@]:1}")
+fi; if [[ $value = false ]]; then value=true; fi; fi; parsed_params+=("$match")
+parsed_values+=("$value"); }; required() { local initial_left=("${left[@]}")
+local node_idx; ((testdepth++)) || true; for node_idx in "$@"; do
+if ! "node_$node_idx"; then left=("${initial_left[@]}"); ((testdepth--)) || true
+return 1; fi; done; if [[ $((--testdepth)) -eq 0 ]]; then
+left=("${initial_left[@]}"); for node_idx in "$@"; do "node_$node_idx"; done; fi
+return 0; }; optional() { local node_idx; for node_idx in "$@"; do
+"node_$node_idx"; done; return 0; }; switch() { local i
+for i in "${!left[@]}"; do local l=${left[$i]}
+if [[ ${parsed_params[$l]} = "$2" ]]; then
+left=("${left[@]:0:$i}" "${left[@]:((i+1))}")
+[[ $testdepth -gt 0 ]] && return 0; if [[ $3 = true ]]; then
+eval "((var_$1++))" || true; else eval "var_$1=true"; fi; return 0; fi; done
+return 1; }; stdout() { printf -- "cat <<'EOM'\n%s\nEOM\n" "$1"; }; stderr() {
+printf -- "cat <<'EOM' >&2\n%s\nEOM\n" "$1"; }; error() {
+[[ -n $1 ]] && stderr "$1"; stderr "$usage"; _return 1; }; _return() {
+printf -- "exit %d\n" "$1"; exit "$1"; }; set -e; trimmed_doc=${DOC:1:333}
+usage=${DOC:37:28}; digest=36916; shorts=(-h -d -n -p)
+longs=(--help --dev --no-prompt --no-deps); argcounts=(0 0 0 0); node_0(){
+switch __help 0; }; node_1(){ switch __dev 1; }; node_2(){ switch __no_prompt 2
+}; node_3(){ switch __no_deps 3; }; node_4(){ optional 0 1 2 3; }; node_5(){
+optional 4; }; node_6(){ required 5; }; node_7(){ required 6; }
+cat <<<' docopt_exit() { [[ -n $1 ]] && printf "%s\n" "$1" >&2
+printf "%s\n" "${DOC:37:28}" >&2; exit 1; }'; unset var___help var___dev \
+var___no_prompt var___no_deps; parse 7 "$@"; local prefix=${DOCOPT_PREFIX:-''}
+unset "${prefix}__help" "${prefix}__dev" "${prefix}__no_prompt" \
+"${prefix}__no_deps"; eval "${prefix}"'__help=${var___help:-false}'
+eval "${prefix}"'__dev=${var___dev:-false}'
+eval "${prefix}"'__no_prompt=${var___no_prompt:-false}'
+eval "${prefix}"'__no_deps=${var___no_deps:-false}'; local docopt_i=1
+[[ $BASH_VERSION =~ ^4.3 ]] && docopt_i=2; for ((;docopt_i>0;docopt_i--)); do
+declare -p "${prefix}__help" "${prefix}__dev" "${prefix}__no_prompt" \
+"${prefix}__no_deps"; done; }
+# docopt parser above, complete command for generating this parser is `docopt.sh installer`
+
+function prompt_confirm {
+  if [ "$no_prompt" == "true" ]; then
+    return
+  fi
+
+  read -rp "Continue? [y/N]? " choice
+
+  case "$choice" in
+    y|Y ) return ;;
+    * ) exit;;
+  esac
+}
+
+function show_banner {
+  echo ""
+  echo "   ____                           ____ _                 _ "
+  echo "  / ___|___         ___  _ __    / ___| | ___  _   _  __| |"
+  echo " | |   / _ \ _____ / _ \| '_ \  | |   | |/ _ \| | | |/ _' |"
+  echo " | |__| (_) |_____| (_) | |_) | | |___| | (_) | |_| | (_| |"
+  echo "  \____\___/       \___/| .__/   \____|_|\___/ \__,_|\__,_|"
+  echo "                        |_|"
+  echo ""
+}
+
+function install_docker {
+  sudo apt-get remove docker docker-engine docker.io containerd runc
+  sudo apt-get install -yq \
+    apt-transport-https \
+    ca-certificates \
+    gnupg \
+    lsb-release
+  curl -fsSL https://download.docker.com/linux/debian/gpg | sudo gpg --dearmor -o /usr/share/keyrings/docker-archive-keyring.gpg
+  echo \
+  "deb [arch=amd64 signed-by=/usr/share/keyrings/docker-archive-keyring.gpg] https://download.docker.com/linux/debian \
+  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
+  sudo apt-get update
+  sudo apt-get install -yq docker-ce docker-ce-cli containerd.io
+}
+
+function install_requirements {
+  if [ -f "/etc/debian_version" ]; then
+    echo "Detected Debian based distribution, attempting to install system requirements..."
+
+    sudo apt update && sudo apt install -y \
+      passwdqc \
+      pwgen
+
+    echo "Install Docker (https://docs.docker.com/engine/install/debian/)?"
+    prompt_confirm
+    install_docker
+  else
+    echo "Sorry, we only support Debian based distributions at the moment"
+    echo "You'll have to install the requirements manually for your distribution"
+    echo "See https://git.autonomic.zone/coop-cloud/abra#requirements for more"
+  fi
+}
+
+function install_abra_release {
+  mkdir -p "$HOME/.local/bin"
+  curl "$ABRA_SRC" > "$HOME/.local/bin/abra"
+  chmod +x "$HOME/.local/bin/abra"
+  echo "abra installed to $HOME/.local/bin/abra"
+}
+
+function install_abra_dev {
+  mkdir -p "$ABRA_DIR/"
+  if [[ ! -d "$ABRA_DIR/src" ]]; then
+    git clone "$GIT_URL" "$ABRA_DIR/src"
+  fi
+  (cd "$ABRA_DIR/src" && git pull origin main && cd - || exit)
+  mkdir -p "$HOME/.local/bin"
+  ln -sf "$ABRA_DIR/src/abra" "$HOME/.local/bin/abra"
+  echo "abra installed to $HOME/.local/bin/abra (development bleeding edge)"
+}
+
+function run_installation {
+  show_banner
+
+  DOCOPT_PREFIX=installer_
+  DOCOPT_ADD_HELP=false
+  eval "$(docopt "$@")"
+
+  dev="$installer___dev"
+  no_prompt="$installer___no_prompt"
+  no_deps="$installer___no_deps"
+
+  if [ "$no_deps" == "false" ]; then
+    install_requirements
+  fi
+
+  if [ "$dev" == "true" ]; then
+    install_abra_dev
+  else
+    install_abra_release
+  fi
+}
+
+run_installation "$@"
+exit 0
--- a/deploy/install.abra.coopcloud.tech/nginx.conf
+++ b/deploy/install.abra.coopcloud.tech/nginx.conf
@ -0,0 +1,10 @@
+server {
+  listen         80 default_server;
+  server_name    install.abra.autonomic.zone install.abra.coopcloud.tech;
+
+  location / {
+    root /var/www/abra-installer;
+    add_header Content-Type text/plain;
+    index installer;
+  }
+}
--- a/18
+++ b/18
@ -1,18 +0,0 @@
-#!/bin/bash
-
-ABRA_VERSION="0.1.2"
-ABRA_SRC="https://git.autonomic.zone/autonomic-cooperative/abra/raw/tag/$ABRA_VERSION/abra"
-
-function install_abra {
-  mkdir -p "$HOME/.local/bin"
-	curl "$ABRA_SRC" > "$HOME/.local/bin/abra"
-  chmod +x "$HOME/.local/bin/abra"
-  echo "abra installed to $HOME/.loca/bin/abra"
-}
-
-function run_installation {
-  install_abra
-}
-
-run_installation
-exit 0
--- a/59
+++ b/59
@ -0,0 +1,59 @@
+.PHONY: test shellcheck docopt release-installer build push deploy-docopt symlink
+
+test:
+	@sudo DOCKER_CONTEXT=default docker run \
+		-v $$(pwd):/workdir \
+		--privileged \
+		-d \
+		--name=abra-test-dind \
+		-e DOCKER_TLS_CERTDIR="" \
+		decentral1se/docker-dind-bats-kcov \
+	@DOCKER_CONTEXT=default sudo docker exec \
+		-it \
+		abra-test-dind \
+		sh -c "cd /workdir && bats /workdir/tests"
+	@DOCKER_CONTEXT=default sudo docker stop abra-test-dind
+	@DOCKER_CONTEXT=default sudo docker rm abra-test-dind
+
+shellcheck:
+	@docker run \
+		-it \
+		--rm \
+		-v $$(pwd):/workdir \
+		koalaman/shellcheck-alpine \
+		sh -c "shellcheck /workdir/abra && \
+		       shellcheck /workdir/bin/*.sh && \
+		       shellcheck /workdir/deploy/install.abra.coopcloud.tech/installer"
+
+docopt:
+	@if [ ! -d ".venv" ]; then \
+		python3 -m venv .venv && \
+		.venv/bin/pip install -U pip setuptools wheel && \
+		.venv/bin/pip install docopt-sh; \
+	fi
+	.venv/bin/docopt.sh abra
+
+deploy-docopt:
+	@if [ ! -d ".venv" ]; then \
+		python3 -m venv .venv && \
+		.venv/bin/pip install -U pip setuptools wheel && \
+		.venv/bin/pip install docopt-sh; \
+	fi
+	.venv/bin/docopt.sh deploy/install.abra.coopcloud.tech/installer
+
+release-installer:
+	@DOCKER_CONTEXT=swarm.autonomic.zone \
+		docker stack rm abra-installer-script && \
+		cd deploy/install.abra.coopcloud.tech && \
+		DOCKER_CONTEXT=swarm.autonomic.zone docker stack deploy -c compose.yml abra-installer-script
+
+build:
+	@docker build -t thecoopcloud/abra .
+
+push: build
+	@docker push thecoopcloud/abra
+
+symlink:
+	@mkdir -p ~/.abra/servers/ && \
+	ln -srf tests/default ~/.abra/servers && \
+	ln -srf tests/apps/* ~/.abra/apps
--- a/test/abra.yml
+++ b/test/abra.yml
@ -1,7 +0,0 @@
---
-name: mystack
-secrets:
-  foo:
-  bar:
-  baz:
-    length: 128
--- a/tests/01_environment.bats
+++ b/tests/01_environment.bats
@ -0,0 +1,17 @@
+#!/usr/bin/env bats
+
+@test "docker is available" {
+    command -v docker
+}
+
+@test "abra is executable" {
+    ./abra --help
+}
+
+@test "git is available" {
+    command -v git
+}
+
+@test "running in a container" {
+    ls /etc/alpine-release
+}
--- a/tests/02_abra.bats
+++ b/tests/02_abra.bats
@ -0,0 +1,56 @@
+#!/usr/bin/env bats
+
+setup() {
+    export ABRA_DIR=$(mktemp -d)
+    mkdir -p $ABRA_DIR/servers/default
+}
+
+teardown() {
+    rm -rf "$ABRA_DIR"
+}
+
+abra() {
+    ./abra -d $@
+}
+
+@test "abra server (add|rm)" {
+    abra server add swarm.test.com
+    docker context ls | grep swarm.test.com
+    [ -d $ABRA_DIR/servers/swarm.test.com ]
+    abra server swarm.test.com rm
+
+    abra server add swarm.test.com foobar 12345
+    [ -d $ABRA_DIR/servers/swarm.test.com ]
+    abra server swarm.test.com rm
+}
+
+@test "abra server init" {
+    abra server default init
+}
+
+@test "abra app (new|rm)" {
+    abra app new --server default --domain traefik.test.com --app-name traefik_test_com traefik
+    [ -f $ABRA_DIR/servers/default/traefik_test_com.env ]
+
+    # interactive prompt
+    echo "y" | abra app traefik_test_com delete
+    [ ! -f $ABRA_DIR/servers/default/traefik_test_com.env ]
+
+    # --no-prompt
+    abra app new --server default --domain traefik_test_com --app-name traefik_test_com traefik
+    abra --no-prompt app traefik_test_com delete
+    [ ! -f $ABRA_DIR/servers/default/traefik_test_com.env ]
+}
+
+@test "abra app <domain> secret (insert|generate|rm)" {
+    abra app new --server default --domain traefik_test_com --app-name traefik_test_com traefik
+
+    abra app traefik_test_com secret insert foobar v1 "foobar"
+
+    # interactive prompt
+    echo "y" | abra app traefik_test_com secret rm foobar
+
+    abra app traefik_test_com secret insert foobar v1 "foobar"
+    #  prompt
+    abra --no-prompt app traefik_test_com secret rm foobar
+}
--- a/tests/apps/works/compose.yml
+++ b/tests/apps/works/compose.yml
@ -0,0 +1,84 @@
+---
+
+# The goal of this compose file is to have a testing ground for understanding
+# what cases we need to handle to get stable deployments. For that, we need to
+# work with healthchecks and deploy configurations quite closely. If you run
+# the `make symlink` target then this will be loaded into a "fake" app on your
+# local machine which you can deploy with `abra`.
+
+version: "3.8"
+services:
+  r1_should_work:
+    image: redis:alpine
+    deploy:
+      update_config:
+        failure_action: rollback
+        order: start-first
+      rollback_config:
+        order: start-first
+      restart_policy:
+        max_attempts: 1
+    healthcheck:
+      test: redis-cli ping
+      interval: 2s
+      retries: 3
+      start_period: 1s
+      timeout: 3s
+
+  r2_broken_health_check:
+    image: redis:alpine
+    deploy:
+      update_config:
+        failure_action: rollback
+        order: start-first
+      rollback_config:
+        order: start-first
+      restart_policy:
+        max_attempts: 3
+    healthcheck:
+      test: foobar
+      interval: 2s
+      retries: 3
+      start_period: 1s
+      timeout: 3s
+
+  r3_no_health_check:
+    image: redis:alpine
+    deploy:
+      update_config:
+        failure_action: rollback
+        order: start-first
+      rollback_config:
+        order: start-first
+      restart_policy:
+        max_attempts: 3
+
+  r4_disabled_health_check:
+    image: redis:alpine
+    deploy:
+      update_config:
+        failure_action: rollback
+        order: start-first
+      rollback_config:
+        order: start-first
+      restart_policy:
+        max_attempts: 3
+    healthcheck:
+      disable: true
+
+  r5_should_also_work:
+    image: redis:alpine
+    deploy:
+      update_config:
+        failure_action: rollback
+        order: start-first
+      rollback_config:
+        order: start-first
+      restart_policy:
+        max_attempts: 1
+    healthcheck:
+      test: redis-cli ping
+      interval: 2s
+      retries: 3
+      start_period: 1s
+      timeout: 3s
--- a/tests/default/works.env
+++ b/tests/default/works.env
@ -0,0 +1 @@
+TYPE=works