Possible fix to invalid length "20"?

Improve recipe new subcommand
Update change log
2021-05-04 08:10:54 +02:00 · 2021-05-03 03:22:32 +02:00 · 2021-05-02 22:18:04 +02:00 · 2021-05-02 22:14:22 +02:00 · 2021-05-02 22:14:15 +02:00 · 2021-05-02 22:11:12 +02:00
18 changed files with 4307 additions and 427 deletions
--- a/.drone.yml
+++ b/.drone.yml
@ -3,21 +3,25 @@ kind: pipeline
 name: linters
 steps:
  - name: run shellcheck
-    image: koalaman/shellcheck-alpine:latest
+    image: koalaman/shellcheck-alpine:v0.7.1
    commands:
      - shellcheck abra
      - shellcheck bin/*.sh
  - name: run flake8
    image: alpine/flake8:3.9.0
    commands:
      - flake8 --max-line-length 100 bin/app-json.py
  - name: run unit tests
-    image: docker:dind
+    image: decentral1se/docker-dind-bats-kcov
    commands:
      - apk add bats git bash
      - bats tests
  - name: collect code coverage
    failure: ignore # until we fix this
-    image: kcov/kcov:latest
+    image: decentral1se/docker-dind-bats-kcov
    commands:
      - apk add bats git bash
      - kcov . bats tests || true
  - name: send code coverage report to codecov
@ -28,6 +32,18 @@ steps:
        from_secret: codecov_token
        required: true
  - name: notify rocket chat
    image: plugins/slack
    settings:
      webhook:
        from_secret: rc_builds_url
      username: comradebritney
      channel: "internal.builds"
      template: "{{repo.owner}}/{{repo.name}} build failed: {{build.link}}"
    when:
      status:
        - failure
 trigger:
  branch:
    - main
--- a/.gitignore
+++ b/.gitignore
@ -1,2 +1,2 @@
 coverage/
 /.venv
 coverage/
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@ -1,6 +1,114 @@
 > 🔥 🔥 🔥 Please note, while we are still in
 > [public alpha](https://docs.cloud.autonomic.zone/roadmap/), the `abra` release
 > versioning scheme is not following [semver](https://semver.org/) conventions
 > because we are still in the exploratory phases of building this tool. Please
 > read the changes before upgrading your `abra` installation as there are
 > **most likely** breaking changes coming each release. Sorry for any
 > inconvenience caused, we're working hard to make this tool stable. Semver
 > will be respected when we reach public beta. 🔥 🔥 🔥
 # abra x.x.x (UNRELEASED)
- ???
+- Sort `apps.json` when publishing ([39a7fc0](https://git.autonomic.zone/coop-cloud/abra/commit/39a7fc04fb5df1a6d78b84f51838530ab3eb76db))
 - Fix publishing of rating for new apps ([0e28af9](https://git.autonomic.zone/coop-cloud/abra/commit/0e28af9eb1af6c6da705b4614ddd173c60576629))
 - Detect compose filenames in `n+1` release generation ([ffc569e](https://git.autonomic.zone/coop-cloud/abra/commit/ffc569e275df7ca784a4db1a3331e17975fd8c87))
 # abra 0.7.3 (2021-04-28)
 - Only check for pw(q)gen if we're actually trying to use them ([#147](https://git.autonomic.zone/coop-cloud/abra/issues/147))
 - Use apps.coopcloud.tech for app data hosting & download ([75bd599](https://git.autonomic.zone/coop-cloud/abra/commit/75bd599))
 - Choose latest commit messages for new tags ([#144](https://git.autonomic.zone/coop-cloud/abra/issues/144))
 - Handle recipes without an `app` service in `recipe .. release` ([#151](https://git.autonomic.zone/coop-cloud/abra/issues/151))
 # abra 0.7.2 (2021-04-07)
 - Fix installation script development installs (again! Thanks Bash!) ([4747d9b7](https://git.autonomic.zone/coop-cloud/abra/commit/4747d9b7fb5fba914f210b6570bfe2db0b53da23))
 # abra 0.7.1 (2021-04-07)
 - Fix installation script development installs ([8f2fadb3c](https://git.autonomic.zone/coop-cloud/abra/commit/8f2fadb3c43c5915520f5ea531ea3815c2ba8531))
 # abra 0.7.0 (2021-04-07)
 - Add `--force` to the `deploy` command to allow overriding deployment logic ([#105](https://git.autonomic.zone/coop-cloud/abra/issues/105))
 - Handle undeployed apps in version summaries when deploying ([#104](https://git.autonomic.zone/coop-cloud/abra/issues/104))
 - Add `--force` to `undeploy` command ([e5e98d5](https://git.autonomic.zone/coop-cloud/abra/commit/e5e98d5))
 - Rename "app type" back to "stack" in the deployment overview ([54b6acc](https://git.autonomic.zone/coop-cloud/abra/commit/54b6acc))
 - Show context connection details on `abra server ls` ([#110](https://git.autonomic.zone/coop-cloud/abra/issues/110))
 - Allow to debug the SSH connection details on swarm init ([#109](https://git.autonomic.zone/coop-cloud/abra/issues/109))
 - Show correct status for apps deployed on servers with missing context ([#99](https://git.autonomic.zone/coop-cloud/abra/issues/99))
 - Search for subcommands in descending order of how many components there are ([#108](https://git.autonomic.zone/coop-cloud/abra/issues/108))
 - Add specific app version checking command (`abra app <app> version`) ([#108](https://git.autonomic.zone/coop-cloud/abra/issues/108))
 - Add docker version check (guestimating < v19 is a bad idea) ([#15](https://git.autonomic.zone/coop-cloud/abra/issues/15))
 - Fix git branch handling when not passing `-b <branch>` ([#122](https://git.autonomic.zone/coop-cloud/abra/issues/122))
 - Add work-around to correctly git clone non-master default branch app repositories ([#122](https://git.autonomic.zone/coop-cloud/abra/issues/122))
 - Replace `--force` (except for the `deploy` command) with a global `--no-prompt` for avoiding interactive questions ([#118](https://git.autonomic.zone/coop-cloud/abra/issues/118))
 - Use [docker-stack-wait-deploy](https://github.com/vitalets/docker-stack-wait-deploy) inspired logic to deploy apps ([#116](https://git.autonomic.zone/coop-cloud/abra/issues/116))
 - Add a domain polling check when deploying apps ([#113](https://git.autonomic.zone/coop-cloud/abra/issues/113))
 - Recognise when apps are already undeployed with `abra app <app> undeploy` ([#123](https://git.autonomic.zone/coop-cloud/abra/issues/123))
 - Add `abra doctor` command to help diagnose setup issues ([#119](https://git.autonomic.zone/coop-cloud/abra/issues/119))
 - Add apps version and feature catalogue generation script ([#121](https://git.autonomic.zone/coop-cloud/abra/issues/121))
 - New `--skip-version-check` option to `deploy` ([df4e504](https://git.autonomic.zone/coop-cloud/abra/commit/df4e504))
 - Look up local available version from compose files instead of `abra.sh` ([#131](https://git.autonomic.zone/coop-cloud/abra/issues/131))
 - Improve domain polling logging and allow to skip the check altogether with `--no-domain-poll` ([#140](https://git.autonomic.zone/coop-cloud/abra/issues/140), [#141](https://git.autonomic.zone/coop-cloud/abra/issues/141))
 - Support `ABRA_DIR` in the installer script ([4e94a424e94a42](https://git.autonomic.zone/coop-cloud/abra/commit/4e94a424e94a42))
 - Support [abra-hetzner](https://git.autonomic.zone/coop-cloud/abra-hetzner) plugin ([#88](https://git.autonomic.zone/coop-cloud/abra/issues/88))
 # abra 0.6.0 (2021-03-17)
 - Show version and digest of app if labelled ([98e674b8e8](https://git.autonomic.zone/coop-cloud/abra/commit/98e674b8e83458a83dcbf331e8e34c7188559c4a))
 - Implement basic version checking on deployment ([#82](https://git.autonomic.zone/coop-cloud/abra/issues/82))
 - New `app-catalogue.sh` script to auto-generate app list for documentation ([f163d4b](https://git.autonomic.zone/coop-cloud/abra/commit/f163d4b0fa920232e9d995a22d20fe78b174b3a9))
 - Support app service rollbacks with `abra <app> rollback <service>` ([#76](https://git.autonomic.zone/coop-cloud/abra/issues/76))
 - Detect when latest version is deployed and perform a no-op ([#87](https://git.autonomic.zone/coop-cloud/abra/issues/87))
 - Allow cloning of app repos with different main branches using `-b, --branch=<branch>` ([#80](https://git.autonomic.zone/coop-cloud/abra/issues/80))
 - Protect against lengthy app names which gives Docker trouble later on ([#83](https://git.autonomic.zone/coop-cloud/abra/issues/83))
 - Support removal of secrets and volumes when `rm`'ing apps ([#44](https://git.autonomic.zone/coop-cloud/abra/issues/44))
 - Always choose the default IPv4 address with `abra server <host> init` ([#91](https://git.autonomic.zone/coop-cloud/abra/issues/91))
 - Add `--type=<type>` filtering option to `abra <app> ls` ([0828189](https://git.autonomic.zone/coop-cloud/abra/commit/0828189))
 - Check for bash 4+ ([#96](https://git.autonomic.zone/coop-cloud/abra/commit/0828189))
 - Add `--dev` option to installer using `git clone` ([88d2a75](https://git.autonomic.zone/coop-cloud/abra/commit/88d2a75))
 - Support `--dev` on the `abra upgrade` command also ([bcc15ec](https://git.autonomic.zone/coop-cloud/abra/commit/bcc15ec))
 - Vendor [yq](https://github.com/mikefarah/yq/releases) automatically ([3b59adf](https://git.autonomic.zone/coop-cloud/abra/commit/3b59adf))
 - Extend version handling logic to support all underlying services ([#90](https://git.autonomic.zone/coop-cloud/abra/issues/90))
 - Fix development installation script symlink issue ([#98](https://git.autonomic.zone/coop-cloud/abra/issues/98))
 - Add `app-version.sh` script to help packagers version apps ([28618bd](https://git.autonomic.zone/coop-cloud/abra/commit/28618bd))
 - Add git digest to `abra version` output ([8b41416](https://git.autonomic.zone/coop-cloud/abra/commit/8b41416))
 # abra 0.5.0 (2021-03-01)
 - `secret auto` merged into `secret generate` and `app new --auto` is now `app new --secrets` ([#64](https://git.autonomic.zone/coop-cloud/abra/pulls/64))
 - Avoid outputting length during secret generation when not in use ([#67](https://git.autonomic.zone/coop-cloud/abra/issues/67))
 - Support graceful failure when missing secret generation commands ([44d3ac3](https://git.autonomic.zone/coop-cloud/abra/commit/44d3ac3a1cb86edc9b9e91eea1a00e70eae14965))
 - Fix secret detection when using new `.env` file format in apps ([5532452](https://git.autonomic.zone/coop-cloud/abra/commit/55324524ca77141666ffe6cc41b62cc71cf89ace))
 - Support choosing an `$EDITOR` when editing configs ([29cc392](https://git.autonomic.zone/coop-cloud/abra/commit/29cc392dff3e93e48e0e2edd3ce11b405c66a95a))
 - "server" shell completion fixed ([8839bd4](https://git.autonomic.zone/coop-cloud/abra/commit/8839bd45951d00dccf4ef81ece445bcc49e13ee6))
 - Drop `multilogs` command ([#56](https://git.autonomic.zone/coop-cloud/abra/pulls/56))
 - Remove `server use` command ([#51](https://git.autonomic.zone/coop-cloud/abra/issues/51))
 - `new <app>` becomes `new <type>` ([#48](https://git.autonomic.zone/coop-cloud/abra/issues/48))
 - `check` is run on `deploy` now and configurable ([77ba565](https://git.autonomic.zone/coop-cloud/abra/commit/77ba5652b2fe15820f5edfa0f642636f7b8eae7e))
 - App configurations are always updated now ([#42](https://git.autonomic.zone/coop-cloud/abra/issues/42))
 - We use docker format `.env` files (no "export" syntax) from now now ([#55](https://git.autonomic.zone/coop-cloud/abra/pulls/55))
 - Rename `<domain>` option to `<app>` and `APP` variable to `TYPE`, see ([#47](https://git.autonomic.zone/coop-cloud/abra/issues/47))
 - Use Docker-in-Docker (dind), and `dind-bats-kcov` Docker image, for `make test` ([1600b62](https://git.autonomic.zone/coop-cloud/abra/commit/1600b6277fbbffc4c6de1e4ba799c7bbe72ec6a0))
 - Add built-in documentation using `abra help <subcommand>...`, see ([#50](https://git.autonomic.zone/coop-cloud/abra/issues/50))
 - `version` subcommand ([e6b24fe](https://git.autonomic.zone/coop-cloud/abra/commit/e6b24fe))
 - Use `# length=x` comments to generate passwords with `pwgen` and drop `KEY`/`PASSWORD` logic ([#68](https://git.autonomic.zone/coop-cloud/abra/issues/68))
 - Global `--skip-update|-U` / `--skip-check|-C` options to make things quicker ([37e8b00](https://git.autonomic.zone/coop-cloud/abra/commit/37e8b00))
 - `app backup` and `app restore` commands; requires per-app definition ([#70](https://git.autonomic.zone/coop-cloud/abra/issues/70))
 - Rename per-type `abra-commands.sh` to `abra.sh`, and include config versions as type-level instead of app-level config ([#43](https://git.autonomic.zone/coop-cloud/abra/issues/43))
 - Show per-subcommand help by adding `-h/--help` to a command line ([#38](https://git.autonomic.zone/coop-cloud/abra/issues/78))
 # abra 0.4.1 (2020-12-24)
 - Bug-fixes on `app ls --status` & custom commands
 - Add `app ls --server=...` and alias
 # abra 0.4.0 (2020-12-24)
 - New command-line interface based on docopt
 - `~/.abra` directory instead of expecting local `.env` files
 - Integration tests & code coverage
 # abra 0.3.1 (2020-09-27)
@ -8,7 +116,7 @@
 # abra 0.3.0 (2020-09-27)
- Add multilogs stack logs implementation ([#8](https://git.autonomic.zone/compose-stacks/abra/issues/8)
+- Add multilogs stack logs implementation ([#8](https://git.autonomic.zone/compose-stacks/abra/issues/8))
 - Add beginnings of "monorepo" functionality
 # abra 0.2.0 (2020-09-24)
--- a/38
+++ b/38
@ -1,38 +0,0 @@
 .PHONY: test shellcheck docopt kcov codecov
 test:
 	@DOCKER_CONTEXT=default docker run \
 		-it \
 		--rm \
 		-v $$(pwd):/workdir \
 		docker:dind \
 		sh -c "apk add bats git bash && cd /workdir && bats /workdir/test.bats"
 shellcheck:
 	@docker run \
 		-it \
 		--rm \
 		-v $$(pwd):/workdir \
 		koalaman/shellcheck-alpine \
 		shellcheck /workdir/abra
 docopt:
 	@if [ ! -d ".venv" ]; then \
 		python3 -m venv .venv && \
 		.venv/bin/pip install -U pip setuptools wheel && \
 		.venv/bin/pip install docopt-sh; \
 	fi
 	.venv/bin/docopt.sh abra
 kcov:
 	@docker run \
 		-it \
 		--rm \
 		-v $$(pwd):/workdir \
 		kcov/kcov:latest \
 		sh -c "kcov /workdir/coverage /workdir/abra || true"
 codecov: SHELL:=/bin/bash
 codecov:
 	@bash <(curl -s https://codecov.io/bash) \
 		-s coverage -t $$(pass show hosts/swarm.autonomic.zone/drone/codecov/token)
--- a/README.md
+++ b/README.md
@ -3,12 +3,91 @@
 [![Build Status](https://drone.autonomic.zone/api/badges/coop-cloud/abra/status.svg)](https://drone.autonomic.zone/coop-cloud/abra)
 [![codecov](https://codecov.io/gh/Autonomic-Cooperative/abra/branch/main/graph/badge.svg?token=aX3I5NMRsj)](undefined)
-> https://cloud.autonomic.zone
+> https://coopcloud.tech
 The cooperative cloud utility belt 🎩🐇
 `abra` is a command-line tool for managing your own [Co-op Cloud](https://coopcloud.tech). It can provision new servers, create applications, deploy them, run backup and restore operations and a whole lot of other things. It is the go-to tool for day-to-day operations when managing a Co-op Cloud instance.
 ## Change log
 > 🔥 🔥 🔥 Please note, while we are still in [public
 > alpha](https://docs.coopcloud.tech/roadmap/), the `abra` release
 > versioning scheme is not following [semver](https://semver.org/) conventions
 > because we are still in the exploratory phases of building this tool. Please
 > read the changes before upgrading your `abra` installation as there are
 > **most likely** breaking changes coming each release. Sorry for any
 > inconvenience caused, we're working hard to make this tool stable. Semver
 > will be respected when we reach public beta. 🔥 🔥 🔥
 See [CHANGELOG.md](./CHANGELOG.md).
 ## Documentation
 > [docs.coopcloud.tech](https://docs.coopcloud.tech)
 ## Install
 Install the latest stable release:
 ```sh
 curl https://install.abra.autonomic.zone | bash
 ```
 or the bleeding-edge development version:
 ```sh
 curl https://install.abra.autonomic.zone | bash -s -- --dev
 ```
 The source for this script is [here](./deploy/install.abra.autonomic.zone/installer).
 ## Update
 Run `abra upgrade` to automatically download and install the latest release
 version.
 To update the development version, run `abra upgrade --dev`.
 ## Hack
 It's written in Bash version 4 or greater!
 Install it via `curl https://install.abra.autonomic.zone | bash -s -- --dev`, then you can hack on the source in `~/.abra/src`.
 The command-line interface is generated via [docopt](http://docopt.org/). If you add arguments then you need to run `make docopt` ro regenerate the parser.
 Please remember to update the [CHANGELOG](./CHANGELOG.md) when you make a change.
 ## Generating a new apps.json
 You'll need to install the following requirements:
 - [requests](https://docs.python-requests.org/en/master/) (`apt install python3-requests` / `pip install requests`)
 - [skopeo](https://github.com/containers/skopeo) (check [the install docs](https://github.com/containers/skopeo/blob/master/install.md))
 - [jq](https://stedolan.github.io/jq/tutorial/) (`sudo apt-get install jq` or see [the install docs](https://stedolan.github.io/jq/download/))
 - [yq](https://mikefarah.gitbook.io/yq/) (see [the install docs](https://mikefarah.gitbook.io/yq/#install))
 Then run `./bin/app-json.py` ([source](./bin/app-json.py)) and it will spit out the JSON file into [deploy/apps.coopcloud.tech/apps.json](./deploy/apps.coopcloud.tech/apps.json).
 ## Releasing
 ### `abra`
 > [install.abra.autonomic.zone](https://install.abra.autonomic.zone)
 - Change the `x.x.x` header in [CHANGELOG.md](./CHANGELOG.md) to reflect new version and mark date
 - Update the version in [abra](./abra)
 - Update the version in [deploy/install.abra.autonomic.zone/installer](./deploy/install.abra.autonomic.zone/installer)
 - `git commit` the above changes and then tag it with `git tag <your-new-version>`
 - `git push` and `git push --tags`
 - Deploy a new installer script `make release-installer`
 - Tell the world (CoTech forum, Matrix public channel, Autonomic mastodon, etc.)
 ### apps.coopcloud.tech
 > [apps.coopcloud.tech](https://apps.coopcloud.tech)
 ```bash
 $ make release-apps
 ```
--- a/2089
+++ b/2089
--- a/bin/app-catalogue.sh
+++ b/bin/app-catalogue.sh
@ -0,0 +1,103 @@
 #!/bin/bash
 # shellcheck disable=SC2119
 # Usage: ./app-catalogue.sh
 #
 # Gather metadata from Co-op Cloud apps in $ABRA_DIR/apps (default
 # ~/.abra/apps), and format it as a Markdown table for this page:
 # https://docs.cloud.autonomic.zone/apps/
 stack_dir="${ABRA_DIR:-$HOME/.abra}/apps/"
 cd "$stack_dir" || exit
 # load all README files into ENV_FILES array
 mapfile -t readmes < <(find -L . -name "README.md")
 # FIXME 3wc: requires bash 4, use for loop instead
 base_url="https://git.autonomic.zone/coop-cloud"
 cat_apps=()
 cat_development=()
 cat_utilities=()
 cat_graveyard=()
 get_var() {
 	echo "$1" | grep "$2" | sed 's/^[^:]*: //'
 }
 # shellcheck disable=SC2120
 trim() {
 	# accept input as argument or from STDIN, see here:
 	# https://zwbetz.com/passing-input-to-a-bash-function-via-arguments-or-stdin/
 	# shellcheck disable=SC2155
 	local input="$([[ -p /dev/stdin ]] && cat - || echo "$@")"
 	[[ -z "$input" ]] && return 1
 	echo "$input" | tr -d ' '
 }
 # shellcheck disable=SC2120
 prettify() {
 	# as above
 	# shellcheck disable=SC2155
 	local input="$([[ -p /dev/stdin ]] && cat - || echo "$@")"
 	[[ -z "$input" ]] && return 1
 	echo "$input" | sed -e 's/Yes/✅/' -e 's/No/❌/' -e 's/N\/A/⛔/'
 }
 for readme in "${readmes[@]}"; do
 	type="$(basename "${readme%README.md}")"
 	if [ "$type" = "example" ]; then
 		continue
 	fi
 	title="$(grep '^# ' "$type/README.md" | sed 's/^# //' )"
 	# find section between 'metadata' and 'endmetadata' comments
 	metadata="$(awk '/-- metadata --/,/-- endmetadata --/' "$type/README.md")"
 	status="$(get_var "$metadata" "Status")"
 	category="$(get_var "$metadata" "Category" | cut -d',' -f2 | trim)"
 	if [ -z "$category" ]; then
 		echo "ERROR: missing category for $type"
 		continue
 	fi
 	image="$(get_var "$metadata" "Image" | cut -d',' -f2 | trim)"
 	healthcheck="$(get_var "$metadata" "Healthcheck" | prettify)"
 	backups="$(get_var "$metadata" "Backups" | prettify)"
 	email="$(get_var "$metadata" "Email" | prettify)"
 	tests="$(get_var "$metadata" "Tests" | prettify)"
 	sso="$(get_var "$metadata" "SSO" | prettify)"
 	row="| [$title]($base_url/$type) | $status | $image | $healthcheck | $backups | $email | $tests | $sso |"
 	category_lower="$(echo "$category" | tr '[:upper:]' '[:lower:]')"
 	eval "cat_$category_lower+=( '$row' )"
 done
 headers="
 | **Name** | **Status** | **Image** | **Healtcheck** | **Backups** | **Email** | **CI** | **Single-Sign-On** |
 | --- | --- | --- | --- | --- | --- | --- | --- |"
 echo "## Applications"
 echo "$headers"
 printf '%s\n' "${cat_apps[@]}" | sort
 echo
 echo "## Developer tools"
 echo "$headers"
 printf '%s\n' "${cat_development[@]}" | sort
 echo
 echo "## Utilities"
 echo "$headers"
 printf '%s\n' "${cat_utilities[@]}" | sort
 echo
 echo "## Graveyard"
 echo "$headers"
 printf '%s\n' "${cat_graveyard[@]}" | sort
--- a/bin/app-json.py
+++ b/bin/app-json.py
@ -0,0 +1,298 @@
 #!/usr/bin/env python3
 # Usage: ./app-json.py
 #
 # Gather metadata from Co-op Cloud apps in $ABRA_DIR/apps (default
 # ~/.abra/apps), and format it as JSON so that it can be hosted here:
 # https://apps.coopcloud.tech
 from json import dump
 from logging import DEBUG, basicConfig, getLogger
 from os import chdir, listdir, mkdir
 from os.path import basename, exists, expanduser
 from pathlib import Path
 from re import findall, search
 from shlex import split
 from subprocess import DEVNULL, check_output
 from sys import exit
 from requests import get
 HOME_PATH = expanduser("~/")
 CLONES_PATH = Path(f"{HOME_PATH}/.abra/apps").absolute()
 YQ_PATH = Path(f"{HOME_PATH}/.abra/vendor/yq")
 SCRIPT_PATH = Path(__file__).absolute().parent
 REPOS_TO_SKIP = (
    "abra",
    "abra-apps",
    "abra-gandi",
    "abra-hetzner",
    "backup-bot",
    "coopcloud.tech",
    "coturn",
    "docker-cp-deploy",
    "docker-dind-bats-kcov",
    "docs.coopcloud.tech",
    "example",
    "gardening",
    "organising",
    "pyabra",
    "radicle-seed-node",
    "stack-ssh-deploy",
    "swarm-cronjob",
 )
 log = getLogger(__name__)
 basicConfig()
 log.setLevel(DEBUG)
 def _run_cmd(cmd, shell=False, **kwargs):
    """Run a shell command."""
    args = [split(cmd)]
    if shell:
        args = [cmd]
        kwargs = {"shell": shell}
    try:
        return check_output(*args, **kwargs).decode("utf-8").strip()
    except Exception as exception:
        log.error(f"Failed to run {cmd}, saw {str(exception)}")
        exit(1)
 def get_repos_json():
    """ Retrieve repo list from Gitea """
    url = "https://git.autonomic.zone/api/v1/orgs/coop-cloud/repos"
    log.info(f"Retrieving {url}")
    repos = []
    response = True
    page = 1
    try:
        while response:
            log.info(f"Trying to fetch page {page}")
            response = get(url + f"?page={page}", timeout=10).json()
            repos.extend(response)
            page += 1
        return repos
    except Exception as exception:
        log.error(f"Failed to retrieve {url}, saw {str(exception)}")
        exit(1)
 def get_published_apps_json():
    """Retrieve already published apps json."""
    url = "https://apps.coopcloud.tech"
    log.info(f"Retrieving {url}")
    try:
        return get(url, timeout=5).json()
    except Exception as exception:
        log.error(f"Failed to retrieve {url}, saw {str(exception)}")
        return {}
 def clone_all_apps(repos_json):
    """Clone all Co-op Cloud apps to ~/.abra/apps."""
    if not exists(CLONES_PATH):
        mkdir(CLONES_PATH)
    repos = [[p["name"], p["ssh_url"]] for p in repos_json]
    for name, url in repos:
        if name in REPOS_TO_SKIP:
            continue
        if not exists(f"{CLONES_PATH}/{name}"):
            log.info(f"Retrieving {url}")
            _run_cmd(f"git clone {url} {CLONES_PATH}/{name}")
            chdir(f"{CLONES_PATH}/{name}")
            if not int(_run_cmd("git branch --list | wc -l", shell=True)):
                log.info(f"Guessing main branch is HEAD for {name}")
                _run_cmd("git checkout main")
        else:
            log.info(f"Updating {name}")
            chdir(f"{CLONES_PATH}/{name}")
            _run_cmd("git fetch -a")
 def generate_apps_json(repos_json):
    """Generate the abra-apps.json application versions file."""
    apps_json = {}
    cached_apps_json = get_published_apps_json()
    for app in listdir(CLONES_PATH):
        if app in REPOS_TO_SKIP:
            log.info(f"Skipping {app}")
            continue
        repo_details = next(filter(lambda x: x["name"] == app, repos_json), {})
        app_path = f"{CLONES_PATH}/{app}"
        chdir(app_path)
        metadata = get_app_metadata(app_path)
        name = metadata.pop("name", "")
        log.info(f"Processing {app}")
        apps_json[app] = {
            "name": name,
            "category": metadata.get("category", ""),
            "repository": repo_details.get("clone_url", ""),
            "default_branch": repo_details.get("default_branch", ""),
            "description": repo_details.get("description", ""),
            "website": repo_details.get("website", ""),
            "features": metadata,
            "versions": get_app_versions(app_path, cached_apps_json),
            "icon": repo_details.get("avatar_url", ""),
        }
    return apps_json
 def get_app_metadata(app_path):
    """Parse metadata from app repo README files."""
    metadata = {}
    chdir(app_path)
    try:
        with open(f"{app_path}/README.md", "r") as handle:
            log.info(f"{app_path}/README.md")
            contents = handle.read()
    except Exception:
        log.info(f"No {app_path}/README.md discovered, moving on")
        return {}
    try:
        for match in findall(r"\*\*.*\s\*", contents):
            title = search(r"(?<=\*\*).*(?=\*\*)", match).group().lower()
            if title == "image":
                value = {
                    "image": search(r"(?<=`).*(?=`)", match).group(),
                    "url": search(r"(?<=\().*(?=\))", match).group(),
                    "rating": match.split(",")[1].strip(),
                    "source": match.split(",")[-1].replace("*", "").strip(),
                }
            elif title == "status":
                value = {"❶💚": 1, "❷💛": 2, "❸🍎": 3, "❹💣": 4, "?": 5, "": 5}[
                    match.split(":")[-1].replace("*", "").strip()
                ]
            else:
                value = match.split(":")[-1].replace("*", "").strip()
            metadata[title] = value
        metadata["name"] = findall(r"^# (.*)", contents)[0]
    except (IndexError, AttributeError):
        log.info(f"Can't parse {app_path}/README.md")
        return {}
    finally:
        _run_cmd("git checkout HEAD")
    log.info(f"Parsed {metadata}")
    return metadata
 def get_app_versions(app_path, cached_apps_json):
    versions = {}
    chdir(app_path)
    tags = _run_cmd("git tag --list").split()
    if not tags:
        log.info("No tags discovered, moving on")
        return {}
    initial_branch = _run_cmd("git rev-parse --abbrev-ref HEAD")
    app_name = basename(app_path)
    try:
        existing_tags = cached_apps_json[app_name]["versions"].keys()
    except KeyError:
        existing_tags = []
    for tag in tags:
        _run_cmd(f"git checkout {tag}", stderr=DEVNULL)
        services_cmd = f"{YQ_PATH} e '.services | keys | .[]' compose*.yml"
        services = _run_cmd(services_cmd, shell=True).split()
        parsed_services = []
        service_versions = {}
        for service in services:
            if service in ("null", "---"):
                continue
            if (
                tag in existing_tags
                and service in cached_apps_json[app_name]["versions"][tag]
            ):
                log.info(f"Skipping {tag} because we've already processed it")
                existing_versions = cached_apps_json[app_name]["versions"][tag][service]
                service_versions[service] = existing_versions
                _run_cmd(f"git checkout {initial_branch}")
                continue
            if service in parsed_services:
                log.info(f"Skipped {service}, we've already parsed it locally")
                continue
            services_cmd = f"{YQ_PATH} e '.services.{service}.image' compose*.yml"
            images = _run_cmd(services_cmd, shell=True).split()
            for image in images:
                if image in ("null", "---"):
                    continue
                images_cmd = f"skopeo inspect docker://{image} | jq '.Digest'"
                output = _run_cmd(images_cmd, shell=True)
                service_version_info = {
                    "image": image.split(":")[0],
                    "tag": image.split(":")[-1],
                    "digest": output.split(":")[-1][:8],
                }
                log.info(f"Parsed {service_version_info}")
                service_versions[service] = service_version_info
            parsed_services.append(service)
        versions[tag] = service_versions
    _run_cmd(f"git checkout {initial_branch}")
    return versions
 def main():
    """Run the script."""
    repos_json = get_repos_json()
    clone_all_apps(repos_json)
    target = f"{SCRIPT_PATH}/../deploy/apps.coopcloud.tech/apps.json"
    with open(target, "w", encoding="utf-8") as handle:
        dump(
            generate_apps_json(repos_json),
            handle,
            ensure_ascii=False,
            indent=4,
            sort_keys=True,
        )
    log.info(f"Successfully generated {target}")
 main()
--- a/completion/abra.bash
+++ b/completion/abra.bash
@ -91,7 +91,7 @@ _abra_complete()
        case "$cmd" in
            server)
                # Offer exactly one server name completion.
-                if (( COMP_CWORD == cmd_index + 2 )); then
+                if (( COMP_CWORD == cmd_index + 1 )); then
                    _abra_complete_servers "$cur"
                fi
                ;;
--- a/deploy/apps.coopcloud.tech/apps.json
+++ b/deploy/apps.coopcloud.tech/apps.json
--- a/deploy/apps.coopcloud.tech/compose.yml
+++ b/deploy/apps.coopcloud.tech/compose.yml
@ -0,0 +1,41 @@
 ---
 version: "3.8"
 services:
  app:
    image: "nginx:stable"
    configs:
      - source: abra_conf
        target: /etc/nginx/conf.d/abra.conf
      - source: abra_apps_json
        target: /var/www/abra-apps/apps.json
    volumes:
      - "public:/var/www/abra-apps"
    networks:
      - proxy
    deploy:
      update_config:
        failure_action: rollback
        order: start-first
      labels:
        - "traefik.enable=true"
        - "traefik.http.services.abra-apps.loadbalancer.server.port=80"
        - "traefik.http.routers.abra-apps.rule=Host(`apps.coopcloud.tech`, `abra-apps.cloud.autonomic.zone`)"
        - "traefik.http.routers.abra-apps.entrypoints=web-secure"
        - "traefik.http.routers.abra-apps.tls.certresolver=production"
        - "traefik.http.routers.abra-apps.middlewares=abra-apps-redirect"
        - "traefik.http.middlewares.abra-apps-redirect.headers.SSLForceHost=true"
        - "traefik.http.middlewares.abra-apps-redirect.headers.SSLHost=apps.coopcloud.tech"
 configs:
  abra_apps_json:
    file: apps.json
  abra_conf:
    file: nginx.conf
 networks:
  proxy:
    external: true
 volumes:
  public:
--- a/deploy/apps.coopcloud.tech/nginx.conf
+++ b/deploy/apps.coopcloud.tech/nginx.conf
@ -0,0 +1,10 @@
 server {
  listen         80 default_server;
  server_name    apps.coopcloud.tech;
  location / {
    root /var/www/abra-apps;
    add_header Content-Type application/json;
    index apps.json;
  }
 }
--- a/deploy/install.abra.autonomic.zone/compose.yml
+++ b/deploy/install.abra.autonomic.zone/compose.yml
@ -2,7 +2,7 @@
 version: "3.8"
 services:
-  abra_installer:
+  app:
    image: "nginx:stable"
    configs:
      - source: abra_conf
@ -26,10 +26,8 @@ services:
 configs:
  abra_installer:
    name: abra_installer_v1
    file: installer
  abra_conf:
    name: abra_conf_v1
    file: nginx.conf
 networks:
--- a/deploy/install.abra.autonomic.zone/installer
+++ b/deploy/install.abra.autonomic.zone/installer
@ -0,0 +1,35 @@
 #!/bin/bash
 ABRA_VERSION="0.7.3"
 GIT_URL="https://git.autonomic.zone/coop-cloud/abra"
 ABRA_SRC="$GIT_URL/raw/tag/$ABRA_VERSION/abra"
 ABRA_DIR="${ABRA_DIR:-$HOME/.abra/}"
 function install_abra_release {
  mkdir -p "$HOME/.local/bin"
  curl "$ABRA_SRC" > "$HOME/.local/bin/abra"
  chmod +x "$HOME/.local/bin/abra"
  echo "abra installed to $HOME/.local/bin/abra"
 }
 function install_abra_dev {
  mkdir -p "$ABRA_DIR/"
  if [[ ! -d "$ABRA_DIR/src" ]]; then
    git clone "$GIT_URL" "$ABRA_DIR/src"
  fi
  (cd "$ABRA_DIR/src" && git pull origin main && cd - || exit)
  mkdir -p "$HOME/.local/bin"
  ln -sf "$ABRA_DIR/src/abra" "$HOME/.local/bin/abra"
  echo "abra installed to $HOME/.local/bin/abra (development bleeding edge)"
 }
 function run_installation {
  if [ "$1" = "--dev" ]; then
    install_abra_dev
  else
    install_abra_release
  fi
 }
 run_installation "$@"
 exit 0
--- a/deploy/install.abra.autonomic.zone/nginx.conf
+++ b/deploy/install.abra.autonomic.zone/nginx.conf
--- a/installer/installer
+++ b/installer/installer
@ -1,18 +0,0 @@
 #!/bin/bash
 ABRA_VERSION="0.3.1"
 ABRA_SRC="https://git.autonomic.zone/coop-cloud/abra/raw/tag/$ABRA_VERSION/abra"
 function install_abra {
  mkdir -p "$HOME/.local/bin"
 	curl "$ABRA_SRC" > "$HOME/.local/bin/abra"
  chmod +x "$HOME/.local/bin/abra"
  echo "abra installed to $HOME/.loca/bin/abra"
 }
 function run_installation {
  install_abra
 }
 run_installation
 exit 0
--- a/58
+++ b/58
@ -0,0 +1,58 @@
 .PHONY: test shellcheck docopt kcov codecov release-installer
 test:
 	@sudo DOCKER_CONTEXT=default docker run \
 		-v $$(pwd):/workdir \
 		--privileged \
 		-d \
 		--name=abra-test-dind \
 		-e DOCKER_TLS_CERTDIR="" \
 		decentral1se/docker-dind-bats-kcov
 	@DOCKER_CONTEXT=default docker exec \
 		-it \
 		abra-test-dind \
 		sh -c "cd /workdir && bats /workdir/tests"
 	@DOCKER_CONTEXT=default docker stop abra-test-dind
 	@DOCKER_CONTEXT=default docker rm abra-test-dind
 shellcheck:
 	@docker run \
 		-it \
 		--rm \
 		-v $$(pwd):/workdir \
 		koalaman/shellcheck-alpine \
 		shellcheck /workdir/abra && \
 		shellcheck /workdir/bin/*.sh
 docopt:
 	@if [ ! -d ".venv" ]; then \
 		python3 -m venv .venv && \
 		.venv/bin/pip install -U pip setuptools wheel && \
 		.venv/bin/pip install docopt-sh; \
 	fi
 	.venv/bin/docopt.sh abra
 kcov:
 	@docker run \
 		-it \
 		--rm \
 		-v $$(pwd):/workdir \
 		kcov/kcov:latest \
 		sh -c "kcov /workdir/coverage /workdir/abra || true"
 codecov: SHELL:=/bin/bash
 codecov:
 	@bash <(curl -s https://codecov.io/bash) \
 		-s coverage -t $$(pass show hosts/swarm.autonomic.zone/drone/codecov/token)
 release-installer:
 	@DOCKER_CONTEXT=swarm.autonomic.zone \
 		docker stack rm abra-installer-script && \
 		cd deploy/install.abra.autonomic.zone && \
 		DOCKER_CONTEXT=swarm.autonomic.zone docker stack deploy -c compose.yml abra-installer-script
 release-apps:
 	@DOCKER_CONTEXT=swarm.autonomic.zone \
 		docker stack rm abra-apps-json && \
 		cd deploy/apps.coopcloud.tech && \
 		DOCKER_CONTEXT=swarm.autonomic.zone docker stack deploy -c compose.yml abra-apps-json
--- a/tests/02_abra.bats
+++ b/tests/02_abra.bats
@ -1,49 +1,56 @@
 #!/usr/bin/env bats
 setup() {
-    mkdir -p ~/.abra/servers/default
+    export ABRA_DIR=$(mktemp -d)
    mkdir -p $ABRA_DIR/servers/default
 }
 teardown() {
-    rm -rf ~/.abra/servers/default
+    rm -rf "$ABRA_DIR"
    rm -rf ~/.abra/servers/swarm.test.com
 }
-@test "abra server add/rm works" {
+abra() {
-    ./abra server add swarm.test.com
+    ./abra -d $@
 }
@test "abra server (add|rm)" {
    abra server add swarm.test.com
    docker context ls | grep swarm.test.com
-    [ -d ~/.abra/servers/swarm.test.com ]
+    [ -d $ABRA_DIR/servers/swarm.test.com ]
-    ./abra server rm swarm.test.com
+    abra server swarm.test.com rm
-    ./abra server add swarm.test.com foobar 12345
+    abra server add swarm.test.com foobar 12345
-    [ -d ~/.abra/servers/swarm.test.com ]
+    [ -d $ABRA_DIR/servers/swarm.test.com ]
-    ./abra server rm swarm.test.com
+    abra server swarm.test.com rm
 }
-@test "abra app new/rm works" {
+@test "abra server init" {
-    ./abra app new --server default --domain traefik.test.com traefik
+    abra server default init
-    [ -f ~/.abra/servers/default/traefik.test.com.env ]
+}
@test "abra app (new|rm)" {
    abra app new --server default --domain traefik.test.com --app-name traefik_test_com traefik
    [ -f $ABRA_DIR/servers/default/traefik_test_com.env ]
    # interactive prompt
-    echo "y" | ./abra app traefik.test.com delete
+    echo "y" | abra app traefik_test_com delete
-    [ ! -f ~/.abra/servers/default/traefik.test.com.env ]
+    [ ! -f $ABRA_DIR/servers/default/traefik_test_com.env ]
-    # --force
+    # --no-prompt
-    ./abra app new --server default --domain traefik.test.com traefik
+    abra app new --server default --domain traefik_test_com --app-name traefik_test_com traefik
-    ./abra app traefik.test.com delete --force
+    abra --no-prompt app traefik_test_com delete
-    [ ! -f ~/.abra/servers/default/traefik.test.com.env ]
+    [ ! -f $ABRA_DIR/servers/default/traefik_test_com.env ]
 }
@test "abra app <domain> secret (insert|generate|rm)" {
-    # TODO 3wc: mock `server new` so we don't endlessly re-test it
+    abra app new --server default --domain traefik_test_com --app-name traefik_test_com traefik
    ./abra app new --server default --domain traefik.test.com traefik
-    ./abra app traefik.test.com secret insert foobar v1 "foobar"
+    abra app traefik_test_com secret insert foobar v1 "foobar"
    # interactive prompt
-    echo "y" | ./abra app traefik.test.com secret rm foobar
+    echo "y" | abra app traefik_test_com secret rm foobar
-    ./abra app traefik.test.com secret insert foobar v1 "foobar"
+    abra app traefik_test_com secret insert foobar v1 "foobar"
    #  prompt
-    ./abra app traefik.test.com secret rm foobar --force
+    abra --no-prompt app traefik_test_com secret rm foobar
 }