Lower Docker req to 19, make sure we test locally

Ref #15

``` $ docker --version
Docker version 19.03.11, build 42e35e6
```

But abra works. So I think 19 might be sufficient?

Also, wondering if we should add `DOCKER_CONTEXT=default` to that check,
to make sure we're not querying a remote server. (Semi-related question
- should we also check remote Docker versions?)

.drone.yml

@@ -6,12 +6,6 @@ steps:
     image: koalaman/shellcheck-alpine:v0.7.1
     commands:
       - shellcheck abra
-      - shellcheck bin/*.sh
-
-  - name: run flake8
-    image: alpine/flake8:3.9.0
-    commands:
-      - flake8 --max-line-length 100 bin/app-json.py
 
   - name: run unit tests
     image: decentral1se/docker-dind-bats-kcov

.gitignore

@@ -1,2 +1,2 @@
-/.venv
 coverage/
+/.venv

CHANGELOG.md

@@ -9,12 +9,6 @@
 
 # abra x.x.x (UNRELEASED)
 
-# abra 0.7.1 (2021-04-07)
-
-- Fix installation script development installs ([8f2fadb3c](https://git.autonomic.zone/coop-cloud/abra/commit/8f2fadb3c43c5915520f5ea531ea3815c2ba8531))
-
-# abra 0.7.0 (2021-04-07)
-
 - Add `--force` to the `deploy` command to allow overriding deployment logic ([#105](https://git.autonomic.zone/coop-cloud/abra/issues/105))
 - Handle undeployed apps in version summaries when deploying ([#104](https://git.autonomic.zone/coop-cloud/abra/issues/104))
 - Add `--force` to `undeploy` command ([e5e98d5](https://git.autonomic.zone/coop-cloud/abra/commit/e5e98d5))
@@ -24,20 +18,7 @@
 - Show correct status for apps deployed on servers with missing context ([#99](https://git.autonomic.zone/coop-cloud/abra/issues/99))
 - Search for subcommands in descending order of how many components there are ([#108](https://git.autonomic.zone/coop-cloud/abra/issues/108))
 - Add specific app version checking command (`abra app <app> version`) ([#108](https://git.autonomic.zone/coop-cloud/abra/issues/108))
-- Add docker version check (guestimating < v19 is a bad idea) ([#15](https://git.autonomic.zone/coop-cloud/abra/issues/15))
+- Add docker version check (guestimating < v20 is a bad idea) ([#15](https://git.autonomic.zone/coop-cloud/abra/issues/15))
-- Fix git branch handling when not passing `-b <branch>` ([#122](https://git.autonomic.zone/coop-cloud/abra/issues/122))
-- Add work-around to correctly git clone non-master default branch app repositories ([#122](https://git.autonomic.zone/coop-cloud/abra/issues/122))
-- Replace `--force` (except for the `deploy` command) with a global `--no-prompt` for avoiding interactive questions ([#118](https://git.autonomic.zone/coop-cloud/abra/issues/118))
-- Use [docker-stack-wait-deploy](https://github.com/vitalets/docker-stack-wait-deploy) inspired logic to deploy apps ([#116](https://git.autonomic.zone/coop-cloud/abra/issues/116))
-- Add a domain polling check when deploying apps ([#113](https://git.autonomic.zone/coop-cloud/abra/issues/113))
-- Recognise when apps are already undeployed with `abra app <app> undeploy` ([#123](https://git.autonomic.zone/coop-cloud/abra/issues/123))
-- Add `abra doctor` command to help diagnose setup issues ([#119](https://git.autonomic.zone/coop-cloud/abra/issues/119))
-- Add apps version and feature catalogue generation script ([#121](https://git.autonomic.zone/coop-cloud/abra/issues/121))
-- New `--skip-version-check` option to `deploy` ([df4e504](https://git.autonomic.zone/coop-cloud/abra/commit/df4e504)
-- Look up local available version from compose files instead of `abra.sh` ([#131](https://git.autonomic.zone/coop-cloud/abra/issues/131))
-- Improve domain polling logging and allow to skip the check altogether with `--no-domain-poll` ([#140](https://git.autonomic.zone/coop-cloud/abra/issues/140), [#141](https://git.autonomic.zone/coop-cloud/abra/issues/141))
-- Support `ABRA_DIR` in the installer script ([4e94a424e94a42](https://git.autonomic.zone/coop-cloud/abra/commit/4e94a424e94a42))
-- Support [abra-hetzner](https://git.autonomic.zone/coop-cloud/abra-hetzner) plugin ([#88](https://git.autonomic.zone/coop-cloud/abra/issues/88))
 
 # abra 0.6.0 (2021-03-17)
 

README.md

@@ -40,7 +40,7 @@ or the bleeding-edge development version:
 curl https://install.abra.autonomic.zone | bash -s -- --dev
 ```
 
-The source for this script is [here](./deploy/install.abra.autonomic.zone/installer).
+The source for this script is [here](./installer/installer).
 
 ## Update
 
@@ -59,33 +59,11 @@ The command-line interface is generated via [docopt](http://docopt.org/). If you
 
 Please remember to update the [CHANGELOG](./CHANGELOG.md) when you make a change.
 
-## Generating a new abra-apps.json
+## Release
-
-You'll need to install the following requirements:
-
-- [requests](https://docs.python-requests.org/en/master/) (`apt install python3-requests` / `pip install requests`)
-- [skopeo](https://github.com/containers/skopeo) (check [the install docs](https://github.com/containers/skopeo/blob/master/install.md))
-- [jq](https://stedolan.github.io/jq/tutorial/) (`sudo apt-get install jq` or see [the install docs](https://stedolan.github.io/jq/download/))
-- [yq](https://mikefarah.gitbook.io/yq/) (see [the install docs](https://mikefarah.gitbook.io/yq/#install))
-
-Then run `./bin/app-json.py` ([source](./bin/app-json.py)) and it will spit out the JSON file into [deploy/abra-apps.cloud.autonomic.zone/abra-apps.json](./deploy/abra-apps.cloud.autonomic.zone/abra-apps.json).
-
-## Releasing
-
-### `abra`
 
 - Change the `x.x.x` header in [CHANGELOG.md](./CHANGELOG.md) to reflect new version and mark date
-- Update the version in [abra](./abra)
+- Update versions in [installer/installer](./installer/installer) and [abra](./abra)
-- Update the version in [deploy/install.abra.autonomic.zone/installer](./deploy/install.abra.autonomic.zone/installer)
 - `git commit` the above changes and then tag it with `git tag <your-new-version>`
-- `git push` and `git push --tags`
+- Then `git push` and `git push --tags`
 - Deploy a new installer script `make release-installer`
-- Tell the world (CoTech forum, Matrix public channel, Autonomic mastodon, etc.)
+- Tell the world
-
-### abra-apps.cloud.autonomic.zone
-
-> [abra-apps.cloud.autonomic.zone](https://abra-apps.cloud.autonomic.zone)
-
-```bash
-$ make release-abra-apps
-```

app-catalogue.sh

@@ -1,7 +1,4 @@
 #!/bin/bash
-
-# shellcheck disable=SC2119
-
 # Usage: ./app-catalogue.sh
 #
 # Gather metadata from Co-op Cloud apps in $ABRA_DIR/apps (default
@@ -10,7 +7,7 @@
 
 stack_dir="${ABRA_DIR:-$HOME/.abra}/apps/"
 
-cd "$stack_dir" || exit
+cd "$stack_dir"
 
 # load all README files into ENV_FILES array
 mapfile -t readmes < <(find -L . -name "README.md")

app-version.sh

@@ -0,0 +1,42 @@
+#!/bin/bash
+
+# Usage: ./app-version.sh <image> <service>
+# Example: ./app-version.sh drone/drone:1.10.1 app
+#
+# Accepts a full format hub.docker.com image tag which it pulls locally and
+# generates output which can be used to put in the abra.sh for app packaging.
+# Requires the yq program https://mikefarah.gitbook.io/yq/
+
+error() {
+  echo "$(tput setaf 1)ERROR: $*$(tput sgr0)"
+  exit 1
+}
+
+IMAGE="$1"
+SERVICE="$2"
+
+if ! docker pull -q "$IMAGE" > /dev/null 2>&1; then
+	error "Failed to download image, is the tag correct?"
+fi
+
+version=$(echo "$IMAGE" | cut -d ':' -f2)
+digest=$(docker image inspect -f "{{.Id}}" "$IMAGE" | cut -d ':' -f2- | cut -c 1-8)
+
+echo "--- Add the following to your abra.sh ---"
+echo "export ABRA_TYPE_${SERVICE^^}_VERSION=${version}"
+echo "export ABRA_TYPE_${SERVICE^^}_DIGEST=${digest}"
+
+version_lookup="ABRA_TYPE_${SERVICE^^}_VERSION"
+digest_lookup="ABRA_TYPE_${SERVICE^^}_DIGEST"
+
+label='- "coop-cloud.${STACK_NAME}.'
+label+="${SERVICE}"
+label+='.version=${'
+label+="${version_lookup}"
+label+='}-${'
+label+="${digest_lookup}"
+label+='}"'
+
+echo
+echo "--- And don't forget to label the actual service in the compose file ---"
+echo "$label"

bin/app-json.py

@@ -1,253 +0,0 @@
-#!/usr/bin/env python3
-
-# Usage: ./app-json.py
-#
-# Gather metadata from Co-op Cloud apps in $ABRA_DIR/apps (default
-# ~/.abra/apps), and format it as JSON so that it can be hosted here:
-# https://abra-apps.cloud.autonomic.zone
-
-from json import dump
-from logging import DEBUG, basicConfig, getLogger
-from os import chdir, listdir, mkdir
-from os.path import basename, exists, expanduser
-from pathlib import Path
-from re import findall, search
-from shlex import split
-from subprocess import check_output, DEVNULL
-from sys import exit
-
-from requests import get
-
-HOME_PATH = expanduser("~/")
-CLONES_PATH = Path(f"{HOME_PATH}/.abra/apps").absolute()
-YQ_PATH = Path(f"{HOME_PATH}/.abra/vendor/yq")
-SCRIPT_PATH = Path(__file__).absolute().parent
-REPOS_TO_SKIP = (
-    "abra",
-    "backup-bot",
-    "cloud.autonomic.zone",
-    "docs.cloud.autonomic.zone",
-    "example",
-    "organising",
-    "pyabra",
-    "stack-ssh-deploy",
-    "radicle-seed-node",
-    "coturn"
-)
-
-log = getLogger(__name__)
-basicConfig()
-log.setLevel(DEBUG)
-
-
-def _run_cmd(cmd, shell=False, **kwargs):
-    """Run a shell command."""
-    args = [split(cmd)]
-
-    if shell:
-        args = [cmd]
-        kwargs = {"shell": shell}
-
-    try:
-        return check_output(*args, **kwargs).decode("utf-8").strip()
-    except Exception as exception:
-        log.error(f"Failed to run {cmd}, saw {str(exception)}")
-        exit(1)
-
-
-def get_published_apps_json():
-    """Retrieve already published apps json."""
-    url = "https://abra-apps.cloud.autonomic.zone"
-
-    log.info(f"Retrieving {url}")
-
-    try:
-        return get(url, timeout=5).json()
-    except Exception as exception:
-        log.error(f"Failed to retrieve {url}, saw {str(exception)}")
-        return {}
-
-
-def clone_all_apps():
-    """Clone all Co-op Cloud apps to ~/.abra/apps."""
-    if not exists(CLONES_PATH):
-        mkdir(CLONES_PATH)
-    url = "https://git.autonomic.zone/api/v1/orgs/coop-cloud/repos"
-
-    log.info(f"Retrieving {url}")
-
-    try:
-        response = get(url, timeout=10)
-    except Exception as exception:
-        log.error(f"Failed to retrieve {url}, saw {str(exception)}")
-        exit(1)
-
-    repos = [[p["name"], p["ssh_url"]] for p in response.json()]
-
-    for name, url in repos:
-        if name in REPOS_TO_SKIP:
-            continue
-
-        if not exists(f"{CLONES_PATH}/{name}"):
-            log.info(f"Retrieving {url}")
-            _run_cmd(f"git clone {url} {CLONES_PATH}/{name}")
-
-            chdir(f"{CLONES_PATH}/{name}")
-            if not int(_run_cmd("git branch --list | wc -l", shell=True)):
-                log.info(f"Guessing main branch is HEAD for {name}")
-                _run_cmd("git checkout main")
-        else:
-            log.info(f"Updating {name}")
-            chdir(f"{CLONES_PATH}/{name}")
-            _run_cmd("git fetch -a")
-
-
-def generate_apps_json():
-    """Generate the abra-apps.json application versions file."""
-    apps_json = {}
-    cached_apps_json = get_published_apps_json()
-
-    for app in listdir(CLONES_PATH):
-        if app in REPOS_TO_SKIP:
-            log.info(f"Skipping {app}")
-            continue
-
-        app_path = f"{CLONES_PATH}/{app}"
-        chdir(app_path)
-
-        log.info(f"Processing {app}")
-        apps_json[app] = {
-            "category": "apps",
-            "repository": f"https://git.autonomic.zone/coop-cloud/{app}.git",
-            # Note(decentral1se): please note that the app features do not
-            # correspond to version tags. We simply parse the latest features
-            # list from HEAD. This may lead to unexpected situations where
-            # users believe X feature is available under Y version but it is
-            # not.
-            "features": get_app_features(app_path),
-            "versions": get_app_versions(app_path, cached_apps_json),
-        }
-
-    return apps_json
-
-
-def get_app_features(app_path):
-    """Parse features from app repo README files."""
-    features = {}
-
-    chdir(app_path)
-
-    with open(f"{app_path}/README.md", "r") as handle:
-        log.info(f"{app_path}/README.md")
-        contents = handle.read()
-
-    try:
-        for match in findall(r"\*\*.*\s\*", contents):
-            title = search(r"(?<=\*\*).*(?=\*\*)", match).group().lower()
-
-            if title == "image":
-                value = {
-                    "image": search(r"(?<=`).*(?=`)", match).group(),
-                    "url": search(r"(?<=\().*(?=\))", match).group(),
-                    "rating": match.split(",")[1].strip(),
-                    "source": match.split(",")[-1].replace("*", "").strip(),
-                }
-            else:
-                value = match.split(":")[-1].replace("*", "").strip()
-
-            features[title] = value
-    except (IndexError, AttributeError):
-        log.info(f"Can't parse {app_path}/README.md")
-        return {}
-    finally:
-        _run_cmd("git checkout HEAD")
-
-    log.info(f"Parsed {features}")
-
-    return features
-
-
-def get_app_versions(app_path, cached_apps_json):
-    versions = {}
-
-    chdir(app_path)
-
-    tags = _run_cmd("git tag --list").split()
-
-    if not tags:
-        log.info("No tags discovered, moving on")
-        return {}
-
-    initial_branch = _run_cmd("git rev-parse --abbrev-ref HEAD")
-
-    app_name = basename(app_path)
-
-    try:
-        existing_tags = cached_apps_json[app_name]["versions"].keys()
-    except KeyError:
-        existing_tags = []
-
-    for tag in tags:
-        _run_cmd(f"git checkout {tag}",
-                 stderr=DEVNULL)
-
-        services_cmd = f"{YQ_PATH} e '.services | keys | .[]' compose*.yml"
-        services = _run_cmd(services_cmd, shell=True).split()
-
-        parsed_services = []
-        service_versions = {}
-        for service in services:
-            if service in ("null", "---"):
-                continue
-
-            if tag in existing_tags:
-                log.info(f"Skipping {tag} because we've already processed it")
-                existing_versions = cached_apps_json[app_name]["versions"][tag][service]
-                service_versions[service] = existing_versions
-                _run_cmd(f"git checkout {initial_branch}")
-                continue
-
-            if service in parsed_services:
-                log.info(f"Skipped {service}, we've already parsed it locally")
-                continue
-
-            services_cmd = f"{YQ_PATH} e '.services.{service}.image' compose*.yml"
-            images = _run_cmd(services_cmd, shell=True).split()
-
-            for image in images:
-                if image in ("null", "---"):
-                    continue
-
-                images_cmd = f"skopeo inspect docker://{image} | jq '.Digest'"
-                output = _run_cmd(images_cmd, shell=True)
-
-                service_version_info = {
-                    "image": image.split(":")[0],
-                    "tag": image.split(":")[-1],
-                    "digest": output.split(":")[-1][:8],
-                }
-
-                log.info(f"Parsed {service_version_info}")
-                service_versions[service] = service_version_info
-
-            parsed_services.append(service)
-
-        versions[tag] = service_versions
-
-    _run_cmd(f"git checkout {initial_branch}")
-
-    return versions
-
-
-def main():
-    """Run the script."""
-    clone_all_apps()
-
-    target = f"{SCRIPT_PATH}/../deploy/abra-apps.cloud.autonomic.zone/abra-apps.json"
-    with open(target, "w", encoding="utf-8") as handle:
-        dump(generate_apps_json(), handle, ensure_ascii=False, indent=4)
-
-    log.info(f"Successfully generated {target}")
-
-
-main()

deploy/abra-apps.cloud.autonomic.zone/compose.yml

@@ -1,38 +0,0 @@
----
-version: "3.8"
-
-services:
-  app:
-    image: "nginx:stable"
-    configs:
-      - source: abra_conf
-        target: /etc/nginx/conf.d/abra.conf
-      - source: abra_apps_json
-        target: /var/www/abra-apps/abra-apps.json
-    volumes:
-      - "public:/var/www/abra-apps"
-    networks:
-      - proxy
-    deploy:
-      update_config:
-        failure_action: rollback
-        order: start-first
-      labels:
-        - "traefik.enable=true"
-        - "traefik.http.services.abra-apps.loadbalancer.server.port=80"
-        - "traefik.http.routers.abra-apps.rule=Host(`abra-apps.cloud.autonomic.zone`)"
-        - "traefik.http.routers.abra-apps.entrypoints=web-secure"
-        - "traefik.http.routers.abra-apps.tls.certresolver=production"
-
-configs:
-  abra_apps_json:
-    file: abra-apps.json
-  abra_conf:
-    file: nginx.conf
-
-networks:
-  proxy:
-    external: true
-
-volumes:
-  public:

deploy/abra-apps.cloud.autonomic.zone/nginx.conf

@@ -1,10 +0,0 @@
-server {
-  listen         80 default_server;
-  server_name    abra-apps.cloud.autonomic.zone;
-
-  location / {
-    root /var/www/abra-apps;
-    add_header Content-Type application/json;
-    index abra-apps.json;
-  }
-}

installer/compose.yml

@@ -2,7 +2,7 @@
 version: "3.8"
 
 services:
-  app:
+  abra_installer:
     image: "nginx:stable"
     configs:
       - source: abra_conf

installer/installer

@@ -1,9 +1,8 @@
 #!/bin/bash
 
-ABRA_VERSION="0.7.1"
+ABRA_VERSION="0.6.0"
 GIT_URL="https://git.autonomic.zone/coop-cloud/abra"
 ABRA_SRC="$GIT_URL/raw/tag/$ABRA_VERSION/abra"
-ABRA_DIR="${ABRA_DIR:+$HOME/.abra/}"
 
 function install_abra_release {
   mkdir -p "$HOME/.local/bin"
@@ -13,13 +12,13 @@ function install_abra_release {
 }
 
 function install_abra_dev {
-  mkdir -p "$ABRA_DIR/"
+  mkdir -p "$HOME/.abra/"
-  if [[ ! -d "$ABRA_DIR/src" ]]; then
+  if [[ ! -d "$HOME/.abra/src" ]]; then
-    git clone "$GIT_URL" "$ABRA_DIR/src"
+    git clone "$GIT_URL" "$HOME/.abra/src"
   fi
-  (cd "$ABRA_DIR/src" && git pull origin main && cd - || exit)
+  ( cd "$HOME/.abra/src" && git pull origin main && cd -)
   mkdir -p "$HOME/.local/bin"
-  ln -sf "$ABRA_DIR/src/abra" "$HOME/.local/bin/abra"
+  ln -sf "$HOME/.abra/src/abra" "$HOME/.local/bin/abra"
   echo "abra installed to $HOME/.local/bin/abra (development bleeding edge)"
 }
 

makefile

@@ -21,8 +21,7 @@ shellcheck:
 		--rm \
 		-v $$(pwd):/workdir \
 		koalaman/shellcheck-alpine \
-		shellcheck /workdir/abra && \
+		shellcheck /workdir/abra
-		shellcheck /workdir/bin/*.sh
 
 docopt:
 	@if [ ! -d ".venv" ]; then \
@@ -46,13 +45,6 @@ codecov:
 		-s coverage -t $$(pass show hosts/swarm.autonomic.zone/drone/codecov/token)
 
 release-installer:
-	@DOCKER_CONTEXT=swarm.autonomic.zone \
+	@docker stack rm abra-installer-script && \
-		docker stack rm abra-installer-script && \
+		cd installer && \
-		cd deploy/install.abra.autonomic.zone && \
+		docker stack deploy -c compose.yml abra-installer-script
-		DOCKER_CONTEXT=swarm.autonomic.zone docker stack deploy -c compose.yml abra-installer-script
-
-release-abra-apps:
-	@DOCKER_CONTEXT=swarm.autonomic.zone \
-		docker stack rm abra-apps-json && \
-		cd deploy/abra-apps.cloud.autonomic.zone && \
-		DOCKER_CONTEXT=swarm.autonomic.zone docker stack deploy -c compose.yml abra-apps-json

tests/02_abra.bats

@@ -9,48 +9,44 @@ teardown() {
     rm -rf "$ABRA_DIR"
 }
 
-abra() {
-    ./abra -d $@
-}
-
 @test "abra server (add|rm)" {
-    abra server add swarm.test.com
+    ./abra server add swarm.test.com
     docker context ls | grep swarm.test.com
     [ -d $ABRA_DIR/servers/swarm.test.com ]
-    abra server swarm.test.com rm
+    ./abra server swarm.test.com rm
 
-    abra server add swarm.test.com foobar 12345
+    ./abra server add swarm.test.com foobar 12345
     [ -d $ABRA_DIR/servers/swarm.test.com ]
-    abra server swarm.test.com rm
+    ./abra server swarm.test.com rm
 }
 
 @test "abra server init" {
-    abra server default init
+    ./abra server default init
 }
 
 @test "abra app (new|rm)" {
-    abra app new --server default --domain traefik.test.com --app-name traefik_test_com traefik
+    ./abra app new --server default --domain traefik.test.com --app-name traefik_test_com traefik
     [ -f $ABRA_DIR/servers/default/traefik_test_com.env ]
 
     # interactive prompt
-    echo "y" | abra app traefik_test_com delete
+    echo "y" | ./abra app traefik_test_com delete
     [ ! -f $ABRA_DIR/servers/default/traefik_test_com.env ]
 
-    # --no-prompt
+    # --force
-    abra app new --server default --domain traefik_test_com --app-name traefik_test_com traefik
+    ./abra app new --server default --domain traefik_test_com --app-name traefik_test_com traefik
-    abra --no-prompt app traefik_test_com delete
+    ./abra app traefik_test_com delete --force
     [ ! -f $ABRA_DIR/servers/default/traefik_test_com.env ]
 }
 
 @test "abra app <domain> secret (insert|generate|rm)" {
-    abra app new --server default --domain traefik_test_com --app-name traefik_test_com traefik
+    ./abra app new --server default --domain traefik_test_com --app-name traefik_test_com traefik
 
-    abra app traefik_test_com secret insert foobar v1 "foobar"
+    ./abra app traefik_test_com secret insert foobar v1 "foobar"
 
     # interactive prompt
-    echo "y" | abra app traefik_test_com secret rm foobar
+    echo "y" | ./abra app traefik_test_com secret rm foobar
 
-    abra app traefik_test_com secret insert foobar v1 "foobar"
+    ./abra app traefik_test_com secret insert foobar v1 "foobar"
     #  prompt
-    abra --no-prompt app traefik_test_com secret rm foobar
+    ./abra app traefik_test_com secret rm foobar --force
 }