feat(deploy): run the hooks after one wait for all apps

Reviewed-on: #15

Dockerfile

@@ -26,14 +26,16 @@ RUN python -m venv /opt/alakazam \
   && /opt/alakazam/bin/pip install --no-cache-dir -r /tmp/requirements.txt
 
 FROM python:3.11-slim
+# git: alakazam syncs the recipe repos itself. openssh-client: abra reaches the servers over
+# ssh. argon2: recipes hash secrets locally in their abra.sh, vaultwarden's admin token does
 RUN apt-get update \
-  && apt-get install -y --no-install-recommends git openssh-client make \
+  && apt-get install -y --no-install-recommends git openssh-client make argon2 \
   && rm -rf /var/lib/apt/lists/* \
   && git config --global --add safe.directory '*'
 
 COPY --from=abra /out/abra /usr/local/bin/abra
 COPY --from=deps /opt/alakazam /opt/alakazam
-COPY alakazam.py combine.yml /opt/alakazam/
+COPY alakazam.py /opt/alakazam/
 RUN printf '#!/bin/sh\nexec /opt/alakazam/bin/python /opt/alakazam/alakazam.py "$@"\n' \
   > /usr/local/bin/alakazam \
   && chmod +x /usr/local/bin/alakazam

README.md

@@ -185,6 +185,22 @@ Without `-s` all secrets of the recipes selected by `-r` are purged.
 alakazam example.com.yml ps --wait --timeout 300
 ```
 
+### Excluding Recipes
+
+`-er`/`--exclude-recipe` leaves single recipes out of a run entirely:
+
+```
+alakazam -er traefik example.com.yml clean-deploy -n
+```
+
+### Hiding Secrets
+
+`secrets` and `setup` print the values they generate, which is right at a terminal and wrong in a pipeline whose logs are kept. `--hide-secrets` replaces them with `[hidden]`:
+
+```
+alakazam --hide-secrets example.com.yml clean-deploy -n
+```
+
 ### Run CMDs
 
 Escaping can be akward:
@@ -211,12 +227,7 @@ Configuration files support templating with Jinja2 and global variables, facilit
 1. **`alaka.yml`**/**`alaka-*.yml`**: Contains global configurations for a hierarchical layer, affecting all subordinate apps.
 2. **`example.com.yml`** (`<your-domain>.yml`): A minimalist configuration for each instance, specifying the apps to be installed, the instance domain and optionally instance specific configurations.
     - an **instance** means a collection of apps that are integrated with each other (one server / vm can contain multiple instances)
-3. **`combine.yml`**: Manages configurations required to integrate multiple apps with each other.
-    - This configuration should not be touched by the operator.
-    - At the moment it's part of the `alakazam` repository.
-    - In future it should be split into the recipes repositories and maintained by the recipe maintainer.
-        - We started to move each app entry as `alaconnect.yml` into the respective repositories.
-4. **`config-sets.yml`**: Optional file placed at the `root` path. Defines named sets of per-app configurations that can be activated or deactivated per instance or group, avoiding duplication across instance files. See [examples/config-sets.yml](./examples/config-sets.yml).
+3. **`config-sets.yml`**/**`config-sets-*.yml`**: Optional files at the `root` path. They define named sets of per-app configurations, either switched on per instance or group, or applying by themselves as an integration between apps. See [Config-Sets](#config-sets).
 
 ### Global Settings
 
@@ -243,7 +254,7 @@ uptime_kuma:
 
 ### App Configuration
 
-`alaka.yml`/`alaka-*.yml`, `example.com.yml` and `combine.yml`/`alaconnect.yml` contain a similar configuration structure.
+`alaka.yml`/`alaka-*.yml`, `example.com.yml` and the config-sets contain a similar configuration structure.
 For each app/recipe the following `<app_configurations>` can be used:
 
 - **`uncomment`/`comment`**: Manages the active status of lines in configuration files, such as uncommenting needed variables .
@@ -258,14 +269,47 @@ For each app/recipe the following `<app_configurations>` can be used:
 - **`dependency`**: Names the apps that have to be set up before this one. See [Dependencies](#dependencies).
 - **`secrets`**: Inserts specific values (i.E. smtp passwords) into secrets; future updates will support encrypted file usage.
 - **`secret-hooks`**: Run `abra.sh` commands locally or local scripts for secrets that need to be generated.
-- **`subdomain`**: Specifies the subdomain scheme for individual recipes and apps. (not available in `combine.yml`/`alaconnect.yml`)
+- **`shared_secrets`**: Shares a secret between two apps, as `<source_app>: {<target_secret>: <source_secret>}`.
+- **`subdomain`**: Specifies the subdomain scheme for individual recipes and apps. (not available in config-sets)
     - i.e. `cloud.example.com` for nextcloud
-- **`version`**: Controls the recipe version to deploy; if unspecified, the latest version is used. (not available in `combine.yml`/`alaconnect.yml`)
+- **`version`**: Controls the recipe version to deploy; if unspecified, the latest version is used. (not available in config-sets)
+    - a commit can be pinned with `<release>@<commit>`, for example `12.0.2+2026.5.2@be9ebb3`
+    - only the commit is deployed; the release in front of the `@` is there so that a dependency bot can follow the recipe's tags and offer patch updates
 
-The `combine.yml`/`alaconnect.yml` configuration additionally contains:
+### Config-Sets
 
-- **`shared_secrets`**: Specifies secret sharing between apps.
-     - `<source_secret_name>:<target_secret_name>`
+A config-set is a named bundle of per-app configuration, defined in `config-sets.yml` or any `config-sets-*.yml` at the `root` path. The files are merged in the order `config-sets.yml`, then the `config-sets-*.yml` alphabetically, the same precedence the `alaka*.yml` of a directory have.
+
+A set applies in one of two ways.
+
+**Switched on per instance or group**, via `CONFIG-SETS` in any `alaka.yml` or `example.com.yml`:
+
+```yaml
+CONFIG-SETS:
+    bbb: true
+```
+
+This is for bundles of per-app configuration that an operator turns on deliberately, a feature for instance. See [examples/config-sets.yml](./examples/config-sets.yml).
+
+**By itself**, when it carries `apply-when` and every app named there is part of the instance:
+
+```yaml
+integrate-authentik-nextcloud:
+    apply-when: [authentik, nextcloud]
+    authentik:
+        uncomment: [compose.nextcloud.yml]
+    nextcloud:
+        uncomment: [compose.authentik.yml]
+        shared_secrets:
+            authentik:
+                nextcloud_secret: authentik_secret
+```
+
+That is what an integration between two apps is: configuration belonging to both of them that is pointless unless both are there. `apply-when` names the condition rather than deriving it from the configured apps, so an integration can configure only one side: `[matrix-synapse, traefik]` while configuring `traefik` alone. An instance can still switch such a set off with `CONFIG-SETS: {integrate-authentik-nextcloud: false}`.
+
+Merge priority: automatic sets < group < sets switched on explicitly < instance. An integration is the lowest layer of all, so a group configuration can adjust it — the precedence the `combine.yml` it replaces had. Its list entries land behind the ones an app configures for itself, which is what keeps an integration hook such as `app set_authentik` running after the `app core_install` that sets the app up in the first place.
+
+[examples/config-sets-combine.yml](./examples/config-sets-combine.yml) holds the integrations alakazam used to ship as `combine.yml`. Copy it into your `root` path to keep them.
 
 ### Dependencies
 
@@ -328,22 +372,9 @@ These configurations are designed to modularize and simplify the management of a
         <app_configurations>
     ```
 
-2. **`combine.yml`**
-    - This file orchestrates the integration between multiple apps within the same instance (`example.com.yml`), applying `<target_app_configurations>` to ensure seamless functionality between connected applications.
-    - The configuration for each app combination is structured as follows, again `<target_app_configurations>` is following the guidelines set in the [App Configuration](#app-configuration) section:
-
-    ```
-    <target_app_recipe>:
-        <source_app_recipe>:
-            <target_app_configurations>
-    ```
-
-    - In future each `<source_app_recipe>` entry will be placed in the `alaconnect.yml` inside the `<target_app_recipe>` folder.
-
-3. **`config-sets.yml`**
-    - Optional file at the `root` path that defines named sets of per-app configurations. Each top-level key is a config-set name; its value is a map of `<app_recipe>` to `<app_configurations>`.
-    - Activate or disable sets per instance or group via `CONFIG-SETS` in any `alaka.yml` or `example.com.yml`. Merge priority: group < config-set < instance.
-    - See [examples/config-sets.yml](./examples/config-sets.yml).
+2. **`config-sets.yml`**/**`config-sets-*.yml`**
+    - Optional files at the `root` path. Each top-level key is a config-set name; its value is a map of `<app_recipe>` to `<app_configurations>`.
+    - See [Config-Sets](#config-sets).
 
 
 ### Templating Configurations

alakazam.py

@@ -4,7 +4,7 @@ import os
 import json
 import logging
 from pathlib import Path
-from typing import List, Dict, Any, Iterator, Optional, Union, Set, Tuple
+from typing import List, Dict, Any, Iterable, Iterator, Optional, Union, Set, Tuple
 import subprocess
 import re
 
@@ -23,8 +23,9 @@ from packaging import version
 from uptime_kuma_api import UptimeKumaApi, MonitorType
 from time import sleep, time
 
-COMBINE_PATH = os.path.dirname(os.path.realpath(__file__)) + "/combine.yml"
 NON_APP_KEYS = {'CONFIG-SETS', 'GLOBALS'}
+# key of a config-set that names the apps whose presence makes the set apply by itself
+APPLY_WHEN = 'apply-when'
 # INSTANCE_CONFIGS: dict: contains all app organized by recipe names and instance domains
 # The structure of the dictionary is as follows:
 # {
@@ -59,7 +60,7 @@ SETTINGS_PATH = ""  # path to the alakazam settings file
 GROUP_PATH = None  # path this run was invoked with, either an instance file or a group directory
 HIDE_SECRETS = False  # keep secret values out of the output, for runs whose logs are archived
 ABRA_DIR = None  # path to the abra data directory
-FETCH_MAX_AGE = 600 # seconds; recipe repos are refetched at most once per hour
+FETCH_MAX_AGE = 3600 # seconds; recipe repos are refetched at most once per hour
 ABRA_RETRIES = 6 # number of attempts per abra command that failed to reach the server
 ABRA_RETRY_DELAY = 2 # seconds before the first retry, doubled for every further one
 # lower case markers in the abra output that indicate an unreachable server instead of a failed operation
@@ -87,6 +88,7 @@ PENDING_STATUS_RE = re.compile(r".*(starting|unknown|unhealthy).*")
 INIT_CONTAINER_MARKER = "-init"
 PS_WAIT_INTERVAL = 10 # seconds between two health checks of 'ps --wait'
 PS_WAIT_TIMEOUT = 900 # seconds 'ps --wait' waits for all services to become healthy
+PS_WAIT_SETTLE = 30 # seconds before a health check, so that replaced containers are gone
 # defaults of a readiness hook, each of them overridable per hook in the app configuration
 READINESS_RETRIES = 20 # attempts before a readiness hook is considered failed
 READINESS_INTERVAL = 60 # seconds between two attempts
@@ -95,6 +97,8 @@ READINESS_INITIAL_DELAY = 0 # seconds to wait before the first attempt
 SECRET_VARIANTS = ("conf", "secret-hooks", "exchange", "generate")
 # group configuration files, inherited down the directory hierarchy
 GROUP_CONFIG_RE = re.compile(r'^alaka(-.*)?\.ya?ml$')
+# config-set files, read from the root path only
+CONFIG_SET_RE = re.compile(r'^config-sets(-.*)?\.ya?ml$')
 # instance configuration files, named '<domain>.yml' after the instance they configure
 INSTANCE_CONFIG_RE = re.compile(r'^(?:[A-Za-z0-9](?:[A-Za-z0-9\-]{0,61}[A-Za-z0-9])?\.)+[A-Za-z]{2,6}(?:\.yaml|\.yml)$')
 ROOT_PATH = None  # resolved root path from alakazam.yml
@@ -180,7 +184,7 @@ def get_relevant_config_paths(root_path: Path, group_path: Path, exclude_paths:
     """
     Collects every configuration file that can influence the result of this run: the 'alaka*.yml'
     inheritance chain from the root path down to the group path, the instance configurations below
-    it, the config sets of the root path and the connection configuration shipped with alakazam.
+    it, and every config-set file in the root path.
 
     Args:
         root_path (Path): The alakazam root path
@@ -190,7 +194,7 @@ def get_relevant_config_paths(root_path: Path, group_path: Path, exclude_paths:
     Returns:
         list: The configuration files to validate, without duplicates
     """
-    paths = [Path(COMBINE_PATH), root_path / "config-sets.yml"]
+    paths = [root_path / file for file in os.listdir(root_path) if CONFIG_SET_RE.match(file)]
     directory = group_path if group_path.is_dir() else group_path.parent
     for ancestor in reversed([directory, *directory.parents]):
         if ancestor != root_path and root_path not in ancestor.parents:
@@ -404,48 +408,105 @@ def merge_all_group_configs(root_path: Path) -> Dict[str, Dict[str, Any]]:
 
 
 
-def get_config_set_app_configs(
-    active_config_sets: Dict[str, Any], config_sets: Dict[str, Any]
-) -> Dict[str, Any]:
+def read_config_sets(root_path: Path) -> Dict[str, Any]:
     """
-    Merges per-app configurations from all active config-sets into a single dictionary.
-    Config-sets disabled with False are skipped; unknown set names log a warning.
-    When multiple active sets define config for the same app, later sets take precedence.
+    Reads and merges every 'config-sets*.yml' of the root path.
+
+    'config-sets.yml' comes first as the base, the 'config-sets-*.yml' specialisations follow in
+    alphabetical order, the same precedence the 'alaka*.yml' of a directory have. Config-sets live
+    in the root path only, so that an integration reads the same for every instance below it.
+
+    Args:
+        root_path (Path): The alakazam root path
+
+    Returns:
+        dict: The merged config-set definitions
+    """
+    files = sorted((f for f in os.listdir(root_path) if CONFIG_SET_RE.match(f)),
+                   key=lambda f: (not f.startswith("config-sets."), f))
+    merged: Dict[str, Any] = {}
+    defined_by: Dict[Tuple[str, ...], Tuple[str, Any]] = {}
+    for file in files:
+        config = read_config(str(root_path / file)) or {}
+        warn_on_key_collisions(config, defined_by, file, str(root_path))
+        merged = merge_dict(merged, config)
+    return merged
+
+
+def get_config_set_app_configs(
+    active_config_sets: Dict[str, Any], config_sets: Dict[str, Any], instance_apps: Iterable[str]
+) -> Tuple[Dict[str, Any], Dict[str, Any]]:
+    """
+    Merges per-app configurations from all applying config-sets into a single dictionary.
+
+    A set applies either because the instance enabled it under CONFIG-SETS, or by itself when it
+    carries an 'apply-when' and every app named there is part of the instance. That is what an
+    integration between two apps is: configuration that belongs to both of them and is pointless
+    unless both are there. Setting such a set to False for an instance still turns it off.
+
+    The two kinds are returned apart because they sit at different heights: an integration is the
+    lowest layer of all, below even the group configuration, so that a group can adjust it. That is
+    the precedence the combine.yml this replaces had. Among the explicit ones, later sets win.
 
     Args:
         active_config_sets (dict): Merged CONFIG-SETS dict (set name -> True or False).
-        config_sets (dict): Full config-set definitions loaded from config-sets.yml.
+        config_sets (dict): Full config-set definitions from the root path.
+        instance_apps (iterable): The app names configured for this instance.
 
     Returns:
-        dict: A merged dictionary of per-app configurations from all active config-sets.
+        tuple: The per-app configurations of the automatic sets and of the explicitly enabled ones
     """
+    automatic_app_configs: Dict[str, Any] = {}
     config_set_app_configs: Dict[str, Any] = {}
+    automatic = set()
+    for name, definition in config_sets.items():
+        if not (apply_when := (definition or {}).get(APPLY_WHEN)):
+            continue
+        automatic.add(name)
+        if active_config_sets.get(name) is False:
+            logging.debug(f"config-set '{name}' applies but is switched off for this instance")
+            continue
+        if not all(app in instance_apps for app in apply_when):
+            continue
+        logging.debug(f"config-set '{name}' applies, {', '.join(apply_when)} are all configured")
+        apps = {app: config for app, config in definition.items() if app != APPLY_WHEN}
+        automatic_app_configs = merge_dict(automatic_app_configs, apps)
     for config_set_name, config_set_enabled in active_config_sets.items():
-        if config_set_enabled:
-            if config_set_name not in config_sets:
-                logging.warning(
-                    f"Config-set '{config_set_name}' is enabled but not defined in config-sets.yml. Skipping."
-                )
-            else:
-                config_set_app_configs = merge_dict(config_set_app_configs, config_sets[config_set_name] or {})
-    return config_set_app_configs
+        if not config_set_enabled or config_set_name in automatic:
+            continue
+        if config_set_name not in config_sets:
+            logging.warning(
+                f"Config-set '{config_set_name}' is enabled but not defined in the root path. Skipping."
+            )
+            continue
+        config_set_app_configs = merge_dict(config_set_app_configs, config_sets[config_set_name] or {})
+    return automatic_app_configs, config_set_app_configs
 
 
-def substitute_jinja_variable(jinja_dict, subs_dict) -> None:
+def substitute_jinja_variable(node, subs_dict):
     """
-    This function recursively traverses the given jinja_dict and wherever it finds a jinja template variable, it replaces it with the corresponding value from the subs_dict.
+    Recursively substitutes jinja template variables inside node with values from subs_dict.
+    Dicts and lists are walked and updated in place. The rendered value is also returned so a
+    recursive call can assign a replaced scalar.
 
     Args:
-        jinja_dict (dict): The dictionary which may contain jinja template variables. Can be a nested dictionary.
-        subs_dict (dict): The dictionary containing the substitutions for the jinja template variables. 
+        node: A dict, list, or scalar which may contain jinja template variables, at any nesting.
+        subs_dict (dict): The dictionary containing the substitutions for the jinja template variables.
+
+    Returns:
+        The same dict/list (mutated in place), or the rendered scalar.
     """
-    for key, value in jinja_dict.items():
-        if isinstance(value, dict):  # If value itself is dictionary
-            substitute_jinja_variable(value, subs_dict)  # Recursive call
-        else:
-            if "{{" in str(value) and "}}" in str(value):  # If value is a jinja template
-                template = Template(str(value))
-                jinja_dict[key] = template.render(subs_dict)
+    if isinstance(node, dict):
+        items = node.items()
+    elif isinstance(node, list):
+        items = enumerate(node)
+    else:
+        if "{{" in str(node) and "}}" in str(node):  # If node is a jinja template
+            return Template(str(node)).render(subs_dict)
+        return node
+    for key, item in items:
+        node[key] = substitute_jinja_variable(item, subs_dict)
+    return node
 
 
 def merge_instance_configs(group_config: Dict[str, Any], instance_domain: str, instance_config: Dict[str, Any], config_sets: Optional[Dict[str, Any]] = None) -> Dict[str, Any]:
@@ -474,28 +535,67 @@ def merge_instance_configs(group_config: Dict[str, Any], instance_domain: str, i
         group_config.get('CONFIG-SETS') or {},
         instance_config.get('CONFIG-SETS') or {},
     )
-    config_set_app_configs = get_config_set_app_configs(active_config_sets, config_sets)
     instance_apps = {k: v for k, v in instance_config.items() if k not in NON_APP_KEYS}
+    automatic_app_configs, config_set_app_configs = get_config_set_app_configs(active_config_sets, config_sets, instance_apps)
     # Warn about config-set entries that target apps not listed in the instance config.
-    for config_set_app in set(config_set_app_configs) - set(instance_apps):
+    for config_set_app in (set(automatic_app_configs) | set(config_set_app_configs)) - set(instance_apps):
         logging.warning(
             f"Config-set specifies config for '{config_set_app}' but '{config_set_app}' is not listed "
             f"in the instance config for '{instance_domain}'. Skipping."
         )
     merged_config = {}
     for app in instance_apps:
+        automatic_app_config = automatic_app_configs.get(app) or {}
         group_app_config = group_config.get(app) or {}
         config_set_app_config = config_set_app_configs.get(app) or {}
         instance_app_config = instance_apps[app] or {}
-        # Priority: group < config-set < instance
-        merged_config[app] = merge_dict(merge_dict(group_app_config, config_set_app_config), instance_app_config)
+        # Priority: automatic set < group < config-set < instance. The automatic sets are merged
+        # last so that everything else outranks them, and with the list order reversed so that
+        # their entries land behind the ones the app configures for itself. An integration that
+        # adds 'set_authentik' must run after the 'core_install' that sets the app up at all.
+        merged_config[app] = merge_dict(
+            automatic_app_config,
+            merge_dict(merge_dict(group_app_config, config_set_app_config), instance_app_config),
+            reverse_list_order=True)
         merged_config[app]['app_domain'] = map_subdomain(app, instance_domain, merged_config[app])
         if not merged_config[app].get('server'):
             merged_config[app]['server'] = server
     substitute_jinja_variable(merged_config, global_vars)
+    # after the substitution, so that a templated version is resolved too
+    for app, app_config in merged_config.items():
+        if app_config.get('version'):
+            app_config['version'] = resolve_version(app_config['version'], app)
     return merged_config
 
 
+def resolve_version(version: Any, app: str) -> Any:
+    """
+    Reduces a configured version to the part that is deployed.
+
+    A version may name a commit behind an '@', as in '12.0.2+2026.5.2@be9ebb3'. What stands before
+    it is the release the commit is based on: it carries no meaning for the deployment and exists
+    so that a dependency bot can follow the recipe's tags and offer patch updates. Only the commit
+    is deployed. Everything without an '@' is passed through untouched.
+
+    Args:
+        version: The configured version, any type the configuration may hold
+        app (str): The app the version belongs to, for the error message
+
+    Returns:
+        The commit behind the '@', or the version unchanged
+
+    Raises:
+        click.ClickException: If the '@' is there but no commit follows it
+    """
+    if not isinstance(version, str) or "@" not in version:
+        return version
+    base, _, commit = version.rpartition("@")
+    if not commit:
+        raise click.ClickException(f"version '{version}' of {app} ends in '@' without a commit")
+    logging.debug(f"{app}: deploying commit {commit}, based on {base}")
+    return commit
+
+
 def map_subdomain(recipe: str, instance_domain: str, app_config: Dict[str, Any]) -> str:
     """
     Maps a subdomain for an app based on the recipe, instance domain, and specific app configuration.
@@ -551,40 +651,30 @@ def get_merged_instance_configs(config_path: Path, group_configs: Dict[str, Any]
     return instances
 
 
-def merge_connection_configs(configs: Dict[str, Any]) -> Dict[str, Any]:
+def exclude_from_configs(configs: Dict[str, Dict[str, Any]], recipes: Tuple[str]) -> Dict[str, Dict[str, Any]]:
     """
-    Merge connection configurations from the 'combine.yml' to extend instance configurations with inter-app secrets and settings.
-    This involves integrating shared secrets and other connection-specific settings between applications within the same instance.
+    Removes the given recipes from every instance configuration.
+
+    An excluded recipe is not merely left undeployed, it is invisible for the whole run: it is
+    neither listed, nor configured, nor purged, nor waited for. That is what keeps a rebuild from
+    touching an app whose state has to survive it, such as the certificates of a reverse proxy.
 
     Args:
-        configs (dict): The initial instance configurations before integrating connection-specific adjustments.
+        configs (dict): Instances as keys and their app configurations as values
+        recipes (tuple): Recipe names to leave out, all recipes are kept if empty
 
     Returns:
-        dict: The updated instance configurations after applying the connection settings.
+        dict: The configurations without those recipes
     """
-    connection_config = read_config(COMBINE_PATH)
-    extend_shared_secrets(connection_config)
-    merged_configs = configs.copy()
-    for _, instance_config in merged_configs.items():
-        for target_app, source_apps in connection_config.items():
-            for source_app, target_conf in source_apps.items():
-                if target_app in instance_config and source_app in instance_config:
-                    instance_config[target_app] = merge_dict(target_conf, instance_config[target_app], reverse_list_order=True)
-    return merged_configs
-
-
-def extend_shared_secrets(connection_config: Dict[str, Any]) -> None:
-    """
-    Extends connection configurations by embedding source app details into the shared secrets configuration.
-    This modifies the existing connection configurations in place, adding a layer of source app information to shared secrets.
-
-    Args:
-        connection_config (dict): Connection configurations which involve shared secrets.
-    """
-    for _, source_apps in connection_config.items(): 
-        for source_app, target_conf in source_apps.items():
-            if shared_secrets:= target_conf.get('shared_secrets'):
-                target_conf['shared_secrets'] = {source_app: shared_secrets}
+    if not recipes:
+        return configs
+    for recipe in recipes:
+        if not any(recipe in apps for apps in configs.values()):
+            logging.warning(f"'{recipe}' is excluded but not configured for this path")
+    return {
+        instance: {name: config for name, config in apps.items() if name not in recipes}
+        for instance, apps in configs.items()
+    }
 
 
 def is_secret_command(args: Tuple[str, ...]) -> bool:
@@ -636,7 +726,7 @@ def run_streamed(command: List[str]) -> subprocess.CompletedProcess:
     return subprocess.CompletedProcess(command, process.returncode, b"".join(lines), b"")
 
 
-def abra(*args: str, machine_output: bool = False, ignore_error: bool = False, stream: bool = False) -> Union[str,Dict]:
+def abra(*args: str, machine_output: bool = False, ignore_error: bool = False, stream: bool = False, secret: bool = False) -> Union[str,Dict]:
     """
     Execute the 'abra' command with the specified arguments. This function acts as a wrapper around the 'abra' CLI tool. It allows for capturing the output and optionally returning it as machine-readable JSON.
 
@@ -645,6 +735,7 @@ def abra(*args: str, machine_output: bool = False, ignore_error: bool = False, s
         machine_output (bool): If True, expects the output in JSON format and parses it before returning.
         ignore_error (bool): If True, suppresses the raising of errors on non-zero return codes, otherwise an exception is raised.
         stream (bool): If True, echoes the output while the command runs instead of only returning it afterwards. Cannot be combined with machine_output, which needs clean JSON on stdout.
+        secret (bool): If True, the output of this command may carry secret values, which HIDE_SECRETS then keeps out of the stream, the log and the error message.
 
     Returns:
         str or dict: Returns the output from the 'abra' command. If machine_output is True, returns a dictionary, otherwise returns raw output as a string.
@@ -661,7 +752,10 @@ def abra(*args: str, machine_output: bool = False, ignore_error: bool = False, s
         command.append("-m")
     # the command line of 'app secret insert' ends in the secret itself, and the output of
     # 'app secret generate' contains the generated values
-    quiet = HIDE_SECRETS and is_secret_command(args)
+    quiet = HIDE_SECRETS and (secret or is_secret_command(args))
+    if quiet:
+        # echoing it live would put the very values that are being hidden into the output
+        stream = False
     if not quiet:
         logging.debug(f"run command: {' '.join(command)}")
     for attempt in range(1, ABRA_RETRIES + 1):
@@ -675,12 +769,15 @@ def abra(*args: str, machine_output: bool = False, ignore_error: bool = False, s
         delay = ABRA_RETRY_DELAY * 2 ** (attempt - 1)
         logging.warning(f"attempt {attempt}/{ABRA_RETRIES} of '{' '.join(command)}' failed to reach the server, retry in {delay}s")
         sleep(delay)
-    if process.stderr and ignore_error:
+    if process.stderr and ignore_error and not quiet:
         logging.warning(process.stderr.decode())
     if process.stdout and not stream and not quiet:
         logging.debug(process.stdout.decode())
     if process.returncode and not ignore_error:
         #breakpoint()
+        if quiet:
+            raise RuntimeError(
+                f'{" ".join(args[:3])} failed, its output is withheld by --hide-secrets')
         raise RuntimeError(
             f'{" ".join(command)} \n STDOUT: \n {process.stdout.decode()} \n STDERR: {process.stderr.decode()}')
     if machine_output:
@@ -829,6 +926,71 @@ def new_app(recipe: str, domain: str, server: str, version: str) -> None:
     logging.info(f'{recipe} created on {server} at {domain}')
 
 
+def format_env_value(value: Any, default_quote: str = '') -> Any:
+    """
+    Renders a configuration value as the text that follows "KEY=" in an .env file.
+
+    A value keeps the quotes of its YAML scalar, because abra expands a "$" inside double quotes and
+    reads single quotes literally. Quotes of the same style within the value, which jinja can bring
+    in through a global, would end the line early. The other style is used where it carries the same
+    meaning, which needs no escape and is understood by every reader of the file. Only a value that
+    relies on expansion or on escapes of its own keeps its style and has its quotes escaped.
+
+    Args:
+        value (Any): The value to render, with or without the quotes of its YAML scalar
+        default_quote (str): Quote style to wrap a value that brings none of its own
+
+    Returns:
+        Any: The value as it can be written after "KEY=", non-strings unchanged
+    """
+    if not isinstance(value, str):
+        return value
+    quote = value[0] if len(value) > 1 and value[0] == value[-1] and value[0] in '"\'' else ''
+    inner = value[1:-1] if quote else value
+    quote = quote or default_quote
+    if not quote or quote not in inner:
+        return f'{quote}{inner}{quote}' if quote else inner
+    # "$" expands only inside double quotes and a backslash only escapes there, so a value holding
+    # either means what its own style says and cannot be moved to the other one
+    other = '"' if quote == "'" else "'"
+    if other not in inner and '$' not in inner and '\\' not in inner:
+        return f'{other}{inner}{other}'
+    if quote == "'":
+        logging.warning(f"the value {value} holds a single quote inside single quotes, which ends "
+                        "it early when it is read back, write it in double quotes instead")
+        return f"'{inner}'"
+    # the already escaped quotes are normalised first, so that they are not escaped a second time
+    return '"{}"'.format(inner.replace('\\"', '"').replace('"', '\\"'))
+
+
+def set_extending_key(path: Path, key: str, value: str) -> None:
+    """
+    Sets a "key=value" line in the .env file at path, for keys that appear more than
+    once in the file (e.g. COMPOSE_FILE lines, which accumulate via bash expansion).
+    Unlike dotenv.set_key(), which would overwrite every line sharing that key.
+    - if that exact line already exists uncommented, nothing to do.
+    - if it exists but commented, uncomment it in place.
+    - otherwise, append it as a new line at the end of the file.
+
+    Args:
+        path (str): Path to the .env configuration file to update.
+        key (str): The env var key.
+        value (str): The value the key should be set to.
+    """
+    targetline = '{}={}'.format(key, format_env_value(value, default_quote='"'))
+    with open(path, "r") as file:
+        lines = file.readlines()
+    for i, line in enumerate(lines):
+        if line.lstrip("#").strip() == targetline:
+            if line.lstrip().startswith("#"):
+                lines[i] = f"{targetline}\n"
+                with open(path, "w") as file:
+                    file.writelines(lines)
+            return
+    with open(path, "a") as file:
+        file.write(f"{targetline}\n")
+
+
 def update_configs(path: Path, config: Dict[str, Any]) -> None:
     """
     Update the .env configuration files at the specified path according to the provided configuration dictionary.
@@ -843,12 +1005,37 @@ def update_configs(path: Path, config: Dict[str, Any]) -> None:
     if comment_keys := config.get("comment"):
         comment(comment_keys, path, True)
     if envs := config.get("env"):
-        uncomment(envs.keys(), path)
+        # A value that references its own key via shell expansion (e.g.
+        # "$COMPOSE_FILE:compose.mariadb.yml") is meant to extend an existing accumulation of
+        # values, not replace it. Those keys must be kept out of
+        # uncomment()/set_key(), which would otherwise affect every line sharing the key.
+        extending_keys = {key for key, value in envs.items() if f"${key}" in str(value)}
+        uncomment(envs.keys() - extending_keys, path)
         for key, value in envs.items():
             logging.debug(f'set {key}={value} in {path}')
             if isinstance(value, dict):
                 value=json.dumps(value)
-            dotenv.set_key(path, key, value, quote_mode="never")
+            if key in extending_keys:
+                set_extending_key(path, key, value)
+            else:
+                dotenv.set_key(path, key, format_env_value(value), quote_mode="never")
+
+
+def get_missing_secrets(domain: str) -> List[str]:
+    """
+    Lists the secrets a recipe declares that the server does not hold.
+
+    This is the same condition abra checks before it deploys, where a missing secret ends the run
+    with "secret not generated". Reading it beforehand is what lets a caller fill the gap instead.
+
+    Args:
+        domain (str): The app domain to check
+
+    Returns:
+        list: The names of the missing secrets, empty when the app is complete
+    """
+    stored_secrets = abra("app", "secret", "ls", domain, machine_output=True)
+    return [s['name'] for s in stored_secrets or [] if not str2bool(s['created on server'])]
 
 
 def generate_all_secrets(domain: str) -> None:
@@ -1221,7 +1408,8 @@ def run_local_script(tokens: List[str], app_domain: str, server: str, instance_d
         return
     if dry_run:
         return
-    result = subprocess.run([str(script_path)] + args, env=env)
+    # the script may print a secret it just created, so its output is captured and dropped
+    result = subprocess.run([str(script_path)] + args, env=env, capture_output=HIDE_SECRETS)
     if result.returncode != 0:
         message = f"Script '{cmd_display}' exited with code {result.returncode}"
         if strict:
@@ -1233,6 +1421,9 @@ def run_secret_hooks(domain: str, app_config: Dict[str, Any], instance_domain: s
     """
     Run local abra.sh commands or local scripts to generate secrets.
 
+    A secret hook may print the value it just created, which some recipes do on purpose because
+    only a hash of it is stored. Under HIDE_SECRETS that output is withheld, failures included.
+
     Args:
         domain (str): The app domain into which the secrets are to be inserted.
         app_config (dict): A dictionary containing the secrets hooks and their corresponding values to insert.
@@ -1251,7 +1442,7 @@ def run_secret_hooks(domain: str, app_config: Dict[str, Any], instance_domain: s
             else:
                 print(f"Run '{cmd}' in {domain}", flush=True)
                 try:
-                    abra("app", "cmd", "--local", domain, cmd, ignore_error=not strict, stream=True)
+                    abra("app", "cmd", "--local", domain, cmd, ignore_error=not strict, stream=True, secret=True)
                 except RuntimeError as e:
                     raise click.ClickException(f"secret hook '{cmd}' failed for {domain}: {e}")
 
@@ -1314,22 +1505,25 @@ def update_secret(domain: str, secret_name: str, secret: str, was_deployed: bool
 def uncomment(keys: List[str], path: str, match_all: bool = False) -> None:
     """
     Uncomments lines in a configuration file that contain specified keys.
-    If 'match_all' is True, it matches against the entire line, otherwise, it matches only against the key.
+    If 'match_all' is True, it matches against the entire line (substring match), otherwise, it
+    matches only against the key exactly.
 
     Args:
         keys (list of str): The keys corresponding to the lines to be uncommented.
         path (str): Path to the file where lines will be uncommented.
-        match_all (bool): Whether to match the keys against the entire line or just the beginning.
+        match_all (bool): Whether to match the keys against the entire line or just the key.
     """
     logging.debug(f'Uncomment {keys} in {path}')
     with open(path, "r") as file:
         lines = file.readlines()
     with open(path, "w") as file:
         for line in lines:
-            line_match = line.split("=")[0]  # Match only keys
             if match_all:
-                line_match = line
-            if ('=' in line) and any(key in line_match for key in keys):
+                matched = ('=' in line) and any(key in line for key in keys)
+            else:
+                line_key = line.lstrip("#").split("=", 1)[0].strip()  # Match only keys
+                matched = ('=' in line) and (line_key in keys)
+            if matched:
                 line = line.lstrip("#").lstrip()
             file.write(line)
 
@@ -1337,22 +1531,26 @@ def uncomment(keys: List[str], path: str, match_all: bool = False) -> None:
 def comment(keys: List[str], path: str, match_all: bool = False) -> None:
     """
     Comments lines in a configuration file that contain specified keys.
-    If 'match_all' is True, it matches against the entire line, otherwise, it matches only against the key.
+    If 'match_all' is True, it matches against the entire line (substring match), otherwise, it
+    matches only against the key, exactly (so a key like "COMPOSE" doesn't also match a line whose
+    key is "COMPOSE_FILE").
 
     Args:
         keys (list of str): The keys corresponding to the lines to be commented.
         path (str): Path to the file where lines will be commented.
-        match_all (bool): Whether to match the keys against the entire line or just the beginning.
+        match_all (bool): Whether to match the keys against the entire line or just the key.
     """
     logging.debug(f'Comment {keys} in {path}')
     with open(path, "r") as file:
         lines = file.readlines()
     with open(path, "w") as file:
         for line in lines:
-            line_match = line.split("=")[0]  # Match only keys
             if match_all:
-                line_match = line
-            if any(key in line_match for key in keys):
+                matched = any(key in line for key in keys)
+            else:
+                line_key = line.lstrip("#").split("=", 1)[0].strip()  # Match only keys
+                matched = line_key in keys
+            if matched:
                 line = line.lstrip("#").lstrip()
                 line = f"#{line}"
             file.write(line)
@@ -1526,9 +1724,10 @@ def execute_cmds(app_config: Dict[str, Any], commands: Tuple[str] = tuple(), ini
 @click.group(context_settings={"help_option_names": ['-h', '--help']})
 @click.option('-l', '--log', 'loglevel', help='Desired logging level ("debug", "info", "warning", "error", "critical")')
 @click.option('-e', '--exclude', help='Path to a directory that contains a group of instance configurations to be excluded.', multiple=True, type=click.Path(exists=True))
+@click.option('exclude_recipes', '-er', '--exclude-recipe', multiple=True, metavar='<RecipeName>', help='Leave these recipes untouched, this option can be specified multiple times.')
 @click.option('hide_secrets', '--hide-secrets', is_flag=True, help='Keep secret values out of the output, for runs whose logs are kept.')
 @click.argument('group_path', type=click.Path(exists=True))
-def cli(loglevel: str, group_path: str, exclude: Tuple[str], hide_secrets: bool) -> None:
+def cli(loglevel: str, group_path: str, exclude: Tuple[str], exclude_recipes: Tuple[str], hide_secrets: bool) -> None:
     """
     Alakazam is a meta-configuration app-connector and an abra wrapper, designed as a proof-of-concept to simplify the management of environment configuration files across multiple instances.
 
@@ -1566,11 +1765,15 @@ def cli(loglevel: str, group_path: str, exclude: Tuple[str], hide_secrets: bool)
         exclude_paths.append(str(ABRA_DIR))
     preflight_configs(get_relevant_config_paths(ROOT_PATH, _group_path, exclude_paths))
     all_group_configs = merge_all_group_configs(ROOT_PATH)
-    config_sets = read_config(str(ROOT_PATH / "config-sets.yml"))
+    config_sets = read_config_sets(ROOT_PATH)
     instance_configs = get_merged_instance_configs(_group_path, all_group_configs, exclude_paths, config_sets)
-    INSTANCE_CONFIGS = merge_connection_configs(instance_configs)
+    INSTANCE_CONFIGS = instance_configs
+    # dropping the recipes here rather than at every filter is what makes the exclusion complete:
+    # get_apps(), create_secrets(), configure_apps() and the rest all read INSTANCE_CONFIGS.
+    # ALL_CONFIGS stays whole, it is the cross-instance view that backup looks up its bot in
+    INSTANCE_CONFIGS = exclude_from_configs(INSTANCE_CONFIGS, exclude_recipes)
     all_configs = get_merged_instance_configs(ROOT_PATH, all_group_configs, exclude_paths, config_sets)
-    ALL_CONFIGS = merge_connection_configs(all_configs)
+    ALL_CONFIGS = all_configs
     fetch_recipes(INSTANCE_CONFIGS)
 
         
@@ -1720,8 +1923,10 @@ def get_deployed_apps(apps: Tuple[str]) -> Dict[str, str]:
 @click.option('-e', '--execute-hooks', is_flag=True, help='run post-deployment commands.')
 @click.option('-f', '--force', is_flag=True, help='force redeployment even if the application is already deployed.')
 @click.option('-c', '--converge-checks', is_flag=True, help='perform convergence checks during deployment.')
+@click.option('--timeout', default=PS_WAIT_TIMEOUT, show_default=True, metavar='<Seconds>', help='how long to wait for all services to become healthy.')
+@click.option('--settle', default=PS_WAIT_SETTLE, show_default=True, metavar='<Seconds>', help='how long to give the replaced services to be taken down before their health is read.')
 @click.option('noninteractive', '-n', '--non-interactive', is_flag=True, help='Run this command non-interactively')
-def deploy(recipes: Tuple[str], execute_hooks: bool, force: bool, converge_checks: bool, noninteractive: bool) -> None:
+def deploy(recipes: Tuple[str], execute_hooks: bool, force: bool, converge_checks: bool, timeout: int, settle: int, noninteractive: bool) -> None:
     """
     Deploys applications as specified in the configuration.
     """
@@ -1732,19 +1937,33 @@ def deploy(recipes: Tuple[str], execute_hooks: bool, force: bool, converge_check
     print_all_apps(instance_apps)
     if not instance_apps or not (noninteractive or input(f"Do you really want to deploy these apps? Type YES: ") == "YES"):
         return
-    deploy_apps(instance_apps, execute_hooks=execute_hooks, force=force, converge_checks=converge_checks)
+    deploy_apps(instance_apps, execute_hooks=execute_hooks, force=force, converge_checks=converge_checks,
+                timeout=timeout, settle=settle)
 
 
-def deploy_apps(instance_apps: Dict[str, List[List]], execute_hooks: bool = False, force: bool = False, converge_checks: bool = False) -> None:
+def deploy_apps(instance_apps: Dict[str, List[List]], execute_hooks: bool = False, force: bool = False,
+                converge_checks: bool = False, wait: bool = True, timeout: int = PS_WAIT_TIMEOUT,
+                settle: int = PS_WAIT_SETTLE) -> None:
     """
     Deploys the given apps with the version taken from their configuration.
 
+    The apps do not wait for each other, so the run blocks until every service of all of them is
+    healthy, and only then are the deploy-hooks executed: a hook needs the app it addresses to be
+    up again. A caller that arranges the waiting itself passes wait=False.
+
     Args:
         instance_apps (dict): Instances as keys and lists of lists [app name, domain, ...] as values
         execute_hooks (bool): run the deploy-hooks from the configuration afterwards.
         force (bool): deploy even if the app is already deployed.
         converge_checks (bool): perform convergence checks during deployment.
+        wait (bool): block until every deployed service is healthy.
+        timeout (int): Seconds to wait for all services to become healthy.
+        settle (int): Seconds to give the services to be replaced before their health is read.
+
+    Raises:
+        click.ClickException: If a service is still unhealthy when the timeout is reached
     """
+    deployed = []
     for instance, apps_to_deploy in instance_apps.items():
         for app, domain, *_ in apps_to_deploy:
             app_config = get_value(INSTANCE_CONFIGS, instance, app)
@@ -1756,16 +1975,22 @@ def deploy_apps(instance_apps: Dict[str, List[List]], execute_hooks: bool = Fals
                 cmd.append("--chaos")
             if force:
                 cmd.append("--force")
-            if not execute_hooks and not converge_checks:
+            if not converge_checks:
                 cmd.append("--no-converge-checks")
             cmd.append(domain)
             if version not in ['latest', 'chaos']:
                 cmd.append(version)
             print(f'deploy {domain} with version "{version}"')
             print(abra("app", *cmd))
-            if execute_hooks:
-                logging.info(f'execute commands for {domain}')
-                execute_cmds(app_config, deploy=True, instance_domain=instance)
+            deployed.append((instance, app, domain, app_config))
+    if not deployed:
+        return
+    if wait:
+        wait_for_healthy_apps(tuple({app for _, app, *_ in deployed}), timeout, settle=settle)
+    if execute_hooks:
+        for instance, app, domain, app_config in deployed:
+            logging.info(f'execute commands for {domain}')
+            execute_cmds(app_config, deploy=True, instance_domain=instance)
 
 
 @cli.command()
@@ -1776,12 +2001,15 @@ def deploy_apps(instance_apps: Dict[str, List[List]], execute_hooks: bool = Fals
 @click.option('--redeploy-wait', type=int, default=20, metavar='<SECONDS>',
               help='seconds to wait between undeploy and deploy in --redeploy mode (default: 20).')
 @click.option('-c', '--converge-checks', is_flag=True, help='perform convergence checks during deployment.')
+@click.option('--timeout', default=PS_WAIT_TIMEOUT, show_default=True, metavar='<Seconds>', help='how long to wait for all services to become healthy.')
+@click.option('--settle', default=PS_WAIT_SETTLE, show_default=True, metavar='<Seconds>', help='how long to give the replaced services to be taken down before their health is read.')
 @click.option('noninteractive', '-n', '--non-interactive', is_flag=True, help='Run this command non-interactively')
 def upgrade(recipes: Tuple[str], execute_hooks: bool, dry_run: bool, redeploy: bool,
-            converge_checks: bool, noninteractive: bool, redeploy_wait: int) -> None:
+            converge_checks: bool, timeout: int, settle: int, noninteractive: bool, redeploy_wait: int) -> None:
     """
     Upgrades specified applications by executing the upgrade commands via the 'abra' command-line interface.
     It checks the current deployment status of the apps and performs upgrades only where necessary, with options to execute additional commands or perform a dry run. It either took the target version from the configuration or it uses the latest available version.
+    The apps are upgraded without waiting for each other, then the run blocks until every service of them is healthy, and only then the upgrade hooks are executed.
     """
     deployed_instance_apps = get_apps_by_deployment(recipes, deployed=True)
     upgrade_cmds = []
@@ -1805,7 +2033,7 @@ def upgrade(recipes: Tuple[str], execute_hooks: bool, dry_run: bool, redeploy: b
             elif version_type == 'unknown':
                 logging.warning(f"Cold not detect deployed version of {domain}")
                 continue
-            if not execute_hooks and not converge_checks:
+            if not converge_checks:
                 upgrade_cmd.append("--no-converge-checks")
             upgrade_cmd.append(domain)
             deployed_version_type = classify_version(deployed_version)
@@ -1822,7 +2050,7 @@ def upgrade(recipes: Tuple[str], execute_hooks: bool, dry_run: bool, redeploy: b
             app_details.append(upgrade_version)
             upgrade_apps.append(app_details)
             logging.info(f'upgrade {app}: {domain} from version {deployed_version} to version "{upgrade_version}"')
-            upgrade_cmds.append((app_config, upgrade_cmd, instance))
+            upgrade_cmds.append((app, app_config, upgrade_cmd, instance))
             if version_type == 'version':
                 release_note_cmd = upgrade_cmd.copy()
                 release_note_cmd.insert(1, '-r')
@@ -1830,15 +2058,26 @@ def upgrade(recipes: Tuple[str], execute_hooks: bool, dry_run: bool, redeploy: b
                 if (not (note:= release_notes.get(app))) or len(note) <= len(release_note):
                     release_notes[app] = release_note
         deployed_instance_apps[instance] = upgrade_apps
-    print_all_apps(deployed_instance_apps)
+    print_all_apps(deployed_instance_apps, headers=("from", "to"))
     if not noninteractive:
         input("Press any key to show release notes.")
     for app, note in release_notes.items():
         print(app)
         print(note)
-    if not dry_run and noninteractive or input(f"Do you really want to upgrade these apps? Type YES: ") == "YES":
-        for app_config, upgrade_cmd, instance_domain in upgrade_cmds:
+    if dry_run:
+        if execute_hooks:
+            # shown instead of run, so that a dry run also answers what the -e would have done
+            for _, app_config, _, instance_domain in upgrade_cmds:
+                execute_cmds(app_config, upgrade=True, dry_run=True, instance_domain=instance_domain)
+        return
+    if noninteractive or input("Do you really want to upgrade these apps? Type YES: ") == "YES":
+        for app, app_config, upgrade_cmd, instance_domain in upgrade_cmds:
             app_domain = app_config.get('app_domain')
+            # a recipe can bring a new secret along, which abra refuses to deploy without. The app
+            # is running, so nothing else can be missing: it had all of them when it was deployed
+            if missing := get_missing_secrets(app_domain):
+                print(f"{app_domain} is missing {len(missing)} secret(s): {', '.join(sorted(missing))}")
+                create_secrets((app,), instances=(instance_domain,))
             if redeploy:
                 upgrade_cmd.pop(0)
                 upgrade_cmd.insert(0, 'deploy')
@@ -1848,8 +2087,14 @@ def upgrade(recipes: Tuple[str], execute_hooks: bool, dry_run: bool, redeploy: b
                 sleep(redeploy_wait)
             print(f'deploy {app_domain}')
             print(abra("app", *upgrade_cmd))
-            if execute_hooks:
-                logging.info(f'execute commands for {app_domain}')
+        if not upgrade_cmds:
+            return
+        # the deployments were started without waiting for each other, so the services are still
+        # settling here and a hook would reach a container that is not up again yet
+        wait_for_healthy_apps(tuple({app for app, *_ in upgrade_cmds}), timeout, settle=settle)
+        if execute_hooks:
+            for app, app_config, _, instance_domain in upgrade_cmds:
+                logging.info(f"execute commands for {app_config.get('app_domain')}")
                 execute_cmds(app_config, upgrade=True, instance_domain=instance_domain)
 
 
@@ -1971,23 +2216,22 @@ def ls(recipes: Tuple[str], deployed: bool, undeployed: bool, domains: bool, ser
     else:
         instance_apps = get_apps(recipes)
     if domains:
-        print(list(zip(*sum(instance_apps.values(), [])))[1])
+        if apps_domains := sum(instance_apps.values(), []):
+            print(list(zip(*apps_domains))[1])
     else:
         print_all_apps(instance_apps)
 
 
-def print_all_apps(instance_apps: Dict[str, List[List]]) -> None:
+def print_all_apps(instance_apps: Dict[str, List[List]], headers: Tuple[str, str] = ("", "")) -> None:
     """
-    Prints a detailed list of all specified applications including their instances and domains. 
+    Prints one table per instance, listing its apps with their domain and whatever follows.
 
     Args:
-        apps (list): List of applications to print details for.
-
-    Returns:
-        dict: Returns a dictionary with application details grouped by instance.
+        instance_apps (dict): Instances as keys and lists of lists [app name, domain, ...] as values
+        headers (tuple): Titles for the two columns after the domain, unlabelled by default
     """
     for instance, apps in instance_apps.items():
-        print(tabulate(apps, headers=["",instance, "", ""], tablefmt='rounded_outline'))
+        print(tabulate(apps, headers=["", instance, *headers], tablefmt='rounded_outline'))
         print()
 
 def sort_apps_by_dependency(apps: List[str], instance_config: Dict[str, Any]) -> List[str]:
@@ -2295,8 +2539,10 @@ def format_recipe_secrets(recipe_secrets: Dict[str, List[str]]) -> str:
 @click.option('-e', '--execute-hooks', is_flag=True, help='run post-deployment commands.')
 @click.option('-c', '--converge-checks', is_flag=True, help='perform convergence checks during deployment.')
 @click.option('only', '-o', '--only', multiple=True, metavar='<conf|secret-hooks|exchange|generate>', help='Only run the listed secret variants when creating the secrets anew, all of them by default. Comma separated or specified multiple times.')
+@click.option('--timeout', default=PS_WAIT_TIMEOUT, show_default=True, metavar='<Seconds>', help='how long to wait for all services to become healthy.')
+@click.option('--settle', default=PS_WAIT_SETTLE, show_default=True, metavar='<Seconds>', help='how long to give the replaced services to be taken down before their health is read.')
 @click.option('noninteractive', '-n', '--non-interactive', is_flag=True, help='Run this command non-interactively')
-def reinsert_secrets(secrets: Tuple[str], execute_hooks: bool, converge_checks: bool, only: Tuple[str], noninteractive: bool) -> None:
+def reinsert_secrets(secrets: Tuple[str], execute_hooks: bool, converge_checks: bool, only: Tuple[str], timeout: int, settle: int, noninteractive: bool) -> None:
     """
     Rotates secrets: undeploys the affected apps, removes the given secrets, creates them anew and redeploys the apps that were deployed before.
 
@@ -2312,7 +2558,8 @@ def reinsert_secrets(secrets: Tuple[str], execute_hooks: bool, converge_checks:
     undeploy_apps(deployed_apps)
     purge_app_secrets(recipe_secrets)
     create_secrets(recipes, variants=variants)
-    deploy_apps(deployed_apps, execute_hooks=execute_hooks, converge_checks=converge_checks)
+    deploy_apps(deployed_apps, execute_hooks=execute_hooks, converge_checks=converge_checks,
+                timeout=timeout, settle=settle)
 
 
 @cli.command()
@@ -2393,22 +2640,31 @@ def print_containers(containers: List[Tuple[str, Dict[str, Any]]], verbose: bool
     print(tabulate(rows, 'firstrow', 'rounded_outline'), flush=True)
 
 
-def wait_for_healthy_apps(recipes: Tuple[str], timeout: int, verbose: bool = False) -> None:
+def wait_for_healthy_apps(recipes: Tuple[str], timeout: int, verbose: bool = False, settle: int = 0) -> None:
     """
     Blocks until no service of the selected apps is starting, unknown or unhealthy any more.
 
-    The status is checked right away and then every PS_WAIT_INTERVAL seconds until the timeout
-    is reached. Having nothing deployed is a failure rather than an early success: waiting for
-    the health of an environment that does not exist would report a broken deployment as green.
+    The status is checked after the settle delay and then every PS_WAIT_INTERVAL seconds until the
+    timeout is reached. Having nothing deployed is a failure rather than an early success: waiting
+    for the health of an environment that does not exist would report a broken deployment as green.
+
+    The delay matters wherever apps are being replaced: such a service keeps reporting its old
+    container for a moment, and that one is healthy, it has been running all along. Reading the
+    health right away would pass on the deployment that is about to go away. An environment that is
+    built from scratch has nothing to replace and leaves the delay at zero.
 
     Args:
         recipes (tuple): Recipes to wait for, all of them if empty
         timeout (int): Seconds to wait before giving up
         verbose (bool): Add the chaos, state and image columns to the final table
+        settle (int): Seconds to give replaced services to be taken down before the first check
 
     Raises:
         click.ClickException: If nothing is deployed, or if services are still not healthy when the timeout is reached
     """
+    if settle:
+        print(f"waiting {settle}s for the replaced services to be taken down", flush=True)
+        sleep(settle)
     deadline = time() + timeout
     while True:
         instance_apps = get_apps_by_deployment(recipes, deployed=True)
@@ -2525,7 +2781,8 @@ def setup_environment(recipes: Tuple[str], converge_checks: bool = False, timeou
         for level in group_apps_by_dependency(list(domains), instance_config):
             create_secrets(tuple(level), instances=(instance,), strict=True,
                            variants={"secret-hooks", "exchange", "generate"})
-            deploy_apps({instance: [[app, domains[app]] for app in level]}, converge_checks=converge_checks)
+            deploy_apps({instance: [[app, domains[app]] for app in level]},
+                        converge_checks=converge_checks, wait=False)
             for app in level:
                 run_readiness_hooks(domains[app], instance_config[app])
     wait_for_healthy_apps(recipes, timeout)

combine.yml

@@ -1,302 +0,0 @@
-authentik:
-    nextcloud:
-        uncomment:
-            - compose.nextcloud.yml
-            - NEXTCLOUD_DOMAIN
-            - SECRET_NEXTCLOUD_ID_VERSION
-            - SECRET_NEXTCLOUD_SECRET_VERSION
-            - nextcloud.png
-    wordpress:
-        uncomment:
-            - compose.wordpress.yml
-            - WORDPRESS_DOMAIN
-            - WORDPRESS_GROUP
-            - SECRET_WORDPRESS_ID_VERSION
-            - SECRET_WORDPRESS_SECRET_VERSION
-            - wordpress.png
-    matrix-synapse:
-        uncomment:
-            - compose.matrix.yml
-            - ELEMENT_DOMAIN
-            - MATRIX_DOMAIN
-            - SECRET_MATRIX_ID_VERSION
-            - SECRET_MATRIX_SECRET_VERSION
-            - matrix.svg
-        secrets:
-            matrix_id: matrix
-    wekan:
-        uncomment:
-            - compose.wekan.yml
-            - WEKAN_DOMAIN
-            - SECRET_WEKAN_ID_VERSION
-            - SECRET_WEKAN_SECRET_VERSION
-            - wekan.png
-        secrets:
-            wekan_id: wekan
-    vikunja:
-        uncomment:
-            - compose.vikunja.yml
-            - VIKUNJA_DOMAIN
-            - SECRET_VIKUNJA_ID_VERSION
-            - SECRET_VIKUNJA_SECRET_VERSION
-            - vikunja.svg
-        secrets:
-            vikunja_id: vikunja
-    kimai:
-        uncomment:
-            - compose.kimai.yml
-            - KIMAI_DOMAIN
-            - SECRET_KIMAI_ID_VERSION
-            - SECRET_KIMAI_SECRET_VERSION
-            - kimai_logo.png
-            - KIMAI_GROUP
-    zammad:
-        uncomment:
-            - compose.zammad.yml
-            - ZAMMAD_DOMAIN
-            - zammad.svg
-    monitoring-ng:
-        uncomment:
-            - compose.monitoring.yml
-            - MONITORING_DOMAIN
-            - SECRET_MONITORING_ID_VERSION
-            - SECRET_MONITORING_SECRET_VERSION
-            - monitoring.svg
-    outline:
-        uncomment:
-            - compose.outline.yml
-            - OUTLINE_DOMAIN
-            - SECRET_OUTLINE_ID_VERSION
-            - SECRET_OUTLINE_SECRET_VERSION
-            - outline.png
-        secrets:
-            outline_id: outline
-    rallly:
-        uncomment:  
-            - compose.rallly.yml
-            - RALLLY_DOMAIN
-            - SECRET_RALLLY_ID_VERSION
-            - SECRET_RALLLY_SECRET_VERSION
-            - rallly.png
-        secrets:
-            rallly_id: rallly
-    hedgedoc:
-        uncomment:  
-            - compose.hedgedoc.yml
-            - HEDGEDOC_DOMAIN
-            - SECRET_HEDGEDOC_ID_VERSION
-            - SECRET_HEDGEDOC_SECRET_VERSION
-            - hedgedoc.png
-        secrets:
-            hedgedoc_id: hedgedoc
-    pretix:
-        env:
-            APPLICATIONS: 
-                Pretix:
-                    url: https://pretix.example.com/control/
-                    group:
-            EXTRA_ICONS:
-                Pretix: ~/.abra/recipes/authentik/icons/pretix.svg
-    vaultwarden:
-        env:
-            APPLICATIONS: 
-                Vaultwarden:
-                    url: https://vaultwarden.example.com/
-                    group:
-            EXTRA_ICONS:
-                Vaultwarden: ~/.abra/recipes/authentik/icons/vaultwarden.svg
-    mila:
-        uncomment:
-            - compose.mila.yml
-            - MILA_DOMAIN
-            - MILA_GROUP
-            - SECRET_MILA_ID_VERSION
-            - SECRET_MILA_SECRET_VERSION
-            - mila.svg
-        secrets:
-            mila_id: mila
-kimai:
-    authentik:
-        uncomment:
-            - SSO_ENABLED
-            - SSO_PROVIDER_URL
-            - SSO_SAML_URL
-            - SSO_LOGOUT_URL
-        secret_hooks:
-            - insert_authentik_certificate
-        dependency: [authentik]
-zammad:
-    authentik:
-        uncomment:
-            - SSO_PROVIDER_DOMAIN
-            - IDP_SSO_TARGET_URL
-            - IDP_SLO_SERVICE_URL
-        initial-hooks:
-            - local enable_authentik_sso
-        dependency: [authentik]
-nextcloud:
-    authentik:
-        uncomment:
-            - compose.authentik.yml
-            - AUTHENTIK_USER_PREFIX
-            - AUTHENTIK_DOMAIN
-            - SECRET_AUTHENTIK_SECRET_VERSION
-            - SECRET_AUTHENTIK_ID_VERSION
-        initial-hooks:
-            - app set_authentik
-        shared_secrets:
-            nextcloud_secret: authentik_secret
-            nextcloud_id: authentik_id
-    onlyoffice:
-        uncomment:
-            - compose.onlyoffice.yml
-            - ONLYOFFICE_URL
-            - SECRET_ONLYOFFICE_JWT_VERSION
-        initial-hooks:
-            - app install_onlyoffice
-    collabora:
-        uncomment:
-            - COLLABORA_URL
-        initial-hooks:
-            - app install_collabora
-onlyoffice:
-    nextcloud:
-        uncomment:
-            - compose.jwt.yml
-            - SECRET_JWT_SECRET_VERSION
-        shared_secrets:
-            onlyoffice_jwt: jwt_secret
-outline:
-    authentik:
-        env: 
-            OIDC_CLIENT_ID: outline
-            OIDC_AUTH_URI: https://authentik.example.com/application/o/authorize/
-            OIDC_TOKEN_URI: https://authentik.example.com/application/o/token/
-            OIDC_USERINFO_URI: https://authentik.example.com/application/o/userinfo/
-            OIDC_DISPLAY_NAME: "Authentik"
-        uncomment:
-            - compose.oidc.yml
-            - OIDC_ENABLED
-            - OIDC_USERNAME_CLAIM
-            - OIDC_SCOPES
-            - SECRET_OIDC_CLIENT_SECRET_VERSION
-        shared_secrets:
-            outline_secret: oidc_client_secret
-wordpress:
-    authentik:
-        uncomment:
-            - compose.authentik.yml
-            - AUTHENTIK_DOMAIN
-            - SECRET_AUTHENTIK_SECRET_VERSION
-            - SECRET_AUTHENTIK_ID_VERSION
-            - LOGIN_TYPE
-        initial-hooks:
-            - app set_authentik
-        shared_secrets:
-            wordpress_secret: authentik_secret
-            wordpress_id: authentik_id
-vikunja:
-    authentik:
-        env:
-            OAUTH_NAME: authentik
-            OAUTH_URL: https://authentik.example.com/application/o/vikunja/
-            OAUTH_LOGOUT_URL: https://authentik.example.com/application/o/vikunja/end-session/
-            # TODO: set CLIENT_ID as secret
-            OAUTH_CLIENT_ID: vikunja
-        uncomment:
-            - compose.oauth.yml
-            - OAUTH_ENABLED
-            - SECRET_OAUTH_SECRET_VERSION
-        shared_secrets:
-            #vikunja_id: oauth_id
-            vikunja_secret: oauth_secret
-matrix-synapse:
-    authentik:
-        env:
-            KEYCLOAK_ID: authentik
-            KEYCLOAK_NAME: sso
-            KEYCLOAK_URL: https://authentik.example.com/application/o/matrix/
-            # TODO: correct client domain?
-            KEYCLOAK_CLIENT_DOMAIN: https://element-web.example.com
-            KEYCLOAK_ALLOW_EXISTING_USERS: "true"
-            # TODO: set CLIENT_ID as secret
-            KEYCLOAK_CLIENT_ID: matrix
-        uncomment:
-            - compose.keycloak.yml
-            - KEYCLOAK_ENABLED
-            - KEYCLOAK_CLIENT_ID
-            - SECRET_KEYCLOAK_CLIENT_SECRET_VERSION
-        shared_secrets:
-            matrix_secret: keycloak_client_secret
-        dependency: [authentik]
-traefik:
-    matrix-synapse:
-        uncomment:
-            - compose.matrix.yml
-            - MATRIX_FEDERATION_ENABLED
-rallly:
-    authentik:
-        env:
-            OIDC_NAME: "Authentik"
-            OIDC_DISCOVERY_URL: "https://authentik.example.com/application/o/rallly/.well-known/openid-configuration"
-            OIDC_ISSUER_URL: "https://authentik.example.com/application/o/rallly/"
-            OIDC_CLIENT_ID: rallly
-        uncomment: 
-            - compose.oidc.yml
-            - SECRET_OIDC_CLIENT_SECRET_VERSION
-            - OIDC_ENABLED
-        shared_secrets:
-            rallly_secret: oidc_client_secret
-wekan:
-    authentik:
-        env: 
-            OAUTH2_ENABLED: "true"
-            OAUTH2_SERVER_URL: https://authentik.example.com
-            # TODO: set CLIENT_ID as secret
-            OAUTH2_CLIENT_ID: wekan
-        uncomment:
-            - OAUTH2_LOGIN_STYLE
-            - OAUTH2_AUTH_ENDPOINT
-            - OAUTH2_USERINFO_ENDPOINT
-            - OAUTH2_TOKEN_ENDPOINT
-            - OAUTH2_REQUEST_PERMISSIONS
-            - OAUTH2_ID_MAP
-            - OAUTH2_USERNAME_MAP
-            - OAUTH2_FULLNAME_MAP
-            - OAUTH2_EMAIL_MAP
-            - PROPAGATE_OIDC_DATA
-            - OIDC_REDIRECTION_ENABLED
-        shared_secrets:
-            wekan_secret: oauth2_secret
-hedgedoc:
-    authentik:
-        env:
-            CMD_OAUTH2_USER_PROFILE_URL: https://authentik.example.com/application/o/userinfo/
-            CMD_OAUTH2_USER_PROFILE_USERNAME_ATTR: preferred_username
-            CMD_OAUTH2_USER_PROFILE_DISPLAY_NAME_ATTR: name
-            CMD_OAUTH2_USER_PROFILE_EMAIL_ATTR: email
-            CMD_OAUTH2_TOKEN_URL: https://authentik.example.com/application/o/token/
-            CMD_OAUTH2_AUTHORIZATION_URL: https://authentik.example.com/application/o/authorize/
-            # TODO: set CLIENT_ID as secret
-            CMD_OAUTH2_CLIENT_ID: hedgedoc
-            CMD_OAUTH2_PROVIDERNAME: Authentik
-        uncomment: 
-            - compose.oauth.yml
-            - SECRET_OAUTH_KEY_VERSION
-        shared_secrets:
-            hedgedoc_secret: oauth_key
-        dependency: [authentik]
-mila:
-    authentik:
-        env:
-            OIDC_CLIENT_ID: mila
-            OIDC_BASE_URL: https://authentik.example.com/application/o/mila
-            OIDC_REDIRECT_URI: https://mila.example.com/auth/user/oidc/callback
-            OIDC_ADMIN_GROUP_NAME: mv_admin
-            OIDC_GROUPS_CLAIM: groups
-            OIDC_ONLY: "true"
-        uncomment:
-            - compose.oidc.yml
-            - SECRET_OIDC_CLIENT_SECRET_VERSION
-        shared_secrets:
-            mila_secret: oidc_client_secret

examples/config-sets-combine.yml

@@ -0,0 +1,395 @@
+# Integrations, converted from the combine.yml that alakazam used to ship.
+# A set applies as soon as every app of its 'apply-when' is part of an instance.
+# Copy this file into your alakazam root path, next to alaka-versions.yml.
+
+integrate-authentik-nextcloud:
+    apply-when:
+      - authentik
+      - nextcloud
+    authentik:
+        uncomment:
+          - compose.nextcloud.yml
+          - NEXTCLOUD_DOMAIN
+          - SECRET_NEXTCLOUD_ID_VERSION
+          - SECRET_NEXTCLOUD_SECRET_VERSION
+          - nextcloud.png
+    nextcloud:
+        uncomment:
+          - compose.authentik.yml
+          - AUTHENTIK_USER_PREFIX
+          - AUTHENTIK_DOMAIN
+          - SECRET_AUTHENTIK_SECRET_VERSION
+          - SECRET_AUTHENTIK_ID_VERSION
+        initial-hooks:
+          - app set_authentik
+        shared_secrets:
+            authentik:
+                nextcloud_secret: authentik_secret
+                nextcloud_id: authentik_id
+integrate-authentik-wordpress:
+    apply-when:
+      - authentik
+      - wordpress
+    authentik:
+        uncomment:
+          - compose.wordpress.yml
+          - WORDPRESS_DOMAIN
+          - WORDPRESS_GROUP
+          - SECRET_WORDPRESS_ID_VERSION
+          - SECRET_WORDPRESS_SECRET_VERSION
+          - wordpress.png
+    wordpress:
+        uncomment:
+          - compose.authentik.yml
+          - AUTHENTIK_DOMAIN
+          - SECRET_AUTHENTIK_SECRET_VERSION
+          - SECRET_AUTHENTIK_ID_VERSION
+          - LOGIN_TYPE
+        initial-hooks:
+          - app set_authentik
+        shared_secrets:
+            authentik:
+                wordpress_secret: authentik_secret
+                wordpress_id: authentik_id
+integrate-authentik-matrix-synapse:
+    apply-when:
+      - authentik
+      - matrix-synapse
+    authentik:
+        uncomment:
+          - compose.matrix.yml
+          - ELEMENT_DOMAIN
+          - MATRIX_DOMAIN
+          - SECRET_MATRIX_ID_VERSION
+          - SECRET_MATRIX_SECRET_VERSION
+          - matrix.svg
+        secrets:
+            matrix_id: matrix
+    matrix-synapse:
+        env:
+            KEYCLOAK_ID: authentik
+            KEYCLOAK_NAME: sso
+            KEYCLOAK_URL: https://authentik.example.com/application/o/matrix/
+            # TODO: correct client domain?
+            KEYCLOAK_CLIENT_DOMAIN: https://element-web.example.com
+            KEYCLOAK_ALLOW_EXISTING_USERS: "true"
+            # TODO: set CLIENT_ID as secret
+            KEYCLOAK_CLIENT_ID: matrix
+        uncomment:
+          - compose.keycloak.yml
+          - KEYCLOAK_ENABLED
+          - KEYCLOAK_CLIENT_ID
+          - SECRET_KEYCLOAK_CLIENT_SECRET_VERSION
+        shared_secrets:
+            authentik:
+                matrix_secret: keycloak_client_secret
+        dependency: [authentik]
+integrate-authentik-wekan:
+    apply-when:
+      - authentik
+      - wekan
+    authentik:
+        uncomment:
+          - compose.wekan.yml
+          - WEKAN_DOMAIN
+          - SECRET_WEKAN_ID_VERSION
+          - SECRET_WEKAN_SECRET_VERSION
+          - wekan.png
+        secrets:
+            wekan_id: wekan
+    wekan:
+        env:
+            OAUTH2_ENABLED: "true"
+            OAUTH2_SERVER_URL: https://authentik.example.com
+            # TODO: set CLIENT_ID as secret
+            OAUTH2_CLIENT_ID: wekan
+        uncomment:
+          - OAUTH2_LOGIN_STYLE
+          - OAUTH2_AUTH_ENDPOINT
+          - OAUTH2_USERINFO_ENDPOINT
+          - OAUTH2_TOKEN_ENDPOINT
+          - OAUTH2_REQUEST_PERMISSIONS
+          - OAUTH2_ID_MAP
+          - OAUTH2_USERNAME_MAP
+          - OAUTH2_FULLNAME_MAP
+          - OAUTH2_EMAIL_MAP
+          - PROPAGATE_OIDC_DATA
+          - OIDC_REDIRECTION_ENABLED
+        shared_secrets:
+            authentik:
+                wekan_secret: oauth2_secret
+integrate-authentik-vikunja:
+    apply-when:
+      - authentik
+      - vikunja
+    authentik:
+        uncomment:
+          - compose.vikunja.yml
+          - VIKUNJA_DOMAIN
+          - SECRET_VIKUNJA_ID_VERSION
+          - SECRET_VIKUNJA_SECRET_VERSION
+          - vikunja.svg
+        secrets:
+            vikunja_id: vikunja
+    vikunja:
+        env:
+            OAUTH_NAME: authentik
+            OAUTH_URL: https://authentik.example.com/application/o/vikunja/
+            OAUTH_LOGOUT_URL: https://authentik.example.com/application/o/vikunja/end-session/
+            # TODO: set CLIENT_ID as secret
+            OAUTH_CLIENT_ID: vikunja
+        uncomment:
+          - compose.oauth.yml
+          - OAUTH_ENABLED
+          - SECRET_OAUTH_SECRET_VERSION
+        shared_secrets:
+            #vikunja_id: oauth_id
+            authentik:
+                vikunja_secret: oauth_secret
+integrate-authentik-kimai:
+    apply-when:
+      - authentik
+      - kimai
+    authentik:
+        uncomment:
+          - compose.kimai.yml
+          - KIMAI_DOMAIN
+          - SECRET_KIMAI_ID_VERSION
+          - SECRET_KIMAI_SECRET_VERSION
+          - kimai_logo.png
+          - KIMAI_GROUP
+    kimai:
+        uncomment:
+          - SSO_ENABLED
+          - SSO_PROVIDER_URL
+          - SSO_SAML_URL
+          - SSO_LOGOUT_URL
+          - SSO_ADMIN_GROUP_NAME
+        secret_hooks:
+          - insert_authentik_certificate
+        dependency: [authentik]
+integrate-authentik-zammad:
+    apply-when:
+      - authentik
+      - zammad
+    authentik:
+        uncomment:
+          - compose.zammad.yml
+          - ZAMMAD_DOMAIN
+          - zammad.svg
+    zammad:
+        uncomment:
+          - SSO_PROVIDER_DOMAIN
+          - IDP_SSO_TARGET_URL
+          - IDP_SLO_SERVICE_URL
+        initial-hooks:
+          - local enable_authentik_sso
+        dependency: [authentik]
+integrate-authentik-monitoring-ng:
+    apply-when:
+      - authentik
+      - monitoring-ng
+    authentik:
+        uncomment:
+          - compose.monitoring.yml
+          - MONITORING_DOMAIN
+          - SECRET_MONITORING_ID_VERSION
+          - SECRET_MONITORING_SECRET_VERSION
+          - monitoring.svg
+        secrets:
+            monitoring_id: monitoring
+    monitoring-ng:
+        env:
+            GF_SERVER_ROOT_URL: https://monitoring-ng.example.com
+            OIDC_CLIENT_ID: monitoring
+            OIDC_AUTH_URL: https://authentik.example.com/application/o/authorize/
+            OIDC_API_URL: https://authentik.example.com/application/o/userinfo/
+            OIDC_TOKEN_URL: https://authentik.example.com/application/o/token/
+        uncomment:
+          - compose.prometheus.yml
+          - PROMETHEUS_RETENTION_TIME
+          - compose.loki.yml
+          - LOKI_RETENTION_PERIOD
+          - LOKI_STORAGE_FILESYSTEM
+          - compose.grafana.yml
+          - OIDC_ENABLED
+          - SECRET_GRAFANA_ADMIN_PASSWORD_VERSION
+          - SECRET_GRAFANA_OIDC_CLIENT_SECRET_VERSION
+          - SECRET_GRAFANA_SMTP_PASSWORD_VERSION
+        shared_secrets:
+            authentik:
+                monitoring_secret: grafana_oidc_client_secret
+        dependency: [authentik]
+integrate-authentik-outline:
+    apply-when:
+      - authentik
+      - outline
+    authentik:
+        uncomment:
+          - compose.outline.yml
+          - OUTLINE_DOMAIN
+          - SECRET_OUTLINE_ID_VERSION
+          - SECRET_OUTLINE_SECRET_VERSION
+          - outline.png
+        secrets:
+            outline_id: outline
+    outline:
+        env:
+            OIDC_CLIENT_ID: outline
+            OIDC_AUTH_URI: https://authentik.example.com/application/o/authorize/
+            OIDC_TOKEN_URI: https://authentik.example.com/application/o/token/
+            OIDC_USERINFO_URI: https://authentik.example.com/application/o/userinfo/
+            OIDC_DISPLAY_NAME: "Authentik"
+        uncomment:
+          - compose.oidc.yml
+          - OIDC_ENABLED
+          - OIDC_USERNAME_CLAIM
+          - OIDC_SCOPES
+          - SECRET_OIDC_CLIENT_SECRET_VERSION
+        shared_secrets:
+            authentik:
+                outline_secret: oidc_client_secret
+integrate-authentik-rallly:
+    apply-when:
+      - authentik
+      - rallly
+    authentik:
+        uncomment:
+          - compose.rallly.yml
+          - RALLLY_DOMAIN
+          - SECRET_RALLLY_ID_VERSION
+          - SECRET_RALLLY_SECRET_VERSION
+          - rallly.png
+        secrets:
+            rallly_id: rallly
+    rallly:
+        env:
+            OIDC_NAME: "Authentik"
+            OIDC_DISCOVERY_URL: "https://authentik.example.com/application/o/rallly/.well-known/openid-configuration"
+            OIDC_ISSUER_URL: "https://authentik.example.com/application/o/rallly/"
+            OIDC_CLIENT_ID: rallly
+        uncomment:
+          - compose.oidc.yml
+          - SECRET_OIDC_CLIENT_SECRET_VERSION
+          - OIDC_ENABLED
+        shared_secrets:
+            authentik:
+                rallly_secret: oidc_client_secret
+integrate-authentik-hedgedoc:
+    apply-when:
+      - authentik
+      - hedgedoc
+    authentik:
+        uncomment:
+          - compose.hedgedoc.yml
+          - HEDGEDOC_DOMAIN
+          - SECRET_HEDGEDOC_ID_VERSION
+          - SECRET_HEDGEDOC_SECRET_VERSION
+          - hedgedoc.png
+        secrets:
+            hedgedoc_id: hedgedoc
+    hedgedoc:
+        env:
+            CMD_OAUTH2_USER_PROFILE_URL: https://authentik.example.com/application/o/userinfo/
+            CMD_OAUTH2_USER_PROFILE_USERNAME_ATTR: preferred_username
+            CMD_OAUTH2_USER_PROFILE_DISPLAY_NAME_ATTR: name
+            CMD_OAUTH2_USER_PROFILE_EMAIL_ATTR: email
+            CMD_OAUTH2_TOKEN_URL: https://authentik.example.com/application/o/token/
+            CMD_OAUTH2_AUTHORIZATION_URL: https://authentik.example.com/application/o/authorize/
+            # TODO: set CLIENT_ID as secret
+            CMD_OAUTH2_CLIENT_ID: hedgedoc
+            CMD_OAUTH2_PROVIDERNAME: Authentik
+        uncomment:
+          - compose.oauth.yml
+          - SECRET_OAUTH_KEY_VERSION
+        shared_secrets:
+            authentik:
+                hedgedoc_secret: oauth_key
+        dependency: [authentik]
+integrate-authentik-pretix:
+    apply-when:
+      - authentik
+      - pretix
+    authentik:
+        env:
+            APPLICATIONS:
+                Pretix:
+                    url: https://pretix.example.com/control/
+                    group:
+            EXTRA_ICONS:
+                Pretix: ~/.abra/recipes/authentik/icons/pretix.svg
+integrate-authentik-vaultwarden:
+    apply-when:
+      - authentik
+      - vaultwarden
+    authentik:
+        env:
+            APPLICATIONS:
+                Vaultwarden:
+                    url: https://vaultwarden.example.com/
+                    group:
+            EXTRA_ICONS:
+                Vaultwarden: ~/.abra/recipes/authentik/icons/vaultwarden.svg
+integrate-authentik-mila:
+    apply-when:
+      - authentik
+      - mila
+    authentik:
+        uncomment:
+          - compose.mila.yml
+          - MILA_DOMAIN
+          - MILA_GROUP
+          - SECRET_MILA_ID_VERSION
+          - SECRET_MILA_SECRET_VERSION
+          - mila.svg
+        secrets:
+            mila_id: mila
+    mila:
+        env:
+            OIDC_CLIENT_ID: mila
+            OIDC_BASE_URL: https://authentik.example.com/application/o/mila
+            OIDC_REDIRECT_URI: https://mila.example.com/auth/user/oidc/callback
+            OIDC_ADMIN_GROUP_NAME: mv_admin
+            OIDC_GROUPS_CLAIM: groups
+            OIDC_ONLY: "true"
+        uncomment:
+          - compose.oidc.yml
+          - SECRET_OIDC_CLIENT_SECRET_VERSION
+        shared_secrets:
+            authentik:
+                mila_secret: oidc_client_secret
+integrate-nextcloud-onlyoffice:
+    apply-when:
+      - nextcloud
+      - onlyoffice
+    nextcloud:
+        uncomment:
+          - compose.onlyoffice.yml
+          - ONLYOFFICE_URL
+          - SECRET_ONLYOFFICE_JWT_VERSION
+        initial-hooks:
+          - app install_onlyoffice
+    onlyoffice:
+        uncomment:
+          - compose.jwt.yml
+          - SECRET_JWT_SECRET_VERSION
+        shared_secrets:
+            nextcloud:
+                onlyoffice_jwt: jwt_secret
+integrate-collabora-nextcloud:
+    apply-when:
+      - collabora
+      - nextcloud
+    nextcloud:
+        uncomment:
+          - COLLABORA_URL
+        initial-hooks:
+          - app install_collabora
+integrate-matrix-synapse-traefik:
+    apply-when:
+      - matrix-synapse
+      - traefik
+    traefik:
+        uncomment:
+          - compose.matrix.yml
+          - MATRIX_FEDERATION_ENABLED

requirements.txt

@@ -7,4 +7,4 @@ python-dotenv==1.0.0
 icecream==2.1.3
 packaging==24.0 
 GitPython==3.1.43
-uptime_kuma_api==1.2.1
+uptime-kuma-api2==2.7.0

tests/test_config_sets.py

@@ -0,0 +1,191 @@
+"""Tests for config-sets, in particular the ones that apply by themselves."""
+
+import logging
+import os
+import sys
+
+import pytest
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+import alakazam
+from alakazam import get_config_set_app_configs, merge_instance_configs, read_config_sets
+
+INTEGRATION = {
+    "integrate-authentik-nextcloud": {
+        "apply-when": ["authentik", "nextcloud"],
+        "authentik": {"uncomment": ["compose.nextcloud.yml"]},
+        "nextcloud": {"uncomment": ["compose.authentik.yml"],
+                      "shared_secrets": {"authentik": {"nextcloud_secret": "authentik_secret"}}},
+    }
+}
+
+
+def apply(config_sets, instance_apps, active=None):
+    """The effective configuration, with the automatic sets below the explicit ones."""
+    automatic, explicit = get_config_set_app_configs(active or {}, config_sets, instance_apps)
+    return alakazam.merge_dict(automatic, explicit)
+
+
+class TestApplyWhen:
+    def test_it_applies_when_every_app_is_there(self):
+        result = apply(INTEGRATION, ["authentik", "nextcloud", "traefik"])
+        assert result["authentik"]["uncomment"] == ["compose.nextcloud.yml"]
+        assert result["nextcloud"]["shared_secrets"] == {"authentik": {"nextcloud_secret": "authentik_secret"}}
+
+    def test_it_stays_out_when_one_app_is_missing(self):
+        assert apply(INTEGRATION, ["authentik", "traefik"]) == {}
+
+    def test_apply_when_is_not_passed_on_as_an_app(self):
+        """It is a condition, not configuration, and would end up in an .env otherwise."""
+        assert "apply-when" not in apply(INTEGRATION, ["authentik", "nextcloud"])
+
+    def test_an_instance_can_switch_it_off(self):
+        assert apply(INTEGRATION, ["authentik", "nextcloud"],
+                     active={"integrate-authentik-nextcloud": False}) == {}
+
+    def test_a_one_sided_integration_configures_only_one_app(self):
+        """4 of the 17 pairs in the shipped combine.yml were one-sided."""
+        sets = {"integrate-matrix-synapse-traefik": {
+            "apply-when": ["matrix-synapse", "traefik"],
+            "traefik": {"uncomment": ["compose.matrix.yml"]}}}
+        assert list(apply(sets, ["matrix-synapse", "traefik"])) == ["traefik"]
+        assert apply(sets, ["traefik"]) == {}
+
+
+class TestExplicitSets:
+    SET = {"bbb": {"authentik": {"env": {"A": "b"}}}}
+
+    def test_an_enabled_set_applies(self):
+        assert apply(self.SET, ["authentik"], active={"bbb": True})["authentik"]["env"] == {"A": "b"}
+
+    def test_a_set_that_is_not_enabled_does_not_apply(self):
+        assert apply(self.SET, ["authentik"]) == {}
+
+    def test_an_unknown_name_is_reported(self, caplog):
+        with caplog.at_level(logging.WARNING):
+            apply(self.SET, ["authentik"], active={"typo": True})
+        assert "'typo' is enabled but not defined" in caplog.text
+
+    def test_an_explicit_set_refines_an_automatic_one(self):
+        """Automatic integrations are the base, an explicit set may still override them."""
+        sets = dict(INTEGRATION)
+        sets["override"] = {"authentik": {"env": {"SOURCE": "explicit"}}}
+        result = apply(sets, ["authentik", "nextcloud"], active={"override": True})
+        assert result["authentik"]["env"] == {"SOURCE": "explicit"}
+        assert result["authentik"]["uncomment"] == ["compose.nextcloud.yml"]
+
+
+class TestReadConfigSets:
+    def write(self, root, name, text):
+        (root / name).write_text(text)
+
+    def test_the_base_file_is_read(self, tmp_path):
+        self.write(tmp_path, "config-sets.yml", "bbb:\n    authentik:\n        env:\n            A: b\n")
+        assert read_config_sets(tmp_path)["bbb"]["authentik"]["env"] == {"A": "b"}
+
+    def test_further_files_are_merged(self, tmp_path):
+        self.write(tmp_path, "config-sets.yml", "bbb:\n    authentik: {}\n")
+        self.write(tmp_path, "config-sets-authentik.yml", "integrate:\n    apply-when: [a, b]\n")
+        assert sorted(read_config_sets(tmp_path)) == ["bbb", "integrate"]
+
+    def test_the_base_file_is_merged_first(self, tmp_path):
+        self.write(tmp_path, "config-sets.yml", "bbb:\n    authentik:\n        env:\n            A: base\n")
+        self.write(tmp_path, "config-sets-later.yml", "bbb:\n    authentik:\n        env:\n            A: later\n")
+        assert read_config_sets(tmp_path)["bbb"]["authentik"]["env"]["A"] == "later"
+
+    def test_a_root_without_config_sets(self, tmp_path):
+        assert read_config_sets(tmp_path) == {}
+
+    def test_a_key_defined_twice_is_reported(self, tmp_path, caplog):
+        self.write(tmp_path, "config-sets.yml", "bbb:\n    authentik:\n        env:\n            A: base\n")
+        self.write(tmp_path, "config-sets-later.yml", "bbb:\n    authentik:\n        env:\n            A: later\n")
+        with caplog.at_level(logging.WARNING):
+            read_config_sets(tmp_path)
+        assert "bbb.authentik.env.A" in caplog.text
+
+
+class TestIntegrationReachesTheMergedConfig:
+    """The instance config is what every command reads, the integration has to arrive there."""
+
+    def merge(self, apps):
+        return merge_instance_configs({}, "example.com", {app: None for app in apps}, INTEGRATION)
+
+    def test_both_apps_get_their_side(self):
+        merged = self.merge(["authentik", "nextcloud"])
+        assert merged["authentik"]["uncomment"] == ["compose.nextcloud.yml"]
+        assert merged["nextcloud"]["uncomment"] == ["compose.authentik.yml"]
+
+    def test_a_lone_app_gets_nothing(self):
+        assert "uncomment" not in self.merge(["authentik"])["authentik"]
+
+
+class TestUnknownAppKeys:
+    """A misspelt app key drops its configuration, which is only visible if it is reported."""
+
+    def test_a_typo_in_an_automatic_set_is_reported(self, caplog):
+        sets = {"integrate": {"apply-when": ["authentik", "nextcloud"],
+                              "nextcoud": {"env": {"A": "b"}}}}
+        with caplog.at_level(logging.WARNING):
+            merge_instance_configs({}, "example.com", {"authentik": None, "nextcloud": None}, sets)
+        assert "'nextcoud' is not listed" in caplog.text
+
+    def test_a_typo_in_an_explicit_set_is_reported(self, caplog):
+        sets = {"bbb": {"nextcoud": {"env": {"A": "b"}}}}
+        with caplog.at_level(logging.WARNING):
+            merge_instance_configs({}, "example.com",
+                                   {"nextcloud": None, "CONFIG-SETS": {"bbb": True}}, sets)
+        assert "'nextcoud' is not listed" in caplog.text
+
+    def test_a_correct_app_key_is_not_reported(self, caplog):
+        sets = {"integrate": {"apply-when": ["authentik", "nextcloud"],
+                              "nextcloud": {"env": {"A": "b"}}}}
+        with caplog.at_level(logging.WARNING):
+            merge_instance_configs({}, "example.com", {"authentik": None, "nextcloud": None}, sets)
+        assert caplog.text == ""
+
+
+class TestPrecedence:
+    """An integration is the lowest layer, as combine.yml was before it."""
+
+    def test_the_group_configuration_beats_an_integration(self):
+        group = {"nextcloud": {"env": {"SOURCE": "group"}}}
+        sets = {"integrate": {"apply-when": ["authentik", "nextcloud"],
+                              "nextcloud": {"env": {"SOURCE": "integration"}}}}
+        merged = merge_instance_configs(group, "example.com", {"authentik": None, "nextcloud": None}, sets)
+        assert merged["nextcloud"]["env"]["SOURCE"] == "group"
+
+    def test_the_instance_beats_everything(self):
+        group = {"nextcloud": {"env": {"SOURCE": "group"}}}
+        sets = {"integrate": {"apply-when": ["authentik", "nextcloud"],
+                              "nextcloud": {"env": {"SOURCE": "integration"}}}}
+        merged = merge_instance_configs(
+            group, "example.com", {"authentik": None, "nextcloud": {"env": {"SOURCE": "instance"}}}, sets)
+        assert merged["nextcloud"]["env"]["SOURCE"] == "instance"
+
+    def test_an_integration_still_adds_what_nobody_else_sets(self):
+        group = {"nextcloud": {"env": {"OTHER": "group"}}}
+        sets = {"integrate": {"apply-when": ["authentik", "nextcloud"],
+                              "nextcloud": {"env": {"SOURCE": "integration"}}}}
+        merged = merge_instance_configs(group, "example.com", {"authentik": None, "nextcloud": None}, sets)
+        assert merged["nextcloud"]["env"] == {"SOURCE": "integration", "OTHER": "group"}
+
+
+class TestHookOrder:
+    """An integration adds to an app that already configures itself, and must come after it."""
+
+    SETS = {"integrate-authentik-wordpress": {
+        "apply-when": ["authentik", "wordpress"],
+        "wordpress": {"initial-hooks": ["app set_authentik"]}}}
+
+    def test_the_integration_hook_runs_last(self):
+        """set_authentik configures SSO in a WordPress that core_install has to create first."""
+        group = {"wordpress": {"initial-hooks": ["app core_install", "app enable_auto_updates"]}}
+        merged = merge_instance_configs(group, "example.com", {"authentik": None, "wordpress": None}, self.SETS)
+        assert merged["wordpress"]["initial-hooks"] == [
+            "app core_install", "app enable_auto_updates", "app set_authentik"]
+
+    def test_an_instance_hook_also_comes_first(self):
+        instance = {"authentik": None, "wordpress": {"initial-hooks": ["app from_instance"]}}
+        merged = merge_instance_configs({}, "example.com", instance, self.SETS)
+        assert merged["wordpress"]["initial-hooks"] == ["app from_instance", "app set_authentik"]

tests/test_deploy_hooks.py

@@ -0,0 +1,132 @@
+"""Tests for what a deploy run does: deploy everything, wait for all of it, then hook."""
+
+import os
+import sys
+
+import pytest
+from click.testing import CliRunner
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+import alakazam
+from alakazam import deploy_apps
+
+CONFIG = {"a.org": {
+    "nextcloud": {"app_domain": "cloud.a.org", "server": "a.org", "version": "2.0.0"},
+    "wekan": {"app_domain": "boards.a.org", "server": "a.org", "version": "2.0.0"},
+}}
+
+
+@pytest.fixture
+def steps(monkeypatch):
+    """Records what a deploy run does, in order, against an abra that answers every call."""
+    steps = []
+    monkeypatch.setattr(alakazam, "INSTANCE_CONFIGS", CONFIG)
+    monkeypatch.setattr(alakazam, "sleep", lambda seconds: steps.append(f"sleep:{seconds}"))
+    monkeypatch.setattr(alakazam, "execute_cmds",
+                        lambda config, **kw: steps.append(f"hook:{config['app_domain']}"))
+    monkeypatch.setattr(alakazam, "get_apps", lambda recipes, **kw: {
+        "a.org": [[app, config["app_domain"]] for app, config in CONFIG["a.org"].items()
+                  if not recipes or app in recipes]})
+    monkeypatch.setattr(alakazam, "get_apps_by_deployment", lambda recipes, deployed=True: {
+        "a.org": [[app, config["app_domain"]] + (["2.0.0"] if deployed else [])
+                  for app, config in CONFIG["a.org"].items() if not recipes or app in recipes]})
+
+    def abra(*args, **kwargs):
+        if args[:2] == ("app", "ps"):
+            steps.append(f"ps:{args[2]}")
+            return {"app": {"service": "app", "status": "healthy", "version": "2.0.0",
+                            "chaos": "false", "state": "running", "image": "img"}}
+        steps.append(" ".join(a for a in args if a))
+        return ""
+
+    monkeypatch.setattr(alakazam, "abra", abra)
+    return steps
+
+
+def selected(steps, prefix):
+    return [step for step in steps if step.startswith(prefix)]
+
+
+def run(args):
+    result = CliRunner().invoke(alakazam.deploy, ["-n"] + args, standalone_mode=False)
+    assert result.exit_code == 0, result.exception
+    return result
+
+
+class TestDeployHooks:
+    def test_the_hooks_do_not_switch_on_converge_checks(self, steps):
+        """Waiting for each app in turn is what the separate flag is for."""
+        run(["-e"])
+        assert all("--no-converge-checks" in s for s in selected(steps, "app deploy"))
+
+    def test_converge_checks_come_from_their_own_flag_only(self, steps):
+        run(["-c"])
+        assert not any("--no-converge-checks" in s for s in selected(steps, "app deploy"))
+
+    def test_every_app_is_deployed_before_the_first_hook(self, steps):
+        run(["-e"])
+        deploys = selected(steps, "app deploy")
+        assert max(steps.index(s) for s in deploys) < steps.index("hook:cloud.a.org")
+
+    def test_the_hooks_run_after_the_wait(self, steps):
+        """A hook would otherwise reach a container the deployment has not brought up yet."""
+        run(["-e"])
+        assert steps.index("ps:cloud.a.org") < steps.index("hook:cloud.a.org")
+        assert steps.index("ps:boards.a.org") < steps.index("hook:boards.a.org")
+
+    def test_the_health_is_only_read_after_the_services_settled(self, steps):
+        """A replaced service still reports its old container, which has been healthy all along."""
+        run(["-e"])
+        assert steps.index(f"sleep:{alakazam.PS_WAIT_SETTLE}") < steps.index("ps:cloud.a.org")
+        assert max(steps.index(s) for s in selected(steps, "app deploy")) \
+            < steps.index(f"sleep:{alakazam.PS_WAIT_SETTLE}")
+
+    def test_without_the_flag_nothing_is_hooked(self, steps):
+        run([])
+        assert selected(steps, "hook:") == []
+
+    def test_the_wait_happens_without_the_flag_too(self, steps):
+        """It is what tells a deploy run whether the apps came up at all."""
+        run([])
+        assert selected(steps, "ps:") == ["ps:cloud.a.org", "ps:boards.a.org"]
+
+    def test_a_filtered_run_waits_for_its_own_apps_only(self, steps):
+        run(["-e", "-r", "wekan"])
+        assert selected(steps, "ps:") == ["ps:boards.a.org"]
+        assert selected(steps, "hook:") == ["hook:boards.a.org"]
+
+
+class TestDeployAppsWithoutWaiting:
+    """setup arranges its own waiting, per dependency level and once for the whole environment."""
+
+    def test_the_wait_can_be_opted_out_of(self, steps):
+        deploy_apps({"a.org": [["nextcloud", "cloud.a.org"]]}, wait=False)
+        assert selected(steps, "ps:") == []
+        assert selected(steps, "sleep:") == []
+
+    def test_the_hooks_still_run_without_a_wait(self, steps):
+        deploy_apps({"a.org": [["nextcloud", "cloud.a.org"]]}, execute_hooks=True, wait=False)
+        assert selected(steps, "hook:") == ["hook:cloud.a.org"]
+
+    def test_nothing_to_deploy_skips_the_wait(self, steps):
+        """Waiting on a run that did not happen would report on unrelated apps."""
+        deploy_apps({})
+        assert steps == []
+
+
+class TestSettleOption:
+    def test_the_delay_can_be_set(self, steps):
+        run(["--settle", "5"])
+        assert "sleep:5" in steps
+        assert f"sleep:{alakazam.PS_WAIT_SETTLE}" not in steps
+
+    def test_zero_skips_the_delay_without_skipping_the_wait(self, steps):
+        """A converge-checked run has waited already, the health still has to be confirmed."""
+        run(["--settle", "0"])
+        assert selected(steps, "sleep:") == []
+        assert selected(steps, "ps:") == ["ps:cloud.a.org", "ps:boards.a.org"]
+
+    def test_the_default_is_used_when_it_is_not_given(self, steps):
+        run([])
+        assert f"sleep:{alakazam.PS_WAIT_SETTLE}" in steps

tests/test_env_updates.py

@@ -0,0 +1,144 @@
+"""Tests for how update_configs() applies a config's env/comment/uncomment keys to a .env file."""
+
+import logging
+import os
+import sys
+
+from dotenv import dotenv_values
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from alakazam import format_env_value, set_extending_key, update_configs
+
+
+class TestUnknownConfigKeys:
+    def test_an_unknown_key_never_reaches_the_env_file(self, tmp_path):
+        """A promotion mechanism sets 'hold: <reason>' on a recipe, which must pass through untouched."""
+        env = tmp_path / "app.env"
+        env.write_text("#SMTP_HOST=mail.example.com\n")
+        update_configs(env, {"hold": "waiting for the upstream fix", "env": {"SMTP_HOST": "mail.a.org"}})
+        content = env.read_text()
+        assert "SMTP_HOST=mail.a.org" in content
+        assert "hold" not in content
+        assert "waiting for the upstream fix" not in content
+
+    def test_an_unknown_key_alone_changes_nothing(self, tmp_path):
+        env = tmp_path / "app.env"
+        env.write_text("#SMTP_HOST=mail.example.com\n")
+        update_configs(env, {"hold": "waiting for the upstream fix"})
+        assert env.read_text() == "#SMTP_HOST=mail.example.com\n"
+
+
+class TestExtendingEnvKeys:
+    """
+    Some env keys are repeated on multiple lines, relying on bash accumulation
+    ("COMPOSE_FILE=\"$COMPOSE_FILE:compose.mariadb.yml\"") to build up a value.
+    A config that sets such a key should extend that accumulation, not replace
+    or uncomment every line sharing the key.
+    """
+
+    ACCUMULATING_ENV = (
+        'COMPOSE_FILE="compose.yml"\n'
+        'COMPOSE_FILE="$COMPOSE_FILE:compose.mariadb.yml"\n'
+        '# COMPOSE_FILE="$COMPOSE_FILE:compose.onlyoffice.yml"\n'
+    )
+
+    def test_a_key_sharing_a_prefix_is_not_matched_by_substring(self, tmp_path):
+        """COMPOSE must not be treated as matching lines whose key is COMPOSE_FILE."""
+        env = tmp_path / "app.env"
+        env.write_text(self.ACCUMULATING_ENV)
+        update_configs(env, {"env": {"COMPOSE": "compose.custom.yml"}})  # not self-referencing
+        lines = env.read_text().splitlines()
+        assert lines[:-1] == self.ACCUMULATING_ENV.splitlines()  # every original line untouched
+        assert lines[-1] == 'COMPOSE=compose.custom.yml'  # set as its own, unrelated key
+
+    def test_a_self_referencing_value_is_appended_without_touching_other_lines(self, tmp_path):
+        env = tmp_path / "app.env"
+        env.write_text(self.ACCUMULATING_ENV)
+        update_configs(env, {"env": {"COMPOSE_FILE": "$COMPOSE_FILE:../../customoverride.yml"}})
+        lines = env.read_text().splitlines()
+        assert lines[:-1] == self.ACCUMULATING_ENV.splitlines()  # every original line untouched
+        assert lines[-1] == 'COMPOSE_FILE="$COMPOSE_FILE:../../customoverride.yml"'  # appended
+
+    def test_setting_the_same_self_referencing_value_twice_is_idempotent(self, tmp_path):
+        env = tmp_path / "app.env"
+        env.write_text(self.ACCUMULATING_ENV)
+        config = {"env": {"COMPOSE_FILE": "$COMPOSE_FILE:../../customoverride.yml"}}
+        update_configs(env, config)
+        update_configs(env, config)
+        lines = env.read_text().splitlines()
+        assert lines.count('COMPOSE_FILE="$COMPOSE_FILE:../../customoverride.yml"') == 1
+
+    def test_a_self_referencing_value_matching_an_existing_commented_line_uncomments_it_in_place(self, tmp_path):
+        env = tmp_path / "app.env"
+        env.write_text(self.ACCUMULATING_ENV)
+        update_configs(env, {"env": {"COMPOSE_FILE": "$COMPOSE_FILE:compose.onlyoffice.yml"}})
+        lines = env.read_text().splitlines()
+        assert lines[:-1] == self.ACCUMULATING_ENV.splitlines()[:-1]  # untouched, incl. the already-active line
+        assert lines[-1] == 'COMPOSE_FILE="$COMPOSE_FILE:compose.onlyoffice.yml"'  # uncommented in place
+        assert len(lines) == len(self.ACCUMULATING_ENV.splitlines())  # no new line appended
+
+
+class TestQuotedEnvValues:
+    """
+    A value keeps the quotes of its YAML scalar, and jinja substitutes globals that carry quotes of
+    their own. Quotes of the same style then end the line early where it is read back, so which
+    style a configuration happens to use must not change the value that arrives in the .env file.
+    """
+
+    def test_both_quoting_styles_produce_the_same_line(self, tmp_path):
+        env = tmp_path / "app.env"
+        env.write_text("")
+        update_configs(env, {"env": {
+            "DOUBLE": '"["turn:a.org:443?transport=tcp"]"',
+            "SINGLE": '\'["turn:a.org:443?transport=tcp"]\'',
+        }})
+        assert env.read_text().splitlines() == [
+            'DOUBLE=\'["turn:a.org:443?transport=tcp"]\'',
+            'SINGLE=\'["turn:a.org:443?transport=tcp"]\'',
+        ]
+
+    def test_an_inner_quote_does_not_end_the_line(self, tmp_path):
+        """Otherwise the parser stops after "[" and reads the rest of the line as further keys."""
+        env = tmp_path / "app.env"
+        env.write_text("")
+        update_configs(env, {"env": {"TURN_URIS": '"["turn:a.org:443"]"'}})
+        assert dotenv_values(env) == {"TURN_URIS": '["turn:a.org:443"]'}
+
+    def test_the_other_style_is_preferred_over_escaping(self):
+        """It needs no escape at all, so it survives a reader that resolves none."""
+        assert format_env_value('"["a"]"') == '\'["a"]\''
+        assert format_env_value("'it's here'") == '"it\'s here"'
+
+    def test_a_value_that_expands_keeps_its_style_and_escapes(self):
+        """Single quotes would stop abra from expanding the "$", which changes what the app gets."""
+        assert format_env_value('"a "b" $VAR"') == '"a \\"b\\" $VAR"'
+
+    def test_an_already_escaped_quote_is_not_escaped_again(self):
+        assert format_env_value(r'"a \" b"') == r'"a \" b"'
+
+    def test_a_value_holding_both_styles_is_reported(self, caplog):
+        """Neither style can hold it, single quotes have no escape in abra's parser."""
+        with caplog.at_level(logging.WARNING):
+            format_env_value('\'it\'s "quoted"\'')
+        assert "single quote" in caplog.text
+
+    def test_a_value_without_quotes_is_written_unchanged(self):
+        assert format_env_value("plain value") == "plain value"
+
+    def test_a_non_string_passes_through(self):
+        assert format_env_value(8080) == 8080
+
+    def test_a_quoted_extending_value_is_not_wrapped_twice(self, tmp_path):
+        """Two layers of quotes read back as an empty string, which drops every compose file."""
+        env = tmp_path / "app.env"
+        env.write_text("")
+        set_extending_key(env, "COMPOSE_FILE", '"$COMPOSE_FILE:compose.mariadb.yml"')
+        assert dotenv_values(env)["COMPOSE_FILE"] == "$COMPOSE_FILE:compose.mariadb.yml"
+
+    def test_an_unquoted_extending_value_is_still_quoted_for_expansion(self, tmp_path):
+        """The quotes have to be double ones, abra expands the "$" only inside those."""
+        env = tmp_path / "app.env"
+        env.write_text("")
+        set_extending_key(env, "COMPOSE_FILE", "$COMPOSE_FILE:compose.mariadb.yml")
+        assert env.read_text() == 'COMPOSE_FILE="$COMPOSE_FILE:compose.mariadb.yml"\n'

tests/test_exclude_recipes.py

@@ -0,0 +1,89 @@
+"""Tests for leaving single recipes out of a run."""
+
+import logging
+import os
+import sys
+
+import pytest
+from click.testing import CliRunner
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+import alakazam
+from alakazam import exclude_from_configs
+
+CONFIGS = {
+    "a.org": {"authentik": {"app_domain": "login.a.org"}, "traefik": {"app_domain": "a.org"}},
+    "b.org": {"traefik": {"app_domain": "b.org"}, "nextcloud": {"app_domain": "cloud.b.org"}},
+}
+
+
+class TestExcludeFromConfigs:
+    def test_nothing_excluded_returns_the_configs_unchanged(self):
+        assert exclude_from_configs(CONFIGS, ()) is CONFIGS
+
+    def test_the_recipe_is_gone_from_every_instance(self):
+        result = exclude_from_configs(CONFIGS, ("traefik",))
+        assert sorted(result["a.org"]) == ["authentik"]
+        assert sorted(result["b.org"]) == ["nextcloud"]
+
+    def test_several_recipes(self):
+        result = exclude_from_configs(CONFIGS, ("traefik", "nextcloud"))
+        assert result["b.org"] == {}
+
+    def test_the_original_is_not_modified(self):
+        exclude_from_configs(CONFIGS, ("traefik",))
+        assert "traefik" in CONFIGS["a.org"]
+
+    def test_an_unknown_recipe_is_reported(self, caplog):
+        """A typo would otherwise exclude nothing and look like it worked."""
+        with caplog.at_level(logging.WARNING):
+            exclude_from_configs(CONFIGS, ("treafik",))
+        assert "'treafik' is excluded but not configured" in caplog.text
+
+    def test_a_known_recipe_is_not_reported(self, caplog):
+        with caplog.at_level(logging.WARNING):
+            exclude_from_configs(CONFIGS, ("traefik",))
+        assert caplog.text == ""
+
+
+class TestExcludeReachesTheCommands:
+    """One filter in cli() has to cover every consumer of INSTANCE_CONFIGS."""
+
+    @pytest.fixture
+    def env(self, tmp_path, monkeypatch):
+        root = tmp_path / "root"
+        (root / "group").mkdir(parents=True)
+        (root / "alaka.yml").write_text(
+            "authentik:\n    version: 1.0.0\ntraefik:\n    version: 2.0.0\n")
+        (root / "group" / "example.com.yml").write_text("authentik:\ntraefik:\n")
+        (root / "alakazam.yml").write_text(f"root: {root}\n")
+        monkeypatch.setattr(alakazam, "get_settings_path", lambda: str(root / "alakazam.yml"))
+        monkeypatch.setattr(alakazam, "get_abra_dir", lambda: root / ".abra")
+        monkeypatch.setattr(alakazam, "fetch_recipes", lambda *a, **k: None)
+        monkeypatch.setattr(alakazam, "abra", lambda *a, **k: "")
+        return root
+
+    def invoke(self, root, args):
+        return CliRunner().invoke(alakazam.cli, args + [str(root / "group"), "ls"])
+
+    def test_without_the_flag_both_recipes_are_listed(self, env):
+        result = self.invoke(env, [])
+        assert result.exit_code == 0, result.output
+        assert "authentik" in result.output
+        assert "traefik" in result.output
+
+    def test_the_excluded_recipe_disappears(self, env):
+        result = self.invoke(env, ["-er", "traefik"])
+        assert result.exit_code == 0, result.output
+        assert "authentik" in result.output
+        assert "traefik" not in result.output
+
+    def test_the_long_option_works_too(self, env):
+        result = self.invoke(env, ["--exclude-recipe", "traefik"])
+        assert "traefik" not in result.output
+
+    def test_it_can_be_given_more_than_once(self, env):
+        result = self.invoke(env, ["-er", "traefik", "-er", "authentik"])
+        assert "traefik" not in result.output
+        assert "authentik" not in result.output

tests/test_hide_secrets.py

@@ -98,3 +98,89 @@ class TestDebugLog:
             insert_secret("login.a.org", "db_password", VALUE)
         assert "db_password" in caplog.text
         assert VALUE not in caplog.text
+
+
+class TestLocalHookOutput:
+    """A recipe may print the secret it created, vaultwarden's admin token does exactly that."""
+
+    TOKEN = "vaultwarden-admin-token-in-plain"
+
+    def install(self, monkeypatch, fail=False):
+        seen = {}
+
+        class Process:
+            returncode = 1 if fail else 0
+            stdout = TestLocalHookOutput.TOKEN.encode()
+            stderr = b""
+
+        def run(cmd, capture_output=False, **kwargs):
+            seen["capture_output"] = capture_output
+            return Process()
+
+        monkeypatch.setattr(alakazam.subprocess, "run", run)
+        def streamed(cmd):
+            seen["streamed"] = True
+            return Process()
+
+        monkeypatch.setattr(alakazam, "run_streamed", streamed)
+        return seen
+
+    def hook(self, strict=False):
+        alakazam.run_secret_hooks("login.a.org", {"secret_hooks": ["insert_admin_token"], "server": "a.org"}, strict=strict)
+
+    def test_the_hook_is_streamed_when_not_hiding(self, monkeypatch):
+        monkeypatch.setattr(alakazam, "HIDE_SECRETS", False)
+        seen = self.install(monkeypatch)
+        self.hook()
+        assert seen.get("streamed")
+
+    def test_the_hook_is_not_streamed_when_hiding(self, monkeypatch, capsys):
+        monkeypatch.setattr(alakazam, "HIDE_SECRETS", True)
+        seen = self.install(monkeypatch)
+        self.hook()
+        assert not seen.get("streamed")
+        assert self.TOKEN not in capsys.readouterr().out
+
+    def test_a_failing_hook_does_not_leak_through_the_error(self, monkeypatch):
+        """The captured output lands in the exception, which is where it would escape."""
+        monkeypatch.setattr(alakazam, "HIDE_SECRETS", True)
+        self.install(monkeypatch, fail=True)
+        with pytest.raises(alakazam.click.ClickException) as excinfo:
+            self.hook(strict=True)
+        assert self.TOKEN not in excinfo.value.message
+        assert "withheld by --hide-secrets" in excinfo.value.message
+
+    def test_a_failing_hook_still_reports_its_output_when_not_hiding(self, monkeypatch):
+        monkeypatch.setattr(alakazam, "HIDE_SECRETS", False)
+        self.install(monkeypatch, fail=True)
+        with pytest.raises(alakazam.click.ClickException) as excinfo:
+            self.hook(strict=True)
+        assert self.TOKEN in excinfo.value.message
+
+
+class TestLocalScriptOutput:
+    def install(self, monkeypatch, tmp_path):
+        script = tmp_path / "hook.sh"
+        script.write_text("#!/bin/sh\necho secret\n")
+        script.chmod(0o755)
+        seen = {}
+
+        class Result:
+            returncode = 0
+
+        monkeypatch.setattr(alakazam, "resolve_path", lambda p, base=None: script)
+        monkeypatch.setattr(alakazam.subprocess, "run",
+                            lambda cmd, **kw: seen.update(kw) or Result())
+        return seen
+
+    def test_the_script_output_is_captured_when_hiding(self, monkeypatch, tmp_path):
+        monkeypatch.setattr(alakazam, "HIDE_SECRETS", True)
+        seen = self.install(monkeypatch, tmp_path)
+        alakazam.run_local_script(["script", "hook.sh"], "login.a.org", "a.org", "a.org")
+        assert seen["capture_output"] is True
+
+    def test_the_script_output_is_passed_through_when_not_hiding(self, monkeypatch, tmp_path):
+        monkeypatch.setattr(alakazam, "HIDE_SECRETS", False)
+        seen = self.install(monkeypatch, tmp_path)
+        alakazam.run_local_script(["script", "hook.sh"], "login.a.org", "a.org", "a.org")
+        assert seen["capture_output"] is False

tests/test_jinja_substitution.py

@@ -0,0 +1,33 @@
+"""Tests for substitute_jinja_variable()'s recursion into dicts, lists, and scalars."""
+
+import os
+import sys
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+from alakazam import substitute_jinja_variable
+
+
+class TestSubstituteJinjaVariable:
+    def test_a_scalar_in_a_dict_substitutes(self):
+        config = {"nextcloud": {"env": {"DOMAIN": "{{ domain }}"}}}
+        substitute_jinja_variable(config, {"domain": "nc.example.com"})
+        assert config["nextcloud"]["env"]["DOMAIN"] == "nc.example.com"
+
+    def test_a_scalar_in_a_list_substitutes(self):
+        config = {"nextcloud": {"initial-hooks": [
+            "app install_apps",
+            "app set_app_config files default_quota {{ quota }}",
+            "app run_occ",
+        ]}}
+        substitute_jinja_variable(config, {"quota": "0"})
+        hooks = config["nextcloud"]["initial-hooks"]
+        assert isinstance(hooks, list) and len(hooks) == 3
+        assert hooks[0] == "app install_apps"
+        assert "0" in hooks[1]
+        assert hooks[2] == "app run_occ"
+
+    def test_a_scalar_in_a_list_without_a_template_is_left_untouched(self):
+        config = {"nextcloud": {"initial-hooks": ["app run_occ 'app:install groupfolders'"]}}
+        substitute_jinja_variable(config, {"quota": "0"})
+        assert config["nextcloud"]["initial-hooks"] == ["app run_occ 'app:install groupfolders'"]

tests/test_local_scripts.py

@@ -53,7 +53,7 @@ class TestRunLocalScript:
         script = make_script(tmp_path)
         calls = []
         monkeypatch.setattr(alakazam.subprocess, "run",
-                            lambda cmd, env: calls.append((cmd, env)) or FakeProcess(0))
+                            lambda cmd, env, **kwargs: calls.append((cmd, env)) or FakeProcess(0))
         run_local_script(["script", str(script), "arg1"], "login.a.org", "a.org", "a.org")
         [(cmd, env)] = calls
         assert cmd == [str(script), "arg1"]
@@ -168,7 +168,8 @@ class TestInstanceDomainThreading:
         monkeypatch.setattr(alakazam, "abra", lambda *a, **k: "")
         calls = []
         monkeypatch.setattr(alakazam, "execute_cmds", lambda app_config, **k: calls.append(k))
-        deploy_apps({"a.org": [["authentik", "login.a.org"]]}, execute_hooks=True)
+        # the health check is not what this is about, and it would need a whole abra to answer
+        deploy_apps({"a.org": [["authentik", "login.a.org"]]}, execute_hooks=True, wait=False)
         [kwargs] = calls
         assert kwargs["deploy"] is True
         assert kwargs["instance_domain"] == "a.org"

tests/test_preflight.py

@@ -109,9 +109,11 @@ class TestRelevantConfigPaths:
         assert config_root / "group" / "alaka-versions.yml" in paths
         assert config_root / "group" / "example.com.yml" in paths
 
-    def test_covers_the_connection_configuration(self, config_root):
+    def test_covers_every_config_set_file(self, config_root):
+        (config_root / "config-sets-authentik.yml").write_text("integrate:\n    apply-when: [a]\n")
         paths = get_relevant_config_paths(config_root, config_root / "group", [])
-        assert alakazam.Path(alakazam.COMBINE_PATH) in paths
+        assert config_root / "config-sets.yml" in paths
+        assert config_root / "config-sets-authentik.yml" in paths
 
     def test_a_single_instance_file_pulls_in_its_ancestors(self, config_root):
         paths = get_relevant_config_paths(config_root, config_root / "group" / "example.com.yml", [])

tests/test_retry.py

@@ -1,4 +1,4 @@
-"""Tests for which abra failures may be retried, and which configuration keys are ignored."""
+"""Tests for which abra failures may be retried."""
 
 import os
 import sys
@@ -9,7 +9,7 @@ import pytest
 sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
 import alakazam
-from alakazam import abra, is_connection_error, update_configs
+from alakazam import abra, is_connection_error
 
 
 class FakeProcess:
@@ -83,21 +83,3 @@ class TestConnectionErrors:
     ])
     def test_operational_failures_are_not_retried(self, output):
         assert not is_connection_error(output)
-
-
-class TestUnknownConfigKeys:
-    def test_an_unknown_key_never_reaches_the_env_file(self, tmp_path):
-        """A promotion mechanism sets 'hold: <reason>' on a recipe, which must pass through untouched."""
-        env = tmp_path / "app.env"
-        env.write_text("#SMTP_HOST=mail.example.com\n")
-        update_configs(env, {"hold": "waiting for the upstream fix", "env": {"SMTP_HOST": "mail.a.org"}})
-        content = env.read_text()
-        assert "SMTP_HOST=mail.a.org" in content
-        assert "hold" not in content
-        assert "waiting for the upstream fix" not in content
-
-    def test_an_unknown_key_alone_changes_nothing(self, tmp_path):
-        env = tmp_path / "app.env"
-        env.write_text("#SMTP_HOST=mail.example.com\n")
-        update_configs(env, {"hold": "waiting for the upstream fix"})
-        assert env.read_text() == "#SMTP_HOST=mail.example.com\n"

tests/test_upgrade_hooks.py

@@ -0,0 +1,179 @@
+"""Tests for what an upgrade run does: deploy everything, wait for all of it, then hook."""
+
+import os
+import sys
+
+import pytest
+from click.testing import CliRunner
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+import alakazam
+
+CONFIG = {"a.org": {
+    "nextcloud": {"app_domain": "cloud.a.org", "server": "a.org", "version": "2.0.0",
+                  "upgrade-hooks": ["app occ upgrade"]},
+    "wekan": {"app_domain": "boards.a.org", "server": "a.org", "version": "2.0.0"},
+}}
+
+# kept before any fixture replaces it, for the tests that check what a dry run actually prints
+REAL_EXECUTE_CMDS = alakazam.execute_cmds
+
+
+@pytest.fixture
+def versions():
+    """The version each app is deployed at, a test changes it to steer what needs upgrading."""
+    return {"nextcloud": "1.0.0", "wekan": "1.0.0"}
+
+
+@pytest.fixture
+def steps(monkeypatch, versions):
+    """Records what an upgrade run does, in order, against an abra that answers every call."""
+    steps = []
+    monkeypatch.setattr(alakazam, "INSTANCE_CONFIGS", CONFIG)
+    monkeypatch.setattr(alakazam, "sleep", lambda seconds: steps.append(f"sleep:{seconds}"))
+    monkeypatch.setattr(alakazam, "get_missing_secrets", lambda domain: [])
+    # a dry run reaches the same function, so the two have to be told apart here
+    monkeypatch.setattr(alakazam, "execute_cmds", lambda config, dry_run=False, **kw: steps.append(
+        f"{'show' if dry_run else 'hook'}:{config['app_domain']}"))
+
+    # the recipe filter has to be honoured here, it is what the wait narrows itself down with
+    def apps_by_deployment(recipes, deployed=True):
+        return {"a.org": [[app, CONFIG["a.org"][app]["app_domain"], version]
+                          for app, version in versions.items() if not recipes or app in recipes]}
+
+    def abra(*args, **kwargs):
+        if args[:2] == ("app", "ps"):
+            steps.append(f"ps:{args[2]}")
+            return {"app": {"service": "app", "status": "healthy", "version": "2.0.0",
+                            "chaos": "false", "state": "running", "image": "img"}}
+        # the planning phase asks for the release notes with the same command plus -r
+        if "-r" not in args:
+            steps.append(" ".join(a for a in args if a))
+        return ""
+
+    monkeypatch.setattr(alakazam, "get_apps_by_deployment", apps_by_deployment)
+    monkeypatch.setattr(alakazam, "abra", abra)
+    return steps
+
+
+def selected(steps, prefix):
+    return [step for step in steps if step.startswith(prefix)]
+
+
+def run(args, **kwargs):
+    result = CliRunner().invoke(alakazam.upgrade, args, standalone_mode=False, **kwargs)
+    assert result.exit_code == 0, result.exception
+    return result
+
+
+class TestUpgradeHooks:
+    def test_the_hooks_do_not_switch_on_converge_checks(self, steps):
+        """Waiting for each app in turn is what the separate flag is for."""
+        run(["-n", "-e"])
+        assert all("--no-converge-checks" in s for s in selected(steps, "app upgrade"))
+
+    def test_converge_checks_come_from_their_own_flag_only(self, steps):
+        run(["-n", "-c"])
+        assert not any("--no-converge-checks" in s for s in selected(steps, "app upgrade"))
+
+    def test_every_app_is_deployed_before_the_first_hook(self, steps):
+        run(["-n", "-e"])
+        deploys = selected(steps, "app upgrade")
+        assert max(steps.index(s) for s in deploys) < steps.index("hook:cloud.a.org")
+
+    def test_the_hooks_run_after_the_wait(self, steps):
+        """A hook would otherwise reach a container the upgrade has not brought up again yet."""
+        run(["-n", "-e"])
+        assert steps.index("ps:cloud.a.org") < steps.index("hook:cloud.a.org")
+        assert steps.index("ps:boards.a.org") < steps.index("hook:boards.a.org")
+
+    def test_the_health_is_only_read_after_the_services_settled(self, steps):
+        """A replaced service still reports its old container, which has been healthy all along."""
+        run(["-n", "-e"])
+        settle = steps.index(f"sleep:{alakazam.PS_WAIT_SETTLE}")
+        assert max(steps.index(s) for s in selected(steps, "app upgrade")) < settle
+        assert settle < steps.index("ps:cloud.a.org")
+
+    def test_without_the_flag_nothing_is_hooked(self, steps):
+        run(["-n"])
+        assert selected(steps, "hook:") == []
+
+    def test_the_wait_happens_without_the_flag_too(self, steps):
+        """It is what tells an upgrade run whether the apps came back at all."""
+        run(["-n"])
+        assert selected(steps, "ps:") == ["ps:cloud.a.org", "ps:boards.a.org"]
+
+    def test_a_filtered_run_waits_for_its_own_apps_only(self, steps):
+        run(["-n", "-e", "-r", "wekan"])
+        assert selected(steps, "ps:") == ["ps:boards.a.org"]
+        assert selected(steps, "hook:") == ["hook:boards.a.org"]
+
+    def test_an_app_already_at_its_version_is_neither_waited_for_nor_hooked(self, steps, versions):
+        versions["nextcloud"] = "2.0.0"
+        run(["-n", "-e"])
+        assert selected(steps, "ps:") == ["ps:boards.a.org"]
+        assert selected(steps, "hook:") == ["hook:boards.a.org"]
+
+    def test_an_app_with_an_unreadable_version_is_neither_waited_for_nor_hooked(self, steps, versions):
+        versions["nextcloud"] = "whatever"
+        run(["-n", "-e"])
+        assert selected(steps, "ps:") == ["ps:boards.a.org"]
+        assert selected(steps, "hook:") == ["hook:boards.a.org"]
+
+    def test_nothing_to_upgrade_skips_the_wait_entirely(self, steps, versions):
+        """Waiting on a run that did not happen would report on unrelated apps."""
+        versions.update({"nextcloud": "2.0.0", "wekan": "2.0.0"})
+        run(["-n", "-e"])
+        assert steps == []
+
+
+class TestDryRun:
+    """A dry run prints what would happen. Touching the server at all makes the flag pointless."""
+
+    def executed(self, steps):
+        return [step for step in steps if not step.startswith("show:")]
+
+    def test_an_interactive_dry_run_upgrades_nothing(self, steps):
+        """It still shows the release notes, so it asks a question that must not start anything."""
+        run(["--dry-run", "-e"], input="\nYES\n")
+        assert self.executed(steps) == []
+
+    def test_a_non_interactive_dry_run_upgrades_nothing(self, steps):
+        run(["-n", "--dry-run", "-e"])
+        assert self.executed(steps) == []
+
+    def test_an_interactive_dry_run_asks_nothing_about_upgrading(self, steps):
+        assert "Do you really want to upgrade" not in run(["--dry-run"], input="\n").output
+
+    def test_the_version_jump_is_shown_and_labelled(self, steps):
+        output = run(["--dry-run"], input="\n").output
+        assert "from" in output and "to" in output
+        assert "1.0.0" in output and "2.0.0" in output
+
+    def test_the_hooks_are_shown_when_they_would_run(self, steps, monkeypatch):
+        monkeypatch.setattr(alakazam, "execute_cmds", REAL_EXECUTE_CMDS)
+        assert "Run 'app occ upgrade' in cloud.a.org" in run(["--dry-run", "-e"], input="\n").output
+
+    def test_the_hooks_stay_out_without_the_flag(self, steps, monkeypatch):
+        """They would not run either, so showing them would misreport the run."""
+        monkeypatch.setattr(alakazam, "execute_cmds", REAL_EXECUTE_CMDS)
+        assert "app occ upgrade" not in run(["--dry-run"], input="\n").output
+
+    def test_a_shown_hook_is_not_executed(self, steps):
+        run(["--dry-run", "-e"], input="\n")
+        assert selected(steps, "show:") == ["show:cloud.a.org", "show:boards.a.org"]
+        assert selected(steps, "hook:") == []
+
+
+class TestSettleOption:
+    def test_the_delay_can_be_set(self, steps):
+        run(["-n", "--settle", "5"])
+        assert "sleep:5" in steps
+        assert f"sleep:{alakazam.PS_WAIT_SETTLE}" not in steps
+
+    def test_zero_skips_the_delay_without_skipping_the_wait(self, steps):
+        """A converge-checked run has waited already, the health still has to be confirmed."""
+        run(["-n", "--settle", "0"])
+        assert selected(steps, "sleep:") == []
+        assert selected(steps, "ps:") == ["ps:cloud.a.org", "ps:boards.a.org"]

tests/test_upgrade_secrets.py

@@ -0,0 +1,88 @@
+"""Tests for filling in secrets a recipe added, before the upgrade that would fail on them."""
+
+import os
+import sys
+
+import pytest
+from click.testing import CliRunner
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+import alakazam
+from alakazam import get_missing_secrets
+
+CONFIG = {"a.org": {"nextcloud": {"app_domain": "cloud.a.org", "server": "a.org", "version": "2.0.0"}}}
+
+
+class TestGetMissingSecrets:
+    def install(self, monkeypatch, stored):
+        monkeypatch.setattr(alakazam, "abra", lambda *a, **k: stored)
+
+    def test_a_complete_app_reports_nothing(self, monkeypatch):
+        self.install(monkeypatch, [{"name": "db_password", "created on server": "true"}])
+        assert get_missing_secrets("cloud.a.org") == []
+
+    def test_the_missing_names_are_returned(self, monkeypatch):
+        self.install(monkeypatch, [
+            {"name": "db_password", "created on server": "true"},
+            {"name": "oidc_secret", "created on server": "false"},
+        ])
+        assert get_missing_secrets("cloud.a.org") == ["oidc_secret"]
+
+    def test_an_app_without_any_secrets(self, monkeypatch):
+        """abra answers with an empty document rather than a list when nothing is stored."""
+        self.install(monkeypatch, {})
+        assert get_missing_secrets("cloud.a.org") == []
+
+
+class TestUpgradeFillsThemIn:
+    @pytest.fixture
+    def steps(self, monkeypatch):
+        steps = []
+        monkeypatch.setattr(alakazam, "INSTANCE_CONFIGS", CONFIG)
+        monkeypatch.setattr(alakazam, "sleep", lambda s: None)
+        monkeypatch.setattr(alakazam, "get_apps_by_deployment",
+                            lambda recipes, deployed=True: {"a.org": [["nextcloud", "cloud.a.org", "1.0.0"]]})
+        monkeypatch.setattr(alakazam, "create_secrets",
+                            lambda recipes, **kw: steps.append(f"secrets:{recipes[0]}"))
+        return steps
+
+    def install_abra(self, monkeypatch, steps, missing):
+        def abra(*args, **kwargs):
+            if args[:3] == ("app", "secret", "ls"):
+                return [{"name": n, "created on server": str(n not in missing).lower()}
+                        for n in ("db_password", "oidc_secret")]
+            if args[:2] == ("app", "ps"):
+                return {"app": {"service": "app", "status": "healthy", "version": "2.0.0",
+                                "chaos": "false", "state": "running", "image": "img"}}
+            steps.append(" ".join(a for a in args if a))
+            return ""
+        monkeypatch.setattr(alakazam, "abra", abra)
+
+    def invoke(self):
+        return CliRunner().invoke(alakazam.upgrade, ["-n"], standalone_mode=False)
+
+    def test_a_missing_secret_is_created_before_the_upgrade(self, monkeypatch, steps):
+        self.install_abra(monkeypatch, steps, missing={"oidc_secret"})
+        result = self.invoke()
+        assert result.exit_code == 0, result.exception
+        # the planning phase fetches release notes with the same command plus -r, skip that one
+        upgrade = next(i for i, s in enumerate(steps)
+                       if s.startswith("app upgrade") and " -r " not in s)
+        assert steps.index("secrets:nextcloud") < upgrade
+
+    def test_the_missing_name_is_reported(self, monkeypatch, steps):
+        """Silently repairing secrets would hide which one the recipe added."""
+        self.install_abra(monkeypatch, steps, missing={"oidc_secret"})
+        assert "missing 1 secret(s): oidc_secret" in self.invoke().output
+
+    def test_a_complete_app_does_not_run_the_secrets(self, monkeypatch, steps):
+        self.install_abra(monkeypatch, steps, missing=set())
+        self.invoke()
+        assert not any(s.startswith("secrets:") for s in steps)
+
+    def test_a_dry_run_creates_nothing(self, monkeypatch, steps):
+        """The check sits behind the confirmation, so --dry-run must not reach it."""
+        self.install_abra(monkeypatch, steps, missing={"oidc_secret"})
+        CliRunner().invoke(alakazam.upgrade, ["-n", "--dry-run"], standalone_mode=False)
+        assert not any(s.startswith("secrets:") for s in steps)

tests/test_versions_pinned.py

@@ -0,0 +1,79 @@
+"""Tests for a version that pins a commit while naming the release it is based on."""
+
+import logging
+import os
+import sys
+
+import click
+import pytest
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+import alakazam
+from alakazam import classify_version, merge_instance_configs, resolve_version
+
+
+class TestResolveVersion:
+    def test_a_pinned_commit_wins_over_the_release(self):
+        assert resolve_version("12.0.2+2026.5.2@be9ebb3", "nextcloud") == "be9ebb3"
+
+    @pytest.mark.parametrize("version", ["1.2.3", "12.0.2+2026.5.2", "chaos", "latest", "be9ebb3"])
+    def test_a_version_without_a_commit_is_untouched(self, version):
+        assert resolve_version(version, "nextcloud") == version
+
+    @pytest.mark.parametrize("version", [None, 1.2, True])
+    def test_non_strings_are_untouched(self, version):
+        """The configuration is user supplied and does not have to hold a string."""
+        assert resolve_version(version, "nextcloud") is version
+
+    def test_a_trailing_at_is_rejected(self):
+        """Silently deploying the release instead of the intended commit would be worse."""
+        with pytest.raises(click.ClickException) as excinfo:
+            resolve_version("12.0.2@", "nextcloud")
+        assert "without a commit" in excinfo.value.message
+        assert "nextcloud" in excinfo.value.message
+
+    def test_the_release_is_kept_in_the_debug_log(self, caplog):
+        """It is the only place the base release survives, and CI failures are read there."""
+        with caplog.at_level(logging.DEBUG):
+            resolve_version("12.0.2+2026.5.2@be9ebb3", "nextcloud")
+        assert "based on 12.0.2+2026.5.2" in caplog.text
+
+    def test_the_last_at_separates(self):
+        assert resolve_version("a@b@be9ebb3", "nextcloud") == "be9ebb3"
+
+
+class TestClassification:
+    def test_the_resolved_commit_classifies_as_a_hash(self):
+        """upgrade branches on this, a release would take the wrong abra command."""
+        assert classify_version(resolve_version("12.0.2+2026.5.2@be9ebb3", "nextcloud")) == "hash"
+
+    def test_an_unpinned_release_still_classifies_as_a_version(self):
+        assert classify_version(resolve_version("12.0.2+2026.5.2", "nextcloud")) == "version"
+
+
+class TestMergedConfig:
+    """The resolution has to happen once, where the configuration is built."""
+
+    def merge(self, version):
+        return merge_instance_configs(
+            {}, "example.com", {"nextcloud": {"version": version}}, {})
+
+    def test_the_merged_config_carries_the_commit(self):
+        assert self.merge("12.0.2+2026.5.2@be9ebb3")["nextcloud"]["version"] == "be9ebb3"
+
+    def test_a_plain_version_survives_the_merge(self):
+        assert self.merge("12.0.2")["nextcloud"]["version"] == "12.0.2"
+
+    def test_an_app_without_a_version(self):
+        assert "version" not in self.merge(None)["nextcloud"] or \
+            self.merge(None)["nextcloud"]["version"] is None
+
+    def test_a_templated_version_is_resolved_after_substitution(self):
+        """Jinja runs first, so a version coming from GLOBALS is pinned as well."""
+        merged = merge_instance_configs(
+            {"GLOBALS": {"pin": "12.0.2+2026.5.2@be9ebb3"}},
+            "example.com",
+            {"nextcloud": {"version": "{{pin}}"}},
+            {})
+        assert merged["nextcloud"]["version"] == "be9ebb3"