feat(config)!: replace combine.yml with auto-applying config-sets

BREAKING CHANGE: the integrations alakazam shipped as combine.yml are gone
from the repository. Copy examples/config-sets-combine.yml into the root path
to keep them, where they can be pinned and adjusted along with the versions.

An integration is now a config-set carrying 'apply-when', which names the apps
whose presence makes it apply. Naming the condition rather than deriving it
from the configured apps is what lets a set configure only one side, as four
of the seventeen pairs in combine.yml did.

Dockerfile

@@ -26,14 +26,16 @@ RUN python -m venv /opt/alakazam \
   && /opt/alakazam/bin/pip install --no-cache-dir -r /tmp/requirements.txt
 
 FROM python:3.11-slim
+# git: alakazam syncs the recipe repos itself. openssh-client: abra reaches the servers over
+# ssh. argon2: recipes hash secrets locally in their abra.sh, vaultwarden's admin token does
 RUN apt-get update \
-  && apt-get install -y --no-install-recommends git openssh-client make \
+  && apt-get install -y --no-install-recommends git openssh-client make argon2 \
   && rm -rf /var/lib/apt/lists/* \
   && git config --global --add safe.directory '*'
 
 COPY --from=abra /out/abra /usr/local/bin/abra
 COPY --from=deps /opt/alakazam /opt/alakazam
-COPY alakazam.py combine.yml /opt/alakazam/
+COPY alakazam.py /opt/alakazam/
 RUN printf '#!/bin/sh\nexec /opt/alakazam/bin/python /opt/alakazam/alakazam.py "$@"\n' \
   > /usr/local/bin/alakazam \
   && chmod +x /usr/local/bin/alakazam

README.md

@@ -185,6 +185,22 @@ Without `-s` all secrets of the recipes selected by `-r` are purged.
 alakazam example.com.yml ps --wait --timeout 300
 ```
 
+### Excluding Recipes
+
+`-er`/`--exclude-recipe` leaves single recipes out of a run entirely:
+
+```
+alakazam -er traefik example.com.yml clean-deploy -n
+```
+
+### Hiding Secrets
+
+`secrets` and `setup` print the values they generate, which is right at a terminal and wrong in a pipeline whose logs are kept. `--hide-secrets` replaces them with `[hidden]`:
+
+```
+alakazam --hide-secrets example.com.yml clean-deploy -n
+```
+
 ### Run CMDs
 
 Escaping can be akward:
@@ -211,12 +227,7 @@ Configuration files support templating with Jinja2 and global variables, facilit
 1. **`alaka.yml`**/**`alaka-*.yml`**: Contains global configurations for a hierarchical layer, affecting all subordinate apps.
 2. **`example.com.yml`** (`<your-domain>.yml`): A minimalist configuration for each instance, specifying the apps to be installed, the instance domain and optionally instance specific configurations.
     - an **instance** means a collection of apps that are integrated with each other (one server / vm can contain multiple instances)
-3. **`combine.yml`**: Manages configurations required to integrate multiple apps with each other.
-    - This configuration should not be touched by the operator.
-    - At the moment it's part of the `alakazam` repository.
-    - In future it should be split into the recipes repositories and maintained by the recipe maintainer.
-        - We started to move each app entry as `alaconnect.yml` into the respective repositories.
-4. **`config-sets.yml`**: Optional file placed at the `root` path. Defines named sets of per-app configurations that can be activated or deactivated per instance or group, avoiding duplication across instance files. See [examples/config-sets.yml](./examples/config-sets.yml).
+3. **`config-sets.yml`**/**`config-sets-*.yml`**: Optional files at the `root` path. They define named sets of per-app configurations, either switched on per instance or group, or applying by themselves as an integration between apps. See [Config-Sets](#config-sets).
 
 ### Global Settings
 
@@ -243,29 +254,60 @@ uptime_kuma:
 
 ### App Configuration
 
-`alaka.yml`/`alaka-*.yml`, `example.com.yml` and `combine.yml`/`alaconnect.yml` contain a similar configuration structure.
+`alaka.yml`/`alaka-*.yml`, `example.com.yml` and the config-sets contain a similar configuration structure.
 For each app/recipe the following `<app_configurations>` can be used:
 
 - **`uncomment`/`comment`**: Manages the active status of lines in configuration files, such as uncommenting needed variables .
     - it matches against parts of the line (i.E. `compose.smtp.yml`)
     - this is useful for env variables that are used multiple times like `COMPOSE_FILE`
 - **`env`**: Sets values for environment variables.
-- **`*-hooks`**: Specifies `abra.sh` commands to run at specific stages.
+- **`*-hooks`**: Specifies `abra.sh` commands or local scripts to run at specific stages.
     - **`initial-hooks`**: commands for initialisation
     - **`deploy-hooks`**: commands that should be run after each deployment
     - **`upgrade-hooks`**: commands that should be run after each upgrade
 - **`readiness-hooks`**: Commands that decide whether an app is usable yet, repeated until one succeeds. See [Readiness Hooks](#readiness-hooks).
 - **`dependency`**: Names the apps that have to be set up before this one. See [Dependencies](#dependencies).
 - **`secrets`**: Inserts specific values (i.E. smtp passwords) into secrets; future updates will support encrypted file usage.
-- **`secret-hooks`**: Run `abra.sh` commands locally for secrets that need to be generated.
-- **`subdomain`**: Specifies the subdomain scheme for individual recipes and apps. (not available in `combine.yml`/`alaconnect.yml`)
+- **`secret-hooks`**: Run `abra.sh` commands locally or local scripts for secrets that need to be generated.
+- **`shared_secrets`**: Shares a secret between two apps, as `<source_app>: {<target_secret>: <source_secret>}`.
+- **`subdomain`**: Specifies the subdomain scheme for individual recipes and apps. (not available in config-sets)
     - i.e. `cloud.example.com` for nextcloud
-- **`version`**: Controls the recipe version to deploy; if unspecified, the latest version is used. (not available in `combine.yml`/`alaconnect.yml`)
+- **`version`**: Controls the recipe version to deploy; if unspecified, the latest version is used. (not available in config-sets)
+    - a commit can be pinned with `<release>@<commit>`, for example `12.0.2+2026.5.2@be9ebb3`
+    - only the commit is deployed; the release in front of the `@` is there so that a dependency bot can follow the recipe's tags and offer patch updates
 
-The `combine.yml`/`alaconnect.yml` configuration additionally contains:
+### Config-Sets
 
-- **`shared_secrets`**: Specifies secret sharing between apps.
-     - `<source_secret_name>:<target_secret_name>`
+A config-set is a named bundle of per-app configuration, defined in `config-sets.yml` or any `config-sets-*.yml` at the `root` path. The files are merged in the order `config-sets.yml`, then the `config-sets-*.yml` alphabetically, the same precedence the `alaka*.yml` of a directory have.
+
+A set applies in one of two ways.
+
+**Switched on per instance or group**, via `CONFIG-SETS` in any `alaka.yml` or `example.com.yml`:
+
+```yaml
+CONFIG-SETS:
+    bbb: true
+```
+
+**By itself**, when it carries `apply-when` and every app named there is part of the instance:
+
+```yaml
+integrate-authentik-nextcloud:
+    apply-when: [authentik, nextcloud]
+    authentik:
+        uncomment: [compose.nextcloud.yml]
+    nextcloud:
+        uncomment: [compose.authentik.yml]
+        shared_secrets:
+            authentik:
+                nextcloud_secret: authentik_secret
+```
+
+That is what an integration between two apps is: configuration belonging to both of them that is pointless unless both are there. `apply-when` names the condition rather than deriving it from the configured apps, so an integration can configure only one side: `[matrix-synapse, traefik]` while configuring `traefik` alone. An instance can still switch such a set off with `CONFIG-SETS: {integrate-authentik-nextcloud: false}`.
+
+Merge priority: automatic sets < group < sets switched on explicitly < instance. An integration is the lowest layer of all, so a group configuration can adjust it — the precedence the `combine.yml` it replaces had. Its list entries land behind the ones an app configures for itself, which is what keeps an integration hook such as `app set_authentik` running after the `app core_install` that sets the app up in the first place.
+
+[examples/config-sets-combine.yml](./examples/config-sets-combine.yml) holds the integrations alakazam used to ship as `combine.yml`. Copy it into your `root` path to keep them.
 
 ### Dependencies
 
@@ -277,6 +319,7 @@ nextcloud:
 ```
 
 Alakazam builds a directed graph from those entries and processes the apps in topological order. For the configuration, the secrets and the deployment alike. Apps unrelated by a dependency keep their configuration order. [`setup`](#setup-and-clean-deploy) additionally groups them into levels: everything without a dependency forms the first level, apps depending only on those the second, and so on.
+
 ### Readiness Hooks
 
 An app can be deployed and healthy while still not being usable — authentik accepts connections long before it has applied its blueprints. A readiness hook is an `abra.sh` command of the recipe that answers that question and exits zero once the app is ready:
@@ -294,6 +337,24 @@ authentik:
 
 Readiness hooks run as part of [`setup`](#setup-and-clean-deploy) only, right after the app they belong to is deployed and before anything that depends on it. `deploy` does not run them, so it keeps returning as soon as the deployment is through.
 
+### \*-Hooks Command Formats
+
+**Abra command** — runs an abra.sh command inside a container:
+
+```yaml
+initial-hooks:
+  - app set_default_quota
+```
+
+**Local script** — runs a script on the local machine (for custom actions that aren't generic enough to be implemented as abra.sh commands):
+
+```yaml
+initial-hooks:
+  - script ./scripts/script.sh arg1
+```
+
+Relative paths resolve from the `root` path. The script receives `ALAKAZAM_APP_DOMAIN`, `ALAKAZAM_APP_SERVER`, and `ALAKAZAM_INSTANCE_DOMAIN` as environment variables.
+
 ### Configuration Structure
 
 Configuration can be simplified into a single `example.com.yml` or expanded into multiple layered `alaka.yml`/`alaka-*.yml` files for complex deployments. This allows for easy maintenance of multiple instances or groups.
@@ -309,22 +370,9 @@ These configurations are designed to modularize and simplify the management of a
         <app_configurations>
     ```
 
-2. **`combine.yml`**
-    - This file orchestrates the integration between multiple apps within the same instance (`example.com.yml`), applying `<target_app_configurations>` to ensure seamless functionality between connected applications.
-    - The configuration for each app combination is structured as follows, again `<target_app_configurations>` is following the guidelines set in the [App Configuration](#app-configuration) section:
-
-    ```
-    <target_app_recipe>:
-        <source_app_recipe>:
-            <target_app_configurations>
-    ```
-
-    - In future each `<source_app_recipe>` entry will be placed in the `alaconnect.yml` inside the `<target_app_recipe>` folder.
-
-3. **`config-sets.yml`**
-    - Optional file at the `root` path that defines named sets of per-app configurations. Each top-level key is a config-set name; its value is a map of `<app_recipe>` to `<app_configurations>`.
-    - Activate or disable sets per instance or group via `CONFIG-SETS` in any `alaka.yml` or `example.com.yml`. Merge priority: group < config-set < instance.
-    - See [examples/config-sets.yml](./examples/config-sets.yml).
+2. **`config-sets.yml`**/**`config-sets-*.yml`**
+    - Optional files at the `root` path. Each top-level key is a config-set name; its value is a map of `<app_recipe>` to `<app_configurations>`.
+    - See [Config-Sets](#config-sets).
 
 
 ### Templating Configurations

alakazam.py

@@ -4,7 +4,7 @@ import os
 import json
 import logging
 from pathlib import Path
-from typing import List, Dict, Any, Iterator, Optional, Union, Set, Tuple
+from typing import List, Dict, Any, Iterable, Iterator, Optional, Union, Set, Tuple
 import subprocess
 import re
 
@@ -23,8 +23,9 @@ from packaging import version
 from uptime_kuma_api import UptimeKumaApi, MonitorType
 from time import sleep, time
 
-COMBINE_PATH = os.path.dirname(os.path.realpath(__file__)) + "/combine.yml"
 NON_APP_KEYS = {'CONFIG-SETS', 'GLOBALS'}
+# key of a config-set that names the apps whose presence makes the set apply by itself
+APPLY_WHEN = 'apply-when'
 # INSTANCE_CONFIGS: dict: contains all app organized by recipe names and instance domains
 # The structure of the dictionary is as follows:
 # {
@@ -57,6 +58,7 @@ ALL_CONFIGS = {}
 SETTINGS = {}
 SETTINGS_PATH = ""  # path to the alakazam settings file
 GROUP_PATH = None  # path this run was invoked with, either an instance file or a group directory
+HIDE_SECRETS = False  # keep secret values out of the output, for runs whose logs are archived
 ABRA_DIR = None  # path to the abra data directory
 FETCH_MAX_AGE = 600 # seconds; recipe repos are refetched at most once per hour
 ABRA_RETRIES = 6 # number of attempts per abra command that failed to reach the server
@@ -75,8 +77,11 @@ CONNECTION_ERRORS = (
 )
 # marker in the abra output that indicates a secret which is not stored on the server
 MISSING_SECRET_ERROR = "doesn't exist on server"
-# marker in the abra output for a generate run that had nothing left to do, which abra exits 1 on
+# abra subcommands that take a secret as an argument or return one
+SECRET_COMMANDS = ("insert", "generate")
+# markers in the abra output for a secret run that had nothing left to do, which abra exits 1 on
 NO_SECRETS_GENERATED = "no secrets generated"
+NO_SECRETS_TO_REMOVE = "no secrets to remove"
 # container statuses that have not settled yet, matched against the abra output
 PENDING_STATUS_RE = re.compile(r".*(starting|unknown|unhealthy).*")
 # marker in a service name for a container that runs once and exits instead of becoming healthy
@@ -91,8 +96,11 @@ READINESS_INITIAL_DELAY = 0 # seconds to wait before the first attempt
 SECRET_VARIANTS = ("conf", "secret-hooks", "exchange", "generate")
 # group configuration files, inherited down the directory hierarchy
 GROUP_CONFIG_RE = re.compile(r'^alaka(-.*)?\.ya?ml$')
+# config-set files, read from the root path only
+CONFIG_SET_RE = re.compile(r'^config-sets(-.*)?\.ya?ml$')
 # instance configuration files, named '<domain>.yml' after the instance they configure
 INSTANCE_CONFIG_RE = re.compile(r'^(?:[A-Za-z0-9](?:[A-Za-z0-9\-]{0,61}[A-Za-z0-9])?\.)+[A-Za-z]{2,6}(?:\.yaml|\.yml)$')
+ROOT_PATH = None  # resolved root path from alakazam.yml
 
 
 class MySafeConstructor(SafeConstructor):
@@ -185,7 +193,7 @@ def get_relevant_config_paths(root_path: Path, group_path: Path, exclude_paths:
     Returns:
         list: The configuration files to validate, without duplicates
     """
-    paths = [Path(COMBINE_PATH), root_path / "config-sets.yml"]
+    paths = [root_path / file for file in os.listdir(root_path) if CONFIG_SET_RE.match(file)]
     directory = group_path if group_path.is_dir() else group_path.parent
     for ancestor in reversed([directory, *directory.parents]):
         if ancestor != root_path and root_path not in ancestor.parents:
@@ -399,31 +407,79 @@ def merge_all_group_configs(root_path: Path) -> Dict[str, Dict[str, Any]]:
 
 
 
-def get_config_set_app_configs(
-    active_config_sets: Dict[str, Any], config_sets: Dict[str, Any]
-) -> Dict[str, Any]:
+def read_config_sets(root_path: Path) -> Dict[str, Any]:
     """
-    Merges per-app configurations from all active config-sets into a single dictionary.
-    Config-sets disabled with False are skipped; unknown set names log a warning.
-    When multiple active sets define config for the same app, later sets take precedence.
+    Reads and merges every 'config-sets*.yml' of the root path.
+
+    'config-sets.yml' comes first as the base, the 'config-sets-*.yml' specialisations follow in
+    alphabetical order, the same precedence the 'alaka*.yml' of a directory have. Config-sets live
+    in the root path only, so that an integration reads the same for every instance below it.
+
+    Args:
+        root_path (Path): The alakazam root path
+
+    Returns:
+        dict: The merged config-set definitions
+    """
+    files = sorted((f for f in os.listdir(root_path) if CONFIG_SET_RE.match(f)),
+                   key=lambda f: (not f.startswith("config-sets."), f))
+    merged: Dict[str, Any] = {}
+    defined_by: Dict[Tuple[str, ...], Tuple[str, Any]] = {}
+    for file in files:
+        config = read_config(str(root_path / file)) or {}
+        warn_on_key_collisions(config, defined_by, file, str(root_path))
+        merged = merge_dict(merged, config)
+    return merged
+
+
+def get_config_set_app_configs(
+    active_config_sets: Dict[str, Any], config_sets: Dict[str, Any], instance_apps: Iterable[str]
+) -> Tuple[Dict[str, Any], Dict[str, Any]]:
+    """
+    Merges per-app configurations from all applying config-sets into a single dictionary.
+
+    A set applies either because the instance enabled it under CONFIG-SETS, or by itself when it
+    carries an 'apply-when' and every app named there is part of the instance. That is what an
+    integration between two apps is: configuration that belongs to both of them and is pointless
+    unless both are there. Setting such a set to False for an instance still turns it off.
+
+    The two kinds are returned apart because they sit at different heights: an integration is the
+    lowest layer of all, below even the group configuration, so that a group can adjust it. That is
+    the precedence the combine.yml this replaces had. Among the explicit ones, later sets win.
 
     Args:
         active_config_sets (dict): Merged CONFIG-SETS dict (set name -> True or False).
-        config_sets (dict): Full config-set definitions loaded from config-sets.yml.
+        config_sets (dict): Full config-set definitions from the root path.
+        instance_apps (iterable): The app names configured for this instance.
 
     Returns:
-        dict: A merged dictionary of per-app configurations from all active config-sets.
+        tuple: The per-app configurations of the automatic sets and of the explicitly enabled ones
     """
+    automatic_app_configs: Dict[str, Any] = {}
     config_set_app_configs: Dict[str, Any] = {}
+    automatic = set()
+    for name, definition in config_sets.items():
+        if not (apply_when := (definition or {}).get(APPLY_WHEN)):
+            continue
+        automatic.add(name)
+        if active_config_sets.get(name) is False:
+            logging.debug(f"config-set '{name}' applies but is switched off for this instance")
+            continue
+        if not all(app in instance_apps for app in apply_when):
+            continue
+        logging.debug(f"config-set '{name}' applies, {', '.join(apply_when)} are all configured")
+        apps = {app: config for app, config in definition.items() if app != APPLY_WHEN}
+        automatic_app_configs = merge_dict(automatic_app_configs, apps)
     for config_set_name, config_set_enabled in active_config_sets.items():
-        if config_set_enabled:
-            if config_set_name not in config_sets:
-                logging.warning(
-                    f"Config-set '{config_set_name}' is enabled but not defined in config-sets.yml. Skipping."
-                )
-            else:
-                config_set_app_configs = merge_dict(config_set_app_configs, config_sets[config_set_name] or {})
-    return config_set_app_configs
+        if not config_set_enabled or config_set_name in automatic:
+            continue
+        if config_set_name not in config_sets:
+            logging.warning(
+                f"Config-set '{config_set_name}' is enabled but not defined in the root path. Skipping."
+            )
+            continue
+        config_set_app_configs = merge_dict(config_set_app_configs, config_sets[config_set_name] or {})
+    return automatic_app_configs, config_set_app_configs
 
 
 def substitute_jinja_variable(jinja_dict, subs_dict) -> None:
@@ -469,8 +525,8 @@ def merge_instance_configs(group_config: Dict[str, Any], instance_domain: str, i
         group_config.get('CONFIG-SETS') or {},
         instance_config.get('CONFIG-SETS') or {},
     )
-    config_set_app_configs = get_config_set_app_configs(active_config_sets, config_sets)
     instance_apps = {k: v for k, v in instance_config.items() if k not in NON_APP_KEYS}
+    automatic_app_configs, config_set_app_configs = get_config_set_app_configs(active_config_sets, config_sets, instance_apps)
     # Warn about config-set entries that target apps not listed in the instance config.
     for config_set_app in set(config_set_app_configs) - set(instance_apps):
         logging.warning(
@@ -479,18 +535,57 @@ def merge_instance_configs(group_config: Dict[str, Any], instance_domain: str, i
         )
     merged_config = {}
     for app in instance_apps:
+        automatic_app_config = automatic_app_configs.get(app) or {}
         group_app_config = group_config.get(app) or {}
         config_set_app_config = config_set_app_configs.get(app) or {}
         instance_app_config = instance_apps[app] or {}
-        # Priority: group < config-set < instance
-        merged_config[app] = merge_dict(merge_dict(group_app_config, config_set_app_config), instance_app_config)
+        # Priority: automatic set < group < config-set < instance. The automatic sets are merged
+        # last so that everything else outranks them, and with the list order reversed so that
+        # their entries land behind the ones the app configures for itself. An integration that
+        # adds 'set_authentik' must run after the 'core_install' that sets the app up at all.
+        merged_config[app] = merge_dict(
+            automatic_app_config,
+            merge_dict(merge_dict(group_app_config, config_set_app_config), instance_app_config),
+            reverse_list_order=True)
         merged_config[app]['app_domain'] = map_subdomain(app, instance_domain, merged_config[app])
         if not merged_config[app].get('server'):
             merged_config[app]['server'] = server
     substitute_jinja_variable(merged_config, global_vars)
+    # after the substitution, so that a templated version is resolved too
+    for app, app_config in merged_config.items():
+        if app_config.get('version'):
+            app_config['version'] = resolve_version(app_config['version'], app)
     return merged_config
 
 
+def resolve_version(version: Any, app: str) -> Any:
+    """
+    Reduces a configured version to the part that is deployed.
+
+    A version may name a commit behind an '@', as in '12.0.2+2026.5.2@be9ebb3'. What stands before
+    it is the release the commit is based on: it carries no meaning for the deployment and exists
+    so that a dependency bot can follow the recipe's tags and offer patch updates. Only the commit
+    is deployed. Everything without an '@' is passed through untouched.
+
+    Args:
+        version: The configured version, any type the configuration may hold
+        app (str): The app the version belongs to, for the error message
+
+    Returns:
+        The commit behind the '@', or the version unchanged
+
+    Raises:
+        click.ClickException: If the '@' is there but no commit follows it
+    """
+    if not isinstance(version, str) or "@" not in version:
+        return version
+    base, _, commit = version.rpartition("@")
+    if not commit:
+        raise click.ClickException(f"version '{version}' of {app} ends in '@' without a commit")
+    logging.debug(f"{app}: deploying commit {commit}, based on {base}")
+    return commit
+
+
 def map_subdomain(recipe: str, instance_domain: str, app_config: Dict[str, Any]) -> str:
     """
     Maps a subdomain for an app based on the recipe, instance domain, and specific app configuration.
@@ -546,40 +641,43 @@ def get_merged_instance_configs(config_path: Path, group_configs: Dict[str, Any]
     return instances
 
 
-def merge_connection_configs(configs: Dict[str, Any]) -> Dict[str, Any]:
+def exclude_from_configs(configs: Dict[str, Dict[str, Any]], recipes: Tuple[str]) -> Dict[str, Dict[str, Any]]:
     """
-    Merge connection configurations from the 'combine.yml' to extend instance configurations with inter-app secrets and settings.
-    This involves integrating shared secrets and other connection-specific settings between applications within the same instance.
+    Removes the given recipes from every instance configuration.
+
+    An excluded recipe is not merely left undeployed, it is invisible for the whole run: it is
+    neither listed, nor configured, nor purged, nor waited for. That is what keeps a rebuild from
+    touching an app whose state has to survive it, such as the certificates of a reverse proxy.
 
     Args:
-        configs (dict): The initial instance configurations before integrating connection-specific adjustments.
+        configs (dict): Instances as keys and their app configurations as values
+        recipes (tuple): Recipe names to leave out, all recipes are kept if empty
 
     Returns:
-        dict: The updated instance configurations after applying the connection settings.
+        dict: The configurations without those recipes
     """
-    connection_config = read_config(COMBINE_PATH)
-    extend_shared_secrets(connection_config)
-    merged_configs = configs.copy()
-    for _, instance_config in merged_configs.items():
-        for target_app, source_apps in connection_config.items():
-            for source_app, target_conf in source_apps.items():
-                if target_app in instance_config and source_app in instance_config:
-                    instance_config[target_app] = merge_dict(target_conf, instance_config[target_app], reverse_list_order=True)
-    return merged_configs
+    if not recipes:
+        return configs
+    for recipe in recipes:
+        if not any(recipe in apps for apps in configs.values()):
+            logging.warning(f"'{recipe}' is excluded but not configured for this path")
+    return {
+        instance: {name: config for name, config in apps.items() if name not in recipes}
+        for instance, apps in configs.items()
+    }
 
 
-def extend_shared_secrets(connection_config: Dict[str, Any]) -> None:
+def is_secret_command(args: Tuple[str, ...]) -> bool:
     """
-    Extends connection configurations by embedding source app details into the shared secrets configuration.
-    This modifies the existing connection configurations in place, adding a layer of source app information to shared secrets.
+    Checks whether an abra command carries a secret value in its arguments or in its output.
 
     Args:
-        connection_config (dict): Connection configurations which involve shared secrets.
+        args (tuple): The arguments passed to abra()
+
+    Returns:
+        bool: True for the commands that insert or generate a secret
     """
-    for _, source_apps in connection_config.items(): 
-        for source_app, target_conf in source_apps.items():
-            if shared_secrets:= target_conf.get('shared_secrets'):
-                target_conf['shared_secrets'] = {source_app: shared_secrets}
+    return args[:2] == ("app", "secret") and len(args) > 2 and args[2] in SECRET_COMMANDS
 
 
 def is_connection_error(output: str) -> bool:
@@ -618,7 +716,7 @@ def run_streamed(command: List[str]) -> subprocess.CompletedProcess:
     return subprocess.CompletedProcess(command, process.returncode, b"".join(lines), b"")
 
 
-def abra(*args: str, machine_output: bool = False, ignore_error: bool = False, stream: bool = False) -> Union[str,Dict]:
+def abra(*args: str, machine_output: bool = False, ignore_error: bool = False, stream: bool = False, secret: bool = False) -> Union[str,Dict]:
     """
     Execute the 'abra' command with the specified arguments. This function acts as a wrapper around the 'abra' CLI tool. It allows for capturing the output and optionally returning it as machine-readable JSON.
 
@@ -627,6 +725,7 @@ def abra(*args: str, machine_output: bool = False, ignore_error: bool = False, s
         machine_output (bool): If True, expects the output in JSON format and parses it before returning.
         ignore_error (bool): If True, suppresses the raising of errors on non-zero return codes, otherwise an exception is raised.
         stream (bool): If True, echoes the output while the command runs instead of only returning it afterwards. Cannot be combined with machine_output, which needs clean JSON on stdout.
+        secret (bool): If True, the output of this command may carry secret values, which HIDE_SECRETS then keeps out of the stream, the log and the error message.
 
     Returns:
         str or dict: Returns the output from the 'abra' command. If machine_output is True, returns a dictionary, otherwise returns raw output as a string.
@@ -641,7 +740,14 @@ def abra(*args: str, machine_output: bool = False, ignore_error: bool = False, s
     command = [arg for arg in command if arg]
     if machine_output:
         command.append("-m")
-    logging.debug(f"run command: {' '.join(command)}")
+    # the command line of 'app secret insert' ends in the secret itself, and the output of
+    # 'app secret generate' contains the generated values
+    quiet = HIDE_SECRETS and (secret or is_secret_command(args))
+    if quiet:
+        # echoing it live would put the very values that are being hidden into the output
+        stream = False
+    if not quiet:
+        logging.debug(f"run command: {' '.join(command)}")
     for attempt in range(1, ABRA_RETRIES + 1):
         process = run_streamed(command) if stream else subprocess.run(command, capture_output=True)
         if not process.returncode or attempt == ABRA_RETRIES:
@@ -653,12 +759,15 @@ def abra(*args: str, machine_output: bool = False, ignore_error: bool = False, s
         delay = ABRA_RETRY_DELAY * 2 ** (attempt - 1)
         logging.warning(f"attempt {attempt}/{ABRA_RETRIES} of '{' '.join(command)}' failed to reach the server, retry in {delay}s")
         sleep(delay)
-    if process.stderr and ignore_error:
+    if process.stderr and ignore_error and not quiet:
         logging.warning(process.stderr.decode())
-    if process.stdout and not stream:
+    if process.stdout and not stream and not quiet:
         logging.debug(process.stdout.decode())
     if process.returncode and not ignore_error:
         #breakpoint()
+        if quiet:
+            raise RuntimeError(
+                f'{" ".join(args[:3])} failed, its output is withheld by --hide-secrets')
         raise RuntimeError(
             f'{" ".join(command)} \n STDOUT: \n {process.stdout.decode()} \n STDERR: {process.stderr.decode()}')
     if machine_output:
@@ -829,6 +938,23 @@ def update_configs(path: Path, config: Dict[str, Any]) -> None:
             dotenv.set_key(path, key, value, quote_mode="never")
 
 
+def get_missing_secrets(domain: str) -> List[str]:
+    """
+    Lists the secrets a recipe declares that the server does not hold.
+
+    This is the same condition abra checks before it deploys, where a missing secret ends the run
+    with "secret not generated". Reading it beforehand is what lets a caller fill the gap instead.
+
+    Args:
+        domain (str): The app domain to check
+
+    Returns:
+        list: The names of the missing secrets, empty when the app is complete
+    """
+    stored_secrets = abra("app", "secret", "ls", domain, machine_output=True)
+    return [s['name'] for s in stored_secrets or [] if not str2bool(s['created on server'])]
+
+
 def generate_all_secrets(domain: str) -> None:
     """
     Generates all secrets for the app specified by its domain using the 'abra' command.
@@ -856,7 +982,25 @@ def generate_all_secrets(domain: str) -> None:
             return
         print(f"secrets for {domain} generated")
         for gen_sec in generated_secrets:
-            print(f"\t {gen_sec['name']}: {gen_sec['value']}")
+            value = "[hidden]" if HIDE_SECRETS else gen_sec['value']
+            print(f"\t {gen_sec['name']}: {value}")
+
+
+def resolve_path(path_str: str, base: Optional[Path] = None) -> Path:
+    """
+    Resolve a path string to an absolute Path, expanding ~ and resolving relative paths against a base directory.
+
+    Args:
+        path_str (str): The path string to resolve. May be absolute, relative, or start with ~.
+        base (Path): The base directory for resolving relative paths. Defaults to ROOT_PATH if not provided.
+
+    Returns:
+        Path: The resolved absolute path.
+    """
+    p = Path(path_str).expanduser()
+    if p.is_absolute():
+        return p
+    return ((base or ROOT_PATH) / p).absolute()
 
 
 def get_abra_dir() -> Path:
@@ -881,8 +1025,7 @@ def get_abra_dir() -> Path:
             if config_file.exists():
                 abra_config = read_config(str(config_file))
                 if abra_dir := abra_config.get("abraDir"):
-                    p = Path(abra_dir)
-                    return (current / p).resolve() if not p.is_absolute() else p
+                    return resolve_path(abra_dir, current)
         if current == home:
             break
         current = current.parent
@@ -1154,13 +1297,54 @@ def insert_secrets_from_conf(domain: str, app_config: Dict[str, Any]) -> None:
                 insert_secret(domain, secret_name, secret)
 
 
-def run_secret_hooks(domain: str, app_config: Dict[str, Any], strict: bool = False) -> None:
+def run_local_script(tokens: List[str], app_domain: str, server: str, instance_domain: str, dry_run: bool = False, strict: bool = False) -> None:
     """
-    Run local abra.sh commands to generate secrets.
+    Run a local script hook. Relative paths are resolved against ROOT_PATH for execution. Logs an error and returns early if the script is not executable.
+
+    Args:
+        tokens (list): The hook entry split on whitespace, with tokens[0] confirmed to be 'script', tokens[1] the script path, and tokens[2:] positional arguments.
+        app_domain (str): The app domain, passed as ALAKAZAM_APP_DOMAIN to the script environment.
+        server (str): The server name, passed as ALAKAZAM_APP_SERVER to the script environment.
+        instance_domain (str): The instance domain, passed as ALAKAZAM_INSTANCE_DOMAIN to the script environment.
+        dry_run (bool): If True, prints the command but does not execute it.
+        strict (bool): Abort on a non-executable script or a non-zero exit code instead of only logging it.
+
+    Raises:
+        click.ClickException: If the script is not executable or exits non-zero and strict is set
+    """
+    script_path = resolve_path(tokens[1])
+    args = tokens[2:]
+    env = {**os.environ, "ALAKAZAM_APP_DOMAIN": app_domain, "ALAKAZAM_APP_SERVER": server, "ALAKAZAM_INSTANCE_DOMAIN": instance_domain}
+    cmd_display = " ".join(tokens[1:])
+    print(f"Run local script '{cmd_display}' for {app_domain}")
+    if not os.access(script_path, os.X_OK):
+        message = f"Script is not executable (run: chmod +x {script_path})"
+        if strict:
+            raise click.ClickException(f"script '{cmd_display}' failed for {app_domain}: {message}")
+        logging.error(message)
+        return
+    if dry_run:
+        return
+    # the script may print a secret it just created, so its output is captured and dropped
+    result = subprocess.run([str(script_path)] + args, env=env, capture_output=HIDE_SECRETS)
+    if result.returncode != 0:
+        message = f"Script '{cmd_display}' exited with code {result.returncode}"
+        if strict:
+            raise click.ClickException(f"script '{cmd_display}' failed for {app_domain}: {message}")
+        logging.warning(message)
+
+
+def run_secret_hooks(domain: str, app_config: Dict[str, Any], instance_domain: str = "", strict: bool = False) -> None:
+    """
+    Run local abra.sh commands or local scripts to generate secrets.
+
+    A secret hook may print the value it just created, which some recipes do on purpose because
+    only a hash of it is stored. Under HIDE_SECRETS that output is withheld, failures included.
 
     Args:
         domain (str): The app domain into which the secrets are to be inserted.
         app_config (dict): A dictionary containing the secrets hooks and their corresponding values to insert.
+        instance_domain (str): The instance domain, used to set ALAKAZAM_INSTANCE_DOMAIN for script hooks.
         strict (bool): Abort on a failing hook instead of only logging it.
 
     Raises:
@@ -1169,11 +1353,15 @@ def run_secret_hooks(domain: str, app_config: Dict[str, Any], strict: bool = Fal
     logging.info(f"Run secret hooks for {domain}")
     if secret_hooks := app_config.get("secret_hooks"):
         for cmd in secret_hooks:
-            print(f"Run '{cmd}' in {domain}", flush=True)
-            try:
-                abra("app", "cmd", "--local", domain, cmd, ignore_error=not strict, stream=True)
-            except RuntimeError as e:
-                raise click.ClickException(f"secret hook '{cmd}' failed for {domain}: {e}")
+            tokens = cmd.split()
+            if tokens[0] == "script" and len(tokens) >= 2 and resolve_path(tokens[1]).exists():
+                run_local_script(tokens, domain, app_config['server'], instance_domain, strict=strict)
+            else:
+                print(f"Run '{cmd}' in {domain}", flush=True)
+                try:
+                    abra("app", "cmd", "--local", domain, cmd, ignore_error=not strict, stream=True, secret=True)
+                except RuntimeError as e:
+                    raise click.ClickException(f"secret hook '{cmd}' failed for {domain}: {e}")
 
 
 def unquote_strings(s: str) -> str:
@@ -1208,7 +1396,10 @@ def insert_secret(domain: str, secret_name: str, secret: str) -> None:
     """
     # Fix extra quotes around secrets
     secret = unquote_strings(secret)
-    logging.debug(f"Insert secret {secret_name}: {secret} into {domain}")
+    if HIDE_SECRETS:
+        logging.debug(f"Insert secret {secret_name} into {domain}")
+    else:
+        logging.debug(f"Insert secret {secret_name}: {secret} into {domain}")
     abra("app", "secret", "insert", domain, secret_name, "v1", secret)
 
 
@@ -1389,7 +1580,7 @@ def run_readiness_hooks(domain: str, app_config: Dict[str, Any]) -> None:
         run_readiness_hook(domain, hook)
 
 
-def execute_cmds(app_config: Dict[str, Any], commands: Tuple[str] = tuple(), initial: bool = False, deploy: bool = False, upgrade: bool = False, dry_run: bool = False, chaos: bool = False, strict: bool = False) -> None:
+def execute_cmds(app_config: Dict[str, Any], commands: Tuple[str] = tuple(), initial: bool = False, deploy: bool = False, upgrade: bool = False, dry_run: bool = False, chaos: bool = False, strict: bool = False, instance_domain: str = "") -> None:
     """
     Execute post-deployment commands for an application based on the provided configuration.
     This can include running scripts or commands inside the application's environment.
@@ -1401,8 +1592,9 @@ def execute_cmds(app_config: Dict[str, Any], commands: Tuple[str] = tuple(), ini
         initial (bool): execute initial-hooks
         deploy (bool): execute deploy-hooks
         upgrade (bool): execute upgrade-hooks
-        dry-run(bool): only show cmds, don't execute them
+        dry_run (bool): only show cmds, don't execute them
         strict (bool): abort on a failing command instead of only logging it
+        instance_domain (str): The instance domain, used to set ALAKAZAM_INSTANCE_DOMAIN for script hooks.
 
     Returns:
         None
@@ -1411,6 +1603,7 @@ def execute_cmds(app_config: Dict[str, Any], commands: Tuple[str] = tuple(), ini
         click.ClickException: If a command fails and strict is set
     """
     domain = app_config['app_domain']
+    server = app_config['server']
     all_cmds = []
     if initial and (initial_hooks:= app_config.get('initial-hooks')):
         all_cmds = all_cmds + initial_hooks
@@ -1424,7 +1617,12 @@ def execute_cmds(app_config: Dict[str, Any], commands: Tuple[str] = tuple(), ini
     if chaos:
         chaos_flag = "-C"
     for cmd in all_cmds:
-        print(f"Run '{cmd}' in {domain}:{cmd.split()[0]}", flush=True)
+        tokens = cmd.split()
+        container = tokens[0]
+        if container == "script" and len(tokens) >= 2 and resolve_path(tokens[1]).exists():
+            run_local_script(tokens, domain, server, instance_domain, dry_run, strict)
+            continue
+        print(f"Run '{cmd}' in {domain}:{container}", flush=True)
         if dry_run:
             continue
         try:
@@ -1436,8 +1634,10 @@ def execute_cmds(app_config: Dict[str, Any], commands: Tuple[str] = tuple(), ini
 @click.group(context_settings={"help_option_names": ['-h', '--help']})
 @click.option('-l', '--log', 'loglevel', help='Desired logging level ("debug", "info", "warning", "error", "critical")')
 @click.option('-e', '--exclude', help='Path to a directory that contains a group of instance configurations to be excluded.', multiple=True, type=click.Path(exists=True))
+@click.option('exclude_recipes', '-er', '--exclude-recipe', multiple=True, metavar='<RecipeName>', help='Leave these recipes untouched, this option can be specified multiple times.')
+@click.option('hide_secrets', '--hide-secrets', is_flag=True, help='Keep secret values out of the output, for runs whose logs are kept.')
 @click.argument('group_path', type=click.Path(exists=True))
-def cli(loglevel: str, group_path: str, exclude:Tuple[str]) -> None:
+def cli(loglevel: str, group_path: str, exclude: Tuple[str], exclude_recipes: Tuple[str], hide_secrets: bool) -> None:
     """
     Alakazam is a meta-configuration app-connector and an abra wrapper, designed as a proof-of-concept to simplify the management of environment configuration files across multiple instances.
 
@@ -1448,7 +1648,10 @@ def cli(loglevel: str, group_path: str, exclude:Tuple[str]) -> None:
     global SETTINGS
     global SETTINGS_PATH
     global GROUP_PATH
+    global HIDE_SECRETS
     global ABRA_DIR
+    global ROOT_PATH
+    HIDE_SECRETS = hide_secrets
     if loglevel:
         numeric_level = getattr(logging, loglevel.upper(), None)
         if not isinstance(numeric_level, int):
@@ -1463,22 +1666,24 @@ def cli(loglevel: str, group_path: str, exclude:Tuple[str]) -> None:
         root_path = os.getcwd()
         logging.warning(f"There is no 'root' path defined in '{SETTINGS_PATH}', use current path '{root_path}'instead")
     _group_path = GROUP_PATH = Path(group_path).expanduser().absolute()
-    _root_path = Path(root_path).expanduser()
-    if not _root_path.is_absolute():
-        _root_path = (settings_dir / _root_path).absolute()
-    if not str(_group_path).startswith(str(_root_path)):
-        logging.error(f"{_root_path} does not contain {_group_path}?")
+    ROOT_PATH = resolve_path(root_path, settings_dir)
+    if not str(_group_path).startswith(str(ROOT_PATH)):
+        logging.error(f"{ROOT_PATH} does not contain {_group_path}?")
         exit(1)
     exclude_paths = list(map(lambda p: str(Path(p).absolute()), exclude))
-    if ABRA_DIR.is_relative_to(_root_path) and str(ABRA_DIR) not in exclude_paths:
+    if ABRA_DIR.is_relative_to(ROOT_PATH) and str(ABRA_DIR) not in exclude_paths:
         exclude_paths.append(str(ABRA_DIR))
-    preflight_configs(get_relevant_config_paths(_root_path, _group_path, exclude_paths))
-    all_group_configs = merge_all_group_configs(_root_path)
-    config_sets = read_config(str(_root_path / "config-sets.yml"))
+    preflight_configs(get_relevant_config_paths(ROOT_PATH, _group_path, exclude_paths))
+    all_group_configs = merge_all_group_configs(ROOT_PATH)
+    config_sets = read_config_sets(ROOT_PATH)
     instance_configs = get_merged_instance_configs(_group_path, all_group_configs, exclude_paths, config_sets)
-    INSTANCE_CONFIGS = merge_connection_configs(instance_configs)
-    all_configs = get_merged_instance_configs(_root_path, all_group_configs, exclude_paths, config_sets)
-    ALL_CONFIGS = merge_connection_configs(all_configs)
+    INSTANCE_CONFIGS = instance_configs
+    # dropping the recipes here rather than at every filter is what makes the exclusion complete:
+    # get_apps(), create_secrets(), configure_apps() and the rest all read INSTANCE_CONFIGS.
+    # ALL_CONFIGS stays whole, it is the cross-instance view that backup looks up its bot in
+    INSTANCE_CONFIGS = exclude_from_configs(INSTANCE_CONFIGS, exclude_recipes)
+    all_configs = get_merged_instance_configs(ROOT_PATH, all_group_configs, exclude_paths, config_sets)
+    ALL_CONFIGS = all_configs
     fetch_recipes(INSTANCE_CONFIGS)
 
         
@@ -1561,8 +1766,8 @@ def create_secrets(recipes: Tuple[str], syncvalues: bool = False, variants: Opti
     """
     if variants is None:
         variants = set(SECRET_VARIANTS)
-    for instance, instance_config in INSTANCE_CONFIGS.items():
-        if instances and instance not in instances:
+    for instance_domain, instance_config in INSTANCE_CONFIGS.items():
+        if instances and instance_domain not in instances:
             continue
         instance_apps = instance_config.keys()
         if recipes:
@@ -1579,7 +1784,7 @@ def create_secrets(recipes: Tuple[str], syncvalues: bool = False, variants: Opti
             if "conf" in variants:
                 insert_secrets_from_conf(domain, app_config)
             if "secret-hooks" in variants:
-                run_secret_hooks(domain, app_config, strict=strict)
+                run_secret_hooks(domain, app_config, instance_domain, strict=strict)
         # Pass 2: exchange secrets between apps, then generate any remaining missing secrets.
         for app in selected_apps:
             app_config = instance_config[app]
@@ -1673,7 +1878,7 @@ def deploy_apps(instance_apps: Dict[str, List[List]], execute_hooks: bool = Fals
             print(abra("app", *cmd))
             if execute_hooks:
                 logging.info(f'execute commands for {domain}')
-                execute_cmds(app_config, deploy=True)
+                execute_cmds(app_config, deploy=True, instance_domain=instance)
 
 
 @cli.command()
@@ -1681,9 +1886,12 @@ def deploy_apps(instance_apps: Dict[str, List[List]], execute_hooks: bool = Fals
 @click.option('-e', '--execute-hooks', is_flag=True, help='run post-upgrade commands.')
 @click.option('-d', '--dry-run', is_flag=True, help="don't execute the upgrade process")
 @click.option('-rd', '--redeploy', is_flag=True, help="use undeploy and deploy for the updating process")
+@click.option('--redeploy-wait', type=int, default=20, metavar='<SECONDS>',
+              help='seconds to wait between undeploy and deploy in --redeploy mode (default: 20).')
 @click.option('-c', '--converge-checks', is_flag=True, help='perform convergence checks during deployment.')
 @click.option('noninteractive', '-n', '--non-interactive', is_flag=True, help='Run this command non-interactively')
-def upgrade(recipes: Tuple[str], execute_hooks: bool, dry_run: bool, redeploy: bool, converge_checks: bool, noninteractive: bool) -> None:
+def upgrade(recipes: Tuple[str], execute_hooks: bool, dry_run: bool, redeploy: bool,
+            converge_checks: bool, noninteractive: bool, redeploy_wait: int) -> None:
     """
     Upgrades specified applications by executing the upgrade commands via the 'abra' command-line interface.
     It checks the current deployment status of the apps and performs upgrades only where necessary, with options to execute additional commands or perform a dry run. It either took the target version from the configuration or it uses the latest available version.
@@ -1727,7 +1935,7 @@ def upgrade(recipes: Tuple[str], execute_hooks: bool, dry_run: bool, redeploy: b
             app_details.append(upgrade_version)
             upgrade_apps.append(app_details)
             logging.info(f'upgrade {app}: {domain} from version {deployed_version} to version "{upgrade_version}"')
-            upgrade_cmds.append((app_config, upgrade_cmd))
+            upgrade_cmds.append((app, app_config, upgrade_cmd, instance))
             if version_type == 'version':
                 release_note_cmd = upgrade_cmd.copy()
                 release_note_cmd.insert(1, '-r')
@@ -1742,19 +1950,25 @@ def upgrade(recipes: Tuple[str], execute_hooks: bool, dry_run: bool, redeploy: b
         print(app)
         print(note)
     if not dry_run and noninteractive or input(f"Do you really want to upgrade these apps? Type YES: ") == "YES":
-        for app_config, upgrade_cmd in upgrade_cmds:
+        for app, app_config, upgrade_cmd, instance_domain in upgrade_cmds:
             app_domain = app_config.get('app_domain')
+            # a recipe can bring a new secret along, which abra refuses to deploy without. The app
+            # is running, so nothing else can be missing: it had all of them when it was deployed
+            if missing := get_missing_secrets(app_domain):
+                print(f"{app_domain} is missing {len(missing)} secret(s): {', '.join(sorted(missing))}")
+                create_secrets((app,), instances=(instance_domain,))
             if redeploy:
                 upgrade_cmd.pop(0)
-                print(f'undeploy {app_domain}')
                 upgrade_cmd.insert(0, 'deploy')
+                print(f'undeploy {app_domain}')
                 print(abra("app", "undeploy", '--no-input', app_domain))
-                sleep(20)
+                print(f'waiting {redeploy_wait}s …')
+                sleep(redeploy_wait)
             print(f'deploy {app_domain}')
             print(abra("app", *upgrade_cmd))
             if execute_hooks:
                 logging.info(f'execute commands for {app_domain}')
-                execute_cmds(app_config, upgrade=True)
+                execute_cmds(app_config, upgrade=True, instance_domain=instance_domain)
 
 
 @cli.command()
@@ -1800,7 +2014,7 @@ def cmd(recipes: Tuple[str], commands: Tuple[str], initial: bool, deploy: bool,
     Execute commands for all specified applications based on the provided configuration.
     """
     deployed_domains = get_deployed_apps(recipes)
-    for _, instance_config in INSTANCE_CONFIGS.items():
+    for instance_domain, instance_config in INSTANCE_CONFIGS.items():
         if recipes:
             selected_apps = [app for app in recipes if app in instance_config.keys()]
         else:
@@ -1812,7 +2026,7 @@ def cmd(recipes: Tuple[str], commands: Tuple[str], initial: bool, deploy: bool,
                 print(f"{domain} is not deployed")
                 continue
             logging.info(f'execute commands for {domain}')
-            execute_cmds(app_config, commands, initial, deploy, upgrade, list_cmds, chaos)
+            execute_cmds(app_config, commands=commands, initial=initial, deploy=deploy, upgrade=upgrade, dry_run=list_cmds, chaos=chaos, instance_domain=instance_domain)
 
 
 @cli.command()
@@ -2128,7 +2342,7 @@ def parse_secret_variants(only: Tuple[str]) -> Set[str]:
 def purge_app_secrets(recipe_secrets: Dict[str, List[str]]) -> None:
     """
     Removes the given secrets from every app of the given recipes.
-    A secret that is not stored on the server is skipped, so an aborted run can simply be repeated.
+    A secret that is not stored on the server is skipped, and so is an app that has none at all, so an aborted run can simply be repeated.
     All other errors are not suppressed: a secret that could not be removed would silently keep its old value on a later recreation.
 
     Args:
@@ -2139,7 +2353,13 @@ def purge_app_secrets(recipe_secrets: Dict[str, List[str]]) -> None:
             for _, domain in app_domains:
                 logging.info(f'purge secrets of {domain}')
                 if not secret_names:
-                    abra("app", "secret", "rm", "-a", domain)
+                    try:
+                        abra("app", "secret", "rm", "-a", domain)
+                    except RuntimeError as e:
+                        if NO_SECRETS_TO_REMOVE not in str(e):
+                            raise
+                        print(f"{domain} has no secrets on its server, skip")
+                        continue
                     print(f"Secrets for {domain} purged")
                     continue
                 for secret_name in secret_names:

combine.yml

@@ -1,301 +0,0 @@
-authentik:
-    nextcloud:
-        uncomment:
-            - compose.nextcloud.yml
-            - NEXTCLOUD_DOMAIN
-            - SECRET_NEXTCLOUD_ID_VERSION
-            - SECRET_NEXTCLOUD_SECRET_VERSION
-            - nextcloud.png
-    wordpress:
-        uncomment:
-            - compose.wordpress.yml
-            - WORDPRESS_DOMAIN
-            - WORDPRESS_GROUP
-            - SECRET_WORDPRESS_ID_VERSION
-            - SECRET_WORDPRESS_SECRET_VERSION
-            - wordpress.png
-    matrix-synapse:
-        uncomment:
-            - compose.matrix.yml
-            - ELEMENT_DOMAIN
-            - MATRIX_DOMAIN
-            - SECRET_MATRIX_ID_VERSION
-            - SECRET_MATRIX_SECRET_VERSION
-            - matrix.svg
-        secrets:
-            matrix_id: matrix
-    wekan:
-        uncomment:
-            - compose.wekan.yml
-            - WEKAN_DOMAIN
-            - SECRET_WEKAN_ID_VERSION
-            - SECRET_WEKAN_SECRET_VERSION
-            - wekan.png
-        secrets:
-            wekan_id: wekan
-    vikunja:
-        uncomment:
-            - compose.vikunja.yml
-            - VIKUNJA_DOMAIN
-            - SECRET_VIKUNJA_ID_VERSION
-            - SECRET_VIKUNJA_SECRET_VERSION
-            - vikunja.svg
-        secrets:
-            vikunja_id: vikunja
-    kimai:
-        uncomment:
-            - compose.kimai.yml
-            - KIMAI_DOMAIN
-            - SECRET_KIMAI_ID_VERSION
-            - SECRET_KIMAI_SECRET_VERSION
-            - kimai_logo.png
-    zammad:
-        uncomment:
-            - compose.zammad.yml
-            - ZAMMAD_DOMAIN
-            - zammad.svg
-    monitoring-ng:
-        uncomment:
-            - compose.monitoring.yml
-            - MONITORING_DOMAIN
-            - SECRET_MONITORING_ID_VERSION
-            - SECRET_MONITORING_SECRET_VERSION
-            - monitoring.svg
-    outline:
-        uncomment:
-            - compose.outline.yml
-            - OUTLINE_DOMAIN
-            - SECRET_OUTLINE_ID_VERSION
-            - SECRET_OUTLINE_SECRET_VERSION
-            - outline.png
-        secrets:
-            outline_id: outline
-    rallly:
-        uncomment:  
-            - compose.rallly.yml
-            - RALLLY_DOMAIN
-            - SECRET_RALLLY_ID_VERSION
-            - SECRET_RALLLY_SECRET_VERSION
-            - rallly.png
-        secrets:
-            rallly_id: rallly
-    hedgedoc:
-        uncomment:  
-            - compose.hedgedoc.yml
-            - HEDGEDOC_DOMAIN
-            - SECRET_HEDGEDOC_ID_VERSION
-            - SECRET_HEDGEDOC_SECRET_VERSION
-            - hedgedoc.png
-        secrets:
-            hedgedoc_id: hedgedoc
-    pretix:
-        env:
-            APPLICATIONS: 
-                Pretix:
-                    url: https://pretix.example.com/control/
-                    group:
-            EXTRA_ICONS:
-                Pretix: ~/.abra/recipes/authentik/icons/pretix.svg
-    vaultwarden:
-        env:
-            APPLICATIONS: 
-                Vaultwarden:
-                    url: https://vaultwarden.example.com/
-                    group:
-            EXTRA_ICONS:
-                Vaultwarden: ~/.abra/recipes/authentik/icons/vaultwarden.svg
-    mila:
-        uncomment:
-            - compose.mila.yml
-            - MILA_DOMAIN
-            - MILA_GROUP
-            - SECRET_MILA_ID_VERSION
-            - SECRET_MILA_SECRET_VERSION
-            - mila.svg
-        secrets:
-            mila_id: mila
-kimai:
-    authentik:
-        uncomment:
-            - SSO_ENABLED
-            - SSO_PROVIDER_URL
-            - SSO_SAML_URL
-            - SSO_LOGOUT_URL
-        secret_hooks:
-            - insert_authentik_certificate
-        dependency: [authentik]
-zammad:
-    authentik:
-        uncomment:
-            - SSO_PROVIDER_DOMAIN
-            - IDP_SSO_TARGET_URL
-            - IDP_SLO_SERVICE_URL
-        initial-hooks:
-            - local enable_authentik_sso
-        dependency: [authentik]
-nextcloud:
-    authentik:
-        uncomment:
-            - compose.authentik.yml
-            - AUTHENTIK_USER_PREFIX
-            - AUTHENTIK_DOMAIN
-            - SECRET_AUTHENTIK_SECRET_VERSION
-            - SECRET_AUTHENTIK_ID_VERSION
-        initial-hooks:
-            - app set_authentik
-        shared_secrets:
-            nextcloud_secret: authentik_secret
-            nextcloud_id: authentik_id
-    onlyoffice:
-        uncomment:
-            - compose.onlyoffice.yml
-            - ONLYOFFICE_URL
-            - SECRET_ONLYOFFICE_JWT_VERSION
-        initial-hooks:
-            - app install_onlyoffice
-    collabora:
-        uncomment:
-            - COLLABORA_URL
-        initial-hooks:
-            - app install_collabora
-onlyoffice:
-    nextcloud:
-        uncomment:
-            - compose.jwt.yml
-            - SECRET_JWT_SECRET_VERSION
-        shared_secrets:
-            onlyoffice_jwt: jwt_secret
-outline:
-    authentik:
-        env: 
-            OIDC_CLIENT_ID: outline
-            OIDC_AUTH_URI: https://authentik.example.com/application/o/authorize/
-            OIDC_TOKEN_URI: https://authentik.example.com/application/o/token/
-            OIDC_USERINFO_URI: https://authentik.example.com/application/o/userinfo/
-            OIDC_DISPLAY_NAME: "Authentik"
-        uncomment:
-            - compose.oidc.yml
-            - OIDC_ENABLED
-            - OIDC_USERNAME_CLAIM
-            - OIDC_SCOPES
-            - SECRET_OIDC_CLIENT_SECRET_VERSION
-        shared_secrets:
-            outline_secret: oidc_client_secret
-wordpress:
-    authentik:
-        uncomment:
-            - compose.authentik.yml
-            - AUTHENTIK_DOMAIN
-            - SECRET_AUTHENTIK_SECRET_VERSION
-            - SECRET_AUTHENTIK_ID_VERSION
-            - LOGIN_TYPE
-        initial-hooks:
-            - app set_authentik
-        shared_secrets:
-            wordpress_secret: authentik_secret
-            wordpress_id: authentik_id
-vikunja:
-    authentik:
-        env:
-            OAUTH_NAME: authentik
-            OAUTH_URL: https://authentik.example.com/application/o/vikunja/
-            OAUTH_LOGOUT_URL: https://authentik.example.com/application/o/vikunja/end-session/
-            # TODO: set CLIENT_ID as secret
-            OAUTH_CLIENT_ID: vikunja
-        uncomment:
-            - compose.oauth.yml
-            - OAUTH_ENABLED
-            - SECRET_OAUTH_SECRET_VERSION
-        shared_secrets:
-            #vikunja_id: oauth_id
-            vikunja_secret: oauth_secret
-matrix-synapse:
-    authentik:
-        env:
-            KEYCLOAK_ID: authentik
-            KEYCLOAK_NAME: sso
-            KEYCLOAK_URL: https://authentik.example.com/application/o/matrix/
-            # TODO: correct client domain?
-            KEYCLOAK_CLIENT_DOMAIN: https://element-web.example.com
-            KEYCLOAK_ALLOW_EXISTING_USERS: "true"
-            # TODO: set CLIENT_ID as secret
-            KEYCLOAK_CLIENT_ID: matrix
-        uncomment:
-            - compose.keycloak.yml
-            - KEYCLOAK_ENABLED
-            - KEYCLOAK_CLIENT_ID
-            - SECRET_KEYCLOAK_CLIENT_SECRET_VERSION
-        shared_secrets:
-            matrix_secret: keycloak_client_secret
-        dependency: [authentik]
-traefik:
-    matrix-synapse:
-        uncomment:
-            - compose.matrix.yml
-            - MATRIX_FEDERATION_ENABLED
-rallly:
-    authentik:
-        env:
-            OIDC_NAME: "Authentik"
-            OIDC_DISCOVERY_URL: "https://authentik.example.com/application/o/rallly/.well-known/openid-configuration"
-            OIDC_ISSUER_URL: "https://authentik.example.com/application/o/rallly/"
-            OIDC_CLIENT_ID: rallly
-        uncomment: 
-            - compose.oidc.yml
-            - SECRET_OIDC_CLIENT_SECRET_VERSION
-            - OIDC_ENABLED
-        shared_secrets:
-            rallly_secret: oidc_client_secret
-wekan:
-    authentik:
-        env: 
-            OAUTH2_ENABLED: "true"
-            OAUTH2_SERVER_URL: https://authentik.example.com
-            # TODO: set CLIENT_ID as secret
-            OAUTH2_CLIENT_ID: wekan
-        uncomment:
-            - OAUTH2_LOGIN_STYLE
-            - OAUTH2_AUTH_ENDPOINT
-            - OAUTH2_USERINFO_ENDPOINT
-            - OAUTH2_TOKEN_ENDPOINT
-            - OAUTH2_REQUEST_PERMISSIONS
-            - OAUTH2_ID_MAP
-            - OAUTH2_USERNAME_MAP
-            - OAUTH2_FULLNAME_MAP
-            - OAUTH2_EMAIL_MAP
-            - PROPAGATE_OIDC_DATA
-            - OIDC_REDIRECTION_ENABLED
-        shared_secrets:
-            wekan_secret: oauth2_secret
-hedgedoc:
-    authentik:
-        env:
-            CMD_OAUTH2_USER_PROFILE_URL: https://authentik.example.com/application/o/userinfo/
-            CMD_OAUTH2_USER_PROFILE_USERNAME_ATTR: preferred_username
-            CMD_OAUTH2_USER_PROFILE_DISPLAY_NAME_ATTR: name
-            CMD_OAUTH2_USER_PROFILE_EMAIL_ATTR: email
-            CMD_OAUTH2_TOKEN_URL: https://authentik.example.com/application/o/token/
-            CMD_OAUTH2_AUTHORIZATION_URL: https://authentik.example.com/application/o/authorize/
-            # TODO: set CLIENT_ID as secret
-            CMD_OAUTH2_CLIENT_ID: hedgedoc
-            CMD_OAUTH2_PROVIDERNAME: Authentik
-        uncomment: 
-            - compose.oauth.yml
-            - SECRET_OAUTH_KEY_VERSION
-        shared_secrets:
-            hedgedoc_secret: oauth_key
-        dependency: [authentik]
-mila:
-    authentik:
-        env:
-            OIDC_CLIENT_ID: mila
-            OIDC_BASE_URL: https://authentik.example.com/application/o/mila
-            OIDC_REDIRECT_URI: https://mila.example.com/auth/user/oidc/callback
-            OIDC_ADMIN_GROUP_NAME: mv_admin
-            OIDC_GROUPS_CLAIM: groups
-            OIDC_ONLY: "true"
-        uncomment:
-            - compose.oidc.yml
-            - SECRET_OIDC_CLIENT_SECRET_VERSION
-        shared_secrets:
-            mila_secret: oidc_client_secret

examples/config-sets-combine.yml

@@ -0,0 +1,395 @@
+# Integrations, converted from the combine.yml that alakazam used to ship.
+# A set applies as soon as every app of its 'apply-when' is part of an instance.
+# Copy this file into your alakazam root path, next to alaka-versions.yml.
+
+integrate-authentik-nextcloud:
+    apply-when:
+      - authentik
+      - nextcloud
+    authentik:
+        uncomment:
+          - compose.nextcloud.yml
+          - NEXTCLOUD_DOMAIN
+          - SECRET_NEXTCLOUD_ID_VERSION
+          - SECRET_NEXTCLOUD_SECRET_VERSION
+          - nextcloud.png
+    nextcloud:
+        uncomment:
+          - compose.authentik.yml
+          - AUTHENTIK_USER_PREFIX
+          - AUTHENTIK_DOMAIN
+          - SECRET_AUTHENTIK_SECRET_VERSION
+          - SECRET_AUTHENTIK_ID_VERSION
+        initial-hooks:
+          - app set_authentik
+        shared_secrets:
+            authentik:
+                nextcloud_secret: authentik_secret
+                nextcloud_id: authentik_id
+integrate-authentik-wordpress:
+    apply-when:
+      - authentik
+      - wordpress
+    authentik:
+        uncomment:
+          - compose.wordpress.yml
+          - WORDPRESS_DOMAIN
+          - WORDPRESS_GROUP
+          - SECRET_WORDPRESS_ID_VERSION
+          - SECRET_WORDPRESS_SECRET_VERSION
+          - wordpress.png
+    wordpress:
+        uncomment:
+          - compose.authentik.yml
+          - AUTHENTIK_DOMAIN
+          - SECRET_AUTHENTIK_SECRET_VERSION
+          - SECRET_AUTHENTIK_ID_VERSION
+          - LOGIN_TYPE
+        initial-hooks:
+          - app set_authentik
+        shared_secrets:
+            authentik:
+                wordpress_secret: authentik_secret
+                wordpress_id: authentik_id
+integrate-authentik-matrix-synapse:
+    apply-when:
+      - authentik
+      - matrix-synapse
+    authentik:
+        uncomment:
+          - compose.matrix.yml
+          - ELEMENT_DOMAIN
+          - MATRIX_DOMAIN
+          - SECRET_MATRIX_ID_VERSION
+          - SECRET_MATRIX_SECRET_VERSION
+          - matrix.svg
+        secrets:
+            matrix_id: matrix
+    matrix-synapse:
+        env:
+            KEYCLOAK_ID: authentik
+            KEYCLOAK_NAME: sso
+            KEYCLOAK_URL: https://authentik.example.com/application/o/matrix/
+            # TODO: correct client domain?
+            KEYCLOAK_CLIENT_DOMAIN: https://element-web.example.com
+            KEYCLOAK_ALLOW_EXISTING_USERS: "true"
+            # TODO: set CLIENT_ID as secret
+            KEYCLOAK_CLIENT_ID: matrix
+        uncomment:
+          - compose.keycloak.yml
+          - KEYCLOAK_ENABLED
+          - KEYCLOAK_CLIENT_ID
+          - SECRET_KEYCLOAK_CLIENT_SECRET_VERSION
+        shared_secrets:
+            authentik:
+                matrix_secret: keycloak_client_secret
+        dependency: [authentik]
+integrate-authentik-wekan:
+    apply-when:
+      - authentik
+      - wekan
+    authentik:
+        uncomment:
+          - compose.wekan.yml
+          - WEKAN_DOMAIN
+          - SECRET_WEKAN_ID_VERSION
+          - SECRET_WEKAN_SECRET_VERSION
+          - wekan.png
+        secrets:
+            wekan_id: wekan
+    wekan:
+        env:
+            OAUTH2_ENABLED: "true"
+            OAUTH2_SERVER_URL: https://authentik.example.com
+            # TODO: set CLIENT_ID as secret
+            OAUTH2_CLIENT_ID: wekan
+        uncomment:
+          - OAUTH2_LOGIN_STYLE
+          - OAUTH2_AUTH_ENDPOINT
+          - OAUTH2_USERINFO_ENDPOINT
+          - OAUTH2_TOKEN_ENDPOINT
+          - OAUTH2_REQUEST_PERMISSIONS
+          - OAUTH2_ID_MAP
+          - OAUTH2_USERNAME_MAP
+          - OAUTH2_FULLNAME_MAP
+          - OAUTH2_EMAIL_MAP
+          - PROPAGATE_OIDC_DATA
+          - OIDC_REDIRECTION_ENABLED
+        shared_secrets:
+            authentik:
+                wekan_secret: oauth2_secret
+integrate-authentik-vikunja:
+    apply-when:
+      - authentik
+      - vikunja
+    authentik:
+        uncomment:
+          - compose.vikunja.yml
+          - VIKUNJA_DOMAIN
+          - SECRET_VIKUNJA_ID_VERSION
+          - SECRET_VIKUNJA_SECRET_VERSION
+          - vikunja.svg
+        secrets:
+            vikunja_id: vikunja
+    vikunja:
+        env:
+            OAUTH_NAME: authentik
+            OAUTH_URL: https://authentik.example.com/application/o/vikunja/
+            OAUTH_LOGOUT_URL: https://authentik.example.com/application/o/vikunja/end-session/
+            # TODO: set CLIENT_ID as secret
+            OAUTH_CLIENT_ID: vikunja
+        uncomment:
+          - compose.oauth.yml
+          - OAUTH_ENABLED
+          - SECRET_OAUTH_SECRET_VERSION
+        shared_secrets:
+            #vikunja_id: oauth_id
+            authentik:
+                vikunja_secret: oauth_secret
+integrate-authentik-kimai:
+    apply-when:
+      - authentik
+      - kimai
+    authentik:
+        uncomment:
+          - compose.kimai.yml
+          - KIMAI_DOMAIN
+          - SECRET_KIMAI_ID_VERSION
+          - SECRET_KIMAI_SECRET_VERSION
+          - kimai_logo.png
+          - KIMAI_GROUP
+    kimai:
+        uncomment:
+          - SSO_ENABLED
+          - SSO_PROVIDER_URL
+          - SSO_SAML_URL
+          - SSO_LOGOUT_URL
+          - SSO_ADMIN_GROUP_NAME
+        secret_hooks:
+          - insert_authentik_certificate
+        dependency: [authentik]
+integrate-authentik-zammad:
+    apply-when:
+      - authentik
+      - zammad
+    authentik:
+        uncomment:
+          - compose.zammad.yml
+          - ZAMMAD_DOMAIN
+          - zammad.svg
+    zammad:
+        uncomment:
+          - SSO_PROVIDER_DOMAIN
+          - IDP_SSO_TARGET_URL
+          - IDP_SLO_SERVICE_URL
+        initial-hooks:
+          - local enable_authentik_sso
+        dependency: [authentik]
+integrate-authentik-monitoring-ng:
+    apply-when:
+      - authentik
+      - monitoring-ng
+    authentik:
+        uncomment:
+          - compose.monitoring.yml
+          - MONITORING_DOMAIN
+          - SECRET_MONITORING_ID_VERSION
+          - SECRET_MONITORING_SECRET_VERSION
+          - monitoring.svg
+        secrets:
+            monitoring_id: monitoring
+    monitoring-ng:
+        env:
+            GF_SERVER_ROOT_URL: https://monitoring-ng.example.com
+            OIDC_CLIENT_ID: monitoring
+            OIDC_AUTH_URL: https://authentik.example.com/application/o/authorize/
+            OIDC_API_URL: https://authentik.example.com/application/o/userinfo/
+            OIDC_TOKEN_URL: https://authentik.example.com/application/o/token/
+        uncomment:
+          - compose.prometheus.yml
+          - PROMETHEUS_RETENTION_TIME
+          - compose.loki.yml
+          - LOKI_RETENTION_PERIOD
+          - LOKI_STORAGE_FILESYSTEM
+          - compose.grafana.yml
+          - OIDC_ENABLED
+          - SECRET_GRAFANA_ADMIN_PASSWORD_VERSION
+          - SECRET_GRAFANA_OIDC_CLIENT_SECRET_VERSION
+          - SECRET_GRAFANA_SMTP_PASSWORD_VERSION
+        shared_secrets:
+            authentik:
+                monitoring_secret: grafana_oidc_client_secret
+        dependency: [authentik]
+integrate-authentik-outline:
+    apply-when:
+      - authentik
+      - outline
+    authentik:
+        uncomment:
+          - compose.outline.yml
+          - OUTLINE_DOMAIN
+          - SECRET_OUTLINE_ID_VERSION
+          - SECRET_OUTLINE_SECRET_VERSION
+          - outline.png
+        secrets:
+            outline_id: outline
+    outline:
+        env:
+            OIDC_CLIENT_ID: outline
+            OIDC_AUTH_URI: https://authentik.example.com/application/o/authorize/
+            OIDC_TOKEN_URI: https://authentik.example.com/application/o/token/
+            OIDC_USERINFO_URI: https://authentik.example.com/application/o/userinfo/
+            OIDC_DISPLAY_NAME: "Authentik"
+        uncomment:
+          - compose.oidc.yml
+          - OIDC_ENABLED
+          - OIDC_USERNAME_CLAIM
+          - OIDC_SCOPES
+          - SECRET_OIDC_CLIENT_SECRET_VERSION
+        shared_secrets:
+            authentik:
+                outline_secret: oidc_client_secret
+integrate-authentik-rallly:
+    apply-when:
+      - authentik
+      - rallly
+    authentik:
+        uncomment:
+          - compose.rallly.yml
+          - RALLLY_DOMAIN
+          - SECRET_RALLLY_ID_VERSION
+          - SECRET_RALLLY_SECRET_VERSION
+          - rallly.png
+        secrets:
+            rallly_id: rallly
+    rallly:
+        env:
+            OIDC_NAME: "Authentik"
+            OIDC_DISCOVERY_URL: "https://authentik.example.com/application/o/rallly/.well-known/openid-configuration"
+            OIDC_ISSUER_URL: "https://authentik.example.com/application/o/rallly/"
+            OIDC_CLIENT_ID: rallly
+        uncomment:
+          - compose.oidc.yml
+          - SECRET_OIDC_CLIENT_SECRET_VERSION
+          - OIDC_ENABLED
+        shared_secrets:
+            authentik:
+                rallly_secret: oidc_client_secret
+integrate-authentik-hedgedoc:
+    apply-when:
+      - authentik
+      - hedgedoc
+    authentik:
+        uncomment:
+          - compose.hedgedoc.yml
+          - HEDGEDOC_DOMAIN
+          - SECRET_HEDGEDOC_ID_VERSION
+          - SECRET_HEDGEDOC_SECRET_VERSION
+          - hedgedoc.png
+        secrets:
+            hedgedoc_id: hedgedoc
+    hedgedoc:
+        env:
+            CMD_OAUTH2_USER_PROFILE_URL: https://authentik.example.com/application/o/userinfo/
+            CMD_OAUTH2_USER_PROFILE_USERNAME_ATTR: preferred_username
+            CMD_OAUTH2_USER_PROFILE_DISPLAY_NAME_ATTR: name
+            CMD_OAUTH2_USER_PROFILE_EMAIL_ATTR: email
+            CMD_OAUTH2_TOKEN_URL: https://authentik.example.com/application/o/token/
+            CMD_OAUTH2_AUTHORIZATION_URL: https://authentik.example.com/application/o/authorize/
+            # TODO: set CLIENT_ID as secret
+            CMD_OAUTH2_CLIENT_ID: hedgedoc
+            CMD_OAUTH2_PROVIDERNAME: Authentik
+        uncomment:
+          - compose.oauth.yml
+          - SECRET_OAUTH_KEY_VERSION
+        shared_secrets:
+            authentik:
+                hedgedoc_secret: oauth_key
+        dependency: [authentik]
+integrate-authentik-pretix:
+    apply-when:
+      - authentik
+      - pretix
+    authentik:
+        env:
+            APPLICATIONS:
+                Pretix:
+                    url: https://pretix.example.com/control/
+                    group:
+            EXTRA_ICONS:
+                Pretix: ~/.abra/recipes/authentik/icons/pretix.svg
+integrate-authentik-vaultwarden:
+    apply-when:
+      - authentik
+      - vaultwarden
+    authentik:
+        env:
+            APPLICATIONS:
+                Vaultwarden:
+                    url: https://vaultwarden.example.com/
+                    group:
+            EXTRA_ICONS:
+                Vaultwarden: ~/.abra/recipes/authentik/icons/vaultwarden.svg
+integrate-authentik-mila:
+    apply-when:
+      - authentik
+      - mila
+    authentik:
+        uncomment:
+          - compose.mila.yml
+          - MILA_DOMAIN
+          - MILA_GROUP
+          - SECRET_MILA_ID_VERSION
+          - SECRET_MILA_SECRET_VERSION
+          - mila.svg
+        secrets:
+            mila_id: mila
+    mila:
+        env:
+            OIDC_CLIENT_ID: mila
+            OIDC_BASE_URL: https://authentik.example.com/application/o/mila
+            OIDC_REDIRECT_URI: https://mila.example.com/auth/user/oidc/callback
+            OIDC_ADMIN_GROUP_NAME: mv_admin
+            OIDC_GROUPS_CLAIM: groups
+            OIDC_ONLY: "true"
+        uncomment:
+          - compose.oidc.yml
+          - SECRET_OIDC_CLIENT_SECRET_VERSION
+        shared_secrets:
+            authentik:
+                mila_secret: oidc_client_secret
+integrate-nextcloud-onlyoffice:
+    apply-when:
+      - nextcloud
+      - onlyoffice
+    nextcloud:
+        uncomment:
+          - compose.onlyoffice.yml
+          - ONLYOFFICE_URL
+          - SECRET_ONLYOFFICE_JWT_VERSION
+        initial-hooks:
+          - app install_onlyoffice
+    onlyoffice:
+        uncomment:
+          - compose.jwt.yml
+          - SECRET_JWT_SECRET_VERSION
+        shared_secrets:
+            nextcloud:
+                onlyoffice_jwt: jwt_secret
+integrate-collabora-nextcloud:
+    apply-when:
+      - collabora
+      - nextcloud
+    nextcloud:
+        uncomment:
+          - COLLABORA_URL
+        initial-hooks:
+          - app install_collabora
+integrate-matrix-synapse-traefik:
+    apply-when:
+      - matrix-synapse
+      - traefik
+    traefik:
+        uncomment:
+          - compose.matrix.yml
+          - MATRIX_FEDERATION_ENABLED

tests/test_config_sets.py

@@ -0,0 +1,166 @@
+"""Tests for config-sets, in particular the ones that apply by themselves."""
+
+import logging
+import os
+import sys
+
+import pytest
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+import alakazam
+from alakazam import get_config_set_app_configs, merge_instance_configs, read_config_sets
+
+INTEGRATION = {
+    "integrate-authentik-nextcloud": {
+        "apply-when": ["authentik", "nextcloud"],
+        "authentik": {"uncomment": ["compose.nextcloud.yml"]},
+        "nextcloud": {"uncomment": ["compose.authentik.yml"],
+                      "shared_secrets": {"authentik": {"nextcloud_secret": "authentik_secret"}}},
+    }
+}
+
+
+def apply(config_sets, instance_apps, active=None):
+    """The effective configuration, with the automatic sets below the explicit ones."""
+    automatic, explicit = get_config_set_app_configs(active or {}, config_sets, instance_apps)
+    return alakazam.merge_dict(automatic, explicit)
+
+
+class TestApplyWhen:
+    def test_it_applies_when_every_app_is_there(self):
+        result = apply(INTEGRATION, ["authentik", "nextcloud", "traefik"])
+        assert result["authentik"]["uncomment"] == ["compose.nextcloud.yml"]
+        assert result["nextcloud"]["shared_secrets"] == {"authentik": {"nextcloud_secret": "authentik_secret"}}
+
+    def test_it_stays_out_when_one_app_is_missing(self):
+        assert apply(INTEGRATION, ["authentik", "traefik"]) == {}
+
+    def test_apply_when_is_not_passed_on_as_an_app(self):
+        """It is a condition, not configuration, and would end up in an .env otherwise."""
+        assert "apply-when" not in apply(INTEGRATION, ["authentik", "nextcloud"])
+
+    def test_an_instance_can_switch_it_off(self):
+        assert apply(INTEGRATION, ["authentik", "nextcloud"],
+                     active={"integrate-authentik-nextcloud": False}) == {}
+
+    def test_a_one_sided_integration_configures_only_one_app(self):
+        """4 of the 17 pairs in the shipped combine.yml were one-sided."""
+        sets = {"integrate-matrix-synapse-traefik": {
+            "apply-when": ["matrix-synapse", "traefik"],
+            "traefik": {"uncomment": ["compose.matrix.yml"]}}}
+        assert list(apply(sets, ["matrix-synapse", "traefik"])) == ["traefik"]
+        assert apply(sets, ["traefik"]) == {}
+
+
+class TestExplicitSets:
+    SET = {"bbb": {"authentik": {"env": {"A": "b"}}}}
+
+    def test_an_enabled_set_applies(self):
+        assert apply(self.SET, ["authentik"], active={"bbb": True})["authentik"]["env"] == {"A": "b"}
+
+    def test_a_set_that_is_not_enabled_does_not_apply(self):
+        assert apply(self.SET, ["authentik"]) == {}
+
+    def test_an_unknown_name_is_reported(self, caplog):
+        with caplog.at_level(logging.WARNING):
+            apply(self.SET, ["authentik"], active={"typo": True})
+        assert "'typo' is enabled but not defined" in caplog.text
+
+    def test_an_explicit_set_refines_an_automatic_one(self):
+        """Automatic integrations are the base, an explicit set may still override them."""
+        sets = dict(INTEGRATION)
+        sets["override"] = {"authentik": {"env": {"SOURCE": "explicit"}}}
+        result = apply(sets, ["authentik", "nextcloud"], active={"override": True})
+        assert result["authentik"]["env"] == {"SOURCE": "explicit"}
+        assert result["authentik"]["uncomment"] == ["compose.nextcloud.yml"]
+
+
+class TestReadConfigSets:
+    def write(self, root, name, text):
+        (root / name).write_text(text)
+
+    def test_the_base_file_is_read(self, tmp_path):
+        self.write(tmp_path, "config-sets.yml", "bbb:\n    authentik:\n        env:\n            A: b\n")
+        assert read_config_sets(tmp_path)["bbb"]["authentik"]["env"] == {"A": "b"}
+
+    def test_further_files_are_merged(self, tmp_path):
+        self.write(tmp_path, "config-sets.yml", "bbb:\n    authentik: {}\n")
+        self.write(tmp_path, "config-sets-authentik.yml", "integrate:\n    apply-when: [a, b]\n")
+        assert sorted(read_config_sets(tmp_path)) == ["bbb", "integrate"]
+
+    def test_the_base_file_is_merged_first(self, tmp_path):
+        self.write(tmp_path, "config-sets.yml", "bbb:\n    authentik:\n        env:\n            A: base\n")
+        self.write(tmp_path, "config-sets-later.yml", "bbb:\n    authentik:\n        env:\n            A: later\n")
+        assert read_config_sets(tmp_path)["bbb"]["authentik"]["env"]["A"] == "later"
+
+    def test_a_root_without_config_sets(self, tmp_path):
+        assert read_config_sets(tmp_path) == {}
+
+    def test_a_key_defined_twice_is_reported(self, tmp_path, caplog):
+        self.write(tmp_path, "config-sets.yml", "bbb:\n    authentik:\n        env:\n            A: base\n")
+        self.write(tmp_path, "config-sets-later.yml", "bbb:\n    authentik:\n        env:\n            A: later\n")
+        with caplog.at_level(logging.WARNING):
+            read_config_sets(tmp_path)
+        assert "bbb.authentik.env.A" in caplog.text
+
+
+class TestIntegrationReachesTheMergedConfig:
+    """The instance config is what every command reads, the integration has to arrive there."""
+
+    def merge(self, apps):
+        return merge_instance_configs({}, "example.com", {app: None for app in apps}, INTEGRATION)
+
+    def test_both_apps_get_their_side(self):
+        merged = self.merge(["authentik", "nextcloud"])
+        assert merged["authentik"]["uncomment"] == ["compose.nextcloud.yml"]
+        assert merged["nextcloud"]["uncomment"] == ["compose.authentik.yml"]
+
+    def test_a_lone_app_gets_nothing(self):
+        assert "uncomment" not in self.merge(["authentik"])["authentik"]
+
+
+class TestPrecedence:
+    """An integration is the lowest layer, as combine.yml was before it."""
+
+    def test_the_group_configuration_beats_an_integration(self):
+        group = {"nextcloud": {"env": {"SOURCE": "group"}}}
+        sets = {"integrate": {"apply-when": ["authentik", "nextcloud"],
+                              "nextcloud": {"env": {"SOURCE": "integration"}}}}
+        merged = merge_instance_configs(group, "example.com", {"authentik": None, "nextcloud": None}, sets)
+        assert merged["nextcloud"]["env"]["SOURCE"] == "group"
+
+    def test_the_instance_beats_everything(self):
+        group = {"nextcloud": {"env": {"SOURCE": "group"}}}
+        sets = {"integrate": {"apply-when": ["authentik", "nextcloud"],
+                              "nextcloud": {"env": {"SOURCE": "integration"}}}}
+        merged = merge_instance_configs(
+            group, "example.com", {"authentik": None, "nextcloud": {"env": {"SOURCE": "instance"}}}, sets)
+        assert merged["nextcloud"]["env"]["SOURCE"] == "instance"
+
+    def test_an_integration_still_adds_what_nobody_else_sets(self):
+        group = {"nextcloud": {"env": {"OTHER": "group"}}}
+        sets = {"integrate": {"apply-when": ["authentik", "nextcloud"],
+                              "nextcloud": {"env": {"SOURCE": "integration"}}}}
+        merged = merge_instance_configs(group, "example.com", {"authentik": None, "nextcloud": None}, sets)
+        assert merged["nextcloud"]["env"] == {"SOURCE": "integration", "OTHER": "group"}
+
+
+class TestHookOrder:
+    """An integration adds to an app that already configures itself, and must come after it."""
+
+    SETS = {"integrate-authentik-wordpress": {
+        "apply-when": ["authentik", "wordpress"],
+        "wordpress": {"initial-hooks": ["app set_authentik"]}}}
+
+    def test_the_integration_hook_runs_last(self):
+        """set_authentik configures SSO in a WordPress that core_install has to create first."""
+        group = {"wordpress": {"initial-hooks": ["app core_install", "app enable_auto_updates"]}}
+        merged = merge_instance_configs(group, "example.com", {"authentik": None, "wordpress": None}, self.SETS)
+        assert merged["wordpress"]["initial-hooks"] == [
+            "app core_install", "app enable_auto_updates", "app set_authentik"]
+
+    def test_an_instance_hook_also_comes_first(self):
+        instance = {"authentik": None, "wordpress": {"initial-hooks": ["app from_instance"]}}
+        merged = merge_instance_configs({}, "example.com", instance, self.SETS)
+        assert merged["wordpress"]["initial-hooks"] == ["app from_instance", "app set_authentik"]

tests/test_exclude_recipes.py

@@ -0,0 +1,89 @@
+"""Tests for leaving single recipes out of a run."""
+
+import logging
+import os
+import sys
+
+import pytest
+from click.testing import CliRunner
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+import alakazam
+from alakazam import exclude_from_configs
+
+CONFIGS = {
+    "a.org": {"authentik": {"app_domain": "login.a.org"}, "traefik": {"app_domain": "a.org"}},
+    "b.org": {"traefik": {"app_domain": "b.org"}, "nextcloud": {"app_domain": "cloud.b.org"}},
+}
+
+
+class TestExcludeFromConfigs:
+    def test_nothing_excluded_returns_the_configs_unchanged(self):
+        assert exclude_from_configs(CONFIGS, ()) is CONFIGS
+
+    def test_the_recipe_is_gone_from_every_instance(self):
+        result = exclude_from_configs(CONFIGS, ("traefik",))
+        assert sorted(result["a.org"]) == ["authentik"]
+        assert sorted(result["b.org"]) == ["nextcloud"]
+
+    def test_several_recipes(self):
+        result = exclude_from_configs(CONFIGS, ("traefik", "nextcloud"))
+        assert result["b.org"] == {}
+
+    def test_the_original_is_not_modified(self):
+        exclude_from_configs(CONFIGS, ("traefik",))
+        assert "traefik" in CONFIGS["a.org"]
+
+    def test_an_unknown_recipe_is_reported(self, caplog):
+        """A typo would otherwise exclude nothing and look like it worked."""
+        with caplog.at_level(logging.WARNING):
+            exclude_from_configs(CONFIGS, ("treafik",))
+        assert "'treafik' is excluded but not configured" in caplog.text
+
+    def test_a_known_recipe_is_not_reported(self, caplog):
+        with caplog.at_level(logging.WARNING):
+            exclude_from_configs(CONFIGS, ("traefik",))
+        assert caplog.text == ""
+
+
+class TestExcludeReachesTheCommands:
+    """One filter in cli() has to cover every consumer of INSTANCE_CONFIGS."""
+
+    @pytest.fixture
+    def env(self, tmp_path, monkeypatch):
+        root = tmp_path / "root"
+        (root / "group").mkdir(parents=True)
+        (root / "alaka.yml").write_text(
+            "authentik:\n    version: 1.0.0\ntraefik:\n    version: 2.0.0\n")
+        (root / "group" / "example.com.yml").write_text("authentik:\ntraefik:\n")
+        (root / "alakazam.yml").write_text(f"root: {root}\n")
+        monkeypatch.setattr(alakazam, "get_settings_path", lambda: str(root / "alakazam.yml"))
+        monkeypatch.setattr(alakazam, "get_abra_dir", lambda: root / ".abra")
+        monkeypatch.setattr(alakazam, "fetch_recipes", lambda *a, **k: None)
+        monkeypatch.setattr(alakazam, "abra", lambda *a, **k: "")
+        return root
+
+    def invoke(self, root, args):
+        return CliRunner().invoke(alakazam.cli, args + [str(root / "group"), "ls"])
+
+    def test_without_the_flag_both_recipes_are_listed(self, env):
+        result = self.invoke(env, [])
+        assert result.exit_code == 0, result.output
+        assert "authentik" in result.output
+        assert "traefik" in result.output
+
+    def test_the_excluded_recipe_disappears(self, env):
+        result = self.invoke(env, ["-er", "traefik"])
+        assert result.exit_code == 0, result.output
+        assert "authentik" in result.output
+        assert "traefik" not in result.output
+
+    def test_the_long_option_works_too(self, env):
+        result = self.invoke(env, ["--exclude-recipe", "traefik"])
+        assert "traefik" not in result.output
+
+    def test_it_can_be_given_more_than_once(self, env):
+        result = self.invoke(env, ["-er", "traefik", "-er", "authentik"])
+        assert "traefik" not in result.output
+        assert "authentik" not in result.output

tests/test_hide_secrets.py

@@ -0,0 +1,186 @@
+"""Tests for keeping secret values out of the output of a run whose logs are kept."""
+
+import logging
+import os
+import sys
+
+import pytest
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+import alakazam
+from alakazam import generate_all_secrets, insert_secret, is_secret_command
+
+VALUE = "hunter2-do-not-log-me"
+
+
+@pytest.fixture(params=[False, True], ids=["visible", "hidden"])
+def hide(request, monkeypatch):
+    monkeypatch.setattr(alakazam, "HIDE_SECRETS", request.param)
+    return request.param
+
+
+class TestIsSecretCommand:
+    @pytest.mark.parametrize("args", [
+        ("app", "secret", "insert", "login.a.org", "db_password", "v1", VALUE),
+        ("app", "secret", "generate", "-a", "login.a.org"),
+    ])
+    def test_commands_carrying_a_value(self, args):
+        assert is_secret_command(args)
+
+    @pytest.mark.parametrize("args", [
+        ("app", "secret", "ls", "login.a.org"),
+        ("app", "secret", "rm", "login.a.org", "db_password"),
+        ("app", "ls"),
+        ("app", "secret"),
+    ])
+    def test_commands_without_a_value(self, args):
+        assert not is_secret_command(args)
+
+
+class TestGeneratedValues:
+    def install(self, monkeypatch, created=False):
+        def abra(*args, **kwargs):
+            if args[:3] == ("app", "secret", "ls"):
+                return [{"name": "db_password", "created on server": str(created).lower()}]
+            if args[:3] == ("app", "secret", "generate"):
+                return [{"name": "db_password", "value": VALUE}]
+            raise AssertionError(f"unexpected: {args}")
+        monkeypatch.setattr(alakazam, "abra", abra)
+
+    def test_the_name_is_always_reported(self, hide, monkeypatch, capsys):
+        self.install(monkeypatch)
+        generate_all_secrets("login.a.org")
+        assert "db_password" in capsys.readouterr().out
+
+    def test_the_value_follows_the_switch(self, hide, monkeypatch, capsys):
+        self.install(monkeypatch)
+        generate_all_secrets("login.a.org")
+        out = capsys.readouterr().out
+        assert (VALUE in out) is not hide
+        assert ("[hidden]" in out) is hide
+
+
+class TestDebugLog:
+    """-l debug is the level a pipeline reaches for when something breaks."""
+
+    def run_abra(self, monkeypatch, args):
+        class Process:
+            returncode = 0
+            stdout = f'[{{"name":"db_password","value":"{VALUE}"}}]'.encode()
+            stderr = b""
+        monkeypatch.setattr(alakazam.subprocess, "run", lambda cmd, capture_output: Process())
+        return alakazam.abra(*args)
+
+    def test_the_command_line_is_not_logged_when_hidden(self, monkeypatch, caplog):
+        monkeypatch.setattr(alakazam, "HIDE_SECRETS", True)
+        with caplog.at_level(logging.DEBUG):
+            self.run_abra(monkeypatch, ("app", "secret", "insert", "login.a.org", "db_password", "v1", VALUE))
+        assert VALUE not in caplog.text
+
+    def test_the_generated_output_is_not_logged_when_hidden(self, monkeypatch, caplog):
+        monkeypatch.setattr(alakazam, "HIDE_SECRETS", True)
+        with caplog.at_level(logging.DEBUG):
+            self.run_abra(monkeypatch, ("app", "secret", "generate", "-a", "login.a.org"))
+        assert VALUE not in caplog.text
+
+    def test_other_commands_are_still_logged(self, monkeypatch, caplog):
+        """Hiding secrets must not turn the debug log off altogether."""
+        monkeypatch.setattr(alakazam, "HIDE_SECRETS", True)
+        with caplog.at_level(logging.DEBUG):
+            self.run_abra(monkeypatch, ("app", "ls"))
+        assert "run command" in caplog.text
+
+    def test_insert_logs_the_name_but_not_the_value(self, monkeypatch, caplog):
+        monkeypatch.setattr(alakazam, "HIDE_SECRETS", True)
+        monkeypatch.setattr(alakazam, "abra", lambda *a, **k: "")
+        with caplog.at_level(logging.DEBUG):
+            insert_secret("login.a.org", "db_password", VALUE)
+        assert "db_password" in caplog.text
+        assert VALUE not in caplog.text
+
+
+class TestLocalHookOutput:
+    """A recipe may print the secret it created, vaultwarden's admin token does exactly that."""
+
+    TOKEN = "vaultwarden-admin-token-in-plain"
+
+    def install(self, monkeypatch, fail=False):
+        seen = {}
+
+        class Process:
+            returncode = 1 if fail else 0
+            stdout = TestLocalHookOutput.TOKEN.encode()
+            stderr = b""
+
+        def run(cmd, capture_output=False, **kwargs):
+            seen["capture_output"] = capture_output
+            return Process()
+
+        monkeypatch.setattr(alakazam.subprocess, "run", run)
+        def streamed(cmd):
+            seen["streamed"] = True
+            return Process()
+
+        monkeypatch.setattr(alakazam, "run_streamed", streamed)
+        return seen
+
+    def hook(self, strict=False):
+        alakazam.run_secret_hooks("login.a.org", {"secret_hooks": ["insert_admin_token"], "server": "a.org"}, strict=strict)
+
+    def test_the_hook_is_streamed_when_not_hiding(self, monkeypatch):
+        monkeypatch.setattr(alakazam, "HIDE_SECRETS", False)
+        seen = self.install(monkeypatch)
+        self.hook()
+        assert seen.get("streamed")
+
+    def test_the_hook_is_not_streamed_when_hiding(self, monkeypatch, capsys):
+        monkeypatch.setattr(alakazam, "HIDE_SECRETS", True)
+        seen = self.install(monkeypatch)
+        self.hook()
+        assert not seen.get("streamed")
+        assert self.TOKEN not in capsys.readouterr().out
+
+    def test_a_failing_hook_does_not_leak_through_the_error(self, monkeypatch):
+        """The captured output lands in the exception, which is where it would escape."""
+        monkeypatch.setattr(alakazam, "HIDE_SECRETS", True)
+        self.install(monkeypatch, fail=True)
+        with pytest.raises(alakazam.click.ClickException) as excinfo:
+            self.hook(strict=True)
+        assert self.TOKEN not in excinfo.value.message
+        assert "withheld by --hide-secrets" in excinfo.value.message
+
+    def test_a_failing_hook_still_reports_its_output_when_not_hiding(self, monkeypatch):
+        monkeypatch.setattr(alakazam, "HIDE_SECRETS", False)
+        self.install(monkeypatch, fail=True)
+        with pytest.raises(alakazam.click.ClickException) as excinfo:
+            self.hook(strict=True)
+        assert self.TOKEN in excinfo.value.message
+
+
+class TestLocalScriptOutput:
+    def install(self, monkeypatch, tmp_path):
+        script = tmp_path / "hook.sh"
+        script.write_text("#!/bin/sh\necho secret\n")
+        script.chmod(0o755)
+        seen = {}
+
+        class Result:
+            returncode = 0
+
+        monkeypatch.setattr(alakazam, "resolve_path", lambda p, base=None: script)
+        monkeypatch.setattr(alakazam.subprocess, "run",
+                            lambda cmd, **kw: seen.update(kw) or Result())
+        return seen
+
+    def test_the_script_output_is_captured_when_hiding(self, monkeypatch, tmp_path):
+        monkeypatch.setattr(alakazam, "HIDE_SECRETS", True)
+        seen = self.install(monkeypatch, tmp_path)
+        alakazam.run_local_script(["script", "hook.sh"], "login.a.org", "a.org", "a.org")
+        assert seen["capture_output"] is True
+
+    def test_the_script_output_is_passed_through_when_not_hiding(self, monkeypatch, tmp_path):
+        monkeypatch.setattr(alakazam, "HIDE_SECRETS", False)
+        seen = self.install(monkeypatch, tmp_path)
+        alakazam.run_local_script(["script", "hook.sh"], "login.a.org", "a.org", "a.org")
+        assert seen["capture_output"] is False

tests/test_local_scripts.py

@@ -0,0 +1,184 @@
+"""Tests for local script functionality in hooks."""
+
+import logging
+import os
+import stat
+import sys
+
+import click
+import pytest
+from click.testing import CliRunner
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+import alakazam
+from alakazam import (create_secrets, deploy_apps, execute_cmds, get_abra_dir,
+                       resolve_path, run_local_script, run_secret_hooks)
+
+
+class FakeProcess:
+    def __init__(self, returncode):
+        self.returncode = returncode
+
+
+def make_script(tmp_path, name="script.sh", executable=True):
+    """Write a script file under tmp_path, executable by default."""
+    path = tmp_path / name
+    path.write_text("#!/bin/sh\nexit 0\n")
+    if executable:
+        path.chmod(path.stat().st_mode | stat.S_IXUSR)
+    return path
+
+
+class TestResolvePath:
+    def test_an_absolute_path_passes_through_unchanged(self, tmp_path):
+        absolute = tmp_path / "script.sh"
+        assert resolve_path(str(absolute)) == absolute
+
+    def test_a_tilde_path_expands_against_home(self, monkeypatch, tmp_path):
+        monkeypatch.setenv("HOME", str(tmp_path))
+        assert resolve_path("~/script.sh") == tmp_path / "script.sh"
+
+    def test_a_relative_path_resolves_against_the_given_base(self, tmp_path):
+        base = tmp_path / "instance"
+        assert resolve_path("scripts/script.sh", base) == base / "scripts/script.sh"
+
+    def test_a_relative_path_resolves_against_root_path_by_default(self, monkeypatch, tmp_path):
+        monkeypatch.setattr(alakazam, "ROOT_PATH", tmp_path)
+        assert resolve_path("scripts/script.sh") == tmp_path / "scripts/script.sh"
+
+
+class TestRunLocalScript:
+    def test_the_script_receives_its_env_vars_and_arguments(self, monkeypatch, tmp_path):
+        script = make_script(tmp_path)
+        calls = []
+        monkeypatch.setattr(alakazam.subprocess, "run",
+                            lambda cmd, env, **kwargs: calls.append((cmd, env)) or FakeProcess(0))
+        run_local_script(["script", str(script), "arg1"], "login.a.org", "a.org", "a.org")
+        [(cmd, env)] = calls
+        assert cmd == [str(script), "arg1"]
+        assert env["ALAKAZAM_APP_DOMAIN"] == "login.a.org"
+        assert env["ALAKAZAM_APP_SERVER"] == "a.org"
+        assert env["ALAKAZAM_INSTANCE_DOMAIN"] == "a.org"
+
+    def test_a_non_executable_script_is_not_run(self, monkeypatch, tmp_path, caplog):
+        script = make_script(tmp_path, executable=False)
+        monkeypatch.setattr(alakazam.subprocess, "run",
+                            lambda *a, **k: pytest.fail("a non-executable script must not run"))
+        with caplog.at_level(logging.ERROR):
+            run_local_script(["script", str(script)], "login.a.org", "a.org", "a.org")
+        assert "not executable" in caplog.text
+
+    def test_dry_run_does_not_execute_the_script(self, monkeypatch, tmp_path, capsys):
+        script = make_script(tmp_path)
+        monkeypatch.setattr(alakazam.subprocess, "run",
+                            lambda *a, **k: pytest.fail("dry_run must not execute anything"))
+        run_local_script(["script", str(script)], "login.a.org", "a.org", "a.org", dry_run=True)
+        assert "Run local script" in capsys.readouterr().out
+
+    def test_a_non_zero_exit_only_warns_by_default(self, monkeypatch, tmp_path, caplog):
+        script = make_script(tmp_path)
+        monkeypatch.setattr(alakazam.subprocess, "run", lambda *a, **k: FakeProcess(1))
+        with caplog.at_level(logging.WARNING):
+            run_local_script(["script", str(script)], "login.a.org", "a.org", "a.org")
+        assert "exited with code 1" in caplog.text
+
+    def test_a_non_zero_exit_aborts_in_strict_mode(self, monkeypatch, tmp_path):
+        script = make_script(tmp_path)
+        monkeypatch.setattr(alakazam.subprocess, "run", lambda *a, **k: FakeProcess(1))
+        with pytest.raises(click.ClickException) as excinfo:
+            run_local_script(["script", str(script)], "login.a.org", "a.org", "a.org", strict=True)
+        assert "exited with code 1" in excinfo.value.message
+
+    def test_a_non_executable_script_aborts_in_strict_mode(self, monkeypatch, tmp_path):
+        script = make_script(tmp_path, executable=False)
+        monkeypatch.setattr(alakazam.subprocess, "run",
+                            lambda *a, **k: pytest.fail("a non-executable script must not run"))
+        with pytest.raises(click.ClickException) as excinfo:
+            run_local_script(["script", str(script)], "login.a.org", "a.org", "a.org", strict=True)
+        assert "not executable" in excinfo.value.message
+
+
+class TestScriptHookDispatch:
+    def test_a_script_secret_hook_dispatches_to_run_local_script(self, monkeypatch, tmp_path):
+        script = make_script(tmp_path)
+        calls = []
+        monkeypatch.setattr(alakazam, "run_local_script", lambda *a, **k: calls.append((a, k)))
+        monkeypatch.setattr(alakazam, "abra", lambda *a, **k: pytest.fail("should not call abra"))
+        run_secret_hooks("login.a.org", {"secret_hooks": [f"script {script}"], "server": "a.org"}, "a.org")
+        [(args, kwargs)] = calls
+        assert args == (["script", str(script)], "login.a.org", "a.org", "a.org")
+
+    def test_a_normal_secret_hook_goes_through_abra(self, monkeypatch):
+        calls = []
+        monkeypatch.setattr(alakazam, "abra", lambda *a, **k: calls.append((a, k)) or "")
+        run_secret_hooks("login.a.org", {"secret_hooks": ["insert_cert"], "server": "a.org"})
+        assert calls, "a non-script hook must still run through abra"
+
+    def test_a_script_command_dispatches_to_run_local_script(self, monkeypatch, tmp_path):
+        script = make_script(tmp_path)
+        calls = []
+        monkeypatch.setattr(alakazam, "run_local_script", lambda *a, **k: calls.append((a, k)))
+        monkeypatch.setattr(alakazam, "abra", lambda *a, **k: pytest.fail("should not call abra"))
+        execute_cmds({"app_domain": "login.a.org", "server": "a.org",
+                       "initial-hooks": [f"script {script} arg1"]}, initial=True)
+        [(args, kwargs)] = calls
+        assert args == (["script", str(script), "arg1"], "login.a.org", "a.org", "", False, False)
+
+    def test_a_normal_command_goes_through_abra(self, monkeypatch):
+        calls = []
+        monkeypatch.setattr(alakazam, "abra", lambda *a, **k: calls.append((a, k)) or "")
+        execute_cmds({"app_domain": "login.a.org", "server": "a.org",
+                       "initial-hooks": ["app set_default_quota"]}, initial=True)
+        assert calls, "a non-script command must still run through abra"
+
+    def test_a_script_token_with_a_missing_file_falls_back_to_abra(self, monkeypatch, tmp_path):
+        """A 'script' hook whose path does not exist is not a script hook at all, just an abra
+        command whose container happens to be named 'script'."""
+        missing = tmp_path / "does-not-exist.sh"
+        calls = []
+        monkeypatch.setattr(alakazam, "run_local_script",
+                            lambda *a, **k: pytest.fail("should not run a missing script"))
+        monkeypatch.setattr(alakazam, "abra", lambda *a, **k: calls.append((a, k)) or "")
+        execute_cmds({"app_domain": "login.a.org", "server": "a.org",
+                       "initial-hooks": [f"script {missing}"]}, initial=True)
+        assert calls, "a missing script path must fall back to a normal abra command"
+
+
+class TestInstanceDomainThreading:
+    """create_secrets(), deploy_apps() and the 'cmd' CLI command each thread instance_domain
+    into run_secret_hooks()/execute_cmds() for script hooks."""
+
+    @pytest.fixture
+    def app_config(self, monkeypatch):
+        """A single 'authentik' app on instance 'a.org', shared by tests below."""
+        config = {"app_domain": "login.a.org", "server": "a.org"}
+        monkeypatch.setattr(alakazam, "INSTANCE_CONFIGS", {"a.org": {"authentik": config}})
+        return config
+
+    def test_create_secrets_threads_the_instance_domain(self, monkeypatch, app_config):
+        calls = []
+        monkeypatch.setattr(alakazam, "run_secret_hooks", lambda *a, **k: calls.append((a, k)))
+        create_secrets(recipes=(), variants={"secret-hooks"})
+        [(args, kwargs)] = calls
+        assert args == ("login.a.org", app_config, "a.org")
+
+    def test_deploy_apps_threads_the_instance_domain(self, monkeypatch, app_config):
+        app_config["version"] = "1.0.0"
+        monkeypatch.setattr(alakazam, "abra", lambda *a, **k: "")
+        calls = []
+        monkeypatch.setattr(alakazam, "execute_cmds", lambda app_config, **k: calls.append(k))
+        deploy_apps({"a.org": [["authentik", "login.a.org"]]}, execute_hooks=True)
+        [kwargs] = calls
+        assert kwargs["deploy"] is True
+        assert kwargs["instance_domain"] == "a.org"
+
+    def test_cmd_threads_the_instance_domain(self, monkeypatch, app_config):
+        monkeypatch.setattr(alakazam, "get_deployed_apps", lambda apps: {"login.a.org": "1.0.0"})
+        calls = []
+        monkeypatch.setattr(alakazam, "execute_cmds", lambda app_config, **k: calls.append(k))
+        result = CliRunner().invoke(alakazam.cmd, ["-i"], standalone_mode=False)
+        assert result.exception is None
+        [kwargs] = calls
+        assert kwargs["initial"] is True
+        assert kwargs["instance_domain"] == "a.org"

tests/test_preflight.py

@@ -109,9 +109,11 @@ class TestRelevantConfigPaths:
         assert config_root / "group" / "alaka-versions.yml" in paths
         assert config_root / "group" / "example.com.yml" in paths
 
-    def test_covers_the_connection_configuration(self, config_root):
+    def test_covers_every_config_set_file(self, config_root):
+        (config_root / "config-sets-authentik.yml").write_text("integrate:\n    apply-when: [a]\n")
         paths = get_relevant_config_paths(config_root, config_root / "group", [])
-        assert alakazam.Path(alakazam.COMBINE_PATH) in paths
+        assert config_root / "config-sets.yml" in paths
+        assert config_root / "config-sets-authentik.yml" in paths
 
     def test_a_single_instance_file_pulls_in_its_ancestors(self, config_root):
         paths = get_relevant_config_paths(config_root, config_root / "group" / "example.com.yml", [])

tests/test_purge.py

@@ -9,7 +9,7 @@ import pytest
 sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
 import alakazam
-from alakazam import purge_apps
+from alakazam import purge_app_secrets, purge_apps
 
 INSTANCE_APPS = {"a.org": [["authentik", "login.a.org"]]}
 CONFIG = {"a.org": {"authentik": {"app_domain": "login.a.org", "server": "a.org"}}}
@@ -50,3 +50,43 @@ class TestPurgeApps:
         monkeypatch.setattr(alakazam, "abra", lambda *a, **k: calls.append(a) or "")
         purge_apps(INSTANCE_APPS)
         assert calls == []
+
+
+class TestPurgeAppSecrets:
+    @pytest.fixture
+    def config(self, monkeypatch):
+        monkeypatch.setattr(alakazam, "INSTANCE_CONFIGS", CONFIG)
+
+    def install(self, monkeypatch, error=None):
+        calls = []
+
+        def abra(*args, **kwargs):
+            calls.append(args)
+            if error:
+                raise RuntimeError(error)
+            return ""
+
+        monkeypatch.setattr(alakazam, "abra", abra)
+        return calls
+
+    def test_all_secrets_of_a_recipe_are_removed(self, config, monkeypatch, capsys):
+        calls = self.install(monkeypatch)
+        purge_app_secrets({"authentik": []})
+        assert calls == [("app", "secret", "rm", "-a", "login.a.org")]
+        assert "purged" in capsys.readouterr().out
+
+    def test_an_app_without_secrets_is_not_a_failure(self, config, monkeypatch, capsys):
+        """abra exits non-zero when it found nothing to remove, which says the job is already done."""
+        self.install(monkeypatch, error="FATA no secrets to remove?")
+        purge_app_secrets({"authentik": []})
+        assert "has no secrets on its server" in capsys.readouterr().out
+
+    def test_a_real_failure_still_propagates(self, config, monkeypatch):
+        self.install(monkeypatch, error="FATA error during connect: no route to host")
+        with pytest.raises(RuntimeError):
+            purge_app_secrets({"authentik": []})
+
+    def test_a_named_secret_that_is_absent_is_skipped(self, config, monkeypatch, capsys):
+        self.install(monkeypatch, error="FATA email_pass doesn't exist on server?")
+        purge_app_secrets({"authentik": ["email_pass"]})
+        assert "is not stored on the server" in capsys.readouterr().out

tests/test_strict_hooks.py

@@ -1,6 +1,7 @@
 """Tests for the strict hook mode that lets a scratch build fail on a broken hook."""
 
 import os
+import stat
 import sys
 
 import click
@@ -12,6 +13,14 @@ import alakazam
 from alakazam import execute_cmds, run_secret_hooks
 
 
+def failing_script(tmp_path):
+    """A tiny local script that exits non-zero, for testing 'script' hook failure handling."""
+    script = tmp_path / "fail.sh"
+    script.write_text("#!/bin/sh\nexit 1\n")
+    script.chmod(script.stat().st_mode | stat.S_IXUSR)
+    return script
+
+
 class TestStrictHooks:
     def test_a_secret_hook_failure_is_tolerated_by_default(self, monkeypatch):
         """'secrets' has always continued past a failing hook, that must not change."""
@@ -30,12 +39,36 @@ class TestStrictHooks:
 
     def test_a_command_failure_is_tolerated_by_default(self, monkeypatch):
         monkeypatch.setattr(alakazam, "abra", lambda *a, **k: "")
-        execute_cmds({"app_domain": "login.a.org", "initial-hooks": ["app init"]}, initial=True)
+        execute_cmds({"app_domain": "login.a.org", "server": "a.org", "initial-hooks": ["app init"]}, initial=True)
 
     def test_a_command_failure_aborts_in_strict_mode(self, monkeypatch):
         def failing(*args, ignore_error=False, **kwargs):
             raise RuntimeError("FATA container not found")
         monkeypatch.setattr(alakazam, "abra", failing)
         with pytest.raises(click.ClickException) as excinfo:
-            execute_cmds({"app_domain": "login.a.org", "initial-hooks": ["app init"]}, initial=True, strict=True)
+            execute_cmds({"app_domain": "login.a.org", "server": "a.org", "initial-hooks": ["app init"]}, initial=True, strict=True)
         assert "command 'app init' failed" in excinfo.value.message
+
+    def test_a_script_secret_hook_failure_is_tolerated_by_default(self, tmp_path):
+        """A failing local script must be tolerated by default too, same as a failing abra.sh hook."""
+        script = failing_script(tmp_path)
+        run_secret_hooks("login.a.org", {"secret_hooks": [f"script {script}"], "server": "a.org"})
+
+    def test_a_script_secret_hook_failure_aborts_in_strict_mode(self, tmp_path):
+        script = failing_script(tmp_path)
+        with pytest.raises(click.ClickException) as excinfo:
+            run_secret_hooks("login.a.org", {"secret_hooks": [f"script {script}"], "server": "a.org"}, strict=True)
+        assert str(script) in excinfo.value.message
+        assert "failed" in excinfo.value.message
+
+    def test_a_script_command_failure_is_tolerated_by_default(self, tmp_path):
+        script = failing_script(tmp_path)
+        execute_cmds({"app_domain": "login.a.org", "server": "a.org", "initial-hooks": [f"script {script}"]}, initial=True)
+
+    def test_a_script_command_failure_aborts_in_strict_mode(self, tmp_path):
+        script = failing_script(tmp_path)
+        with pytest.raises(click.ClickException) as excinfo:
+            execute_cmds({"app_domain": "login.a.org", "server": "a.org", "initial-hooks": [f"script {script}"]},
+                         initial=True, strict=True)
+        assert str(script) in excinfo.value.message
+        assert "failed" in excinfo.value.message

tests/test_upgrade_secrets.py

@@ -0,0 +1,85 @@
+"""Tests for filling in secrets a recipe added, before the upgrade that would fail on them."""
+
+import os
+import sys
+
+import pytest
+from click.testing import CliRunner
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+import alakazam
+from alakazam import get_missing_secrets
+
+CONFIG = {"a.org": {"nextcloud": {"app_domain": "cloud.a.org", "server": "a.org", "version": "2.0.0"}}}
+
+
+class TestGetMissingSecrets:
+    def install(self, monkeypatch, stored):
+        monkeypatch.setattr(alakazam, "abra", lambda *a, **k: stored)
+
+    def test_a_complete_app_reports_nothing(self, monkeypatch):
+        self.install(monkeypatch, [{"name": "db_password", "created on server": "true"}])
+        assert get_missing_secrets("cloud.a.org") == []
+
+    def test_the_missing_names_are_returned(self, monkeypatch):
+        self.install(monkeypatch, [
+            {"name": "db_password", "created on server": "true"},
+            {"name": "oidc_secret", "created on server": "false"},
+        ])
+        assert get_missing_secrets("cloud.a.org") == ["oidc_secret"]
+
+    def test_an_app_without_any_secrets(self, monkeypatch):
+        """abra answers with an empty document rather than a list when nothing is stored."""
+        self.install(monkeypatch, {})
+        assert get_missing_secrets("cloud.a.org") == []
+
+
+class TestUpgradeFillsThemIn:
+    @pytest.fixture
+    def steps(self, monkeypatch):
+        steps = []
+        monkeypatch.setattr(alakazam, "INSTANCE_CONFIGS", CONFIG)
+        monkeypatch.setattr(alakazam, "sleep", lambda s: None)
+        monkeypatch.setattr(alakazam, "get_apps_by_deployment",
+                            lambda recipes, deployed=True: {"a.org": [["nextcloud", "cloud.a.org", "1.0.0"]]})
+        monkeypatch.setattr(alakazam, "create_secrets",
+                            lambda recipes, **kw: steps.append(f"secrets:{recipes[0]}"))
+        return steps
+
+    def install_abra(self, monkeypatch, steps, missing):
+        def abra(*args, **kwargs):
+            if args[:3] == ("app", "secret", "ls"):
+                return [{"name": n, "created on server": str(n not in missing).lower()}
+                        for n in ("db_password", "oidc_secret")]
+            steps.append(" ".join(a for a in args if a))
+            return ""
+        monkeypatch.setattr(alakazam, "abra", abra)
+
+    def invoke(self):
+        return CliRunner().invoke(alakazam.upgrade, ["-n"], standalone_mode=False)
+
+    def test_a_missing_secret_is_created_before_the_upgrade(self, monkeypatch, steps):
+        self.install_abra(monkeypatch, steps, missing={"oidc_secret"})
+        result = self.invoke()
+        assert result.exit_code == 0, result.exception
+        # the planning phase fetches release notes with the same command plus -r, skip that one
+        upgrade = next(i for i, s in enumerate(steps)
+                       if s.startswith("app upgrade") and " -r " not in s)
+        assert steps.index("secrets:nextcloud") < upgrade
+
+    def test_the_missing_name_is_reported(self, monkeypatch, steps):
+        """Silently repairing secrets would hide which one the recipe added."""
+        self.install_abra(monkeypatch, steps, missing={"oidc_secret"})
+        assert "missing 1 secret(s): oidc_secret" in self.invoke().output
+
+    def test_a_complete_app_does_not_run_the_secrets(self, monkeypatch, steps):
+        self.install_abra(monkeypatch, steps, missing=set())
+        self.invoke()
+        assert not any(s.startswith("secrets:") for s in steps)
+
+    def test_a_dry_run_creates_nothing(self, monkeypatch, steps):
+        """The check sits behind the confirmation, so --dry-run must not reach it."""
+        self.install_abra(monkeypatch, steps, missing={"oidc_secret"})
+        CliRunner().invoke(alakazam.upgrade, ["-n", "--dry-run"], standalone_mode=False)
+        assert not any(s.startswith("secrets:") for s in steps)

tests/test_versions_pinned.py

@@ -0,0 +1,79 @@
+"""Tests for a version that pins a commit while naming the release it is based on."""
+
+import logging
+import os
+import sys
+
+import click
+import pytest
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+import alakazam
+from alakazam import classify_version, merge_instance_configs, resolve_version
+
+
+class TestResolveVersion:
+    def test_a_pinned_commit_wins_over_the_release(self):
+        assert resolve_version("12.0.2+2026.5.2@be9ebb3", "nextcloud") == "be9ebb3"
+
+    @pytest.mark.parametrize("version", ["1.2.3", "12.0.2+2026.5.2", "chaos", "latest", "be9ebb3"])
+    def test_a_version_without_a_commit_is_untouched(self, version):
+        assert resolve_version(version, "nextcloud") == version
+
+    @pytest.mark.parametrize("version", [None, 1.2, True])
+    def test_non_strings_are_untouched(self, version):
+        """The configuration is user supplied and does not have to hold a string."""
+        assert resolve_version(version, "nextcloud") is version
+
+    def test_a_trailing_at_is_rejected(self):
+        """Silently deploying the release instead of the intended commit would be worse."""
+        with pytest.raises(click.ClickException) as excinfo:
+            resolve_version("12.0.2@", "nextcloud")
+        assert "without a commit" in excinfo.value.message
+        assert "nextcloud" in excinfo.value.message
+
+    def test_the_release_is_kept_in_the_debug_log(self, caplog):
+        """It is the only place the base release survives, and CI failures are read there."""
+        with caplog.at_level(logging.DEBUG):
+            resolve_version("12.0.2+2026.5.2@be9ebb3", "nextcloud")
+        assert "based on 12.0.2+2026.5.2" in caplog.text
+
+    def test_the_last_at_separates(self):
+        assert resolve_version("a@b@be9ebb3", "nextcloud") == "be9ebb3"
+
+
+class TestClassification:
+    def test_the_resolved_commit_classifies_as_a_hash(self):
+        """upgrade branches on this, a release would take the wrong abra command."""
+        assert classify_version(resolve_version("12.0.2+2026.5.2@be9ebb3", "nextcloud")) == "hash"
+
+    def test_an_unpinned_release_still_classifies_as_a_version(self):
+        assert classify_version(resolve_version("12.0.2+2026.5.2", "nextcloud")) == "version"
+
+
+class TestMergedConfig:
+    """The resolution has to happen once, where the configuration is built."""
+
+    def merge(self, version):
+        return merge_instance_configs(
+            {}, "example.com", {"nextcloud": {"version": version}}, {})
+
+    def test_the_merged_config_carries_the_commit(self):
+        assert self.merge("12.0.2+2026.5.2@be9ebb3")["nextcloud"]["version"] == "be9ebb3"
+
+    def test_a_plain_version_survives_the_merge(self):
+        assert self.merge("12.0.2")["nextcloud"]["version"] == "12.0.2"
+
+    def test_an_app_without_a_version(self):
+        assert "version" not in self.merge(None)["nextcloud"] or \
+            self.merge(None)["nextcloud"]["version"] is None
+
+    def test_a_templated_version_is_resolved_after_substitution(self):
+        """Jinja runs first, so a version coming from GLOBALS is pinned as well."""
+        merged = merge_instance_configs(
+            {"GLOBALS": {"pin": "12.0.2+2026.5.2@be9ebb3"}},
+            "example.com",
+            {"nextcloud": {"version": "{{pin}}"}},
+            {})
+        assert merged["nextcloud"]["version"] == "be9ebb3"