build: add argon2 for recipes that hash secrets locally

Reviewed-on: #13

Dockerfile

@@ -26,8 +26,10 @@ RUN python -m venv /opt/alakazam \
   && /opt/alakazam/bin/pip install --no-cache-dir -r /tmp/requirements.txt
 
 FROM python:3.11-slim
+# git: alakazam syncs the recipe repos itself. openssh-client: abra reaches the servers over
+# ssh. argon2: recipes hash secrets locally in their abra.sh, vaultwarden's admin token does
 RUN apt-get update \
-  && apt-get install -y --no-install-recommends git openssh-client make \
+  && apt-get install -y --no-install-recommends git openssh-client make argon2 \
   && rm -rf /var/lib/apt/lists/* \
   && git config --global --add safe.directory '*'
 

README.md

@@ -185,6 +185,14 @@ Without `-s` all secrets of the recipes selected by `-r` are purged.
 alakazam example.com.yml ps --wait --timeout 300
 ```
 
+### Excluding Recipes
+
+`-er`/`--exclude-recipe` leaves single recipes out of a run entirely:
+
+```
+alakazam -er traefik example.com.yml clean-deploy -n
+```
+
 ### Run CMDs
 
 Escaping can be akward:
@@ -250,14 +258,14 @@ For each app/recipe the following `<app_configurations>` can be used:
     - it matches against parts of the line (i.E. `compose.smtp.yml`)
     - this is useful for env variables that are used multiple times like `COMPOSE_FILE`
 - **`env`**: Sets values for environment variables.
-- **`*-hooks`**: Specifies `abra.sh` commands to run at specific stages.
+- **`*-hooks`**: Specifies `abra.sh` commands or local scripts to run at specific stages.
     - **`initial-hooks`**: commands for initialisation
     - **`deploy-hooks`**: commands that should be run after each deployment
     - **`upgrade-hooks`**: commands that should be run after each upgrade
 - **`readiness-hooks`**: Commands that decide whether an app is usable yet, repeated until one succeeds. See [Readiness Hooks](#readiness-hooks).
 - **`dependency`**: Names the apps that have to be set up before this one. See [Dependencies](#dependencies).
 - **`secrets`**: Inserts specific values (i.E. smtp passwords) into secrets; future updates will support encrypted file usage.
-- **`secret-hooks`**: Run `abra.sh` commands locally for secrets that need to be generated.
+- **`secret-hooks`**: Run `abra.sh` commands locally or local scripts for secrets that need to be generated.
 - **`subdomain`**: Specifies the subdomain scheme for individual recipes and apps. (not available in `combine.yml`/`alaconnect.yml`)
     - i.e. `cloud.example.com` for nextcloud
 - **`version`**: Controls the recipe version to deploy; if unspecified, the latest version is used. (not available in `combine.yml`/`alaconnect.yml`)
@@ -277,6 +285,7 @@ nextcloud:
 ```
 
 Alakazam builds a directed graph from those entries and processes the apps in topological order. For the configuration, the secrets and the deployment alike. Apps unrelated by a dependency keep their configuration order. [`setup`](#setup-and-clean-deploy) additionally groups them into levels: everything without a dependency forms the first level, apps depending only on those the second, and so on.
+
 ### Readiness Hooks
 
 An app can be deployed and healthy while still not being usable — authentik accepts connections long before it has applied its blueprints. A readiness hook is an `abra.sh` command of the recipe that answers that question and exits zero once the app is ready:
@@ -294,6 +303,24 @@ authentik:
 
 Readiness hooks run as part of [`setup`](#setup-and-clean-deploy) only, right after the app they belong to is deployed and before anything that depends on it. `deploy` does not run them, so it keeps returning as soon as the deployment is through.
 
+### \*-Hooks Command Formats
+
+**Abra command** — runs an abra.sh command inside a container:
+
+```yaml
+initial-hooks:
+  - app set_default_quota
+```
+
+**Local script** — runs a script on the local machine (for custom actions that aren't generic enough to be implemented as abra.sh commands):
+
+```yaml
+initial-hooks:
+  - script ./scripts/script.sh arg1
+```
+
+Relative paths resolve from the `root` path. The script receives `ALAKAZAM_APP_DOMAIN`, `ALAKAZAM_APP_SERVER`, and `ALAKAZAM_INSTANCE_DOMAIN` as environment variables.
+
 ### Configuration Structure
 
 Configuration can be simplified into a single `example.com.yml` or expanded into multiple layered `alaka.yml`/`alaka-*.yml` files for complex deployments. This allows for easy maintenance of multiple instances or groups.

alakazam.py

@@ -57,6 +57,7 @@ ALL_CONFIGS = {}
 SETTINGS = {}
 SETTINGS_PATH = ""  # path to the alakazam settings file
 GROUP_PATH = None  # path this run was invoked with, either an instance file or a group directory
+HIDE_SECRETS = False  # keep secret values out of the output, for runs whose logs are archived
 ABRA_DIR = None  # path to the abra data directory
 FETCH_MAX_AGE = 600 # seconds; recipe repos are refetched at most once per hour
 ABRA_RETRIES = 6 # number of attempts per abra command that failed to reach the server
@@ -75,8 +76,11 @@ CONNECTION_ERRORS = (
 )
 # marker in the abra output that indicates a secret which is not stored on the server
 MISSING_SECRET_ERROR = "doesn't exist on server"
-# marker in the abra output for a generate run that had nothing left to do, which abra exits 1 on
+# abra subcommands that take a secret as an argument or return one
+SECRET_COMMANDS = ("insert", "generate")
+# markers in the abra output for a secret run that had nothing left to do, which abra exits 1 on
 NO_SECRETS_GENERATED = "no secrets generated"
+NO_SECRETS_TO_REMOVE = "no secrets to remove"
 # container statuses that have not settled yet, matched against the abra output
 PENDING_STATUS_RE = re.compile(r".*(starting|unknown|unhealthy).*")
 # marker in a service name for a container that runs once and exits instead of becoming healthy
@@ -93,6 +97,7 @@ SECRET_VARIANTS = ("conf", "secret-hooks", "exchange", "generate")
 GROUP_CONFIG_RE = re.compile(r'^alaka(-.*)?\.ya?ml$')
 # instance configuration files, named '<domain>.yml' after the instance they configure
 INSTANCE_CONFIG_RE = re.compile(r'^(?:[A-Za-z0-9](?:[A-Za-z0-9\-]{0,61}[A-Za-z0-9])?\.)+[A-Za-z]{2,6}(?:\.yaml|\.yml)$')
+ROOT_PATH = None  # resolved root path from alakazam.yml
 
 
 class MySafeConstructor(SafeConstructor):
@@ -546,6 +551,32 @@ def get_merged_instance_configs(config_path: Path, group_configs: Dict[str, Any]
     return instances
 
 
+def exclude_from_configs(configs: Dict[str, Dict[str, Any]], recipes: Tuple[str]) -> Dict[str, Dict[str, Any]]:
+    """
+    Removes the given recipes from every instance configuration.
+
+    An excluded recipe is not merely left undeployed, it is invisible for the whole run: it is
+    neither listed, nor configured, nor purged, nor waited for. That is what keeps a rebuild from
+    touching an app whose state has to survive it, such as the certificates of a reverse proxy.
+
+    Args:
+        configs (dict): Instances as keys and their app configurations as values
+        recipes (tuple): Recipe names to leave out, all recipes are kept if empty
+
+    Returns:
+        dict: The configurations without those recipes
+    """
+    if not recipes:
+        return configs
+    for recipe in recipes:
+        if not any(recipe in apps for apps in configs.values()):
+            logging.warning(f"'{recipe}' is excluded but not configured for this path")
+    return {
+        instance: {name: config for name, config in apps.items() if name not in recipes}
+        for instance, apps in configs.items()
+    }
+
+
 def merge_connection_configs(configs: Dict[str, Any]) -> Dict[str, Any]:
     """
     Merge connection configurations from the 'combine.yml' to extend instance configurations with inter-app secrets and settings.
@@ -582,6 +613,19 @@ def extend_shared_secrets(connection_config: Dict[str, Any]) -> None:
                 target_conf['shared_secrets'] = {source_app: shared_secrets}
 
 
+def is_secret_command(args: Tuple[str, ...]) -> bool:
+    """
+    Checks whether an abra command carries a secret value in its arguments or in its output.
+
+    Args:
+        args (tuple): The arguments passed to abra()
+
+    Returns:
+        bool: True for the commands that insert or generate a secret
+    """
+    return args[:2] == ("app", "secret") and len(args) > 2 and args[2] in SECRET_COMMANDS
+
+
 def is_connection_error(output: str) -> bool:
     """
     Checks whether the output of an abra command indicates that the server could not be reached.
@@ -641,7 +685,11 @@ def abra(*args: str, machine_output: bool = False, ignore_error: bool = False, s
     command = [arg for arg in command if arg]
     if machine_output:
         command.append("-m")
-    logging.debug(f"run command: {' '.join(command)}")
+    # the command line of 'app secret insert' ends in the secret itself, and the output of
+    # 'app secret generate' contains the generated values
+    quiet = HIDE_SECRETS and is_secret_command(args)
+    if not quiet:
+        logging.debug(f"run command: {' '.join(command)}")
     for attempt in range(1, ABRA_RETRIES + 1):
         process = run_streamed(command) if stream else subprocess.run(command, capture_output=True)
         if not process.returncode or attempt == ABRA_RETRIES:
@@ -655,7 +703,7 @@ def abra(*args: str, machine_output: bool = False, ignore_error: bool = False, s
         sleep(delay)
     if process.stderr and ignore_error:
         logging.warning(process.stderr.decode())
-    if process.stdout and not stream:
+    if process.stdout and not stream and not quiet:
         logging.debug(process.stdout.decode())
     if process.returncode and not ignore_error:
         #breakpoint()
@@ -856,7 +904,25 @@ def generate_all_secrets(domain: str) -> None:
             return
         print(f"secrets for {domain} generated")
         for gen_sec in generated_secrets:
-            print(f"\t {gen_sec['name']}: {gen_sec['value']}")
+            value = "[hidden]" if HIDE_SECRETS else gen_sec['value']
+            print(f"\t {gen_sec['name']}: {value}")
+
+
+def resolve_path(path_str: str, base: Optional[Path] = None) -> Path:
+    """
+    Resolve a path string to an absolute Path, expanding ~ and resolving relative paths against a base directory.
+
+    Args:
+        path_str (str): The path string to resolve. May be absolute, relative, or start with ~.
+        base (Path): The base directory for resolving relative paths. Defaults to ROOT_PATH if not provided.
+
+    Returns:
+        Path: The resolved absolute path.
+    """
+    p = Path(path_str).expanduser()
+    if p.is_absolute():
+        return p
+    return ((base or ROOT_PATH) / p).absolute()
 
 
 def get_abra_dir() -> Path:
@@ -881,8 +947,7 @@ def get_abra_dir() -> Path:
             if config_file.exists():
                 abra_config = read_config(str(config_file))
                 if abra_dir := abra_config.get("abraDir"):
-                    p = Path(abra_dir)
-                    return (current / p).resolve() if not p.is_absolute() else p
+                    return resolve_path(abra_dir, current)
         if current == home:
             break
         current = current.parent
@@ -1154,13 +1219,50 @@ def insert_secrets_from_conf(domain: str, app_config: Dict[str, Any]) -> None:
                 insert_secret(domain, secret_name, secret)
 
 
-def run_secret_hooks(domain: str, app_config: Dict[str, Any], strict: bool = False) -> None:
+def run_local_script(tokens: List[str], app_domain: str, server: str, instance_domain: str, dry_run: bool = False, strict: bool = False) -> None:
     """
-    Run local abra.sh commands to generate secrets.
+    Run a local script hook. Relative paths are resolved against ROOT_PATH for execution. Logs an error and returns early if the script is not executable.
+
+    Args:
+        tokens (list): The hook entry split on whitespace, with tokens[0] confirmed to be 'script', tokens[1] the script path, and tokens[2:] positional arguments.
+        app_domain (str): The app domain, passed as ALAKAZAM_APP_DOMAIN to the script environment.
+        server (str): The server name, passed as ALAKAZAM_APP_SERVER to the script environment.
+        instance_domain (str): The instance domain, passed as ALAKAZAM_INSTANCE_DOMAIN to the script environment.
+        dry_run (bool): If True, prints the command but does not execute it.
+        strict (bool): Abort on a non-executable script or a non-zero exit code instead of only logging it.
+
+    Raises:
+        click.ClickException: If the script is not executable or exits non-zero and strict is set
+    """
+    script_path = resolve_path(tokens[1])
+    args = tokens[2:]
+    env = {**os.environ, "ALAKAZAM_APP_DOMAIN": app_domain, "ALAKAZAM_APP_SERVER": server, "ALAKAZAM_INSTANCE_DOMAIN": instance_domain}
+    cmd_display = " ".join(tokens[1:])
+    print(f"Run local script '{cmd_display}' for {app_domain}")
+    if not os.access(script_path, os.X_OK):
+        message = f"Script is not executable (run: chmod +x {script_path})"
+        if strict:
+            raise click.ClickException(f"script '{cmd_display}' failed for {app_domain}: {message}")
+        logging.error(message)
+        return
+    if dry_run:
+        return
+    result = subprocess.run([str(script_path)] + args, env=env)
+    if result.returncode != 0:
+        message = f"Script '{cmd_display}' exited with code {result.returncode}"
+        if strict:
+            raise click.ClickException(f"script '{cmd_display}' failed for {app_domain}: {message}")
+        logging.warning(message)
+
+
+def run_secret_hooks(domain: str, app_config: Dict[str, Any], instance_domain: str = "", strict: bool = False) -> None:
+    """
+    Run local abra.sh commands or local scripts to generate secrets.
 
     Args:
         domain (str): The app domain into which the secrets are to be inserted.
         app_config (dict): A dictionary containing the secrets hooks and their corresponding values to insert.
+        instance_domain (str): The instance domain, used to set ALAKAZAM_INSTANCE_DOMAIN for script hooks.
         strict (bool): Abort on a failing hook instead of only logging it.
 
     Raises:
@@ -1169,11 +1271,15 @@ def run_secret_hooks(domain: str, app_config: Dict[str, Any], strict: bool = Fal
     logging.info(f"Run secret hooks for {domain}")
     if secret_hooks := app_config.get("secret_hooks"):
         for cmd in secret_hooks:
-            print(f"Run '{cmd}' in {domain}", flush=True)
-            try:
-                abra("app", "cmd", "--local", domain, cmd, ignore_error=not strict, stream=True)
-            except RuntimeError as e:
-                raise click.ClickException(f"secret hook '{cmd}' failed for {domain}: {e}")
+            tokens = cmd.split()
+            if tokens[0] == "script" and len(tokens) >= 2 and resolve_path(tokens[1]).exists():
+                run_local_script(tokens, domain, app_config['server'], instance_domain, strict=strict)
+            else:
+                print(f"Run '{cmd}' in {domain}", flush=True)
+                try:
+                    abra("app", "cmd", "--local", domain, cmd, ignore_error=not strict, stream=True)
+                except RuntimeError as e:
+                    raise click.ClickException(f"secret hook '{cmd}' failed for {domain}: {e}")
 
 
 def unquote_strings(s: str) -> str:
@@ -1208,7 +1314,10 @@ def insert_secret(domain: str, secret_name: str, secret: str) -> None:
     """
     # Fix extra quotes around secrets
     secret = unquote_strings(secret)
-    logging.debug(f"Insert secret {secret_name}: {secret} into {domain}")
+    if HIDE_SECRETS:
+        logging.debug(f"Insert secret {secret_name} into {domain}")
+    else:
+        logging.debug(f"Insert secret {secret_name}: {secret} into {domain}")
     abra("app", "secret", "insert", domain, secret_name, "v1", secret)
 
 
@@ -1389,7 +1498,7 @@ def run_readiness_hooks(domain: str, app_config: Dict[str, Any]) -> None:
         run_readiness_hook(domain, hook)
 
 
-def execute_cmds(app_config: Dict[str, Any], commands: Tuple[str] = tuple(), initial: bool = False, deploy: bool = False, upgrade: bool = False, dry_run: bool = False, chaos: bool = False, strict: bool = False) -> None:
+def execute_cmds(app_config: Dict[str, Any], commands: Tuple[str] = tuple(), initial: bool = False, deploy: bool = False, upgrade: bool = False, dry_run: bool = False, chaos: bool = False, strict: bool = False, instance_domain: str = "") -> None:
     """
     Execute post-deployment commands for an application based on the provided configuration.
     This can include running scripts or commands inside the application's environment.
@@ -1401,8 +1510,9 @@ def execute_cmds(app_config: Dict[str, Any], commands: Tuple[str] = tuple(), ini
         initial (bool): execute initial-hooks
         deploy (bool): execute deploy-hooks
         upgrade (bool): execute upgrade-hooks
-        dry-run(bool): only show cmds, don't execute them
+        dry_run (bool): only show cmds, don't execute them
         strict (bool): abort on a failing command instead of only logging it
+        instance_domain (str): The instance domain, used to set ALAKAZAM_INSTANCE_DOMAIN for script hooks.
 
     Returns:
         None
@@ -1411,6 +1521,7 @@ def execute_cmds(app_config: Dict[str, Any], commands: Tuple[str] = tuple(), ini
         click.ClickException: If a command fails and strict is set
     """
     domain = app_config['app_domain']
+    server = app_config['server']
     all_cmds = []
     if initial and (initial_hooks:= app_config.get('initial-hooks')):
         all_cmds = all_cmds + initial_hooks
@@ -1424,7 +1535,12 @@ def execute_cmds(app_config: Dict[str, Any], commands: Tuple[str] = tuple(), ini
     if chaos:
         chaos_flag = "-C"
     for cmd in all_cmds:
-        print(f"Run '{cmd}' in {domain}:{cmd.split()[0]}", flush=True)
+        tokens = cmd.split()
+        container = tokens[0]
+        if container == "script" and len(tokens) >= 2 and resolve_path(tokens[1]).exists():
+            run_local_script(tokens, domain, server, instance_domain, dry_run, strict)
+            continue
+        print(f"Run '{cmd}' in {domain}:{container}", flush=True)
         if dry_run:
             continue
         try:
@@ -1436,8 +1552,10 @@ def execute_cmds(app_config: Dict[str, Any], commands: Tuple[str] = tuple(), ini
 @click.group(context_settings={"help_option_names": ['-h', '--help']})
 @click.option('-l', '--log', 'loglevel', help='Desired logging level ("debug", "info", "warning", "error", "critical")')
 @click.option('-e', '--exclude', help='Path to a directory that contains a group of instance configurations to be excluded.', multiple=True, type=click.Path(exists=True))
+@click.option('exclude_recipes', '-er', '--exclude-recipe', multiple=True, metavar='<RecipeName>', help='Leave these recipes untouched, this option can be specified multiple times.')
+@click.option('hide_secrets', '--hide-secrets', is_flag=True, help='Keep secret values out of the output, for runs whose logs are kept.')
 @click.argument('group_path', type=click.Path(exists=True))
-def cli(loglevel: str, group_path: str, exclude:Tuple[str]) -> None:
+def cli(loglevel: str, group_path: str, exclude: Tuple[str], exclude_recipes: Tuple[str], hide_secrets: bool) -> None:
     """
     Alakazam is a meta-configuration app-connector and an abra wrapper, designed as a proof-of-concept to simplify the management of environment configuration files across multiple instances.
 
@@ -1448,7 +1566,10 @@ def cli(loglevel: str, group_path: str, exclude:Tuple[str]) -> None:
     global SETTINGS
     global SETTINGS_PATH
     global GROUP_PATH
+    global HIDE_SECRETS
     global ABRA_DIR
+    global ROOT_PATH
+    HIDE_SECRETS = hide_secrets
     if loglevel:
         numeric_level = getattr(logging, loglevel.upper(), None)
         if not isinstance(numeric_level, int):
@@ -1463,21 +1584,23 @@ def cli(loglevel: str, group_path: str, exclude:Tuple[str]) -> None:
         root_path = os.getcwd()
         logging.warning(f"There is no 'root' path defined in '{SETTINGS_PATH}', use current path '{root_path}'instead")
     _group_path = GROUP_PATH = Path(group_path).expanduser().absolute()
-    _root_path = Path(root_path).expanduser()
-    if not _root_path.is_absolute():
-        _root_path = (settings_dir / _root_path).absolute()
-    if not str(_group_path).startswith(str(_root_path)):
-        logging.error(f"{_root_path} does not contain {_group_path}?")
+    ROOT_PATH = resolve_path(root_path, settings_dir)
+    if not str(_group_path).startswith(str(ROOT_PATH)):
+        logging.error(f"{ROOT_PATH} does not contain {_group_path}?")
         exit(1)
     exclude_paths = list(map(lambda p: str(Path(p).absolute()), exclude))
-    if ABRA_DIR.is_relative_to(_root_path) and str(ABRA_DIR) not in exclude_paths:
+    if ABRA_DIR.is_relative_to(ROOT_PATH) and str(ABRA_DIR) not in exclude_paths:
         exclude_paths.append(str(ABRA_DIR))
-    preflight_configs(get_relevant_config_paths(_root_path, _group_path, exclude_paths))
-    all_group_configs = merge_all_group_configs(_root_path)
-    config_sets = read_config(str(_root_path / "config-sets.yml"))
+    preflight_configs(get_relevant_config_paths(ROOT_PATH, _group_path, exclude_paths))
+    all_group_configs = merge_all_group_configs(ROOT_PATH)
+    config_sets = read_config(str(ROOT_PATH / "config-sets.yml"))
     instance_configs = get_merged_instance_configs(_group_path, all_group_configs, exclude_paths, config_sets)
     INSTANCE_CONFIGS = merge_connection_configs(instance_configs)
-    all_configs = get_merged_instance_configs(_root_path, all_group_configs, exclude_paths, config_sets)
+    # dropping the recipes here rather than at every filter is what makes the exclusion complete:
+    # get_apps(), create_secrets(), configure_apps() and the rest all read INSTANCE_CONFIGS.
+    # ALL_CONFIGS stays whole, it is the cross-instance view that backup looks up its bot in
+    INSTANCE_CONFIGS = exclude_from_configs(INSTANCE_CONFIGS, exclude_recipes)
+    all_configs = get_merged_instance_configs(ROOT_PATH, all_group_configs, exclude_paths, config_sets)
     ALL_CONFIGS = merge_connection_configs(all_configs)
     fetch_recipes(INSTANCE_CONFIGS)
 
@@ -1561,8 +1684,8 @@ def create_secrets(recipes: Tuple[str], syncvalues: bool = False, variants: Opti
     """
     if variants is None:
         variants = set(SECRET_VARIANTS)
-    for instance, instance_config in INSTANCE_CONFIGS.items():
-        if instances and instance not in instances:
+    for instance_domain, instance_config in INSTANCE_CONFIGS.items():
+        if instances and instance_domain not in instances:
             continue
         instance_apps = instance_config.keys()
         if recipes:
@@ -1579,7 +1702,7 @@ def create_secrets(recipes: Tuple[str], syncvalues: bool = False, variants: Opti
             if "conf" in variants:
                 insert_secrets_from_conf(domain, app_config)
             if "secret-hooks" in variants:
-                run_secret_hooks(domain, app_config, strict=strict)
+                run_secret_hooks(domain, app_config, instance_domain, strict=strict)
         # Pass 2: exchange secrets between apps, then generate any remaining missing secrets.
         for app in selected_apps:
             app_config = instance_config[app]
@@ -1673,7 +1796,7 @@ def deploy_apps(instance_apps: Dict[str, List[List]], execute_hooks: bool = Fals
             print(abra("app", *cmd))
             if execute_hooks:
                 logging.info(f'execute commands for {domain}')
-                execute_cmds(app_config, deploy=True)
+                execute_cmds(app_config, deploy=True, instance_domain=instance)
 
 
 @cli.command()
@@ -1681,9 +1804,12 @@ def deploy_apps(instance_apps: Dict[str, List[List]], execute_hooks: bool = Fals
 @click.option('-e', '--execute-hooks', is_flag=True, help='run post-upgrade commands.')
 @click.option('-d', '--dry-run', is_flag=True, help="don't execute the upgrade process")
 @click.option('-rd', '--redeploy', is_flag=True, help="use undeploy and deploy for the updating process")
+@click.option('--redeploy-wait', type=int, default=20, metavar='<SECONDS>',
+              help='seconds to wait between undeploy and deploy in --redeploy mode (default: 20).')
 @click.option('-c', '--converge-checks', is_flag=True, help='perform convergence checks during deployment.')
 @click.option('noninteractive', '-n', '--non-interactive', is_flag=True, help='Run this command non-interactively')
-def upgrade(recipes: Tuple[str], execute_hooks: bool, dry_run: bool, redeploy: bool, converge_checks: bool, noninteractive: bool) -> None:
+def upgrade(recipes: Tuple[str], execute_hooks: bool, dry_run: bool, redeploy: bool,
+            converge_checks: bool, noninteractive: bool, redeploy_wait: int) -> None:
     """
     Upgrades specified applications by executing the upgrade commands via the 'abra' command-line interface.
     It checks the current deployment status of the apps and performs upgrades only where necessary, with options to execute additional commands or perform a dry run. It either took the target version from the configuration or it uses the latest available version.
@@ -1727,7 +1853,7 @@ def upgrade(recipes: Tuple[str], execute_hooks: bool, dry_run: bool, redeploy: b
             app_details.append(upgrade_version)
             upgrade_apps.append(app_details)
             logging.info(f'upgrade {app}: {domain} from version {deployed_version} to version "{upgrade_version}"')
-            upgrade_cmds.append((app_config, upgrade_cmd))
+            upgrade_cmds.append((app_config, upgrade_cmd, instance))
             if version_type == 'version':
                 release_note_cmd = upgrade_cmd.copy()
                 release_note_cmd.insert(1, '-r')
@@ -1742,19 +1868,20 @@ def upgrade(recipes: Tuple[str], execute_hooks: bool, dry_run: bool, redeploy: b
         print(app)
         print(note)
     if not dry_run and noninteractive or input(f"Do you really want to upgrade these apps? Type YES: ") == "YES":
-        for app_config, upgrade_cmd in upgrade_cmds:
+        for app_config, upgrade_cmd, instance_domain in upgrade_cmds:
             app_domain = app_config.get('app_domain')
             if redeploy:
                 upgrade_cmd.pop(0)
-                print(f'undeploy {app_domain}')
                 upgrade_cmd.insert(0, 'deploy')
+                print(f'undeploy {app_domain}')
                 print(abra("app", "undeploy", '--no-input', app_domain))
-                sleep(20)
+                print(f'waiting {redeploy_wait}s …')
+                sleep(redeploy_wait)
             print(f'deploy {app_domain}')
             print(abra("app", *upgrade_cmd))
             if execute_hooks:
                 logging.info(f'execute commands for {app_domain}')
-                execute_cmds(app_config, upgrade=True)
+                execute_cmds(app_config, upgrade=True, instance_domain=instance_domain)
 
 
 @cli.command()
@@ -1800,7 +1927,7 @@ def cmd(recipes: Tuple[str], commands: Tuple[str], initial: bool, deploy: bool,
     Execute commands for all specified applications based on the provided configuration.
     """
     deployed_domains = get_deployed_apps(recipes)
-    for _, instance_config in INSTANCE_CONFIGS.items():
+    for instance_domain, instance_config in INSTANCE_CONFIGS.items():
         if recipes:
             selected_apps = [app for app in recipes if app in instance_config.keys()]
         else:
@@ -1812,7 +1939,7 @@ def cmd(recipes: Tuple[str], commands: Tuple[str], initial: bool, deploy: bool,
                 print(f"{domain} is not deployed")
                 continue
             logging.info(f'execute commands for {domain}')
-            execute_cmds(app_config, commands, initial, deploy, upgrade, list_cmds, chaos)
+            execute_cmds(app_config, commands=commands, initial=initial, deploy=deploy, upgrade=upgrade, dry_run=list_cmds, chaos=chaos, instance_domain=instance_domain)
 
 
 @cli.command()
@@ -2128,7 +2255,7 @@ def parse_secret_variants(only: Tuple[str]) -> Set[str]:
 def purge_app_secrets(recipe_secrets: Dict[str, List[str]]) -> None:
     """
     Removes the given secrets from every app of the given recipes.
-    A secret that is not stored on the server is skipped, so an aborted run can simply be repeated.
+    A secret that is not stored on the server is skipped, and so is an app that has none at all, so an aborted run can simply be repeated.
     All other errors are not suppressed: a secret that could not be removed would silently keep its old value on a later recreation.
 
     Args:
@@ -2139,7 +2266,13 @@ def purge_app_secrets(recipe_secrets: Dict[str, List[str]]) -> None:
             for _, domain in app_domains:
                 logging.info(f'purge secrets of {domain}')
                 if not secret_names:
-                    abra("app", "secret", "rm", "-a", domain)
+                    try:
+                        abra("app", "secret", "rm", "-a", domain)
+                    except RuntimeError as e:
+                        if NO_SECRETS_TO_REMOVE not in str(e):
+                            raise
+                        print(f"{domain} has no secrets on its server, skip")
+                        continue
                     print(f"Secrets for {domain} purged")
                     continue
                 for secret_name in secret_names:

combine.yml

@@ -49,6 +49,7 @@ authentik:
             - SECRET_KIMAI_ID_VERSION
             - SECRET_KIMAI_SECRET_VERSION
             - kimai_logo.png
+            - KIMAI_GROUP
     zammad:
         uncomment:
             - compose.zammad.yml

tests/test_exclude_recipes.py

@@ -0,0 +1,89 @@
+"""Tests for leaving single recipes out of a run."""
+
+import logging
+import os
+import sys
+
+import pytest
+from click.testing import CliRunner
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+import alakazam
+from alakazam import exclude_from_configs
+
+CONFIGS = {
+    "a.org": {"authentik": {"app_domain": "login.a.org"}, "traefik": {"app_domain": "a.org"}},
+    "b.org": {"traefik": {"app_domain": "b.org"}, "nextcloud": {"app_domain": "cloud.b.org"}},
+}
+
+
+class TestExcludeFromConfigs:
+    def test_nothing_excluded_returns_the_configs_unchanged(self):
+        assert exclude_from_configs(CONFIGS, ()) is CONFIGS
+
+    def test_the_recipe_is_gone_from_every_instance(self):
+        result = exclude_from_configs(CONFIGS, ("traefik",))
+        assert sorted(result["a.org"]) == ["authentik"]
+        assert sorted(result["b.org"]) == ["nextcloud"]
+
+    def test_several_recipes(self):
+        result = exclude_from_configs(CONFIGS, ("traefik", "nextcloud"))
+        assert result["b.org"] == {}
+
+    def test_the_original_is_not_modified(self):
+        exclude_from_configs(CONFIGS, ("traefik",))
+        assert "traefik" in CONFIGS["a.org"]
+
+    def test_an_unknown_recipe_is_reported(self, caplog):
+        """A typo would otherwise exclude nothing and look like it worked."""
+        with caplog.at_level(logging.WARNING):
+            exclude_from_configs(CONFIGS, ("treafik",))
+        assert "'treafik' is excluded but not configured" in caplog.text
+
+    def test_a_known_recipe_is_not_reported(self, caplog):
+        with caplog.at_level(logging.WARNING):
+            exclude_from_configs(CONFIGS, ("traefik",))
+        assert caplog.text == ""
+
+
+class TestExcludeReachesTheCommands:
+    """One filter in cli() has to cover every consumer of INSTANCE_CONFIGS."""
+
+    @pytest.fixture
+    def env(self, tmp_path, monkeypatch):
+        root = tmp_path / "root"
+        (root / "group").mkdir(parents=True)
+        (root / "alaka.yml").write_text(
+            "authentik:\n    version: 1.0.0\ntraefik:\n    version: 2.0.0\n")
+        (root / "group" / "example.com.yml").write_text("authentik:\ntraefik:\n")
+        (root / "alakazam.yml").write_text(f"root: {root}\n")
+        monkeypatch.setattr(alakazam, "get_settings_path", lambda: str(root / "alakazam.yml"))
+        monkeypatch.setattr(alakazam, "get_abra_dir", lambda: root / ".abra")
+        monkeypatch.setattr(alakazam, "fetch_recipes", lambda *a, **k: None)
+        monkeypatch.setattr(alakazam, "abra", lambda *a, **k: "")
+        return root
+
+    def invoke(self, root, args):
+        return CliRunner().invoke(alakazam.cli, args + [str(root / "group"), "ls"])
+
+    def test_without_the_flag_both_recipes_are_listed(self, env):
+        result = self.invoke(env, [])
+        assert result.exit_code == 0, result.output
+        assert "authentik" in result.output
+        assert "traefik" in result.output
+
+    def test_the_excluded_recipe_disappears(self, env):
+        result = self.invoke(env, ["-er", "traefik"])
+        assert result.exit_code == 0, result.output
+        assert "authentik" in result.output
+        assert "traefik" not in result.output
+
+    def test_the_long_option_works_too(self, env):
+        result = self.invoke(env, ["--exclude-recipe", "traefik"])
+        assert "traefik" not in result.output
+
+    def test_it_can_be_given_more_than_once(self, env):
+        result = self.invoke(env, ["-er", "traefik", "-er", "authentik"])
+        assert "traefik" not in result.output
+        assert "authentik" not in result.output

tests/test_hide_secrets.py

@@ -0,0 +1,100 @@
+"""Tests for keeping secret values out of the output of a run whose logs are kept."""
+
+import logging
+import os
+import sys
+
+import pytest
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+import alakazam
+from alakazam import generate_all_secrets, insert_secret, is_secret_command
+
+VALUE = "hunter2-do-not-log-me"
+
+
+@pytest.fixture(params=[False, True], ids=["visible", "hidden"])
+def hide(request, monkeypatch):
+    monkeypatch.setattr(alakazam, "HIDE_SECRETS", request.param)
+    return request.param
+
+
+class TestIsSecretCommand:
+    @pytest.mark.parametrize("args", [
+        ("app", "secret", "insert", "login.a.org", "db_password", "v1", VALUE),
+        ("app", "secret", "generate", "-a", "login.a.org"),
+    ])
+    def test_commands_carrying_a_value(self, args):
+        assert is_secret_command(args)
+
+    @pytest.mark.parametrize("args", [
+        ("app", "secret", "ls", "login.a.org"),
+        ("app", "secret", "rm", "login.a.org", "db_password"),
+        ("app", "ls"),
+        ("app", "secret"),
+    ])
+    def test_commands_without_a_value(self, args):
+        assert not is_secret_command(args)
+
+
+class TestGeneratedValues:
+    def install(self, monkeypatch, created=False):
+        def abra(*args, **kwargs):
+            if args[:3] == ("app", "secret", "ls"):
+                return [{"name": "db_password", "created on server": str(created).lower()}]
+            if args[:3] == ("app", "secret", "generate"):
+                return [{"name": "db_password", "value": VALUE}]
+            raise AssertionError(f"unexpected: {args}")
+        monkeypatch.setattr(alakazam, "abra", abra)
+
+    def test_the_name_is_always_reported(self, hide, monkeypatch, capsys):
+        self.install(monkeypatch)
+        generate_all_secrets("login.a.org")
+        assert "db_password" in capsys.readouterr().out
+
+    def test_the_value_follows_the_switch(self, hide, monkeypatch, capsys):
+        self.install(monkeypatch)
+        generate_all_secrets("login.a.org")
+        out = capsys.readouterr().out
+        assert (VALUE in out) is not hide
+        assert ("[hidden]" in out) is hide
+
+
+class TestDebugLog:
+    """-l debug is the level a pipeline reaches for when something breaks."""
+
+    def run_abra(self, monkeypatch, args):
+        class Process:
+            returncode = 0
+            stdout = f'[{{"name":"db_password","value":"{VALUE}"}}]'.encode()
+            stderr = b""
+        monkeypatch.setattr(alakazam.subprocess, "run", lambda cmd, capture_output: Process())
+        return alakazam.abra(*args)
+
+    def test_the_command_line_is_not_logged_when_hidden(self, monkeypatch, caplog):
+        monkeypatch.setattr(alakazam, "HIDE_SECRETS", True)
+        with caplog.at_level(logging.DEBUG):
+            self.run_abra(monkeypatch, ("app", "secret", "insert", "login.a.org", "db_password", "v1", VALUE))
+        assert VALUE not in caplog.text
+
+    def test_the_generated_output_is_not_logged_when_hidden(self, monkeypatch, caplog):
+        monkeypatch.setattr(alakazam, "HIDE_SECRETS", True)
+        with caplog.at_level(logging.DEBUG):
+            self.run_abra(monkeypatch, ("app", "secret", "generate", "-a", "login.a.org"))
+        assert VALUE not in caplog.text
+
+    def test_other_commands_are_still_logged(self, monkeypatch, caplog):
+        """Hiding secrets must not turn the debug log off altogether."""
+        monkeypatch.setattr(alakazam, "HIDE_SECRETS", True)
+        with caplog.at_level(logging.DEBUG):
+            self.run_abra(monkeypatch, ("app", "ls"))
+        assert "run command" in caplog.text
+
+    def test_insert_logs_the_name_but_not_the_value(self, monkeypatch, caplog):
+        monkeypatch.setattr(alakazam, "HIDE_SECRETS", True)
+        monkeypatch.setattr(alakazam, "abra", lambda *a, **k: "")
+        with caplog.at_level(logging.DEBUG):
+            insert_secret("login.a.org", "db_password", VALUE)
+        assert "db_password" in caplog.text
+        assert VALUE not in caplog.text

tests/test_local_scripts.py

@@ -0,0 +1,184 @@
+"""Tests for local script functionality in hooks."""
+
+import logging
+import os
+import stat
+import sys
+
+import click
+import pytest
+from click.testing import CliRunner
+
+sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
+
+import alakazam
+from alakazam import (create_secrets, deploy_apps, execute_cmds, get_abra_dir,
+                       resolve_path, run_local_script, run_secret_hooks)
+
+
+class FakeProcess:
+    def __init__(self, returncode):
+        self.returncode = returncode
+
+
+def make_script(tmp_path, name="script.sh", executable=True):
+    """Write a script file under tmp_path, executable by default."""
+    path = tmp_path / name
+    path.write_text("#!/bin/sh\nexit 0\n")
+    if executable:
+        path.chmod(path.stat().st_mode | stat.S_IXUSR)
+    return path
+
+
+class TestResolvePath:
+    def test_an_absolute_path_passes_through_unchanged(self, tmp_path):
+        absolute = tmp_path / "script.sh"
+        assert resolve_path(str(absolute)) == absolute
+
+    def test_a_tilde_path_expands_against_home(self, monkeypatch, tmp_path):
+        monkeypatch.setenv("HOME", str(tmp_path))
+        assert resolve_path("~/script.sh") == tmp_path / "script.sh"
+
+    def test_a_relative_path_resolves_against_the_given_base(self, tmp_path):
+        base = tmp_path / "instance"
+        assert resolve_path("scripts/script.sh", base) == base / "scripts/script.sh"
+
+    def test_a_relative_path_resolves_against_root_path_by_default(self, monkeypatch, tmp_path):
+        monkeypatch.setattr(alakazam, "ROOT_PATH", tmp_path)
+        assert resolve_path("scripts/script.sh") == tmp_path / "scripts/script.sh"
+
+
+class TestRunLocalScript:
+    def test_the_script_receives_its_env_vars_and_arguments(self, monkeypatch, tmp_path):
+        script = make_script(tmp_path)
+        calls = []
+        monkeypatch.setattr(alakazam.subprocess, "run",
+                            lambda cmd, env: calls.append((cmd, env)) or FakeProcess(0))
+        run_local_script(["script", str(script), "arg1"], "login.a.org", "a.org", "a.org")
+        [(cmd, env)] = calls
+        assert cmd == [str(script), "arg1"]
+        assert env["ALAKAZAM_APP_DOMAIN"] == "login.a.org"
+        assert env["ALAKAZAM_APP_SERVER"] == "a.org"
+        assert env["ALAKAZAM_INSTANCE_DOMAIN"] == "a.org"
+
+    def test_a_non_executable_script_is_not_run(self, monkeypatch, tmp_path, caplog):
+        script = make_script(tmp_path, executable=False)
+        monkeypatch.setattr(alakazam.subprocess, "run",
+                            lambda *a, **k: pytest.fail("a non-executable script must not run"))
+        with caplog.at_level(logging.ERROR):
+            run_local_script(["script", str(script)], "login.a.org", "a.org", "a.org")
+        assert "not executable" in caplog.text
+
+    def test_dry_run_does_not_execute_the_script(self, monkeypatch, tmp_path, capsys):
+        script = make_script(tmp_path)
+        monkeypatch.setattr(alakazam.subprocess, "run",
+                            lambda *a, **k: pytest.fail("dry_run must not execute anything"))
+        run_local_script(["script", str(script)], "login.a.org", "a.org", "a.org", dry_run=True)
+        assert "Run local script" in capsys.readouterr().out
+
+    def test_a_non_zero_exit_only_warns_by_default(self, monkeypatch, tmp_path, caplog):
+        script = make_script(tmp_path)
+        monkeypatch.setattr(alakazam.subprocess, "run", lambda *a, **k: FakeProcess(1))
+        with caplog.at_level(logging.WARNING):
+            run_local_script(["script", str(script)], "login.a.org", "a.org", "a.org")
+        assert "exited with code 1" in caplog.text
+
+    def test_a_non_zero_exit_aborts_in_strict_mode(self, monkeypatch, tmp_path):
+        script = make_script(tmp_path)
+        monkeypatch.setattr(alakazam.subprocess, "run", lambda *a, **k: FakeProcess(1))
+        with pytest.raises(click.ClickException) as excinfo:
+            run_local_script(["script", str(script)], "login.a.org", "a.org", "a.org", strict=True)
+        assert "exited with code 1" in excinfo.value.message
+
+    def test_a_non_executable_script_aborts_in_strict_mode(self, monkeypatch, tmp_path):
+        script = make_script(tmp_path, executable=False)
+        monkeypatch.setattr(alakazam.subprocess, "run",
+                            lambda *a, **k: pytest.fail("a non-executable script must not run"))
+        with pytest.raises(click.ClickException) as excinfo:
+            run_local_script(["script", str(script)], "login.a.org", "a.org", "a.org", strict=True)
+        assert "not executable" in excinfo.value.message
+
+
+class TestScriptHookDispatch:
+    def test_a_script_secret_hook_dispatches_to_run_local_script(self, monkeypatch, tmp_path):
+        script = make_script(tmp_path)
+        calls = []
+        monkeypatch.setattr(alakazam, "run_local_script", lambda *a, **k: calls.append((a, k)))
+        monkeypatch.setattr(alakazam, "abra", lambda *a, **k: pytest.fail("should not call abra"))
+        run_secret_hooks("login.a.org", {"secret_hooks": [f"script {script}"], "server": "a.org"}, "a.org")
+        [(args, kwargs)] = calls
+        assert args == (["script", str(script)], "login.a.org", "a.org", "a.org")
+
+    def test_a_normal_secret_hook_goes_through_abra(self, monkeypatch):
+        calls = []
+        monkeypatch.setattr(alakazam, "abra", lambda *a, **k: calls.append((a, k)) or "")
+        run_secret_hooks("login.a.org", {"secret_hooks": ["insert_cert"], "server": "a.org"})
+        assert calls, "a non-script hook must still run through abra"
+
+    def test_a_script_command_dispatches_to_run_local_script(self, monkeypatch, tmp_path):
+        script = make_script(tmp_path)
+        calls = []
+        monkeypatch.setattr(alakazam, "run_local_script", lambda *a, **k: calls.append((a, k)))
+        monkeypatch.setattr(alakazam, "abra", lambda *a, **k: pytest.fail("should not call abra"))
+        execute_cmds({"app_domain": "login.a.org", "server": "a.org",
+                       "initial-hooks": [f"script {script} arg1"]}, initial=True)
+        [(args, kwargs)] = calls
+        assert args == (["script", str(script), "arg1"], "login.a.org", "a.org", "", False, False)
+
+    def test_a_normal_command_goes_through_abra(self, monkeypatch):
+        calls = []
+        monkeypatch.setattr(alakazam, "abra", lambda *a, **k: calls.append((a, k)) or "")
+        execute_cmds({"app_domain": "login.a.org", "server": "a.org",
+                       "initial-hooks": ["app set_default_quota"]}, initial=True)
+        assert calls, "a non-script command must still run through abra"
+
+    def test_a_script_token_with_a_missing_file_falls_back_to_abra(self, monkeypatch, tmp_path):
+        """A 'script' hook whose path does not exist is not a script hook at all, just an abra
+        command whose container happens to be named 'script'."""
+        missing = tmp_path / "does-not-exist.sh"
+        calls = []
+        monkeypatch.setattr(alakazam, "run_local_script",
+                            lambda *a, **k: pytest.fail("should not run a missing script"))
+        monkeypatch.setattr(alakazam, "abra", lambda *a, **k: calls.append((a, k)) or "")
+        execute_cmds({"app_domain": "login.a.org", "server": "a.org",
+                       "initial-hooks": [f"script {missing}"]}, initial=True)
+        assert calls, "a missing script path must fall back to a normal abra command"
+
+
+class TestInstanceDomainThreading:
+    """create_secrets(), deploy_apps() and the 'cmd' CLI command each thread instance_domain
+    into run_secret_hooks()/execute_cmds() for script hooks."""
+
+    @pytest.fixture
+    def app_config(self, monkeypatch):
+        """A single 'authentik' app on instance 'a.org', shared by tests below."""
+        config = {"app_domain": "login.a.org", "server": "a.org"}
+        monkeypatch.setattr(alakazam, "INSTANCE_CONFIGS", {"a.org": {"authentik": config}})
+        return config
+
+    def test_create_secrets_threads_the_instance_domain(self, monkeypatch, app_config):
+        calls = []
+        monkeypatch.setattr(alakazam, "run_secret_hooks", lambda *a, **k: calls.append((a, k)))
+        create_secrets(recipes=(), variants={"secret-hooks"})
+        [(args, kwargs)] = calls
+        assert args == ("login.a.org", app_config, "a.org")
+
+    def test_deploy_apps_threads_the_instance_domain(self, monkeypatch, app_config):
+        app_config["version"] = "1.0.0"
+        monkeypatch.setattr(alakazam, "abra", lambda *a, **k: "")
+        calls = []
+        monkeypatch.setattr(alakazam, "execute_cmds", lambda app_config, **k: calls.append(k))
+        deploy_apps({"a.org": [["authentik", "login.a.org"]]}, execute_hooks=True)
+        [kwargs] = calls
+        assert kwargs["deploy"] is True
+        assert kwargs["instance_domain"] == "a.org"
+
+    def test_cmd_threads_the_instance_domain(self, monkeypatch, app_config):
+        monkeypatch.setattr(alakazam, "get_deployed_apps", lambda apps: {"login.a.org": "1.0.0"})
+        calls = []
+        monkeypatch.setattr(alakazam, "execute_cmds", lambda app_config, **k: calls.append(k))
+        result = CliRunner().invoke(alakazam.cmd, ["-i"], standalone_mode=False)
+        assert result.exception is None
+        [kwargs] = calls
+        assert kwargs["initial"] is True
+        assert kwargs["instance_domain"] == "a.org"

tests/test_purge.py

@@ -9,7 +9,7 @@ import pytest
 sys.path.insert(0, os.path.dirname(os.path.dirname(os.path.abspath(__file__))))
 
 import alakazam
-from alakazam import purge_apps
+from alakazam import purge_app_secrets, purge_apps
 
 INSTANCE_APPS = {"a.org": [["authentik", "login.a.org"]]}
 CONFIG = {"a.org": {"authentik": {"app_domain": "login.a.org", "server": "a.org"}}}
@@ -50,3 +50,43 @@ class TestPurgeApps:
         monkeypatch.setattr(alakazam, "abra", lambda *a, **k: calls.append(a) or "")
         purge_apps(INSTANCE_APPS)
         assert calls == []
+
+
+class TestPurgeAppSecrets:
+    @pytest.fixture
+    def config(self, monkeypatch):
+        monkeypatch.setattr(alakazam, "INSTANCE_CONFIGS", CONFIG)
+
+    def install(self, monkeypatch, error=None):
+        calls = []
+
+        def abra(*args, **kwargs):
+            calls.append(args)
+            if error:
+                raise RuntimeError(error)
+            return ""
+
+        monkeypatch.setattr(alakazam, "abra", abra)
+        return calls
+
+    def test_all_secrets_of_a_recipe_are_removed(self, config, monkeypatch, capsys):
+        calls = self.install(monkeypatch)
+        purge_app_secrets({"authentik": []})
+        assert calls == [("app", "secret", "rm", "-a", "login.a.org")]
+        assert "purged" in capsys.readouterr().out
+
+    def test_an_app_without_secrets_is_not_a_failure(self, config, monkeypatch, capsys):
+        """abra exits non-zero when it found nothing to remove, which says the job is already done."""
+        self.install(monkeypatch, error="FATA no secrets to remove?")
+        purge_app_secrets({"authentik": []})
+        assert "has no secrets on its server" in capsys.readouterr().out
+
+    def test_a_real_failure_still_propagates(self, config, monkeypatch):
+        self.install(monkeypatch, error="FATA error during connect: no route to host")
+        with pytest.raises(RuntimeError):
+            purge_app_secrets({"authentik": []})
+
+    def test_a_named_secret_that_is_absent_is_skipped(self, config, monkeypatch, capsys):
+        self.install(monkeypatch, error="FATA email_pass doesn't exist on server?")
+        purge_app_secrets({"authentik": ["email_pass"]})
+        assert "is not stored on the server" in capsys.readouterr().out

tests/test_strict_hooks.py

@@ -1,6 +1,7 @@
 """Tests for the strict hook mode that lets a scratch build fail on a broken hook."""
 
 import os
+import stat
 import sys
 
 import click
@@ -12,6 +13,14 @@ import alakazam
 from alakazam import execute_cmds, run_secret_hooks
 
 
+def failing_script(tmp_path):
+    """A tiny local script that exits non-zero, for testing 'script' hook failure handling."""
+    script = tmp_path / "fail.sh"
+    script.write_text("#!/bin/sh\nexit 1\n")
+    script.chmod(script.stat().st_mode | stat.S_IXUSR)
+    return script
+
+
 class TestStrictHooks:
     def test_a_secret_hook_failure_is_tolerated_by_default(self, monkeypatch):
         """'secrets' has always continued past a failing hook, that must not change."""
@@ -30,12 +39,36 @@ class TestStrictHooks:
 
     def test_a_command_failure_is_tolerated_by_default(self, monkeypatch):
         monkeypatch.setattr(alakazam, "abra", lambda *a, **k: "")
-        execute_cmds({"app_domain": "login.a.org", "initial-hooks": ["app init"]}, initial=True)
+        execute_cmds({"app_domain": "login.a.org", "server": "a.org", "initial-hooks": ["app init"]}, initial=True)
 
     def test_a_command_failure_aborts_in_strict_mode(self, monkeypatch):
         def failing(*args, ignore_error=False, **kwargs):
             raise RuntimeError("FATA container not found")
         monkeypatch.setattr(alakazam, "abra", failing)
         with pytest.raises(click.ClickException) as excinfo:
-            execute_cmds({"app_domain": "login.a.org", "initial-hooks": ["app init"]}, initial=True, strict=True)
+            execute_cmds({"app_domain": "login.a.org", "server": "a.org", "initial-hooks": ["app init"]}, initial=True, strict=True)
         assert "command 'app init' failed" in excinfo.value.message
+
+    def test_a_script_secret_hook_failure_is_tolerated_by_default(self, tmp_path):
+        """A failing local script must be tolerated by default too, same as a failing abra.sh hook."""
+        script = failing_script(tmp_path)
+        run_secret_hooks("login.a.org", {"secret_hooks": [f"script {script}"], "server": "a.org"})
+
+    def test_a_script_secret_hook_failure_aborts_in_strict_mode(self, tmp_path):
+        script = failing_script(tmp_path)
+        with pytest.raises(click.ClickException) as excinfo:
+            run_secret_hooks("login.a.org", {"secret_hooks": [f"script {script}"], "server": "a.org"}, strict=True)
+        assert str(script) in excinfo.value.message
+        assert "failed" in excinfo.value.message
+
+    def test_a_script_command_failure_is_tolerated_by_default(self, tmp_path):
+        script = failing_script(tmp_path)
+        execute_cmds({"app_domain": "login.a.org", "server": "a.org", "initial-hooks": [f"script {script}"]}, initial=True)
+
+    def test_a_script_command_failure_aborts_in_strict_mode(self, tmp_path):
+        script = failing_script(tmp_path)
+        with pytest.raises(click.ClickException) as excinfo:
+            execute_cmds({"app_domain": "login.a.org", "server": "a.org", "initial-hooks": [f"script {script}"]},
+                         initial=True, strict=True)
+        assert str(script) in excinfo.value.message
+        assert "failed" in excinfo.value.message