Alakazam

Proof-of-concept meta-configuration app-connector abra wrapper.

Problem Statement

• managing a lot of env files can be error prone
  • copy pasting env files
• loosing the overview of configuration differences between instances
• updating env files is a manual process of comparing the env files
• no way to define default configurations that should always be applied
• connecting two apps requires manual effort of specific env file changes and sharing secrets

Advantages

• Have a global configuration that applies on every instance:
  • avoid copy pasta errors
  • avoid forgetting important configurations
• Reduce manual configuration overhead for connecting multiple apps
  • exchange domains
  • share secrets
  • set specific env vars
• Have a minimal configuration file for each instance
  • it should only contain parameters that differ from the defaults and the global configuration
• Set a global subdomain convention
  • each app will always have the same subdomain
• Set up an instance with multiple connected apps in one run
• Automatic updates of the env configurations
  • avoid manual env configuration at all

Configuration

Concept

This concept contains the following configuration files. Each configuration can be templated with jinja2 and global variables.

1. ~/.abra/defaults.yml
   • It contains global configurations for each app that are applied on each instance
     • i.e. smtp config, language...
   • It contains a mapping for each app to a subdomain
     • i.e. cloud.example.com for each nextcloud instance
   • This configuration is for the operator to avoid copy pasta errors, reduce manual configurations and keep the same naming convention
2. ./config.yml
   • a minimalist configuration per instance
   • contains at least:
     • the domain (and the server, if they differ)
     • the apps to be installed
   • can optionally contain instance specific configurations for each app, overwriting the defaults.yml configurations
3. combine.yml
   • contains the configuration steps that are required to combine apps with each other
   • This configuration should not be touched by the operator
     • It should be either split into the recipes repositories and maintained by the recipe maintainer or it should be maintained by the alakazam maintainer

App Configuration

defaults.yml, config.yml and combine.yml contain a similar structure to configure the individual apps. For each app the following <app_configurations> steps can be used:

• uncomment:
  • uncomment each matching line
  • this is useful for env variables that are used multiple times like COMPOSE_FILE
• env:
  • set the value for env variables
• execute:
  • abra.sh commands that should be executed after deployment
• secrets:
  • insert a specific value into a secret
  • for secrets that can not be generated, i.e. smtp passwords
• subdomain:
  • contains the subdomain that should be used for a specific app
  • i.e. cloud.example.com for nextcloud
  • (not available in combine.yml)

The combine.yml configuration additionally contains

• shared_secrets:
  • <source_secret_name>:<target_secret_name>
  • a mapping to define which secrets should be shared between two apps

Each configuration file can have a GLOBALS section, that cat be used to template the configuration with global variables.

GLOBALS:
    smtp_user: noreply
    smtp_domain: example.org
authentik:
    env:
        AUTHENTIK_EMAIL__USERNAME: "{{smtp_user}}@{{smtp_domain}}"
nextcloud:
    env: 
        MAIL_FROM_ADDRESS: "{{smtp_user}}"

Configuration Structure

1. defaults.yml

   • there is an entry for each <app_recipe> that should have a global configuration
   • the <app_configuration> is configured according to App Configuration above

   <app_recipe>:
       <app_configurations>
   

2. config.yml

   domain: <instance_domain>
   server: <instance_server>
   apps:
       <app_recipe>:
           <app_configurations>
   

3. combine.yml

   • To combine two deployed apps each <target_app_configurations> of a specific <target_app_recipe> is only applied if the config.yml also contains the belonging <source_app_recipe>

   <target_app_recipe>:
       <source_app_recipe>:
           <target_app_configurations>
   

Process

Create a new instance:

1. The config.yml is created by the operator to define the domain and required apps
2. ./alakazam.py init-server
   • installs traefik and backup-bot-two
3. ./alakazam.py setup-apps
   1. for each app the env files are created
   2. for each app the defaults.yml configs are applied on the env files and secrets are inserted
   3. for each app the config.yml configs are applied on the env files and secrets are inserted
   4. for each app the combine.yml configs are applied on the env files and secrets are inserted
      • the shared_secrets are generated
      • <source_app_recipe>.example.com is replaced in the <target_app_recipe> env file according to subdomain configuration
   5. all the remaining secrets are generated
4. ./alakazam.py deploy-apps
   • each app is deployed
   • the execute commands from defaults.yml, configs.yml and combine.yml are executed

Step 3. (./alakazam.py setup-apps) should be executed every time the config is changed or the recipe env file is updated. It regenerates the env files and creates all the necessary secrets. The env files itself shouldn't be edited manually anymore.

Conceptual Questions

Please note any ideas and arguments you have: https://pad.local-it.org/dB0FsiomRCOCPs7rS40YSQ

• Possible abra integration

  • An independent wrapper for abra, that uses abras --machine output
  • A go binary that is build on top of the abra library
  • An abra command like abra compose

• Put config.yml into ~/.abra/servers/<server>/ or let the user decide?

• Should we move completely away from .env files?

  • This would skip the translation step into env files
  • It reduces the complexity!
  • Recipes couldn't be run with docker compose only anymore
    • Is this still possible?

• Where should the combine.yml be maintained?

  • Separated config for each recipe?
    • better configuration versioning for different recipe versions
    • harder to maintain the recipes: requires knowledge about the connectable apps
  • Global config?