Alakazam

Proof-of-concept meta-configuration app-connector abra wrapper.

Problem Statement

• managing a lot of env files can be error prone
  • copy pasting env files
• loosing the overview of configuration differences between instances
  • env files can be long, how to see what are defaults and what are customizations
• updating env files is a manual process of comparing the env files
  • env files of recipes can change, how to keep up to date with them?
• no way to define default configurations that should always be applied
  • if you have global configs for all your apps you have to copy paste them
• connecting two apps requires manual effort of specific env file changes and sharing secrets
  • i.E. the effort of configuring SSO for multiple apps

Alakazam Advantages

• Have a global configuration that applies on every instance:
  • avoid copy pasta errors
  • avoid forgetting important configurations
• Reduce manual configuration overhead for connecting multiple apps
  • exchange domains
  • share secrets
  • set specific env vars
• Have a minimal configuration file for each instance
  • it should only contain parameters that differ from the defaults and the global configuration
• Set a global subdomain convention
  • each app will always have the same subdomain
• Set up an instance with multiple connected apps in one run
• Automatic updates of the env configurations
  • avoid manual env configuration at all
• Automatic mass updates
• Specify the versions for all your apps
• Hierarchical configuration structure with inheritance
  • don't lose track of large scales of configurations

How does it work

alakazam is a wrapper around abra with the purpose of automating the whole deployment process of multiple apps. It takes yaml configuration files and generates the .env files used by abra. Based on the yaml configuration it's also able to generate, insert and exchange secretes between apps. It can deploy/update multiple apps at once and run post deploy hooks on all these apps. All configurations are structured hierarchically in yaml files, that can inherit properties from the layers above.

Configuration

Concept

Each configuration can be templated with jinja2 and global variables. There are the following three general types of configuration files:

1. alaka.yml
   • It contains global configurations for a specific hierarchical layer
     • all the apps under this layer will inherit the properties defined here.
     • i.e. smtp config, language, app versions
   • It can contain specify the subdomains for specific recipes
     • i.e. cloud.example.com for each nextcloud app
   • This configuration is for the operator to avoid copy pasta errors, reduce manual and duplicate configurations and keep the same naming convention
2. ./example.com.yml (<your-domain>.yml)
   • a minimalist configuration per instance
     • an instance means a collection of apps that are integrated with each other
     • one server / vm can contain multiple instances
   • contains at least:
     • the apps to be installed
   • the filename contains the instance domain, which is used to create all the app subdomains
     • if the server is not extra specified it is also used as server on which the apps should be deployed (see Templating Configurations )
   • can optionally contain instance specific configurations for each app, overwriting the inherited alaka.yml configurations
3. combine.yml
   • contains the configuration steps that are required to combine apps with each other
   • This configuration should not be touched by the operator
     • at the moment it's part of the alakazam repository
     • In future it should be split into the recipes repositories and maintained by the recipe maintainer

App Configuration

alaka.yml, example.com.yml and combine.yml contain a similar structure to configure the individual apps. For each app the following <app_configurations> steps can be used:

• uncomment:
  • uncomment each matching line (i.E. compose.smtp.yml)
  • this is useful for env variables that are used multiple times like COMPOSE_FILE
• comment:
  • the opposite of uncomment, comment unwanted env variables
• env:
  • set the value for env variables (implicit uncommenting variables)
• execute:
  • abra.sh commands that should be executed after deployment
• secrets:
  • insert a specific value into a secret
  • for secrets that can not be generated, i.e. smtp passwords
  • TODO: use secrets from encrypted files to avoid plain text secrets
• subdomain:
  • contains the subdomain that should be used for a specific app
  • i.e. cloud.example.com for nextcloud
  • (not available in combine.yml)
• version
  • contains the recipe version, that should be deployed
  • if not specified, the newest version will be used
  • (not available in combine.yml)

The combine.yml configuration additionally contains

• shared_secrets:
  • <source_secret_name>:<target_secret_name>
  • a mapping to define which secrets should be shared between two apps

Configuration Structure

1. alaka.yml and example.com.yml

   • the <app_recipe> entries in alaka.yml define global app configurations that should be inherited by all underlaying configurations
   • in example.com.yml the entries <app_recipe> define which apps should be installed
   • the <app_configuration> is configured according to App Configuration above

   <app_recipe>:
       <app_configurations>
   

2. combine.yml

   • it contains the <target_app_configurations> that should be applied to the target app if both the <target_app_recipe> and the <source_app_recipe> is part of the same instance (example.com.yml)
   • the <target_app_configurations> is configured according to App Configuration above

   <target_app_recipe>:
       <source_app_recipe>:
           <target_app_configurations>
   

Templating Configurations

Each configuration file can have a GLOBALS section, that cat be used to template the configuration with global variables.

GLOBALS:
    server: example-server.com
    smtp_user: noreply
    smtp_domain: example.org
authentik:
    env:
        AUTHENTIK_EMAIL__USERNAME: "{{smtp_user}}@{{smtp_domain}}"
nextcloud:
    env: 
        MAIL_FROM_ADDRESS: "{{smtp_user}}"

The server variable is used to specify on which server the instance should be deployed. If it's not specified the instance domain from the filename example.com.yml is taken.

Inheriting Configuration Structure

In the simplest case it's possible to have only one example.com.yml and place all the necessary configuration in there. If you want to maintain multiple similar instances you can merge the common app configurations inside a global alaka.yml. If the deployment gets more complex it is possible to group instances inside folder and have an alaka.yml for each group. For even more complexity each group can inherit the app configurations from the upper groups.

The following example shows how this configuration structure can be used to deploy complex systems:

File Structure

Configuration Structure

Tutorial

Install alakazam

TODO

Create a new instance:

1. The example.com.yml is created by the operator to define the domain and required apps
   • a simple example could look like this:

traefik:
backup-bot-two:
authentik:
nextcloud:
onlyoffice:
wordpress:
vikunja:
matrix-synapse:
element-web:
rallly:

2. ./alakazam.py -p example.com.yml config for each app the env files are generated:

   • The app configurations from combine.yml are taken if the source app and the target app are part of the instace
   • the app configuration of all the upper alaka.yml files are merged
   • the inherited configuration is overwritten by the instance configuration example.com.yml and applied to the env file

3. ./alakazam.py -p example.com.yml secrets for each app the secrets are inserted:

   4. for each app the combine.yml configs are applied on the env files and secrets are inserted
      • the shared_secrets are generated
      • <source_app_recipe>.example.com is replaced in the <target_app_recipe> env file according to subdomain configuration
   5. all the remaining secrets are generated

4. ./alakazam.py deploy-apps

   • each app is deployed
   • the execute commands from defaults.yml, configs.yml and combine.yml are executed

Step 3. (./alakazam.py setup-apps) should be executed every time the config is changed or the recipe env file is updated. It regenerates the env files and creates all the necessary secrets. The env files itself shouldn't be edited manually anymore.

Conceptual Questions

• Possible abra integration

  • An independent wrapper for abra, that uses abras --machine output
  • A go binary that is build on top of the abra library
  • An abra command like abra compose

• Put config.yml into ~/.abra/servers/<server>/ or let the user decide?

• Should we move completely away from .env files?

  • This would skip the translation step into env files
  • It reduces the complexity!
  • Recipes couldn't be run with docker compose only anymore
    • Is this still possible?

• Where should the combine.yml be maintained?

  • Separated config for each recipe?
    • better configuration versioning for different recipe versions
    • harder to maintain the recipes: requires knowledge about the connectable apps
  • Global config?