alakazam/README.md

Alakazam

Alakazam is a meta-configuration app-connector and an abra wrapper, designed as a proof-of-concept to simplify the management of environment configuration files across multiple instances.

Problem Statement

Managing numerous environment (env) files can be prone to errors such as:

• Frequent copy-pasting mistakes.
• Difficulty maintaining an overview of configuration differences across instances.
• Laborious manual processes required for updating env files, particularly when underlying recipe env configurations change.
• Challenges in defining default configurations that should consistently apply across all apps and all instances.
• Manual effort required for app integration (e.g., SSO), including configuring environment files and exchanging secrets.

Advantages of Alakazam

• Global Configuration: Prevents errors related to manual copying and pasting and ensures no critical configurations are overlooked.
• Reduced Manual Configuration: Facilitates the connection between multiple applications (i.E. SSO) through domain exchanges, secret sharing, and specific environmental variables.
• Minimal Configuration Files: Each instance configuration only contains parameters that deviate from the defaults or global configurations, simplifying management.
• Standardized Subdomain Conventions: Ensures consistent subdomain use across apps.
• Simplified Setup: Allows for the setup of an instance with multiple connected apps in one run.
• Automatic Env Configuration Updates: Eliminates manual env configuration.
• Automatic Batch Updates: update all apps of a group of instances at once.
• Hierarchical Configuration Structure: Supports configuration inheritance for ease of managing large-scale configurations.

How does it work

alakazam serves as a wrapper for abra, automating the deployment process for multiple apps. It utilizes YAML configuration files to generate the necessary .env files and manage secret exchanges between applications. It supports the deployment or updating of multiple apps simultaneously and can execute post-deploy hooks. All configurations are hierarchically structured in YAML files, which can inherit properties from the layers above.

Configuration

Concept

Configuration files support templating with Jinja2 and global variables, facilitating dynamic adjustments. The primary types of configuration files include:

1. alaka.yml/alaka-*.yml: Contains global configurations for a hierarchical layer, affecting all subordinate apps.
2. example.com.yml (<your-domain>.yml): A minimalist configuration for each instance, specifying the apps to be installed, the instance domain and optionally instance specific configurations.
   • an instance means a collection of apps that are integrated with each other (one server / vm can contain multiple instances)
3. combine.yml: Manages configurations required to integrate multiple apps with each other.
   • This configuration should not be touched by the operator.
   • At the moment it's part of the alakazam repository.
   • In future it should be split into the recipes repositories and maintained by the recipe maintainer.

App Configuration

alaka.yml/alaka-*.yml, example.com.yml and combine.yml contain a similar configuration structure. For each app/recipe the following <app_configurations> can be used:

• uncomment/comment: Manages the active status of lines in configuration files, such as uncommenting needed variables .
  • it matches against parts of the line (i.E. compose.smtp.yml)
  • this is useful for env variables that are used multiple times like COMPOSE_FILE
• env: Sets values for environment variables.
• execute: Specifies abra.sh commands to run post-deployment.
• secrets: Inserts specific values (i.E. smtp passwords) into secrets; future updates will support encrypted file usage.
• subdomain: Specifies the subdomain scheme for individual recipes and apps. (not available in combine.yml)
  • i.e. cloud.example.com for nextcloud
• version: Controls the recipe version to deploy; if unspecified, the latest version is used. (not available in combine.yml)

The combine.yml configuration additionally contains:

• shared_secrets: Specifies secret sharing between apps.
  • <source_secret_name>:<target_secret_name>

Configuration Structure

Configuration can be simplified into a single example.com.yml or expanded into multiple layered alaka.yml/alaka-*.yml files for complex deployments. This allows for easy maintenance of multiple instances or groups. These configurations are designed to modularize and simplify the management of app settings and integrations, making it easier to maintain and scale your deployments.

1. alaka.yml/alaka-*.yml and example.com.yml

   • alaka.yml/alaka-*.yml: Contain global app configurations (<app_recipe>) which are inherited by all configurations within its hierarchical layer. This ensures consistent settings across multiple applications. All alaka-*.yml files inside a directory are merged into one configuration. Avoid overrides between alaka-*.yml inside of one layer.
   • example.com.yml: Specifies which apps should be deployed (<app_recipe> entries), focusing on the individual needs of each instance.
   • Configurations within these files are structured as follows, with specific <app_configuration> settings detailed in the App Configuration section:

   <app_recipe>:
       <app_configurations>
   

2. combine.yml

   • This file orchestrates the integration between multiple apps within the same instance (example.com.yml), applying <target_app_configurations> to ensure seamless functionality between connected applications.
   • The configuration for each app combination is structured as follows, again <target_app_configurations> is following the guidelines set in the App Configuration section:

   <target_app_recipe>:
       <source_app_recipe>:
           <target_app_configurations>
   

Templating Configurations

Each configuration file can include a GLOBALS section that allows for templating with global variables, making it easier to manage repetitive values across different configurations. Here is how you can define and use these global variables:

GLOBALS:
    server: example-server.com
    smtp_user: noreply
    smtp_domain: example.org
authentik:
    env:
        AUTHENTIK_EMAIL__USERNAME: "{{smtp_user}}@{{smtp_domain}}"
nextcloud:
    env: 
        MAIL_FROM_ADDRESS: "{{smtp_user}}"

• server: Specifies the server where the instance will be deployed. If not provided, the instance domain is automatically taken from the example.com.yml file name.

Inheriting Configuration Structure

Alakazam supports a flexible inheritance model for configuration management, allowing for both simplicity and complexity depending on your deployment needs:

• Simple Configuration: For straightforward setups, you can manage everything within a single example.com.yml file, placing all necessary configurations directly in this document.
• Multiple Similar Instances: To manage multiple similar instances efficiently, common application configurations can be centralized in a global alaka.yml. This method reduces duplication by allowing shared settings across multiple instances. The global alaka.yml can also be split into multiple alaka-*.yml files which are merged together.
• Grouped Instances: For more complex setups, instances can be organized into folders, each with its own alaka.yml/alaka-*.yml. This structure allows for group-specific configurations, making it easier to manage settings at different levels of your infrastructure hierarchy.
• Inherited Configurations: In even more complex scenarios, each group can inherit configurations from the levels above it. This cascading setup ensures that changes at a higher level can be automatically applied to all subordinate groups and instances, maintaining consistency and ease of updates across your entire environment.

This hierarchical approach to configuration is designed to scale with your system’s complexity and can be visualized through the following example diagram:

Tutorial

Install alakazam

To install Alakazam, follow these steps to clone the repository and set up the executable:

git clone https://git.coopcloud.tech/moritz/alakazam.git
cd alakazam
ln -s $PWD/alakazam.sh ~/.local/bin/alakazam
# Ensure `~/.local/bin` is in your `$PATH`

Create a new instance:

To set up a new instance with Alakazam, begin by specifying the required applications in example.com.yml. The name of this file determines the domain for the instance. Here's an example configuration:

traefik:
backup-bot-two:
authentik:
nextcloud:
onlyoffice:
wordpress:
vikunja:
matrix-synapse:
element-web:
rallly:

Ensure that DNS records are set up for all your applications. The simplest approach is to create a CNAME record for the wildcard domain *.example.com to cover all subdomains used by the apps.

For a more comprehensive configuration, refer to the example in ./examples, which can also serve as a template for single-instance configurations (example.com.yml).

Steps to initialize the instance:

1. Generate .env Files: ./alakazam -p example.com.yml config
2. Insert Secrets: ./alakazam -p example.com.yml secrets
3. Deploy Applications: ./alakazam -p example.com.yml deploy -r
   • the -r flag executes post-deployment hooks after each deployment

Updating All Instances

To update all instances:

1. Update Environment Files: alakazam -p example.com.yml config to refresh .env files. It's a good practice to keep these files under version control (e.g., in a Git repository at ~/.abra/example.com) and review changes with git diff before proceeding.
2. Upgrade Applications: ./alakazam -p example.com.yml upgrade to update all applications.