update secret generation docs

docs/abra/cheat-sheet.md

@@ -70,7 +70,7 @@ To upgrade `abra` itself, run the following:
 $ abra upgrade
 ```
 
-Optional flags: `--rc`
+Option flags: `--rc`
 
 
 ### Upgrade a recipe
@@ -79,13 +79,19 @@ Optional flags: `--rc`
 $ abra recipe upgrade $RECIPE`
 ```
 
-Optional flags: `-x,y,z/--major,minor,patch`, `--commit`
+Option flags: `-x,y,z/--major,minor,patch`
+
+```
+$ abra recipe sync $RECIPE
+```
+
+Optional flags: `-x,y,z`
 
 ```
 $ abra recipe release $RECIPE [$VERSION]
 ```
 
-Optional flags: `-r/--dry-run`, `-x,y,z`
+Optional flags: `-p/--publish`, `-r/--dry-run`, `-x,y,z`
 
 
 ### Manually restoring app data

docs/abra/hack.md

@@ -121,27 +121,7 @@ cd
 ./run-ci-int
 ```
 
-You can also `cd abra` and run `bats ...` directly to trigger specific subsets
-of tests. You'll need to export the env vars at the bottom of the `run-ci-int`
-script to reproduce the same settings.
-
-```
-export ABRA_DIR="$HOME/.abra_test"
-export TERM=xterm
-export TEST_SERVER=default
-export ABRA_CI=1
-```
-
-And then ensuring a clean state and running with the same flags:
-
-```
-rm -rf "$ABRA_DIR"
-bats -Tp tests/integration --filter-tags \!dns --print-output-on-failure
-```
-
-See the [`run-ci-int`](https://git.coopcloud.tech/toolshed/abra/src/branch/main/scripts/tests/run-ci-int) script for more.
-
-See below for more tips on how to run the tests.
+You can also `cd abra` and run `bats ...` directly to trigger specific subsets of tests. See below for more tips.
 
 ### Running them locally
 
@@ -424,7 +404,11 @@ The project with all changes and discussions is [here](XXX).
 
 A huge thanks to all our `abra` hackers for this release 💖
 
-$git_shortlog_output
+```
+...
+```
+
+$CHANGELOG
 ```
 
 #### General announcement text

docs/maintainers/handbook.md

@@ -353,143 +353,111 @@ Therefore, we maintain an additional version part, in front of the project versi
 
 In all cases, we follow the semver semantics. So, if we upgrade the Gitea recipe from `1.14.3` to `1.15.3`, we still publish `1.1.0+1.15.3` as our recipe version. In this case, we skipped a few patch releases but it was all backwards compatible, so we only increment the minor version part.
 
-## Upstream released a new version, how do I upgrade the recipe?
+## How do I release a new recipe version?
 
-`abra` can help you so you don't have to update the image tags in the compose files manually. Let us for example make an upgrade of [Wordpress](https://git.coopcloud.tech/coop-cloud/wordpress). Run:
+The commands uses for dealing with recipe versioning in `abra` are:
+
+- `abra recipe upgrade`: upgrade the image tags in the compose configs of a recipe
+- `abra recipe sync`: upgrade the deploy labels to match the new recipe version
+- `abra recipe release`: publish a git tag for the recipe repo
+
+The `abra` recipe publishing commands have been designed to complement a semi-automatic workflow. If `abra` breaks or doesn't understand what is going on, you can always finish the process manually with a few Git commands and a bit of luck. We designed `abra` to support this way due to the chaotic nature of container publishing versioning schemes.
+
+Let's take a practical example, publishing a new version of [Wordpress](https://git.coopcloud.tech/coop-cloud/wordpress).
+
+If we run `abra recipe upgrade wordpress` (at time of running), we end up with a prompt to upgrade Wordpress to `5.9.0`. We can skip the database upgrade for now. Here is what that looks like:
 
 ```
-$ abra recipe upgrade wordpress
+➜  ~ abra recipe upgrade wordpress
+? upgrade to which tag? (service: app, image: wordpress, tag: 5.8.3) 5.9.0
+? upgrade to which tag? (service: db, image: mariadb, tag: 10.6) skip
+WARN[0004] not upgrading mariadb, skipping as requested
 ```
 
-If we run `abra recipe upgrade wordpress` (at time of running), we end up with a prompt to upgrade Wordpress to `6.9.1` and mariadb to `12.2`. Commit the changes when asked. Here is what that looks like:
+Now, what happened? `abra` queried the upstream container repositories of all the images listed in the Wordpress recipe configuration and checked if there are new tags available. Once you make some choices on the prompt, `abra` will update the recipe configurations. Let's take a look by running `cd ~/.abra/recipes/wordpress && git diff`:
 
-```
-$ abra recipe upgrade wordpress
-? upgrade to which tag? (service: app, image: wordpress, tag: 6.9.0) 6.9.1
-? upgrade to which tag? (service: db, image: mariadb, tag: 12.1) 12.2
-WARN unable to read tag for image atmoz/sftp, is it missing? skipping upgrade for ftp
-INFO wordpress currently has these unstaged changes 👇
+```diff
 diff --git a/compose.yml b/compose.yml
-index b3c3871..07d0a54 100644
+index 1618ef5..6cd754d 100644
 --- a/compose.yml
 +++ b/compose.yml
 @@ -3,7 +3,7 @@ version: "3.8"
- 
+
  services:
    app:
--    image: "wordpress:6.9.0"
-+    image: "wordpress:6.9.1"
+-    image: "wordpress:5.8.3"
++    image: "wordpress:5.9.0"
      volumes:
        - "wordpress_content:/var/www/html/wp-content/"
      networks:
-@@ -65,7 +65,7 @@ services:
-         - "coop-cloud.${STACK_NAME}.version=2.17.0+6.9.0"
- 
-   db:
--    image: "mariadb:12.1"
-+    image: "mariadb:12.2"
-     volumes:
-       - "mariadb:/var/lib/mysql"
-     networks:
-? commit changes? Yes
-INFO committed changes as 'chore: update image tags'
 ```
 
-Here is a rundown of what happened:
-
-- `abra` queried the upstream container repositories of all the images listed in the Wordpress recipe configuration and checked if there are new tags available
-- Once you make some choices on the prompt, `abra` will update the recipe configurations and show you the diff
-- If you choose to, a new commit containing the changes will be created
-
 !!! warning "Here be versioning dragons"
 
-    As you can see with `atmoz/sftp`, `abra` doesn't understand all image tags unfortunately. There are limitations which we're still running into. You can pass `-a` to have `abra` list all available image tags from the upstream repository and then make a choice manually. See [`tagcmp`](https://git.coopcloud.tech/toolshed/tagcmp) for more info on how we implement image parsing.
+    `abra` doesn't understand all image tags unfortunately. There are limitations which we're still running into. You can pass `-a` to have `abra` list all available image tags from the upstream repository and then make a choice manually. See [`tagcmp`](https://git.coopcloud.tech/toolshed/tagcmp) for more info on how we implement image parsing.
 
-You will probably want to [test the upgrade](#how-are-new-recipe-versions-tested) now. Afterwards, you can release the upgraded recipe as outlined in the next section.
-
-## How do I release a new recipe version?
-
-!!! warning "Watch out for old versions of abra 🚧"
-
-    This section describes the release process with `abra` version `0.13.x`.
-    For older version, you will need to use these commands:
-    ```
-    abra recipe sync <recipe>
-    abra recipe release <recipe> -p
-    ```
-
-After upgrading the image tags and/or making some other changes to the recipe, you may want to release a new version. This can be done with
+Next, we need to update the label in the recipe, we can do that with `abra recipe sync wordpress`. You'll be prompted by a question asking what kind of upgrade this is. Take a moment to read the output and if it still doesn't make sense, read [this](/maintainers/handbook/#how-are-recipes-are-versioned). Since we're upgrading from `5.8.3` -> `5.9.0`, it is a minor release, so we choose `minor`:
 
 ```
-abra recipe release
+➜  wordpress (master) ✗ abra recipe sync wordpress
+...
+INFO[0088] synced label coop-cloud.${STACK_NAME}.version=1.1.0+5.9.0 to service app
 ```
 
-Continuing our Wordpress example above, we can start the release process with `abra recipe release wordpress`. You'll be prompted by a question asking what kind of upgrade this is. Take a moment to read the output and if it still doesn't make sense, read [this](/maintainers/handbook/#how-are-recipes-are-versioned). Since we're upgrading Wordpress from `6.9.0` -> `6.9.1`, and the minor MariaDB upgrade does not change anything in our case, we consider this a `patch` release. Next, you will be prompted for release notes. Since there are no breaking changes or otherwise noteworthy changes in this upgrade, you can leave them empty.
+Once again, we can run `cd ~/.abra/recipes/wordpress && git diff` to see what `abra` has done for us:
 
-Here is what you will see:
+```diff
+diff --git a/compose.yml b/compose.yml
+index 1618ef5..4a08db6 100644
+--- a/compose.yml
++++ b/compose.yml
+@@ -3,7 +3,7 @@ version: "3.8"
 
-```
-$ abra recipe release wordpress
-
-You need to make a decision about what kind of an update this new recipe
-version is. If someone else performs this upgrade, do they have to do some
-migration work or take care of some breaking changes? This can be signaled in
-the version you specify on the recipe deploy label and is called a semantic
-version.
-
-The latest published version is 2.17.0+6.9.0.
-
-┏━━━━━━━━━┳━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━┓
-┃ SERVICE ┃  2.17.0+6.9.0   ┃ PROPOSED CHANGES ┃
-┣━━━━━━━━━╋━━━━━━━━━━━━━━━━━╋━━━━━━━━━━━━━━━━━━┫
-┃ app     ┃ wordpress:6.9.0 ┃ wordpress:6.9.1  ┃
-┃ db      ┃ mariadb:12.1    ┃ mariadb:12.2     ┃
-┗━━━━━━━━━┻━━━━━━━━━━━━━━━━━┻━━━━━━━━━━━━━━━━━━┛
-
-Here is a semver cheat sheet (more on https://semver.org):
-
-    major: new features/bug fixes, backwards incompatible (e.g 1.0.0 -> 2.0.0).
-           the upgrade won't work without some preparation work and others need
-           to take care when performing it. "it could go wrong".
-
-    minor: new features/bug fixes, backwards compatible (e.g. 0.1.0 -> 0.2.0).
-           the upgrade should Just Work and there are no breaking changes in
-           the app and the recipe config. "it should go fine".
-
-    patch: bug fixes, backwards compatible (e.g. 0.0.1 -> 0.0.2). this upgrade
-           should also Just Work and is mostly to do with minor bug fixes
-           and/or security patches. "nothing to worry about".
-
-? select recipe version increment type patch
-INFO synced label coop-cloud.${STACK_NAME}.version=2.17.1+6.9.1 to service app
-? add release note? (leave empty to skip) 
-INFO new release published: https://git.coopcloud.tech/coop-cloud/wordpress.git/src/tag/2.17.1+6.9.1
+ services:
+   app:
+-    image: "wordpress:5.8.3"
++    image: "wordpress:5.9.0"
+     volumes:
+       - "wordpress_content:/var/www/html/wp-content/"
+     networks:
+@@ -48,7 +48,7 @@ services:
+         #- "traefik.http.routers.${STACK_NAME}.rule=HostRegexp(`{subdomain:.+}.${DOMAIN}`, `${DOMAIN}`)"
+         - "traefik.http.routers.${STACK_NAME}.tls.certresolver=${LETS_ENCRYPT_ENV}"
+         - "traefik.http.routers.${STACK_NAME}.entrypoints=web-secure"
+-        - "coop-cloud.${STACK_NAME}.version=1.0.2+5.8.3"
++        - "coop-cloud.${STACK_NAME}.version=1.1.0+5.9.0"
+         - "backupbot.backup=true"
+         - "backupbot.backup.path=/var/www/html"
 ```
 
-Here is a rundown of what happened:
+You'll notice that `abra` figured out how to upgrade the Co-op Cloud version label according to our choice, `1.0.2` -> `1.1.0` is a minor update.
 
-- with the choice of patch upgrade, `abra` determined the new recipe version label to be `2.17.1+6.9.1`
-- the label is updated in `compose.yml`
-- the changes to the label (and release notes if there are any) are committed
-- a new annotated git tag is created, similar to `git tag -am "chore: publish 2.17.1+6.9.1 release" 2.17.1+6.9.1`
-- commit and tag are pushed to the Wordpress recipe repository
+At this point, we're all set, we can run `abra recipe release --publish wordpress`. This will do the following:
+
+1. run `git commit` the new changes
+1. run `git tag -am "chore: publish 1.1.0+5.9.0 release" 1.1.0+5.9.0` to create a new annotated git tag named `1.1.0+5.9.0` (abra only accepts annotated tags)
+1. run `git push` to publish changes to the Wordpress repository
 
 !!! warning "Here be more SSH dragons"
 
     In order to have `abra` publish changes for you automatically, you'll have to have write permissons to the git.coopcloud.tech repository and your account must have a working SSH key configuration. `abra` will use the SSH based URL connection details for Git by automagically creating an `origin-ssh` remote in the repository and pushing to it.
 
-`abra` recipe publishing command have been designed to complement a semi-automatic workflow. If `abra` breaks or doesn't understand what is going on, you can always finish the process manually with a few Git commands and a bit of luck. We designed `abra` to support this way due to the chaotic nature of container publishing versioning schemes.
+Here is the output:
 
-If you do this process manually, two important things to note for the release to be picked up in the catalogue:
+```
+WARN[0000] discovered 1.1.0+5.9.0 as currently synced recipe label
+WARN[0000] previous git tags detected, assuming this is a new semver release
+? current: 1.0.2+5.8.3, new: 1.1.0+5.9.0, correct? Yes
+new release published: https://git.coopcloud.tech/coop-cloud/wordpress/src/tag/1.1.0+5.9.0
+```
 
-- create an annotated git tag using `git tag -a`
-- push the tag with `git push --tags`
+And once more, we can validate this tag has been created with `cd ~/.abra/recipes/wordpress && git tag -l`.
 
 ## How are new recipe versions tested?
 
 This is currently a manual process. Our best estimates are to do a backup and run a test deployment and see how things go. [We are working on improving this](https://git.coopcloud.tech/toolshed/-/projects/31).
 
-Following the [entry above](/maintainers/handbook/#how-do-i-release-a-new-recipe-version), before running `abra recipe release <recipe>`, you can deploy the new version of the recipe. You find an app that relies on this recipe and pass `-C/--chaos` to `deploy` so that it accepts the locally unstaged changes.
+Following the [entry above](/maintainers/handbook/#how-do-i-release-a-new-recipe-version), before running `abra recipe release --publish <recipe>`, you can deploy the new version of the recipe. You find an app that relies on this recipe and pass `-C/--chaos` to `ugrade` so that it accepts the locally unstaged changes.
 
 !!! warning "Here be more SSH dragons"
 
@@ -520,11 +488,13 @@ In the root of your recipe repository, run the following (if the folder doesn't
 mkdir -p release
 ```
 
-And then create or append to the text file `release/next`. This file will be used when running `abra recipe release`. Abra will ask you to use these when running `abra recipe release` and rename it to the release tag, e.g. `release/1.1.0+5.9.0`.
+And then create a text file which corresponds to the version release, e.g. `1.1.0+5.9.0` and write some notes. `abra` will show these when another operator runs `abra app deploy` / `abra app upgrade`.
 
-You can also enter the release notes interactively when running `abra recipe release`.
+You can also add release notes for the next release into a special file `release/next`. This file will be used when running `abra recipe release`.
 
-`abra` will show these notes when another operator runs `abra app deploy` / `abra app upgrade`.
+!!! warning "Watch out for old versions of `abra` 🚧"
+
+    This feature is only available in the >= 0.9.x series of `abra`.
 
 ## How do I know whether to accept version upgrades when running `abra recipe upgrade <something>`?
 
@@ -612,23 +582,6 @@ Add the `# generate=false` comment
 SECRET_JWT_SECRET_VERSION=v1 # generate=false
 ```
 
-## How do I specify the charset for a specific secret generation
-
-!!! warning "Watch out for old versions of `abra` 🚧"
-
-    This feature is only available in the >= 0.10.x series of `abra`.
-
-```
-SECRET_JWT_SECRET_VERSION=v1 # charset=default,special
-```
-
-Options are:
-
-* `special`: [source](https://github.com/decentral1se/passgen/blob/8404cb922dea92efa8c3514f0ec8c37ce12a880f/const.go#L22C29-L22C43)
-* `safespecial`: [source](https://git.coopcloud.tech/toolshed/abra/src/commit/6abaf7a094df1a96599af2c4cbae1769821ad17c/pkg/secret/secret.go#L182)
-* `default,special`: mix of `default` and `special`
-* `default,safespecial`: mix of `default` and `safespecial`
-
 ## How do I change secret generation length?
 
 It is possible to tell `abra` which length it should generate secrets with from your recipe config.
@@ -650,6 +603,13 @@ of passwords which admins have to type out in database shells.
 
 ## How do I change secret generation characters?
 
+!!! warning "Watch out for old versions of `abra` 🚧"
+
+    This feature is only available in the >= 0.10.x series of `abra`. 
+    The generation of `hex` secrets is only available in the >= 0.12.x series of `abra`.
+    The generation of `bytes` secrets is only available in the >= 0.13.x series of `abra`.
+
+
 It is also possible to tell `abra` which characters it should use to generate secrets with from your recipe config.
 
 You do this by adding an additional modifier in the inline comment on the secret definition in the `.env.sample` / `.env` file.
@@ -670,10 +630,52 @@ The possible Values are:
 | `default,special`                            | `abcdefghijkmnopqrstuvwxyzABCDEFGHJKLMNPQRSTUVWXYZ23456789!@#$%^&*_-+=` | Uses uppercase letters, lowercase letters and numbers and special characters              |
 | `default,safespecial`                        | `abcdefghijkmnopqrstuvwxyzABCDEFGHJKLMNPQRSTUVWXYZ23456789!@#%^&*_-+=`  | Uses uppercase letters, lowercase letters and numbers and console safe special characters |
 | `default`                                    | `abcdefghijkmnopqrstuvwxyzABCDEFGHJKLMNPQRSTUVWXYZ23456789`             | Uses uppercase letters, lowercase letters and numbers                                     |
+| `hex`                                        | `0123456789abcdef`                                                      | Uses only hexadecimal characters (0-9, a-f)                                                |
+| `bytes`                                      | N/A (generates random bytes)                                            | Generates random bytes instead of character-based passwords (requires `length` modifier, automatically uses `encoding=base64`) |
 | any other value or not setting one will be treated as `default` | `abcdefghijkmnopqrstuvwxyzABCDEFGHJKLMNPQRSTUVWXYZ23456789`             | Uses uppercase letters, lowercase letters and numbers                                     |
 
 The setting does only apply when you also set a length modifier to the secret (documented [here](/maintainers/handbook/#how-do-i-change-secret-generation-length)), so it is not applicable for the "easy to remember word" style generator that used when you don't set a length.
 
+## How do I encode generated secrets?
+
+!!! warning "Watch out for old versions of `abra` 🚧"
+
+    This feature is only available in the >= 0.13.x series of `abra`.
+
+You can tell `abra` to encode generated secrets using the `encoding` modifier in the inline comment on the secret definition in the `.env.sample` / `.env` file.
+
+Here is an example:
+
+```bash
+SECRET_API_KEY_VERSION=v1 # length=32 encoding=base64
+```
+
+This will generate a 32-character secret and then base64-encode it before storing.
+
+Currently supported encoding options:
+
+- `base64`: Base64-encodes the generated secret value
+
+**Note:** When using `charset=bytes`, the encoding automatically defaults to `base64` even if not explicitly specified, as raw binary data needs to be encoded for safe storage.
+
+## How do I add a prefix to generated secrets?
+
+!!! warning "Watch out for old versions of `abra` 🚧"
+
+    This feature is only available in the >= 0.13.x series of `abra`.
+
+You can tell `abra` to add a prefix to generated secrets using the `prefix` modifier in the inline comment on the secret definition in the `.env.sample` / `.env` file.
+
+Here is an example:
+
+```bash
+SECRET_APP_KEY_VERSION=v1 # length=32 charset=bytes prefix=base64:
+```
+
+This will generate a 32-byte random key, base64-encode it, and prefix it with `base64:`. This is useful for applications like Laravel that expect secrets in a specific format.
+
+The prefix is applied after any encoding transformations.
+
 ## How do I configure backup/restore?
 
 From the perspective of the recipe maintainer, backup/restore is just more