Create HTTP web service which generates Recipes Catalogue dynamically #604

New Issue

2024-04-02T09:04:00Z

basebuilder commented

2024-04-02 09:04:00 +00:00

I mentioned to @decentral1se last week the idea of generating the recipes-catalogue-json in a new fashion which is dynamically via a minimal HTTP server which could be found at recipes.coopcloud.tech/api or such.

If I understand correctly, the current flow revolves around manually running abra catalogue generate and committing that to the repo and pushing it. My proposal, I think, would allow that to just happen automagically whenever a new recipe version is published.

This would benefit from migration to coopcloud.tech/recipes organization detailed in #569 and possibly relate to #599 and also benefit #603

I mentioned to @decentral1se last week the idea of generating the `recipes-catalogue-json` in a new fashion which is *dynamically* via a minimal HTTP server which could be found at `recipes.coopcloud.tech/api` or such. If I understand correctly, the current flow revolves around manually running `abra catalogue generate` and committing that to the repo and pushing it. My proposal, I think, would allow that to *just happen automagically* whenever a new recipe version is published. This would benefit from migration to `coopcloud.tech/recipes` organization detailed in #569 and possibly relate to #599 and also benefit #603

basebuilder added the proposal label 2024-04-02 09:04:00 +00:00

basebuilder referenced this issue

2024-04-02 09:04:49 +00:00

Retiring recipes #603

decentral1se commented

2024-04-02 20:01:42 +00:00

Thanks for opening! Under "Operator Collaboration / General Version Improvement" on coop-cloud/organising#583 and in the last fedi meeting (notes: https://docs.coopcloud.tech/federation/minutes/2024-03-29/) we touched on a few ways that we could remove the need for the catalogue repository requirement on the abra side. But we didn't have an answer for how to still provide pre-computed data for catalogue web UI. This /api could deffo sort this. And I'm curious on the overlap between ideas of @p4u1 and your ideas @basebuilder. Hopefully to be discussed at the next fedi meeting 🎉

Thanks for opening! Under "Operator Collaboration / General Version Improvement" on https://git.coopcloud.tech/coop-cloud/organising/issues/583 and in the last fedi meeting (notes: https://docs.coopcloud.tech/federation/minutes/2024-03-29/) we touched on a few ways that we could remove the need for the catalogue repository requirement on the `abra` side. But we didn't have an answer for how to still provide pre-computed data for catalogue web UI. This `/api` could deffo sort this. And I'm curious on the overlap between ideas of @p4u1 and your ideas @basebuilder. Hopefully to be discussed at the next fedi meeting 🎉

👍 1

3wordchant commented

2024-04-02 20:29:56 +00:00

If I understand correctly, the current flow revolves around manually running abra catalogue generate and committing that to the repo and pushing it. My proposal, I think, would allow that to just happen automagically whenever a new recipe version is published.

It's already automatic for some (many? most?) recipes.

abra recipe sync wordpress / abra recipe release wordpress → generates and pushes a git tag
Drone notices the new tag (because of this config) and triggers a build of auto-recipes-catalogue-json
auto-recipes-catalogue-json rebuilds the catalogue and makes a commit to recipes-catalogue-json
Drone notices new commit (config) and auto-deploys new recipe version to https://recipes.coopcloud.tech/recipes.json

There's also a daily Drone job which runs auto-recipes-catalogue-json which will pick up recipes which don't have the drone/downstream step (either ones created not using abra recipe new / the repo template, or older recipes which didn't have that step added afterwards).

> If I understand correctly, the current flow revolves around manually running `abra catalogue generate` and committing that to the repo and pushing it. My proposal, I think, would allow that to just happen automagically whenever a new recipe version is published. It's already automatic for some (many? most?) recipes. 1. `abra recipe sync wordpress` / `abra recipe release wordpress` → generates and pushes a git tag 2. Drone notices the new tag ([because of this config](https://git.coopcloud.tech/coop-cloud/wordpress/src/branch/main/.drone.yml#L28-L40)) and triggers [a build of `auto-recipes-catalogue-json`](https://build.coopcloud.tech/coop-cloud/auto-recipes-catalogue-json) 3. `auto-recipes-catalogue-json` rebuilds the catalogue and makes a commit to [`recipes-catalogue-json`](https://git.coopcloud.tech/coop-cloud/recipes-catalogue-json) 4. Drone notices new commit ([config](https://git.coopcloud.tech/coop-cloud/recipes-catalogue-json/src/branch/main/.drone.yml#L5-L12)) and auto-deploys new recipe version to https://recipes.coopcloud.tech/recipes.json There's also a daily Drone job which runs `auto-recipes-catalogue-json` which will pick up recipes which don't have the `drone/downstream` step (either ones created not using `abra recipe new` / the repo template, or older recipes which didn't have that step added afterwards).

👀 2

3wordchant commented

2024-04-02 20:39:28 +00:00

The main issues I see with the current set-up are:

It's not documented
It means that setting up a new recipe catalogue, for #139, would currently require configuring up a lot of CD.

Copying the above comment into the TODO on Maintainer Handbook could be a start towards addressing (1).

For (2), I think it would be interesting to understand more about the use-cases for 3rd-party recipe catalogues, and maybe make some more progress on the recipe-generation code itself first (i.e. how could we make abra catalogue generate work if .abra/recipes contains recipes which aren't in git.coopcloud.tech/coop-cloud).

The main issues I see with the current set-up are: 1. It's not documented 2. It means that setting up a new recipe catalogue, for #139, would currently require configuring up a lot of CD. Copying the above comment into [the TODO on Maintainer Handbook](https://docs.coopcloud.tech/maintainers/handbook/#how-does-automatic-catalogue-regeneration-work) could be a start towards addressing (1). For (2), I think it would be interesting to understand more about the use-cases for 3rd-party recipe catalogues, and maybe make some more progress on the recipe-generation code itself first (i.e. how could we make `abra catalogue generate` work if `.abra/recipes` contains recipes which aren't in git.coopcloud.tech/coop-cloud).

basebuilder added this to the (deleted) milestone 2024-04-15 07:09:00 +00:00

decentral1se commented

2025-01-03 20:53:19 +00:00

It's unlikely that we'll be switching over to a whole new system like this so let's close this one off for now. There are more and more pain points coming up with the release process, so I think it will definitely be something to focus on after the next abra release comes out. As mentioned above, this will most likely be a social process of figuring out what people need to be improved here.

It's unlikely that we'll be switching over to a whole new system like this so let's close this one off for now. There are more and more pain points coming up with the release process, so I think it will definitely be something to focus on after the next `abra` release comes out. As mentioned above, this will most likely be a social process of figuring out what people need to be improved here.

decentral1se closed this issue

2025-01-03 20:53:19 +00:00

decentral1se commented

2025-08-29 10:38:31 +00:00

I've heard a few hints at this and if it should be a thing, re-opening!

decentral1se reopened this issue

2025-08-29 10:38:31 +00:00

decentral1se added this to the (deleted) project 2025-08-29 10:38:38 +00:00

decentral1se commented

2025-09-29 12:46:54 +00:00

Starting to play around with something for this: https://git.coopcloud.tech/toolshed/almanac

decentral1se moved this to In Progress in Recipe release stability on 2025-10-01 13:15:15 +00:00