Reinstate automatic catalogue generation #4

Closed
opened 2022-11-13 06:20:43 +00:00 by 3wordchant · 11 comments
Owner

Ideally the catalogue should be regenerated automatically.

Wired: cronjob like we used to have
Inspired: webhook whenever there's a new released version of a recipe

Ideally the catalogue should be regenerated automatically. Wired: cronjob like we used to have Inspired: webhook whenever there's a new released version of a recipe
Author
Owner

Current status:

TODO:

  • Why are commits to recipes-catalogue-json showing up as from me? (Example). Drone is authenticating using a key added to the abra-bot repo, shouldn't it show up as abra-bot? Or maybe Drone is picking up git "author" settings from the source repo (in this case, auto-recipes-catalogue-json)
  • Why is Drone failing to look up digests for a bunch of the recipes? In the same example, collabora (and many other recipes) have switched from a digest to "unknown". Or maybe this isn't too critical, as the image digests seem to be changing upstream anyway 🙄
  • How do we trigger the auto-recipes-catalogue-json build whenever there's a new release of a recipe? Ideally it'd be nice to avoid needing to add to .drone.yml in all the recipe repos; maybe a global webhook linked up to something? (n8n? ducks). This might become easier if all the recipes are in a separate organisation, see coop-cloud/organising#377.
Current status: - We have a docker image for abra again, [`git.coopcloud.tech/coop-cloud/abra`](https://git.coopcloud.tech/coop-cloud/-/packages/container/abra/dev), automatically published to `dev` tag on push (and should make tagged releases whenever we push a tag) - [The `drone-abra` Drone plugin](https://git.coopcloud.tech/coop-cloud/drone-abra/) is up-to-date as well (still needs moving to git.coopcloud.tech from Docker Hub tho) - `auto-apps-json` is [renamed to `auto-recipes-catalogue-json`](https://git.coopcloud.tech/coop-cloud/auto-recipes-catalogue-json/), and configured to do a Drone build to generate and publish the catalogue ([example](https://build.coopcloud.tech/coop-cloud/auto-recipes-catalogue-json/80/1/2)) -- it's currently pushing to [a `testing` branch in `recipes-catalogue-json`](https://git.coopcloud.tech/coop-cloud/recipes-catalogue-json/src/branch/testing). TODO: - Why are commits to `recipes-catalogue-json` showing up as from me? ([Example](https://git.coopcloud.tech/coop-cloud/recipes-catalogue-json/commit/6be5e7ecd556ac3c0ca6b38041dfd1f4ed5be09a)). Drone is authenticating using a key added to the `abra-bot` repo, shouldn't it show up as `abra-bot`? Or maybe Drone is picking up git "author" settings from the source repo (in this case, `auto-recipes-catalogue-json`) - Why is Drone failing to look up digests for a bunch of the recipes? In [the same example](https://git.coopcloud.tech/coop-cloud/recipes-catalogue-json/commit/6be5e7ecd556ac3c0ca6b38041dfd1f4ed5be09a), `collabora` (and many other recipes) have switched from a digest to `"unknown"`. Or maybe this isn't too critical, as the image digests seem to be changing upstream anyway 🙄 - How do we trigger the `auto-recipes-catalogue-json` build whenever there's a new release of a recipe? Ideally it'd be nice to avoid needing to add to `.drone.yml` in all the recipe repos; maybe a global webhook linked up to something? (`n8n`? _ducks_). This might become easier if all the recipes are in a separate organisation, see coop-cloud/organising#377.
Member

Why are commits to recipes-catalogue-json showing up as from me? (Example). Drone is authenticating using a key added to the abra-bot repo, shouldn't it show up as abra-bot? Or maybe Drone is picking up git "author" settings from the source repo (in this case, auto-recipes-catalogue-json)

Did you create the key while authenticated as yourself? Typically you have to create a seperate user and create a key once logged in with it

> Why are commits to recipes-catalogue-json showing up as from me? (Example). Drone is authenticating using a key added to the abra-bot repo, shouldn't it show up as abra-bot? Or maybe Drone is picking up git "author" settings from the source repo (in this case, auto-recipes-catalogue-json) Did you create the key while authenticated as yourself? Typically you have to create a seperate user and create a key once logged in with it
Author
Owner

thanks for the reply @mayel!

Did you create the key while authenticated as yourself? Typically you have to create a seperate user and create a key once logged in with it

I generated a new SSH key using ssh-keygen and added it to the abra-bot user on Gitea.

thanks for the reply @mayel! > Did you create the key while authenticated as yourself? Typically you have to create a seperate user and create a key once logged in with it I generated a new SSH key using `ssh-keygen` and added it to the `abra-bot` user on Gitea.
Member
  • Why is Drone failing to look up digests for a bunch of the recipes? In the same example, collabora (and many other recipes) have switched from a digest to "unknown". Or maybe this isn't too critical, as the image digests seem to be changing upstream anyway 🙄

I think this is related to coop-cloud/organising#368 and it is caused by a rate limit.

> - Why is Drone failing to look up digests for a bunch of the recipes? In [the same example](https://git.coopcloud.tech/coop-cloud/recipes-catalogue-json/commit/6be5e7ecd556ac3c0ca6b38041dfd1f4ed5be09a), `collabora` (and many other recipes) have switched from a digest to `"unknown"`. Or maybe this isn't too critical, as the image digests seem to be changing upstream anyway 🙄 I think this is related to https://git.coopcloud.tech/coop-cloud/organising/issues/368 and it is caused by a rate limit.
Author
Owner

Thanks @moritz! I thought the docker login step would fix that, but perhaps it only gets us a higher rate limit, which still isn't high enough to look up all the images in the catalogue?

I agree with the comment by @decentral1se on Matrix, though:

great stuff 👏 one passing by comment is that i'm thinking we should maybe abandon the digest generation on the catalogue side of the things and instead let people pin the build digests in their recipe configus (this approach was shown somewhere in a ticket)

Maybe we just drop digest generation entirely, for now?

Thanks @moritz! I thought [the `docker login` step](https://git.coopcloud.tech/coop-cloud/auto-recipes-catalogue-json/src/branch/main/.drone.yml#L22) would fix that, but perhaps it only gets us a higher rate limit, which still isn't high enough to look up all the images in the catalogue? I agree with the comment by @decentral1se on Matrix, though: > great stuff 👏 one passing by comment is that i'm thinking we should maybe abandon the digest generation on the catalogue side of the things and instead let people pin the build digests in their recipe configus (this approach was shown somewhere in a ticket) Maybe we just drop digest generation entirely, for now?
Owner

great stuff 👏 one passing by comment is that i'm thinking we should maybe abandon the digest generation on the catalogue side of the things and instead let people pin the build digests in their recipe configus (this approach was shown somewhere in a ticket)

Maybe we just drop digest generation entirely, for now?

coop-cloud/organising#379

> > great stuff 👏 one passing by comment is that i'm thinking we should maybe abandon the digest generation on the catalogue side of the things and instead let people pin the build digests in their recipe configus (this approach was shown somewhere in a ticket) > > Maybe we just drop digest generation entirely, for now? https://git.coopcloud.tech/coop-cloud/organising/issues/379
Owner

How do we trigger the auto-recipes-catalogue-json build whenever there's a new release of a recipe? Ideally it'd be nice to avoid needing to add to .drone.yml in all the recipe repos; maybe a global webhook linked up to something? (n8n? ducks). This might become easier if all the recipes are in a separate organisation, see coop-cloud/organising#377.

I feel like if we had a Not Gnarly way to template the .drone.yml files for all recipes from a central repository, then it wouldn't be too bad? We already have this config for the recipe CI, so adding a separate stanza for per-tag catalogue generation doesn't seem too bad? I think we can get inter-repository webhooks going, so when a tag gets pushed to a recipe repo, it pings the catalogue generation repo. It's drone plugins somewhere afaicr.

> How do we trigger the auto-recipes-catalogue-json build whenever there's a new release of a recipe? Ideally it'd be nice to avoid needing to add to .drone.yml in all the recipe repos; maybe a global webhook linked up to something? (n8n? ducks). This might become easier if all the recipes are in a separate organisation, see coop-cloud/organising#377. I feel like if we had a Not Gnarly way to template the `.drone.yml` files for all recipes from a central repository, then it wouldn't be too bad? We already have this config for the recipe CI, so adding a separate stanza for per-tag catalogue generation doesn't seem too bad? I think we can get inter-repository webhooks going, so when a tag gets pushed to a recipe repo, it pings the catalogue generation repo. It's drone plugins somewhere afaicr.
Author
Owner

Oh yeah forgot to update here, another mass-borking later, and we're already doing this, using the Drone downstream plugin, e.g. for vaultwarden

Oh yeah forgot to update here, another mass-borking later, and we're already doing this, using the Drone `downstream` plugin, e.g. [for vaultwarden](https://git.coopcloud.tech/coop-cloud/vaultwarden/src/branch/main/.drone.yml#L26-L41)
Author
Owner

Could maybe close this off now? It's pretty automagic again.

Could maybe close this off now? It's pretty automagic again.
Owner

🤟

🤟
Owner

We'll soon (again) have automatic recipe generation, see coop-cloud/recipes-catalogue-json#4. The usual process from there is to run abra app upgrade; unsure how to translate this to server-side where there isn't (usually) a copy of ~/.abra.

This issue is not available anymore.

@moritz open again 🙈

>> We'll soon (again) have automatic recipe generation, see coop-cloud/recipes-catalogue-json#4. The usual process from there is to run abra app upgrade; unsure how to translate this to server-side where there isn't (usually) a copy of ~/.abra. > > This issue is not available anymore. @moritz open again 🙈
Sign in to join this conversation.
No Label
No Milestone
No project
No Assignees
4 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: coop-cloud/recipes-catalogue-json#4
No description provided.