Performance issue with retrieving recipes #377
No Assignees
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: toolshed/organising#377
Reference in New Issue
Block a user
No description provided.
Delete Branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Disco bash one-liner/script to move all the recipes to a new gitea organisation, e.g.
coop-cloud-recipes.Perhaps:
coop-cloudnamespace?coop-cloud-chaos-patchsorg to thecoop-cloudorg and have all the co-op cloud docker images there too? (see #369)Yeh, that sounds good! Potentially gnarly bit of coordination 😬
Is there such a thing as use repository tags or is there some other way to filter?
@decentral1se you mean tag/filter to avoid needing a separate organisation, or to help with the one-off move?
If it's the one-off, I was thinking of just reusing the list in the abra source.
If it's instead of a separate organisation, I'm not aware of anything. We could feasibly add hashtags to the repo descriptions and use those in
catalogue generateto skip non-recipe repos, but it wouldn't give the benefit of a separate organisation in terms of making it easier to look on Gitea "what non-recipe repos do we have".I don't mind doing the leg-work, either way.
Coördination-wise, yeah, I guess this could be a fairly breaking change, depending on how well Gitea's magic redirection-from-old-repo-names feature works. If the answer is "badly", we'd need to roll a new
abrarelease, and possibly One, Even More Final abra-bash release, and make a big noise for folks to update.If that's the case an alternative could be moving the non-recipe repos instead? And if Gitea handles renaming an org better than moving a recipe then you could for example rename this one to
recipesand move other repos to new orgs liketooling,organisingandwebsites. Just ideas...Interesting idea, thanks @mayel! Maybe the next step is to do a couple of tests, see whether Gitea handles a repo move or an organisation rename better.
Great plan 💯
Sad news, Gitea doesn't handle either transferring a repo or renaming an org very well with SSH connections -
Gitea: Unauthorizedin both cases.Maybe a new
coop-cloud-organisationorganisation is the move? Or something less repetitive? 😅 Personally just one more seems good.To come on my "would tags work?" thing, I meant this thing called "topics":
Which produces this
?q=recipessearch:So, I guess that means we can filter by that too with
abraand other tooling.Workable?
Not against a big move churn in principle but seems like it wouldn't be much fun.
@3wordchant @mayel any further thoughts on coop-cloud/organising#377 (comment)?
It would indeed be great to get rid of the
CatalogueSkipListhack.consider moving recipe repos to e.g. coop-cloud-recipes orgto Consider moving recipe repos to e.g. coop-cloud-recipes orgGentle reminder @3wordchant @mayel 👀
Thanks for the ping! Yeah I see what you mean about tags / topics, promising so far.
My questions (will research myself unless someone beats me to it):
Maybe tags / topics =
recipe&dont-generateI'm definitely thinking to make a move on this sooner than later following this approach btw!
@3wordchant
Diving into the code finally... opened https://github.com/go-gitea/gitea/issues/27456!
The catalogue generation codes makes one additional request per-repo to the topics endpoint now.
WIP PR is coop-cloud/abra#368, it already works.
I added the
recipetopic to https://git.coopcloud.tech/coop-cloud/example but then indeed, as mentioned in coop-cloud/organising#377 (comment) I needed to special caseexample.Seems fine? I'm happy to do the manual labelling labours.
Moar ping @mayel @3wordchant any further concerns? Can pick this up otherwise.
OK, went ahead with it as it doesn't seem controversial at this point. Tagged every recipe by hand. Was unable to tag https://git.coopcloud.tech/coop-cloud/matomo with
recipetopic? Can anyone do it? Idk maybe a permissions thing 🤷♀️There is a performance cost to this. After commit 76f95fe766 the "retrieving recipe repos list" step of
abra catalogue generatenow takes around 28 seconds instead of 2 seconds. This change also relies on Gitea features and an undocumented convention, which would make rolling alternative app catalogues (see #139) harder.I think this mechanism should rely more on previous
recipes.jsoncontent to generate the updated version - it already includes repository url for each recipe. That way for an alternative catalogue it would be possible to push new repos torecipes.jsonby hand and also have them kept up to date. Checking organization's projects and their topics could then proceed only for those repositories that weren't included before in a previous pass (so to proceed like it does now, but hopefully with less requests).Another solution could be to use a configuration file that lists ignored repositories and organizations that are watched to find recipes. I hear that some configuration file support is coming with #303, but maybe this should be stored somewhere in coop-cloud/recipes-catalogue-json repo instead.
I might be misunderstanding the catalogue though, it's not yet documented throughly (#523). I like the idea of custom catalogues, and although it could be easier for the project to decide on having "a single source of truth", I hope in the end it will actually get easier to decentralize the catalogue too without relying on a deployment identical to the official one. Having a configurable path to
recipes-catalogue-jsonrepo inabraand relying more onrecipes.jsonoverall will go a long way in that direction.Re-opening & renaming for coop-cloud/organising#377 (comment)
Consider moving recipe repos to e.g. coop-cloud-recipes orgto Performance issue with retrieving recipes@wykwit please DM me on Matrix and let's plan out something? It'd be good to have this working fast again and certainly the points you mention are important. Happy to support you making some progress on this.
decentral1se referenced this issue2024-03-27 06:57:14 +00:00
decentral1se referenced this issue2024-03-28 13:54:55 +00:00
coop-cloud/organising#569
Your feedback please!
(Reverting to just listing the single org with only recipes in it.)
decentral1se referenced this issue from toolshed/abra2024-10-21 20:23:46 +00:00