coop-cloud/organising
coop-cloud
/
organising
7
0
Fork 0
Code Issues 156 Pull Requests Projects 2 Releases Activity

Performance issue with retrieving recipes #377

New Issue
Open
opened 2023-01-15 05:39:03 +00:00 by 3wordchant · 20 comments
3wordchant commented 2023-01-15 05:39:03 +00:00
Owner
Copy Link

Disco bash one-liner/script to move all the recipes to a new gitea organisation, e.g. coop-cloud-recipes.

Perhaps:

  1. No more need for the exciting manually-maintained list of non-recipe repos in the coop-cloud namespace?
  2. Easier to see all the non-recipe "operations" recipes at once?
  3. Could move things from coop-cloud-chaos-patchs org to the coop-cloud org and have all the co-op cloud docker images there too? (see #369)
Disco bash one-liner/script to move all the recipes to a new gitea organisation, e.g. `coop-cloud-recipes`. Perhaps: 1. No more need for [the exciting manually-maintained list of non-recipe repos in the `coop-cloud` namespace](https://git.coopcloud.tech/coop-cloud/abra/src/commit/1e83373d35b396241346d078f56408560766ac00/cli/catalogue/catalogue.go#L23-L58)? 2. Easier to see all the non-recipe "operations" recipes at once? 3. Could move things from `coop-cloud-chaos-patchs` org to the `coop-cloud` org and have all the co-op cloud docker images there too? (see #369)
3wordchant added the
help wanted
design
 labels 2023-01-15 05:39:03 +00:00
decentral1se commented 2023-01-16 18:00:16 +00:00
Owner
Copy Link

Yeh, that sounds good! Potentially gnarly bit of coordination 😬

Is there such a thing as use repository tags or is there some other way to filter?

Yeh, that sounds good! Potentially gnarly bit of coordination 😬 Is there such a thing as use repository tags or is there some other way to filter?
3wordchant commented 2023-01-17 21:32:58 +00:00
Author
Owner
Copy Link

@decentral1se you mean tag/filter to avoid needing a separate organisation, or to help with the one-off move?

If it's the one-off, I was thinking of just reusing the list in the abra source.

If it's instead of a separate organisation, I'm not aware of anything. We could feasibly add hashtags to the repo descriptions and use those in catalogue generate to skip non-recipe repos, but it wouldn't give the benefit of a separate organisation in terms of making it easier to look on Gitea "what non-recipe repos do we have".

I don't mind doing the leg-work, either way.

@decentral1se you mean tag/filter to avoid needing a separate organisation, or to help with the one-off move? If it's the one-off, I was thinking of just reusing the list in the abra source. If it's instead of a separate organisation, I'm not aware of anything. We could feasibly add hashtags to the repo descriptions and use those in `catalogue generate` to skip non-recipe repos, but it wouldn't give the benefit of a separate organisation in terms of making it easier to look on Gitea "what non-recipe repos do we have". I don't mind doing the leg-work, either way.
3wordchant commented 2023-01-17 21:34:23 +00:00
Author
Owner
Copy Link

Coördination-wise, yeah, I guess this could be a fairly breaking change, depending on how well Gitea's magic redirection-from-old-repo-names feature works. If the answer is "badly", we'd need to roll a new abra release, and possibly One, Even More Final abra-bash release, and make a big noise for folks to update.

Coördination-wise, yeah, I guess this could be a fairly breaking change, depending on how well Gitea's magic redirection-from-old-repo-names feature works. If the answer is "badly", we'd need to roll a new `abra` release, and possibly One, Even More Final abra-bash release, and make a big noise for folks to update.
mayel commented 2023-01-17 21:57:14 +00:00
Member
Copy Link

I guess this could be a fairly breaking change, depending on how well Gitea's magic redirection-from-old-repo-names feature works. If the answer is "badly"

If that's the case an alternative could be moving the non-recipe repos instead? And if Gitea handles renaming an org better than moving a recipe then you could for example rename this one to recipes and move other repos to new orgs like tooling, organising and websites. Just ideas...

> I guess this could be a fairly breaking change, depending on how well Gitea's magic redirection-from-old-repo-names feature works. If the answer is "badly" If that's the case an alternative could be moving the non-recipe repos instead? And if Gitea handles renaming an org better than moving a recipe then you could for example rename this one to `recipes` and move other repos to new orgs like `tooling`, `organising` and `websites`. Just ideas...
3wordchant commented 2023-01-17 21:58:30 +00:00
Author
Owner
Copy Link

Interesting idea, thanks @mayel! Maybe the next step is to do a couple of tests, see whether Gitea handles a repo move or an organisation rename better.

Interesting idea, thanks @mayel! Maybe the next step is to do a couple of tests, see whether Gitea handles a repo move or an organisation rename better.
🎉 1
decentral1se commented 2023-01-18 10:52:27 +00:00
Owner
Copy Link

Great plan 💯

Great plan 💯
3wordchant commented 2023-01-20 05:42:31 +00:00
Author
Owner
Copy Link

Sad news, Gitea doesn't handle either transferring a repo or renaming an org very well with SSH connections - Gitea: Unauthorized in both cases.

Maybe a new coop-cloud-organisation organisation is the move? Or something less repetitive? 😅 Personally just one more seems good.

Sad news, Gitea doesn't handle either transferring a repo or renaming an org very well with SSH connections - `Gitea: Unauthorized` in both cases. Maybe a new `coop-cloud-organisation` organisation is the move? Or something less repetitive? 😅 Personally just one more seems good.
decentral1se commented 2023-01-21 19:42:49 +00:00
Owner
Copy Link

To come on my "would tags work?" thing, I meant this thing called "topics":

image

Which produces this ?q=recipes search:

https://git.coopcloud.tech/explore/repos?q=recipe&topic=1

So, I guess that means we can filter by that too with abra and other tooling.

Workable?

Not against a big move churn in principle but seems like it wouldn't be much fun.

To come on my "would tags work?" thing, I meant this thing called "topics": ![image](/attachments/cb9e14e8-36d7-4d3d-8510-9c5312da9e72) Which produces this `?q=recipes` search: > https://git.coopcloud.tech/explore/repos?q=recipe&topic=1 So, I guess that means we can filter by that too with `abra` and other tooling. Workable? Not against a big move churn in principle but seems like it wouldn't be much fun.
image.png
9.1 KiB
decentral1se commented 2023-02-08 22:33:29 +00:00
Owner
Copy Link

@3wordchant @mayel any further thoughts on #377 (comment)?

It would indeed be great to get rid of the CatalogueSkipList hack.

@3wordchant @mayel any further thoughts on https://git.coopcloud.tech/coop-cloud/organising/issues/377#issuecomment-15121? It would indeed be great to get rid of the [`CatalogueSkipList`](https://git.coopcloud.tech/coop-cloud/abra/src/commit/1e83373d35b396241346d078f56408560766ac00/cli/catalogue/catalogue.go#L23-L58) hack.
decentral1se changed title from consider moving recipe repos to e.g. coop-cloud-recipes org to Consider moving recipe repos to e.g. coop-cloud-recipes org 2023-02-14 07:26:21 +00:00
decentral1se commented 2023-04-10 21:32:08 +00:00
Owner
Copy Link

Gentle reminder @3wordchant @mayel 👀

Gentle reminder @3wordchant @mayel 👀
3wordchant commented 2023-04-10 21:41:45 +00:00
Author
Owner
Copy Link

Thanks for the ping! Yeah I see what you mean about tags / topics, promising so far.

My questions (will research myself unless someone beats me to it):

  1. Are topics exposed in Gitea's API, i.e. how much code would we have to code to get abra to look at these?
  2. Can we add a topic to coop-cloud/example, and does it show up when we make a new repo from that template?
  3. Is there an easy way of skipping coop-cloud/example itself, or would we need to keep that one special-case in abra?
Thanks for the ping! Yeah I see what you mean about tags / topics, promising so far. My questions (will research myself unless someone beats me to it): 1. Are topics exposed in Gitea's API, i.e. how much code would we have to code to get abra to look at these? 2. Can we add a topic to coop-cloud/example, and does it show up when we make a new repo from that template? 3. Is there an easy way of skipping coop-cloud/example itself, or would we need to keep that one special-case in abra?
decentral1se added this to the Medium/large enhancements project 2023-06-08 09:30:14 +00:00
decentral1se self-assigned this 2023-10-04 13:31:59 +00:00
decentral1se commented 2023-10-04 13:33:10 +00:00
Owner
Copy Link

Is there an easy way of skipping coop-cloud/example itself, or would we need to keep that one special-case in abra?

Maybe tags / topics = recipe & dont-generate

I'm definitely thinking to make a move on this sooner than later following this approach btw!

> Is there an easy way of skipping coop-cloud/example itself, or would we need to keep that one special-case in abra? Maybe tags / topics = `recipe` & `dont-generate` --- I'm definitely thinking to make a move on this sooner than later following this approach btw!
decentral1se commented 2023-10-05 06:40:32 +00:00
Owner
Copy Link

Can we add a topic to coop-cloud/example, and does it show up when we make a new repo from that template?

image

> Can we add a topic to coop-cloud/example, and does it show up when we make a new repo from that template? ![image](/attachments/11fd7fc2-b7de-45cc-92cf-34d07eca19d0)
image.png
23 KiB
decentral1se commented 2023-10-05 07:36:47 +00:00
Owner
Copy Link

@3wordchant

Diving into the code finally... opened https://github.com/go-gitea/gitea/issues/27456!

The catalogue generation codes makes one additional request per-repo to the topics endpoint now.

WIP PR is coop-cloud/abra#368, it already works.

I added the recipe topic to https://git.coopcloud.tech/coop-cloud/example but then indeed, as mentioned in #377 (comment) I needed to special case example.

Seems fine? I'm happy to do the manual labelling labours.

@3wordchant Diving into the code finally... opened https://github.com/go-gitea/gitea/issues/27456! The catalogue generation codes makes one additional request per-repo to the topics endpoint now. WIP PR is https://git.coopcloud.tech/coop-cloud/abra/pulls/368, it already works. I added the `recipe` topic to https://git.coopcloud.tech/coop-cloud/example but then indeed, as mentioned in https://git.coopcloud.tech/coop-cloud/organising/issues/377#issuecomment-16307 I needed to special case `example`. Seems fine? I'm happy to do the manual labelling labours.
decentral1se modified the project from Medium/large enhancements to (deleted) 2023-10-06 09:27:58 +00:00
decentral1se commented 2023-10-15 08:51:21 +00:00
Owner
Copy Link

Moar ping @mayel @3wordchant any further concerns? Can pick this up otherwise.

Moar ping @mayel @3wordchant any further concerns? Can pick this up otherwise.
decentral1se commented 2023-10-16 17:36:32 +00:00
Owner
Copy Link

OK, went ahead with it as it doesn't seem controversial at this point. Tagged every recipe by hand. Was unable to tag https://git.coopcloud.tech/coop-cloud/matomo with recipe topic? Can anyone do it? Idk maybe a permissions thing 🤷‍♀️

OK, went ahead with it as it doesn't seem controversial at this point. Tagged every recipe by hand. Was unable to tag https://git.coopcloud.tech/coop-cloud/matomo with `recipe` topic? Can anyone do it? Idk maybe a permissions thing 🤷‍♀️
wykwit commented 2023-11-05 19:18:34 +00:00
Member
Copy Link

There is a performance cost to this. After commit 76f95fe766 the "retrieving recipe repos list" step of abra catalogue generate now takes around 28 seconds instead of 2 seconds. This change also relies on Gitea features and an undocumented convention, which would make rolling alternative app catalogues (see #139) harder.

I think this mechanism should rely more on previous recipes.json content to generate the updated version - it already includes repository url for each recipe. That way for an alternative catalogue it would be possible to push new repos to recipes.json by hand and also have them kept up to date. Checking organization's projects and their topics could then proceed only for those repositories that weren't included before in a previous pass (so to proceed like it does now, but hopefully with less requests).

Another solution could be to use a configuration file that lists ignored repositories and organizations that are watched to find recipes. I hear that some configuration file support is coming with #303, but maybe this should be stored somewhere in coop-cloud/recipes-catalogue-json repo instead.

I might be misunderstanding the catalogue though, it's not yet documented throughly (#523). I like the idea of custom catalogues, and although it could be easier for the project to decide on having "a single source of truth", I hope in the end it will actually get easier to decentralize the catalogue too without relying on a deployment identical to the official one. Having a configurable path to recipes-catalogue-json repo in abra and relying more on recipes.json overall will go a long way in that direction.

There is a performance cost to this. After commit [76f95fe766](https://git.coopcloud.tech/coop-cloud/abra/commit/76f95fe766ce282229e4b0082007032581c3eb3f) the "retrieving recipe repos list" step of `abra catalogue generate` now takes around 28 seconds instead of 2 seconds. This change also relies on Gitea features and an undocumented convention, which would make rolling alternative app catalogues (see #139) harder. I think this mechanism should rely more on previous `recipes.json` content to generate the updated version - it already includes repository url for each recipe. That way for an alternative catalogue it would be possible to push new repos to `recipes.json` by hand and also have them kept up to date. Checking organization's projects and their topics could then proceed _only_ for those repositories that _weren't included before_ in a previous pass (so to proceed like it does now, but hopefully with less requests). Another solution could be to use a configuration file that lists ignored repositories _and_ organizations that are watched to find recipes. I hear that some configuration file support is coming with #303, but maybe this should be stored somewhere in [coop-cloud/recipes-catalogue-json](https://git.coopcloud.tech/coop-cloud/recipes-catalogue-json) repo instead. I might be misunderstanding the catalogue though, it's not yet documented throughly (#523). I like the idea of custom catalogues, and although it could be easier for the project to decide on having "a single source of truth", I hope in the end it will actually get easier to decentralize the catalogue too without relying on a deployment identical to the official one. Having a configurable path to `recipes-catalogue-json` repo in `abra` and relying more on `recipes.json` overall will go a long way in that direction.
decentral1se commented 2023-11-05 23:08:31 +00:00
Owner
Copy Link

Re-opening & renaming for #377 (comment)

Re-opening & renaming for https://git.coopcloud.tech/coop-cloud/organising/issues/377#issuecomment-18742
decentral1se changed title from Consider moving recipe repos to e.g. coop-cloud-recipes org to Performance issue with retrieving recipes 2023-11-05 23:09:07 +00:00
decentral1se added
abra
performance
 and removed
help wanted
design
 labels 2023-11-05 23:09:33 +00:00
decentral1se commented 2023-11-27 10:42:36 +00:00
Owner
Copy Link

@wykwit please DM me on Matrix and let's plan out something? It'd be good to have this working fast again and certainly the points you mention are important. Happy to support you making some progress on this.

@wykwit please DM me on Matrix and let's plan out something? It'd be good to have this working fast again and certainly the points you mention are important. Happy to support you making some progress on this.
❤️ 1
decentral1se removed their assignment 2023-11-27 10:42:49 +00:00
decentral1se removed this from the (deleted) project 2023-11-27 10:43:12 +00:00
decentral1se commented 2024-03-29 09:09:14 +00:00
Owner
Copy Link

#569

https://git.coopcloud.tech/coop-cloud/organising/issues/569
basebuilder added this to the Project Gitea Re-organisation milestone 2024-04-15 07:06:48 +00:00
Sign in to join this conversation.
No Branch/Tag Specified
Branches Tags
master
Labels
Clear labels   
abra

Everything to do with abra

  
abra-gandi

Abra Gandi plugin

  
awaiting-feedback

Ping/pong on comms

  
backups

   
bug

Something is not working

  
build

Go build related issues

  
ci/cd

Getting the robots into the mix

  
community organising

Opening this thing up

  
contributing

Contributors stuff

  
coopcloud.tech

Our main website

  
democracy

Democratic decision making

  
design

Design thinking required

  
documentation

Let's write things together

  
duplicate

This issue or pull request already exists

  
enhancement

New feature

  
finance

Money things

  
funding

Anything related to grant funding

  
good first issue

Easy start with development

  
help wanted

Need some help

  
installer

Installation related issues

  
kadabra

   
performance

Performance related

  
proposal

Large change which requires feedback & decisin making

  
question

More information is needed

  
recipes.coopcloud.tech

Our recipes catalogue

  
security

Securing our shit

  
test

Unit or integration test suite

  
wontfix

This won't be fixed

No Label
abra
abra-gandi
awaiting-feedback
backups
bug
build
ci/cd
community organising
contributing
coopcloud.tech
democracy
design
documentation
duplicate
enhancement
finance
funding
good first issue
help wanted
installer
kadabra
performance
proposal
question
recipes.coopcloud.tech
security
test
wontfix
Milestone
Clear milestone
No items
No Milestone
Project Gitea Re-organisation
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
4 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: coop-cloud/organising#377
No description provided.
Delete Branch "%!s(<nil>)"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?