How to collect app image version/digests that abra can easily read? #51
Labels
No Label
abra
abra-gandi
awaiting-feedback
backups
bug
build
ci/cd
community organising
contributing
coopcloud.tech
democracy
design
documentation
duplicate
enhancement
finance
funding
good first issue
help wanted
installer
kadabra
performance
proposal
question
recipes.coopcloud.tech
security
test
wontfix
No Milestone
No project
No Assignees
2 Participants
Notifications
Due Date
No due date set.
Dependencies
No dependencies set.
Reference: coop-cloud/organising#51
Loading…
Reference in New Issue
No description provided.
Delete Branch "%!s(<nil>)"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Addressing specifically the "How do we know if there's an upstream upgrade?" part of https://git.autonomic.zone/coop-cloud/organising/issues/48, I have a plan:
#coopcloud-apps:autonomic.zone
docker-hub-rss
instanceThen we're good! We're getting notified when new tags are out in a more public way that can be shared via the room.
Testing this out...
Packaged on https://git.autonomic.zone/coop-cloud/docker-hub-rss.
This regex seems to trim down all the tags to only the semver ones for Gitea.
Running a bot seems to be somewhat involved.
I might just use a local feed reader for now until I get more energy for this.
To clarify, we already have
renovate-bot
doing this for us but it is highly dependant on our Gitea instance and notifications from that (which are easier to miss), so I am investigating how RSS feeds can be used to track updates. With an eye on packagers themselves publishing RSS feeds for co-op cloud apps.Maybe easier alternative to Maubot? https://github.com/matrix-org/go-neb#docker-image
On the way https://git.autonomic.zone/coop-cloud/go-neb!
I wrote some related thoughts in https://git.autonomic.zone/coop-cloud/abra/issues/107. I am wondering if we can drop all the
ABRA_TYPE_<SERVICE>_VERSION
stuff and instead just read the tags from the RSS and generate the digests on the fly (will involve actually downloading the image though!). The only thing making me hesitate is that there doesn't seem to be a decent bash RSS/ATOM parser around?But then we could easily ask things from
abra
side like "what versions of this app/service are available" without having to manually go to docker hub and search it up. And it would making rollback/upgrade more clear - you can actually pick what versions you want. You might even want to doabra app <app> deploy --version=<version>
and skip ahead to the latest (e.g. when it is a few patch upgrades (just bug fixes). This would release the packager as the sole guardian of specifying what version is available -abra
users could have full freedom to pick what version they want.I think integrating RSS as a core part of how we interact with app versions would lay a really solid foundation for how we deal with this stuff going ahead. RSS lines up with the decentralisation theme also.
Just some leading thoughts...
Reading this out loud, maybe it is not the greatest idea. What I'd ideally like is the
abra
users to be able to select from a range of versions. But those versions need to be tested on the packager side that they work and the configs match and so on.So maybe consuming an RSS feed from the package repo side is where we need to go in the end. This is making me think of YunoHost CI/CD where they guarantee some X/Y version backwards compatibility of an app.
I don't know how far we want to go into this all. Might be good to chat about it.
Another angle (we follow the fleet approach):
So we'd dooo maybe
versions.cloud.autononomic.zone
which would be a flat JSON file which would be generated by this daily running cronjob which parses the docker hub stuff and generates it.We're questioning the RSS angle because docker hub rate limits, it doesn't seem to have any auth functionality and there is no decent bash XML parser out of the box.
Still thinking about this...
Docker hub seems to be fairly handy to parse once you have your API key setup.
Use docker-hub-rss and matrix bot to stay on top of upstream upgradesto How to collect all versions of apps we package that can be consumed by abraHow to collect all versions of apps we package that can be consumed by abrato How to collect app image version/digests that abra can easily readHow to collect app image version/digests that abra can easily readto How to collect app image version/digests that abra can easily read?YunoHost have a similar approach now that I remember it:
Are there are two kinds of updates we're maybe talking about here?
For (1), because recipes often include multiple Docker images, which often release updates at different times, a list of "available upstream versions" would get very long, very quickly. E.g. for Rocketchat:
rocket-chat:3.12.1
+mongo:4.4.4
rocket-chat:3.12.1
+mongo:4.2.12
rocket-chat:3.11.2
+mongo:4.4.4
rocket-chat:3.11.2
+mongo:4.2.12
I think the primary audience of (1) would be Co-op Cloud app package maintainers, although people using the app may also want to subscribe to these notifications, depending on how far they trust us to apply upstream updates quickly.
So, I'm not sure (1) needs to be exposed in
abra
, which is currently more intended for users than packagers, and the simple "put your faith in Co-op Cloud to update apps" case rather than the advanced "keep an eye on upstream" use-case.It would be nice to offer app packagers an extra way of following upstream beyond the hairy-to-configure
renovate-bot
, but it doesn't seem like such a huge priority -- apart from the technical challenges in providing an RSS feed,renovate-bot
ends up being a lot more useful.(2) seems to be about specific, tested (either by CI, or ideally through our own use) combinations of image versions.
Following discussion in #48, coop-cloud/abra#90 and coop-cloud/abra#102, I think we're trying to avoid assigning recipes their own version number, and fall back to using the upstream images. So the Rocketchat recipe might have versions like this:
rocket-chat:3.12.1+mongo:4.4.4
rocket-chat:3.11.2+mongo:4.2.12
(or maybe
rocket-chat:3.12.1:d3adb33f+mongo:4.4.4:c3351007
, to include the digests?)It seems useful to end users for both the "is there an update available for my
wordpress.example.com
app" question, and facilitating "oh no, this upgrade was a lemon, I want torollback
to the last one" situation, both of which mean making this information available toabra
.I really like this idea, and if we're agreed that this script is pulling information on recipe versions (rather than upstream image versions from Docker Hub), then maybe it's fairly simple -- just
git clone
/git pull
all the recipe repositories, glean the version information from them (whether that continues to live inABRA_TYPE_APP_VERSION
and friends, or a probably-quickly-too-long git tag), and then assemble anabra-apps.json
with sections like this:This might also save us from having to
git pull
the recipe repos every time we do anything with them , because we can just freshen our local cache ofabra-apps.json
and check if our local version of the Rocketchat recipe is the latest available in the JSON.Bonus points: if it kept a simple record of the old vs. new list of version, this script could also publish an RSS feed of updates -- which we could easily (?) feed into the suggested
#coopcloud-apps
Matrix room:I guess my long comment wasn't long enough 🙈
Lazy alternative idea: the script just uses
yq
to parse thecompose.yml
in each repo to find image+tag, then either download the images to get the digest, or usesskopeo
:No more
ABRA_TYPE_APP_VERSION
/ABRA_TYPE_APP_DIGEST
.Only downside I can think of is that something extra would be needed if we released > 1 version of an app between runs of this script.
Lastly, another option into the mix, RSS feeds from Gitea (for direct subscription, or Matrix bot). Gitea is still spinning its wheels on built-in RSS, in the mean-time there's this delicious workaround: https://code.ka.com.kw/miscellaneous/gitea-feed-workaround
Ahhhh, super good thoughts! Excellent detective work on skopeo / gitea RSS also!
I am basically coming at this proposal as an
abra
user from the perspective that our Gitea fell down the other day and I wanted to rollback a few versions to test if that was the issue and the only way to do that was manually hack the~/.abra/apps/gitea/abra.sh
and then runabra app git.autonomic.zone deploy
to rollback 🙈 😂From that, I think
abra
side definitely needs some idea of which versions can be deployed and rolled back to. Also, I think the packager should have a say in what is supported to make things more stable and facilitate a smaller test space for CI/CD.So, maybe a way to avoid the Docker hub available tags massive list, as is to just do what you said,
git clone
the apps and read the version info from them and then stick that in theabra-apps.json
. To get "what can I rollback to?" we'd need to make the script that generates theabra-app.json
remember all the past versions?I think we're definitely ready for a chat on this and we'll get a plan sorted for it pretty quickly. I am hoping this is the last of the versioning work and then this will just be a solid foundation that won't change going forward.
Another idea is to have a way in the
abra.sh
of a package to write some upper/lower bound of the service. So, we might have something likeABRA_TYPE_APP_VERSION_BOUNDS="~=1.13"
(see this for "wtf is that tilde"). That would still involve theabra-apps.json
generating script to read those bounds from Docker hub and list the versions available within those bounds.Yeah exactly, which is what's leaning me towards "only renovate-bot needs to know about Docker Hub image updates, everything else looks at
abra-apps.json
".Yeah exactly. Two ways I can think of to do that:
app-catalogue-json.sh
looks at the existing, previousabra-apps.json
when it runs, and appends any new versions to e.g.rocketchat.versions
rather than replacing what's already there. (Aforementioned downside being that if there are two new recipe versions between runs ofapp-catalogue-json.sh
then one will be missed..)Each recipe version is recorded as a git tag, and then
app-catalogue-json.sh
doesgit tag -a
in each recipe repository to get the list of versions. To get the image versions, this would either involve agit checkout <tag>
for each listed tag, or stuffing all the image versions into the tag name (as per previous probably-terrible proposal for e.g.rocket-chat:3.12.1:d3adb33f+mongo:4.4.4:c3351007
We could also maybe do a combo of the two:
app-catalogue-json.sh
only does a (slow)git checkout
to find the image versions for versions that aren't already listed inabra-apps.json
.Magic!
git tag
praxis will suit well on the packaging side imho. "Just git tag it and then it will be published by the script" would be ideal. Idk how therocket-chat:3.12.1:d3adb33f+mongo:4.4.4:c3351007
relates to our current reverse-ish DNS labelling scheme in thecompose.yml
definitions (e.g.03c48f99de/compose.yml (L59)
), maybe we want to unify those. Happy to change it.TODO for me then is unpack the glory of the task list in https://codimd.autonomic.zone/d4HZ37NeSRyef-5I6RvUCw into tickets and then we are out of this versioning brain bender! Will close this off then 🥳
OK, I made some tickets for what I grok for now, the rest is a bit fuzzy, we'll get there.