Reproducability between deployments and recipes between multiple admins/devs working on the same server #338

New Issue

Frando · 2022-08-05T15:27:25Z

Frando commented

2022-08-05 15:27:25 +00:00

Hi,
this is a bit more of a meta question, maybe docs, maybe a feature request :)
I started to setup two servers, for two different organizations. I really like the structure for me locally, having the configs for all services for all servers layed out in a tree. Now some of the services will use custom (private for now) recipes. I have put those into repositories on the gitea instances (each of the two orgs already had a gitea instance). Now I want to introduce the setup to other members of both orgs and ideally have more-or-less foolproof identical setup between myself and everyone else working on the servers (before, we just had everything on the servers themselves in compose files and used direct ssh access).

If I see things correctly, there is currently nothing in the configs that would pin that config to a specific recipe repository or version, they just have the TYPE variable that, I assume, identifies the recipe in ~/.abra/recipes (on every admin's local machine).

I have put the configs for each server into a private repo at each org. That would mean that to onboard others, I would have to tell them to a) clone the server config into .abra/servers and b) clone each of the service repos or init them via abra app new if they're from the official registiry. Is that correct?

With this setup, I have a fear of different people working on the server getting "out of sync", leading to "abra deploy" having potentially different results depending on which repos and revisions are checked out on a developer's machine.

Are there previous thoughts on that? I was thinking about maybe adding a repo URL and revision to the config, and pre-checking this (if not in chaos mode) or using that info to auto-clone the correct repos before deploying or so. Maybe that could also be a custom shell script in the server config folder/repo.
My thinking was: If each service defined in .abra/servers/YOURDOMAIN would contain a reference to a Git repo and commit, that would make this folder (if in a git repo) a single source of truth for "that server". A abra server init-recipes or so command could clone/checkout the repos mentioned for that server locally and thus provide easy onboarding and reproducible environments.

Hi, this is a bit more of a meta question, maybe docs, maybe a feature request :) I started to setup two servers, for two different organizations. I really like the structure for me locally, having the configs for all services for all servers layed out in a tree. Now some of the services will use custom (private for now) recipes. I have put those into repositories on the gitea instances (each of the two orgs already had a gitea instance). Now I want to introduce the setup to other members of both orgs and ideally have more-or-less foolproof identical setup between myself and everyone else working on the servers (before, we just had everything on the servers themselves in compose files and used direct ssh access). If I see things correctly, there is currently nothing in the configs that would pin that config to a specific recipe repository or version, they just have the TYPE variable that, I assume, identifies the recipe in ~/.abra/recipes (on every admin's local machine). I have put the configs for each server into a private repo at each org. That would mean that to onboard others, I would have to tell them to a) clone the server config into .abra/servers and b) clone each of the service repos or init them via abra app new if they're from the official registiry. Is that correct? With this setup, I have a fear of different people working on the server getting "out of sync", leading to "abra deploy" having potentially different results depending on which repos and revisions are checked out on a developer's machine. Are there previous thoughts on that? I was thinking about maybe adding a repo URL and revision to the config, and pre-checking this (if not in chaos mode) or using that info to auto-clone the correct repos before deploying or so. Maybe that could also be a custom shell script in the server config folder/repo. My thinking was: If each service defined in `.abra/servers/YOURDOMAIN` would contain a reference to a Git repo and commit, that would make this folder (if in a git repo) a single source of truth for "that server". A `abra server init-recipes` or so command could clone/checkout the repos mentioned for that server locally and thus provide easy onboarding and reproducible environments.

Frando added the

enhancement

abra

labels 2022-08-05 15:27:25 +00:00

decentral1se commented

2022-08-09 13:40:52 +00:00

@Frando

If I see things correctly, there is currently nothing in the configs that would pin that config to a specific recipe repository or version, they just have the TYPE variable that, I assume, identifies the recipe in ~/.abra/recipes (on every admin's local machine).

True! It might be nice alright to have a PINNED_DEPLOY_COMMIT=<sha> (or $some_other_name) which can be put in the .env and abra will read to only deploy that version. Would that match what you're thinking about here?

That would mean that to onboard others, I would have to tell them to a) clone the server config into .abra/servers and b) clone each of the service repos or init them via abra app new if they're from the official registiry. Is that correct?

If the recipes are from the "official" listing then abra will automatically clone them from git.coopcloud.tech/coop-cloud/... when it needs them. Otherwise, yes, they'll have to manually clone them. I think abra will error out with a "i don't know where that recipe is?` message?

Following from #303 & #139, we could imagine having a way to point abra at another recipe listing (e.g. private gitea) where it could clone stuff?

A abra server init-recipes or so command could clone/checkout the repos mentioned for that server locally and thus provide easy onboarding and reproducible environments.

I like this thought!

What we have at Autonomic right is basically this makefile approach which serves us on a team of 8+ or so people working on it together. We haven't need to do more than bootstrapping the ~/.abra/servers/... folder, since abra will auto-clone the required recipes.

@Frando > If I see things correctly, there is currently nothing in the configs that would pin that config to a specific recipe repository or version, they just have the TYPE variable that, I assume, identifies the recipe in ~/.abra/recipes (on every admin's local machine). True! It might be nice alright to have a `PINNED_DEPLOY_COMMIT=<sha>` (or `$some_other_name`) which can be put in the `.env` and abra will read to only deploy that version. Would that match what you're thinking about here? > That would mean that to onboard others, I would have to tell them to a) clone the server config into .abra/servers and b) clone each of the service repos or init them via abra app new if they're from the official registiry. Is that correct? If the recipes are from the "official" listing then `abra` will automatically clone them from `git.coopcloud.tech/coop-cloud/...` when it needs them. Otherwise, yes, they'll have to manually clone them. I think `abra` will error out with a "i don't know where that recipe is?` message? Following from https://git.coopcloud.tech/coop-cloud/organising/issues/303 & https://git.coopcloud.tech/coop-cloud/organising/issues/139, we could imagine having a way to point `abra` at another recipe listing (e.g. private gitea) where it could clone stuff? > A `abra server init-recipes` or so command could clone/checkout the repos mentioned for that server locally and thus provide easy onboarding and reproducible environments. I like this thought! What we have at Autonomic right is basically [this makefile approach](https://docs.coopcloud.tech/operators/handbook/#sharing-abra) which serves us on a team of 8+ or so people working on it together. We haven't need to do more than bootstrapping the `~/.abra/servers/...` folder, since `abra` will auto-clone the required recipes.

decentral1se referenced this issue

2022-08-09 13:45:24 +00:00

Missing FOO server connection details when symlinking ~/.abra/servers/FOO #340

decentral1se commented

2022-08-09 13:45:42 +00:00

~~Related to onboarding: #340~~ Solved that one finally!

~~Related to onboarding: https://git.coopcloud.tech/coop-cloud/organising/issues/340~~ Solved that one finally!

decentral1se referenced this issue

2023-03-16 15:56:24 +00:00

Collective maintenance often means hidden app upgrades #434

decentral1se added this to the Medium/large enhancements project 2023-06-08 09:30:59 +00:00

decentral1se commented