Integration testing #8

New Issue

2020-09-10T23:26:31Z

3wordchant commented

2020-09-10 23:26:31 +00:00

Holy shit, CI testing that sets up a swarm and deploys an app to it

Holy shit, [CI testing that sets up a swarm and deploys an app to it](https://github.com/thomasvincent/wordpress-swarm/blob/master/.travis.yml)

decentral1se commented

2020-09-11 08:12:01 +00:00

Yeah, this would be quite cool to do. That one looks a bit hard to maintain and consider that we'll have a .drone.yml for each application which will easily balloon and be a headache to take care of. Maybe this is a job for another abra-like script. The question is what is appropriate for testing? Spinning things up and checking that the URL comes up and maybe that all containers are definitely deployed? That'd be a good start.

Yeah, this would be quite cool to do. That one looks a bit hard to maintain and consider that we'll have a `.drone.yml` for each application which will easily balloon and be a headache to take care of. Maybe this is a job for another `abra`-like script. The question is what is appropriate for testing? Spinning things up and checking that the URL comes up and maybe that all containers are definitely deployed? That'd be a good start.

👍 1

decentral1se referenced this issue

2020-09-11 08:18:37 +00:00

Re-setup the renovate bot to watch for upgrades #9

3wordchant commented

2020-09-14 00:57:31 +00:00

Plus your idea from the meeting: Yunohost-style central CI server running swarm, where stacks get torn down after runs.

One hurdle I can think of with this idea is that docker stack rm sometimes takes a while to take down containers, which means docker stack rm demo_service && docker volume rm demo_service_vol ... fails whenever I've tried it. sleep 30 could be an adequate workaround. Or maybe we'd be generating a unique stack name for each test run?

Plus your idea from the meeting: Yunohost-style central CI server running swarm, where stacks get torn down after runs. One hurdle I can think of with this idea is that `docker stack rm` sometimes takes a while to take down containers, which means `docker stack rm demo_service && docker volume rm demo_service_vol ...` fails whenever I've tried it. `sleep 30` could be an adequate workaround. Or maybe we'd be generating a unique stack name for each test run?

decentral1se commented

2020-09-14 07:44:09 +00:00

Good point. I guess I wouldn't be too worried about tearing things down immediately. We can just stack rm and then leave the hourly cron (or whatever) to run docker system prune -a to clean up all volumes/networks/images/etc. that are not connected to a running container.

I think the only thing stopping this ticket from being a cake-walk is the fact that managing the docker daemon certificates is still a PITA. We'll need to re-generate them all the time and re-load them into Drone so that it can make contact. Need to automate that somehow.

Good point. I guess I wouldn't be too worried about tearing things down immediately. We can just `stack rm` and then leave the hourly cron (or whatever) to run `docker system prune -a` to clean up all volumes/networks/images/etc. that are not connected to a running container. I think the only thing stopping this ticket from being a cake-walk is the fact that managing the docker daemon certificates is still a PITA. We'll need to re-generate them all the time and re-load them into Drone so that it can make contact. Need to automate that somehow.

3wordchant added the enhancement label 2020-09-24 23:15:15 +00:00

3wordchant commented

2020-09-25 09:58:56 +00:00

Some progress towards this here: https://drone.autonomic.zone/compose-stacks/wordpress/6/1/2

You need to manually deploy traefik on swarm-test (or if we configured the traefik repo the same way as wordpress).

Then, push to the wordpress repo, wait a little while, and you can reach the deployed application at https://wordpress.swarm-test.autonomic.zone

Not flipping this "more magic" switch until we have some authentication on the Wordpress instance by default or a method of regularly tearing down apps.

Also still in search of a Drone plugin to run some basic checks using cURL, or checking docker stack ps on the remote host

Some progress towards this here: https://drone.autonomic.zone/compose-stacks/wordpress/6/1/2 You need to manually deploy `traefik` on `swarm-test` (or if we configured the `traefik` repo the same way as `wordpress`). Then, push to the `wordpress` repo, wait a little while, and you can reach the deployed application at https://wordpress.swarm-test.autonomic.zone Not flipping this "more magic" switch until we have some authentication on the Wordpress instance by default or a method of regularly tearing down apps. Also still in search of a Drone plugin to run some basic checks using cURL, or checking `docker stack ps` on the remote host

decentral1se commented

2020-09-25 10:35:38 +00:00

Also still in search of a Drone plugin to run some basic checks using cURL, or checking docker stack ps on the remote host

We should maybe use this: https://github.com/issuu/sure-deploy for the CI. Comes from some $corp and does a bunch of stuff after deployment to check that things went well. I guess the engineering managers were like "wtf we're not using swarm until there is a --detach=false flag".

Could potentially bake that into stack-ssh-deploy?

> Also still in search of a Drone plugin to run some basic checks using cURL, or checking docker stack ps on the remote host We should maybe use this: https://github.com/issuu/sure-deploy for the CI. Comes from some $corp and does a bunch of stuff after deployment to check that things went well. I guess the engineering managers were like "wtf we're not using swarm until there is a --detach=false flag". Could potentially bake that into stack-ssh-deploy?

👀 1

3wordchant commented

2020-09-25 10:38:52 +00:00

Traefik is now also configured to auto-deploy to swarm-test.autonomic.zone; once autonomic-cooperative/stack-ssh-deploy#1 is fixed then this will give us basic safety checks on our stacks and we can move on to more CI excitement like:

post-deployment tests (could get really fancy and do Cloudron-style headless JS tests)
linting (shellcheck on custom entrypoints and .envrc.sample? yaml-lint on compose files?)
testing backup / restore

Traefik is now [also configured](https://git.autonomic.zone/compose-stacks/traefik/src/branch/master/.drone.yml) to [auto-deploy to swarm-test.autonomic.zone](https://drone.autonomic.zone/compose-stacks/traefik/3/1/2); once autonomic-cooperative/stack-ssh-deploy#1 is fixed then this will give us basic safety checks on our stacks and we can move on to more CI excitement like: - post-deployment tests (could get really fancy and do Cloudron-style headless JS tests) - linting (`shellcheck` on custom entrypoints and `.envrc.sample`? `yaml-lint` on compose files?) - testing backup / restore

decentral1se commented

2020-09-25 12:54:05 +00:00

OK nice. The next thing I will work on is then the issue regarding the fact that when we know want to deploy these stacks onto a fresh slate each time. That is different from before because we usually assume that the stack is already up (along with its secrets) and then we're just updating it. We're going want to deploy as a fresh stack, and update now. Soooo, we need to integrate dummy secret creation for the CI. We're thinking along the lines of integrating abra into the stack-ssh-deploy plugin or something that lines up with what we do ourselves. In the meantime, if you want to just get moving, you can manually create the secrets on the swarm-test machine.

decentral1se referenced this issue from toolshed/stack-ssh-deploy

2020-09-25 12:55:24 +00:00

Add secret generation #2

decentral1se referenced this issue from toolshed/stack-ssh-deploy

2020-09-25 12:56:57 +00:00

Integrate with sure-deploy #3

decentral1se referenced this issue from toolshed/stack-ssh-deploy

2020-09-25 13:00:43 +00:00

Add purge: true field for cleaning up stacks #4

decentral1se commented

2020-09-25 19:10:12 +00:00

OK sitrep is now that we put secret generation and cleaning up into stack-ssh-deploy so this means that we can move on with getting CI setup for all the compose stacks we care about.

OK sitrep is now that we put secret generation and cleaning up into `stack-ssh-deploy` so this means that we can move on with getting CI setup for all the compose stacks we care about.

3wordchant commented

2020-09-27 12:25:52 +00:00

Question: auto-creating networks? e.g. my postfix-relay build is failing because the mail network doesn't exist: https://drone.autonomic.zone/compose-stacks/postfix-relay/2/1/2

Could just create it on the test server, or make it part of abra context init, but wondering if we want to automate this to make it more reproducible.

Question: auto-creating networks? e.g. my `postfix-relay` build is failing because the `mail` network doesn't exist: https://drone.autonomic.zone/compose-stacks/postfix-relay/2/1/2 Could just create it on the test server, or make it part of `abra context init`, but wondering if we want to automate this to make it more reproducible.

decentral1se commented

2020-09-27 16:36:01 +00:00

but wondering if we want to automate this to make it more reproducible.

@3wordchant maybe let's add a config for stack-ssh-deploy:

networks:
  - proxy
  - mail

And just pick the default as --driver overlay until we need something otherwise?

> but wondering if we want to automate this to make it more reproducible. @3wordchant maybe let's add a config for `stack-ssh-deploy`: ```yaml networks: - proxy - mail ``` And just pick the default as `--driver overlay` until we need something otherwise?

❤️ 1

3wordchant commented

2020-09-27 17:18:58 +00:00