diff --git a/docs/abra/hack.md b/docs/abra/hack.md index 64147b9e..459253b0 100644 --- a/docs/abra/hack.md +++ b/docs/abra/hack.md @@ -54,11 +54,42 @@ go test ./pkg/recipe -v -run TestGetVersionLabelLocalDoesNotUseTimeoutLabel ## Integration tests -### Install dependencies +### Running on the CI server -We use [`bats`](https://bats-core.readthedocs.io/en/stable/), you can install -the required dependencies with the following. You also need a working -installation of Docker and Go (not covered in this section). +Based on +[R020](https://docs.coopcloud.tech/federation/resolutions/passed/020/), we have +automated running the integration test suite. Here's the TLDR; + +* We have a donated CI server (tysm `@mirsal` 💝) standing at the ready, + `int.coopcloud.tech`. +* We run the entire integration suite nightly via our Drone CI/CD configuration [here](https://git.coopcloud.tech/coop-cloud/abra/src/branch/main/.drone.yml) (see "`name: integration test`" stanza) + +What follows is a listing of how this was achieved so that we can collectivise +the maintenance. + +On the server, we have: + +* Created an `abra` user with `docker` permissions +* Ran `apt install bats bats-file bats-assert bats-support jq make git golang-1.21 wget bash` +* Installed `bats-core` from source, following the instructions below +* Docker was already installed on the machine, so nothing to do there +* `docker login` with the `thecoopcloud` details so we don't get rate limited + +The drone configuration was wired up as follows: + +* Generated a SSH key and put the public key part in `~/.ssh/authorize_keys` +* Added that public key part as a "deploy key" in the abra repo (so we can do `ssh://` git remote pulls) +* Added the private key part as a Drone secret which is available in build so that the build can SSH over to the server to run commands. That was done like so: `drone secret add --repository coop-cloud/abra --name abra_int_private_key --data @id_ed25519` + +Please ask `@decentral1se` or on the Matrix channels for SSH access to the machine. + +### Running them locally + +#### Install dependencies + +We use [`bats`](https://bats-core.readthedocs.io/en/stable/) to run the tests. +You can install the required dependencies with the following. You also need a +working installation of Docker and Go >= 1.16 (not covered in this section). ``` apt install bats-file bats-assert bats-support jq make git @@ -75,12 +106,14 @@ cd bats-core sudo ./install.sh /usr/local ``` -### Setup Test Server +#### Setup Test Server -For many tests an actual server is needed, where apps can be deployed. You can -either use a local one or a remote test server. +For some tests an actual server is needed, where apps can be deployed. You can +either use a local one or a remote test server. There is also a way to run or +skip tests that require a remote server. This is covered below in the +[filtering tests](#filter-tests_1) section. -#### With remote test server +##### Remote swarm ``` export ABRA_TEST_DOMAIN="test.example.com" @@ -89,14 +122,9 @@ export ABRA_DIR="$HOME/.abra_test" `ABRA_TEST_DOMAIN` should also have a DNS A record for `*.test.example.com` which points to the same server so that the test suite can deploy apps freely. -It's advised that you re-use the same server and therefore the same Traefik -deployment for running your integration tests. The test suite does not deploy -Traefik for you. Then you'll have more stable results. +The test suite does not deploy Traefik for you. -You probably don't want to run the entire test suite though, it takes a while. -Try the following for starters. - -#### With local swarm +##### Local swarm When running the test suite localy you need a running docker swarm setup: @@ -128,10 +156,11 @@ bats -Tp tests/integration/autocomplete.bats ### Tagging tests -When a test actually deploys something to a server, we tag it with the following: +When a test actually deploys something, we tag it as "slow". When the test +requires public DNS, we use "dns". There may be more tags we write more tests. ``` -# bats test_tags=slow +# bats test_tags=slow,dns @test "..." { ... } @@ -166,14 +195,17 @@ bats -Tp tests/integration --filter "validate app argument" You can filter on tags. ``` -bats -Tp tests/integration --filter-tags "\!slow" # only fast tests -bats -Tp tests/integration --filter-tags "slow" # only slow tests +bats -Tp tests/integration --filter-tags \!slow # only fast tests +bats -Tp tests/integration --filter-tags slow # only slow tests +bats -Tp tests/integration --filter-tags slow,\!dns # slow but no DNS tests ``` You can also only run the previously failed tests. ``` -bats -TP tests/integration --filter-status failed +mkdir -p tests/integration/.bats/run-logs +bats -Tp tests/integration # run tests +bats -Tp tests/integration --filter-status failed # re-run only failed ``` ### Debug tests