docs.coopcloud.tech/docs/abra/hack.md

---
title: Hack
---

## Contributing

Welcome to Hacking the Planet with `abra`! We're looking forward to see what
you come up. If you have any questions, don't hesitate to ask 💖 If any of your
changes seems a bit controversial, it's probably to come have a chat first to
avoid heartache.

In general, we're into the idea of "Optimistic Merging" (instead of
"Pessimistic Merging" based on our understanding of
[C4](https://hintjens.gitbooks.io/social-architecture/content/chapter4.html)
(described further down under "Development Process" and also [in this blog
post](http://hintjens.com/blog:106)).

## Quick start

Get a fresh copy of the `abra` source code from [here](https://git.coopcloud.tech/coop-cloud/abra).

Install [direnv](https://direnv.net), run `cp .envrc.sample .envrc`, then run `direnv allow` in this directory. This will set coopcloud repos as private due to [this bug.](https://git.coopcloud.tech/coop-cloud/coopcloud.tech/issues/20#issuecomment-8201). Or you can run `go env -w GOPRIVATE=coopcloud.tech` but I'm not sure how persistent this is.

Install [Go >= 1.16](https://golang.org/doc/install) and then:

- `make build` to build. If this fails, run `go mod tidy`.
- `./abra` to run commands
- `make test` will run tests
- `make install-abra` will install abra to `$GOPATH/bin`
- `make install-kadabra` will install kadabra to `$GOPATH/bin`
- `go get <package>` and `go mod tidy` to add a new dependency

Our [Drone CI configuration](https://git.coopcloud.tech/coop-cloud/abra/src/branch/main/.drone.yml) runs a number of checks on each pushed commit. See the [Makefile](https://git.coopcloud.tech/coop-cloud/abra/src/branch/main/Makefile) for more handy targets.

Please use the [conventional commit format](https://www.conventionalcommits.org/en/v1.0.0/) for your commits so we can automate our change log.

## Unit tests

### Run tests

Run the entire suite.

```
make test
```

### Filter tests

Run a specific test.

```
go test ./pkg/recipe -v -run TestGetVersionLabelLocalDoesNotUseTimeoutLabel
```

## Integration tests

### Running on the CI server

Based on
[R020](https://docs.coopcloud.tech/federation/resolutions/passed/020/), we have
automated running the integration test suite. Here's the TLDR;

* We have a donated CI server (tysm `@mirsal` 💝) standing at the ready,
  `int.coopcloud.tech`.
* We run the entire integration suite nightly via our Drone CI/CD configuration [here](https://git.coopcloud.tech/coop-cloud/abra/src/branch/main/.drone.yml) (see "`name: integration test`" stanza)
* Here is the script that is run on the remote server: [`run-ci-int`](https://git.coopcloud.tech/coop-cloud/abra/src/branch/main/scripts/tests/run-ci-int)

What follows is a listing of how this was achieved so that we can collectivise
the maintenance.

On the server, we have:

* Created an `abra` user with `docker` permissions
* Ran `apt install bats bats-file bats-assert bats-support jq make git golang-1.21 wget bash`
* Installed `bats-core` from source, following the instructions below
* Docker was already installed on the machine, so nothing to do there
* `docker login` with the `thecoopcloud` details so we don't get rate limited

The drone configuration was wired up as follows:

* Generated a SSH key and put the public key part in `~/.ssh/authorize_keys`
* Added that public key part as a "deploy key" in the abra repo (so we can do `ssh://` git remote pulls)
* Added the private key part as a Drone secret which is available in build so that the build can SSH over to the server to run commands. That was done like so: `drone secret add --repository coop-cloud/abra --name abra_int_private_key --data @id_ed25519`
* In order to specify a cron timing, you need to create it with the Drone CLI: `drone cron add "coop-cloud/abra" "integration" @daily --branch main`

Please ask `@decentral1se` or on the Matrix channels for SSH access to the machine.

### Running them locally

#### Install dependencies

We use [`bats`](https://bats-core.readthedocs.io/en/stable/) to run the tests.
You can install the required dependencies with the following. You also need a
working installation of Docker and Go >= 1.16 (not covered in this section).

```
apt install bats-file bats-assert bats-support jq make git
```

Unfortunately, the latest `bats` version in Debian stable does not have the
"filter tests by tags" feature, which is very handy for running a subset of the
tests. For this, we need to install `bats` from source. It's easy.

```
apt purge -y bats
git clone https://github.com/bats-core/bats-core.git
cd bats-core
sudo ./install.sh /usr/local
```

#### Setup Test Server

For some tests an actual server is needed, where apps can be deployed. You can
either use a local one or a remote test server. There is also a way to run or
skip tests that require a remote server. This is covered below in the
[filtering tests](#filter-tests_1) section.

##### Remote swarm

```
export ABRA_TEST_DOMAIN="test.example.com"
export ABRA_DIR="$HOME/.abra_test"
```

`ABRA_TEST_DOMAIN` should also have a DNS A record for `*.test.example.com`
which points to the same server so that the test suite can deploy apps freely.
The test suite does not deploy Traefik for you.

##### Local swarm

When running the test suite localy you need a running docker swarm setup:

```
docker swarm init
docker network create -d overlay proxy
```

To use the local swarm set the foloowing env var:

```
export TEST_SERVER=default
export ABRA_DIR="$HOME/.abra_test"
```

### Run tests

Now you can run the whole test suite:

```
bats -Tp tests/integration
```

Or you can run a single test file:

```
bats -Tp tests/integration/autocomplete.bats
```

### Tagging tests

When a test actually deploys something, we tag it as "slow". When the test
requires public DNS, we use "dns". There may be more tags we write more tests.

```
# bats test_tags=slow,dns
@test "..." {
  ...
}
```

Then we can use [filters](#filter-tests) (see below) to pick out a subset of
tests which do/do not use a live server. Feel free to come up with your own
tags. See the `bats-core`
[docs](https://bats-core.readthedocs.io/en/stable/writing-tests.html#tagging-tests)
for more.

### Filter tests

You can run a specific file.

```
bats -Tp tests/integration/autocomplete.bats
```

For example, if you want to check that all `abra recipe ...` tests remain working.

```
bats -Tp tests/integration/recipe_*
```

You can filter on test names to run specific kinds of tests.

```
bats -Tp tests/integration --filter "validate app argument"
```

You can filter on tags.

```
bats -Tp tests/integration --filter-tags \!slow      # only fast tests
bats -Tp tests/integration --filter-tags slow        # only slow tests
bats -Tp tests/integration --filter-tags slow,\!dns  # slow but no DNS tests
```

You can also only run the previously failed tests.

```
mkdir -p tests/integration/.bats/run-logs
bats -Tp tests/integration                        # run tests
bats -Tp tests/integration --filter-status failed # re-run only failed
```

### Debug tests

If you're running into issues and want to debug stuff, you can pass `-x` to
`bats` to trace all commands run in the test. You can add `echo '...' >&3`
debug statements to your test to output stuff also.

## Using the `abra` public API

Warning, there is currently no stability promise for the `abra` public API! Most of the internals are exposed in order to allow a free hand for developers to try build stuff. If people start to build things then we can start the discussion on what is useful to have open/closed and keep stable etc. Please let us know if you depend on the APIs!

The `pkg.go.dev` documentation is [here](https://pkg.go.dev/coopcloud.tech/abra). Here's a brief example to get you going:

```go
package main

import (
	"context"
	"fmt"
	"log"

	abraClient "coopcloud.tech/abra/pkg/client"
	dockerClient "github.com/docker/docker/client"
)

func getClient(serverName string) (*dockerClient.Client, error) {
	cl, err := abraClient.New(serverName)
	if err != nil {
		return nil, fmt.Errorf("getClient: %s", err)
	}
	return cl, nil
}

func main() {
	cl, err := getClient("foo.example.com")
	if err != nil {
		log.Fatal(err)
	}

    // do stuff with the client...
    // https://pkg.go.dev/github.com/docker/docker/client
}
```

Some tools that are making use of the API so far are:

* [`kadabra`](https://git.coopcloud.tech/coop-cloud/abra/src/branch/main/cmd/kadabra/main.go)

## Cross-compiling

If there's no official release for the architecture you use, you can cross-compile `abra` very easily. Clone the source code from [here](https://git.coopcloud.tech/coop-cloud/abra) and then:

- enter the `abra` directory
- run `git tag -l` to see the list of tags, choose the latest one
- run `git checkout <tag>`, where `<tag>` is the latest version
- run `GOOS=<os> GOARCH=<arch> [GOARM=<arm>] make build`. You only have to use `GOARM` if you're building for ARM, this specifies the ARM version (5,6,7 etc). See [this](https://go.dev/doc/install/source#environment) for a list of all supported OS'es and architectures.

## Building in Docker

If you are living under a curse of constant Go environment problems, it might be easier to build `abra` using Docker:

```
sudo setenforce 0  # SELinux probably won't allow Docker to access files
docker run -it -v $PWD:/abra golang:1.19.6 bash
cd /abra
. .envrc
git config --global --add safe.directory /abra  # work around funky file permissions
make build
```

## Release management

We use [goreleaser](https://goreleaser.com) to help us automate releases. We use [semver](https://semver.org) for versioning all releases of the tool. While we are still in the public beta release phase, we will maintain a `0.y.z-beta` format. Change logs are generated from our commit logs. We are still working this out and aim to refine our release praxis as we go.

For developers, while using this `-beta` format, the `y` part is the "major" version part. So, if you make breaking changes, you increment that and _not_ the `x` part. So, if you're on `0.1.0-beta`, then you'd go to `0.1.1-beta` for a backwards compatible change and `0.2.0-beta` for a backwards incompatible change.

### Making a new release

- Run the [integration test suite](#integration-tests) and the unit tests (`make test`) (takes a while!)
- Change `ABRA_VERSION` in [`scripts/installer/installer`](https://git.coopcloud.tech/coop-cloud/abra/src/branch/main/scripts/installer/installer) to match the new tag (use [semver](https://semver.org))
- Commit that change (e.g. `git commit -m 'chore: publish next tag x.y.z-beta'`)
- Make a new tag (e.g. `git tag -a x.y.z-beta`)
- Push the new tag (e.g. `git push && git push --tags`)
- Wait until the build finishes on [build.coopcloud.tech](https://build.coopcloud.tech/coop-cloud/abra)
- Deploy the new installer script (e.g. `cd ./scripts/installer && make`)
- Check the release worked, (e.g. `abra upgrade; abra -v`)

## Fork maintenance

### `godotenv`

We maintain a fork of [godotenv](https://git.coopcloud.tech/coop-cloud/godotenv) because we need inline comment parsing for environment files. You can upgrade the version here by running `go get git.coopcloud.tech/coop-cloud/godotenv@0<COMMID>` where `<commit>` is the latest commit you want to pin to. See [`abra#391`](https://git.coopcloud.tech/coop-cloud/abra/pulls/391) for more.

### `docker/client`

A number of modules in [pkg/upstream](https://git.coopcloud.tech/coop-cloud/abra/src/branch/main/pkg/upstream) are copy/pasta'd from the upstream [docker/docker/client](https://pkg.go.dev/github.com/docker/docker/client). We had to do this because upstream are not exposing their API as public.

### `github.com/schultz-is/passgen`

Due to [`coop-cloud/organising#358`](https://git.coopcloud.tech/coop-cloud/organising/issues/358).