Co-op Cloud stores per-app configuration in the `$USER/.abra/servers` directory, on whichever machine you're running `abra` on (by default, your own work station). In other words, app configurations are grouped under their relevant server directory. This corresponds to the ordering of the output of `abra app ls`.
`.env` files use the same format as used by Docker (with the `env_file:` statement in a `docker-compose.yml` file, or the `--env-file` option to `docker run`) and `direnv`. There is no `export ...=...` required since `abra` will take care to thread the values into the recipe configuration at deploy time.
`abra` doesn't mind if `~/.abra/servers`, or any of its subdirectories, is a [symlink](https://en.wikipedia.org/wiki/Symlink), so you can keep your app definitions wherever you like!
Just make sure the `~/.abra/servers` is included in the configuration of your favourite backup tool. Because `~/.abra/servers` is a collection of plain-text files, it's easy to keep your backup configuration in a version control system (we use `git`, others would almost certainly work).
This is particularly recommended if you're collaborating with others, so that you can all run `abra app ...` commands without having to maintain your own separate, probably-conflicting, configuration files.
In the simple case where you only have one server configured with `abra`, or everyone in your team is using the same set of servers, you can version-control the whole `~/.abra/servers` directory:
```
cd ~/.abra/servers
git init
git add .
git commit -m "Initial import"
```
!!! warning "Test your revision-control self-discipline"
`abra` does not yet help keep your `~/.abra/server` configurations up-to-date! Make sure to run `git add` / `git commit` after making configuration changes, and `cd ~/.abra/servers && git pull` before running `abra app ...` commands. Patches to add some safety checks and auto-updates would be very welcome! 🙏
In a more complex situation, where you're using Co-op Cloud to manage several servers, and you're collaborating with different people on different servers, you can set up **a separate repository for each subdirectory in `~/.abra/servers`**, or even a mixture of single-server and multi-server repositories:
This will set up symlinks from each directory in your repository to a correspondingly-named directory in `~/.abra/servers`– if your repository has a `swarm.example.com` directory, it'll be linked as `~/.abra/servers/swarm.example.com`.
As with the [simple repository set-up above](#backing-up-your-abra-configuration), `abra` doesn't yet help you update your version control system when you make changes, nor check version control to make sure you have the latest configuration. Make sure to `commit` and `push` after you make any configuration changes, and `pull` before running any `abra app ...` commands.
The plain-text, file-based configuration format means that you could even keep the configuration for different apps on the same server in different repositories, e.g. having `git.example.com` configuration in a separate repository to `wordpress.example.com`, using per-file symlinks.
We don't currently recommend this, because it might set inaccurate expectations about the security model – remember that, by default, **any user who can deploy apps to a Docker Swarm can manage _any_ app in that swarm**.
If you're on an environment where it's hard to run Docker, or command-line programs in general, you might want to install `abra` on a server instead of your local work station.
To install `abra` on the same server where you'll be hosting your apps, just follow [getting started guide](/operators/tutorial#deploy-your-first-app) as normal except for one difference. Instead of providing your SSH connection details when you run `abra server add ...`, just pass `--local`.
This will tell `abra` to look at the Docker system running on the server, instead of a remote one (using the Docker internal `default` context). Once this is wired up, `abra` knows that the deployment target is the local server and not a remote one. This will be handle seamlessly for all other deployments on this server.
Co-op Cloud uses [Docker Secrets](https://docs.docker.com/engine/swarm/secrets/) to handle sensitive data, like database passwords and API keys, securely.
Docker secrets are immutable, which means that their values can't be changed after they're set. To accommodate this, Co-op Cloud uses the established convention of "secret versions". Every time you change (rotate) a secret, you will insert it as a new version. Because secret versions are managed per-instance by the people deploying their apps, secret versions are stored in the `.env` file for each app:
For third-party API tokens, like OAuth client secrets, or keys for services like Mailgun, you will be storing values you already have as the appropriately-named Docker secrets. `abra` provides a convenient interface to the underlying `docker secret create` command:
1. Find out the current version number of the secret, e.g. by running `abra app config <domain>`, and choose a new one. Let's assume it's currently `v1`, so by convention the new secret will be `v2`
2. Generate or insert the new secret: `abra app secret generate <domain> db_password v2` or `abra app secret insert <domain> db_password v2 "foobar"`
3. Edit the app configuration to change which secret version the app will use: `abra app config <domain>`
4. Re-reploy the app with the new secret version: `abra app deploy <domain>`
The Co-op Cloud authors use the [UNIX `pass` tool][pass] to share sensitive data, including Co-op Cloud secrets, and `abra app secret ...` commands include a `--pass` option to automatically manage generated / inserted secrets:
Our understanding of Docker networking is probably wrong. We're working on it. Plz send halp :pray:
### Traefik networking
[Traefik](https://doc.traefik.io/traefik/) is our core web proxy, all traffic on a Co-op Cloud deployment goes through a running Traefik container. When setting up a new Co-op Cloud delpyment, `abra` creates a "global" [overlay network](https://docs.docker.com/network/overlay/) which traefik is hooked up to. This is the network that other apps use to speak to traefik and get traffic routed to them. Not every service in every app is also included in this network and hence not internet-facing (by convention, we name this network `internal`, see more below).
### App networking
By convention, the main `app` service is wired up to the "global" traefik overlay network. This container is the one that should be publicy reachable on the internet. The other services in the app such as the database and caches should be not be publicly reachable or visible to other apps on the same instance.
To deal with this, we make an additional "internal" network for each app which is namespaced to that app. So, if you deploy a Wordpress instance called `my_wordpress_blog` then there will be a network called `my_wordpress_blog_internal` created. This allows all the services in an app to speak to each other but not be reachable on the public internet.
At time of writing (Jan 2022), we think there is a limitation in our design which doesn't support multiple apps sharing the same domain (e.g. `example.com/app1/`&`example.com/app2/`). `abra` treats each domain as unique and as the singler reference for a single app.
This may be possible to overcome if someone really needs it, we encourage people to investigate. We've found that often, there are limitations in the actual software which don't support this anyway and several of the current operators simply use a new domain per app.
If "checksum OK" appears in your terminal - you're good to go!
Otherwise, you have downloaded a corrupted file.
## Creating a new server
`abra server new` can create servers if you have an account with a supported 3rd party integration. We currently support [Servers.coop](https://servers.coop) & [Hetzner](https://hetzner.com). The process of creating a new server usually goes like this:
1. Create an account with a server hosting provider
2. Generate an API client key which you'll give to `abra`
3. Run `abra server new`& fill in the values
`abra` supports creating, listing and removing servers if the 3rd party integration supports it.
If you want to teach `abra` how to support your favourite server hosting provider, we'd glady accept patches.
`abra record ...` can help you manage your DNS entries if you have an account with a supported 3rd party provider. We currently support [Gandi](https://gandi.net). The process of managing DNS with `abra` usually goes like this:
1. Create an account with a DNS service provider
2. Generate an API client key which you'll give to `abra`
3. Run `abra record ls` to check everything works
`abra` supports creating, listing and removing DNS entries if the 3rd party integration supports it.
If you want to teach `abra` how to support your favourite server hosting provider, we'd glady accept patches.
## How do I persist container logs after they go away?
This is a big topic but in general, if you're looking for something quick & easy, you can use the [journald logging driver](https://docs.docker.com/config/containers/logging/journald/). This will hook the container logs into systemd which can handle persistent log collection & managing log file size.
You need to add the following to your `/etc/docker/daemon.json` file on the server:
```json
{
"log-driver": "journald",
"log-opts": {
"labels":"com.docker.swarm.service.name"
}
}
```
And for log size management, edit `/etc/systemd/journal.conf`:
```
[Journal]
Storage=persistent
SystemMaxUse=5G
MaxFileSec=1month
```
Tne restart `docker`&`journald`:
```
systemctl restart docker
systemctl restart systemd-journald
```
Now when you use `docker service logs` or `abra app logs`, it will read from the systemd journald logger seamlessly! Some useful `journalctl` commands are as follows, if you're doing some more fine grained logs investigation:
## How do I specify a custom user/port for SSH connections with `abra`?
`abra` uses plain 'ol SSH under the hood and aims to make use of your existing SSH configurations in `~/.ssh/config` and interfaces with your running `ssh-agent` for password protected secret key files.
The `server add` command listed above assumes that that you make SSH connections on port 22 using your current username. If that is not he case, pass the new values as positional arguments. See `abra server add -h` for more on this.
```bash
abra server add <domain><user><port> -p
```
Running `server add` with `-d/--debug` should help you debug what is going on under the hood. It's best to take a moment to read [this troubleshooting entry](/abra/trouble/#ssh-connection-issues) if you're running into SSH connection issues with `abra`.
If you need to run a command within a running container you can use `abra app run <domain> <service> <command>`. For example, you could run `abra app run cloud.lumbung.space app bash` to open a new bash terminal session inside your remote container.
