Files
abra/doc/DesignAudit.md
T
Linus GasserandClaude Opus 5.5 491ea47f50 Check the registry before suggesting a redeploy
A redeploy only fixes an image if its tag now points to a fixed image.
Patch-pinned tags such as nextcloud:35.0.2-fpm are not rebuilt once the
next patch is out, so suggesting a redeploy was wrong for them.

For findings with strategy refresh, read the tag's current image for
the server's platform and apply the advisory again. If it is still not
fixed, suggest a recipe upgrade instead; if the registry cannot be
reached, mark the suggestion as unchecked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 16:32:47 +01:00

28 KiB

abra audit — design

2026-11-20 · Linus Gasser

Summary

Add a top-level abra audit command. It checks every deployed app on every server against a curated list of advisories, and offers to fix what fails. A maintainer-side abra audit update turns upstream vulnerability sources into proposed advisories.

Problem

Operators of Co-op Cloud servers cannot easily tell whether their apps are safe:

  • Mutable tags. Images such as php or nextcloud are often rebuilt under the same tag. A server can run a vulnerable image while its compose file looks current.
  • Stale base images. An image like nextcloud is built on php. When php is fixed, nextcloud is not fixed until upstream rebuilds it.
  • Old recipe versions. Some recipe versions ship known-vulnerable upstream versions, and nothing tells the operator.
  • Unreliable redeploys. A redeploy should pull the new image. If the registry is overloaded at deploy time, Docker falls back to the bare tag and each node may keep its old local image, with only a warning.

Goals and non-goals

Goals:

  • One command reports, for every deployed app, whether it matches a known advisory.
  • Detect vulnerable images even when the tag has not changed, including vulnerable base images.
  • Propose a fix for each finding and, on request, apply it and verify the result.
  • Help advisory maintainers find new problems from upstream sources, without missing images that were fixed before anyone looked.
  • Work on mixed amd64 and arm64 swarms.

Non-goals:

  • A full CVE scanner. Deep scanning is delegated to Trivy or Grype as an option.
  • Rebuilding upstream images. abra can detect a stale base image but not patch it.
  • Publishing advisories automatically. A human reviews every advisory before operators see it.

Commands

abra audit is a top-level command, because an audit spans servers, apps and recipes. abra app check already exists (it checks .env variables), so that name is not available.

Command Who What it does
abra audit operator Audit all deployed apps on all servers
abra audit app <domain> operator Audit one deployed app
abra audit server <server> operator Audit all apps on one server
abra audit recipe <recipe> maintainer, CI Static check of recipe versions and pinned images; nothing deployed needed
abra audit list anyone List advisories, filtered by --recipe or --severity
abra audit show <id> anyone Details and fix notes for one advisory
abra audit update advisory maintainer Propose new or changed advisories from a CVE, from minimum versions, or from upstream sources

Two aliases are kept, because operators look under app and server first:

  • abra app audit <domain> is the same as abra audit app <domain>.
  • abra server audit <server> is the same as abra audit server <server>.

Storage. The audit needs an explicitly chosen abra directory. It is taken from, in order:

  1. $ABRA_DIR;
  2. the abraDir setting of an abra.yaml file in the current directory or one of its parents, as the rest of abra already supports.

If neither exists, all abra audit commands fail with a clear error. They do not fall back to ~/.abra, unlike the rest of abra. Below, $ABRA_DIR means this directory. Everything the audit stores lives below it:

Path Contents
$ABRA_DIR/servers Server and app .env files, as for the rest of abra
$ABRA_DIR/advisories Checkout of the advisories repo: shared, reviewed data
$ABRA_DIR/.cache/audit/images Local cache of image configs, by digest; from v2 on, safe to delete

Fetching. There is no cache lifetime: every invocation checks whether the advisory list has changed, and pulls it if so. The list is a git repo, so this is a git fetch of a few KB when nothing changed. When the repo cannot be reached, the audit warns, says how old the local copy is, and uses it.

Options for audit, app and server

  -s, --server <name>          only apps on this server
  -r, --recipe <name>          only apps of this recipe
  -a, --advisories <url|path>  advisory source (default: c4dt/advisories)
      --severity <level>       minimum severity: low, medium, high, critical
  -f, --fix                    propose fixes, apply after confirmation
  -n, --no-input               apply fixes without asking
      --refresh-only           only re-pull images, never upgrade the recipe
      --retries <n>            re-checks after a fix, with backoff (default 3)
  -o, --offline                no registry or advisory fetch; recipe checks only
      --deep                   also scan running digests with Trivy or Grype
      --report                 send unknown digests to the advisories repo (opt-in)
  -m, --machine                JSON output for cron and monitoring

abra audit recipe takes only --machine, so CI can fail on findings.

Exit codes

The exit code describes the state after the run, so a finding that was fixed and verified no longer counts.

Code Meaning
0 No findings remain: none were found, or all were fixed and verified
1 Findings remain that were not fixed: no --fix, fix declined, or strategy manual
2 A fix was tried but could not be verified, for example because the registry was unreachable

With several apps, the highest code wins. The report lists every finding with its final status (fixed, affected, unknown, unverified), so fixed findings stay visible in the output and in --machine JSON.

Advisory format

Advisories are YAML files, one per advisory, in a separate git repo under advisories/. They are kept apart from the recipe catalogue because they change on a different schedule and need their own review. The repo starts as c4dt/advisories and can move to coop-cloud/advisories once the format has settled.

id: CCA-2026-0012
title: PHP 8.3 CGI argument injection
severity: critical            # low | medium | high | critical
status: published             # draft | published | withdrawn
refs: [CVE-2026-12345, https://example.org/advisory]
published: 2026-10-20

# Recipe versions: semver range on the recipe part of the tag
recipes:
  nextcloud:
    affected: "<7.0.3+30.0.1"
    fixed: "7.0.3+30.0.1"

# Images: an image is fixed if ANY condition holds
images:
  - match: "nextcloud:30*-apache"
    fixed_from: sha256:2222...      # this digest and all first seen after it
    affected_digests: [sha256:1111...]
  - match: "*"                      # any image...
    base: "php:8.3*"                # ...built on php 8.3
    env: { PHP_VERSION: "<8.3.14" } # read from the image config

fix:
  strategy: refresh             # upgrade | refresh | manual
  notes: "Upstream images rebuilt on 2026-10-18; a redeploy is enough."

Image conditions

Digests are hashes and have no order, so "newer than digest X" cannot be computed from the digest itself. Instead, the format uses what the image says about itself, in order of preference:

Condition Field Reliability Notes
Version in the image env, version Best Tests the fix itself, per platform, without pulling layers
Build time created_after Good, if the image records its real build time Covers rebuilds where no version changes; see below
Base ancestry base, base_digest_from Good Docker Official Images record their base in OCI annotations
Allowlist fixed_digests Exact but closed Later rebuilds need the list updated
Denylist affected_digests Exact but incomplete For one known-bad digest
Digest history fixed_from Good Fallback for images whose build time is unusable; later version

An image that matches match but satisfies no condition is reported as unknown, never as OK.

How the conditions combine, for an image that matches match:

  1. A digest in affected_digests is affected; a digest in fixed_digests is fixed.
  2. env both selects and tests. If the image lacks one of the variables, the entry does not apply, because the image is not built on that runtime. Otherwise the ranges are affected ranges: if every variable is in its range, env says affected, else fixed. Write ">=8.3.0 <8.3.14" to limit the entry to one release series.
  3. version is an affected range on org.opencontainers.image.version.
  4. created_after says fixed for images built at or after that time, and affected for older ones, unless the build time is unusable.
  5. If any condition says fixed, the image is fixed. Otherwise, if one says affected, it is affected. Otherwise it is unknown.

match is a glob where * matches any characters. A pattern with a : is matched against name:tag, otherwise against the name only, using Docker's short names (nextcloud, not docker.io/library/nextcloud). Version ranges are space- or comma-separated constraints (<, <=, >, >=, =, !=) that must all hold, with || between alternatives. A recipe range without an upstream part (<7.0.3) only compares the recipe part of the tag.

Ordering images by version and build time

Most fixes change a version that the image shows, so no digest bookkeeping is needed:

  • Application version. NEXTCLOUD_VERSION in the config, or the annotation org.opencontainers.image.version (for example 35.0.3-apache).
  • Inherited runtime version. Images inherit environment variables from their base, so nextcloud:apache also carries PHP_VERSION=8.5.13. A nextcloud rebuilt on a fixed php shows the new PHP_VERSION, even though NEXTCLOUD_VERSION is unchanged.

That leaves rebuilds where every visible version stays the same, typically a Debian package such as openssl patched in the base image. For these, the build time decides: created_after is the Created time of the first fixed image, and any image of that tag built later counts as fixed.

Is the build time monotonic? Per tag, in practice yes: every rebuild is a new build run with a later timestamp. On Docker Hub on 2026-11-20, php:8.3-apache recorded its real build time (2026-11-19 14:40 UTC), and nextcloud:apache was built 33 minutes later on top of it. Both also carry the annotations org.opencontainers.image.base.name and base.digest. The exceptions are known:

Case Effect Handling
Reproducible builds (SOURCE_DATE_EPOCH, ko, Nix, Bazel) set Created to a commit date or to 1970 A base-only rebuild keeps the same Created, so old and new cannot be told apart Detect it, then report unknown
Upstream moves a tag back to an older image The older Created is reported as affected Correct as is
Clock skew between build machines Minutes Irrelevant next to fix dates days apart

The audit treats Created as unusable, and reports unknown, when:

  • it is before 2000, which catches the 1970 epoch;
  • it is earlier than the Created of the base image named in base.digest, because an image cannot be built before its base;
  • two different digests of the same tag have the same Created.

The advisory can then fall back to fixed_digests, or later to the digest history.

Fix strategies

Strategy Meaning What --fix does
upgrade A newer recipe version fixes it abra app upgrade <domain> <fixed version>
refresh The same tag now points to a fixed image Redeploy the same version, then verify the running digest
manual No fixed image exists yet Report only, with the notes and options such as pinning a digest

Digest history (fallback, later version)

The digest history is only needed for images whose build time is unusable, as listed above. It is deferred to v4, and built only if such images turn out to matter.

The advisories repo would keep an append-only history per image tag, recording when each digest was first seen:

# digests/nextcloud/30-apache.yml   (append-only)
- digest: sha256:1111...
  first_seen: 2026-10-02T08:00:00Z
  config: { created: 2026-10-01, NEXTCLOUD_VERSION: 30.0.3, PHP_VERSION: 8.3.12 }
- digest: sha256:2222...
  first_seen: 2026-10-18T14:00:00Z
  config: { created: 2026-10-18, NEXTCLOUD_VERSION: 30.0.3, PHP_VERSION: 8.3.14 }

It would be filled from three sources:

  1. abra audit update runs regularly from CI and appends every new digest it sees, with its image config. The config is stored because registries may delete old digests once they are untagged.
  2. Backfill from docker-library/repo-info, which keeps years of per-tag digests for official images in git.
  3. Running servers. With --report, abra audit sends digests it finds that are not in the history. This is opt-in.

Images fixed before anyone looked

Suppose a CVE is published and upstream pushes a fixed image under the same tag before update runs. update never sees the vulnerable image, but old installations still run it. This is handled because advisories are driven by the CVE, not by the current image:

  • The CVE creates the advisory, whether or not today's image is clean.
  • A clean current image only sets the strategy to refresh. Its versions become the env condition, or its Created time becomes created_after.
  • On a server, an image with older versions or an earlier build time is reported as affected. Both lead to a refresh and a verification.

update only needs to see the fixed image, and that is the one available when it runs.

Audit and fix

abra audit checks each app in the same way, and a fix counts only after the running digest has been verified.

  1. Read state. From the service spec, read the recipe version label and each image reference image:tag@sha256:.... For per-platform checks, read the node's platform and fetch that image config from the registry.
  2. Match advisories. First recipe ranges, then image conditions.
  3. Classify each finding as affected or unknown. No finding means clean.
  4. Fix (with --fix) according to the advisory's strategy, then verify.
flowchart TD
    read[Read service spec] --> match[Match advisories]
    match --> finding{Finding?}
    finding -- no --> clean[Clean, exit 0]
    finding -- affected or unknown --> strategy[Pick fix strategy]
    strategy -- manual --> report[Report, exit 1]
    strategy -- upgrade or refresh --> apply[Upgrade / refresh]
    apply --> verified{Digest fixed?}
    verified -- yes --> fixed[Fixed, exit 0]
    verified -- no --> retry[Retry, backoff]
    retry -- within --retries --> apply
    retry -- retries used up --> unverified[Unverified, exit 2]

Verification waits for the service to converge, then reads the spec again. It fails if the spec holds a bare tag (the registry lookup failed) or a digest that still matches the advisory. Retries use backoff, for example 30 s, 2 min and 10 min, before giving up with exit code 2.

abra audit update

abra audit update is a maintainer tool. It compares the recipes in the catalogue with a vulnerability, and proposes new or changed advisories for a human to review. It has three modes; the first two come first (see Versions).

abra audit update --cve <CVE-ID>                       # one known vulnerability
abra audit update --require <lib>=<v>[,<v>...]         # minimum patched versions
abra audit update --sources [recipe...]                # scan upstream sources (later)

  --repo <path>        local checkout of the advisories repo (default $ABRA_DIR/advisories)
  --source <list>      with --sources: endoflife,osv,ghsa,kev,repo-info,trivy (default: all but trivy)
  --since <date>       with --sources: only upstream changes after this date (default: last run)
  -n, --no-input       write all proposals as drafts without asking
  --pr                 commit accepted proposals on a branch and open a PR

Sources

Source Gives Becomes
endoflife.date End-of-life dates for runtimes such as php and postgres medium advisory, strategy upgrade
OSV.dev and GitHub Advisory Database CVEs with affected and fixed upstream versions Recipe ranges, mapped through the +UPSTREAM part of recipe tags
CISA KEV CVEs exploited in the wild Severity raised to critical
docker-library/repo-info Digest and package history of official images Digest history; fixed_from for rebuilt images
Trivy (opt-in, slow) CVEs found in current catalogue images Proposals the other sources missed

Steps

  1. Inventory. List every recipe version in the catalogue and its images. Fetch each image config for runtime versions (PHP_VERSION, NODE_VERSION) and find its base image.
  2. Record digests. Append new digests and their configs to the digest history.
  3. Query sources for the upstream apps and runtimes found in step 1.
  4. Classify. For each CVE, write a condition that separates affected from fixed images. Look at the current image only to choose the strategy: refresh if the current image is clean, manual if it is not.
  5. Deduplicate. Match on CVE ID and affected recipe or image. If an advisory already covers the finding, propose a change (wider range, new fixed digest, higher severity) instead of a new entry.
  6. Review. Show each proposal as a YAML diff: accept, edit, skip, or ignore permanently. Ignored findings go to ignore.yml with a reason, so they do not return.
  7. Write accepted proposals with status: draft, optionally on a branch with a PR. Only a human sets status: published.

A human is needed because the sources say what is vulnerable upstream. Only a maintainer can judge whether it matters for the recipe's configuration and which fix strategy applies.

Multiple platforms

abra already works on mixed amd64 and arm64 swarms, because it leaves the platform choice to Docker. The audit must compare digests at the right level.

Current behaviour. abra has no platform-specific code. deploy, upgrade and rollback use ResolveImageAlways (cli/app/deploy.go:157), so Docker resolves image:tag to the index digest that covers all platforms. Each node then pulls its own variant. GetRegistryDigest in pkg/client/registry.go also returns the index digest, so pinned image:tag@sha256:... references work on every platform.

Question Compare Where to read it
Has the tag moved? Index digest Service spec, and a registry HEAD request
Which PHP version is inside? Per-platform image config Pick the node's platform from the index (docker node inspect gives Description.Platform)
Is it built on the fixed base? Per-platform layers Same per-platform manifest

Pitfalls:

  • A new index digest does not prove the fix for your platform. Any platform's rebuild changes the index digest. Some upstreams rebuild arm64 later or not at all, so important fixes are checked per platform.
  • Copied per-platform digests. The Docker Hub web page shows per-platform digests most prominently. A maintainer who pins one breaks other platforms with no matching manifest. abra recipe lint should warn when a pinned digest is not an index but the image is multi-arch.
  • Containers show the image ID. docker inspect on a container gives the config digest, not the manifest digest. Use the service spec for comparisons.

What Docker allows

abra can detect almost any stale or vulnerable layer, but it can only fix an image that upstream has already rebuilt.

Possible How
Know exactly what is running Swarm stores image:tag@sha256:... in the service spec
Tell whether a tag has moved Registry HEAD request for the tag's manifest digest
Read versions without pulling Fetch the image config (a few KB): Created and Env, which often inherits PHP_VERSION or NGINX_VERSION from the base
Find the base image OCI annotations org.opencontainers.image.base.name and base.digest where set; otherwise layer-prefix matching against the base's layers
Go deeper, to libc or openssl SBOM and provenance attestations, which Docker Official Images publish
Not possible Consequence
Patch a base layer of someone else's image If upstream has not rebuilt, the options are waiting, pinning a known-good older digest, or having the recipe build its own image
Trust a redeploy without checking On registry failure Docker deploys the bare tag; every refresh must be verified

Undeploy is not needed. undeploy then deploy has the same registry problem as a plain redeploy, and adds downtime. A redeploy of the same version with ResolveImageAlways already asks the registry again.

Pinned digests make failures visible. With image: nextcloud:30-apache@sha256:... in the recipe, a failed registry lookup fails the deploy instead of running a stale image. The allow_digest branch adds this support to abra recipe upgrade.

Versions

The feature is built in four versions. Each one is usable on its own, and is tried on real servers before the next one starts.

Version Adds Changes servers?
v1 abra audit: report failing apps and suggest commands No
v2 abra audit update --cve and --require: draft advisories for one vulnerability No
v3 Subcommands, options, --fix with verification, deploy warnings Yes, with --fix
v4 Automatic advisory proposals from upstream sources, digest history No

v1: basic audit

Answers "which of my apps need attention, and what do I run?" using only a hand-written advisory list.

  • Command. abra audit, without arguments. It goes through every server in $ABRA_DIR/servers and every deployed app on it, using the same data as abra app ls -S.
  • Advisories. Hand-written YAML in c4dt/advisories, pulled on every run. The first advisory is the nextcloud same-tag rebuild.
  • Conditions. Only recipe ranges (affected, fixed), env, version, created_after (with the checks for unusable build times), fixed_digests and affected_digests. Fields v1 does not know are skipped with a warning, so newer advisories do not break older abra versions.
  • Reads only. Service specs (version label and image digests) and image configs. On a single-node server, the common case, the image config is read from the node itself with docker image inspect, so no registry is needed.
  • Output. Only failing apps, one row each: server, domain, recipe version, advisory, status (affected or unknown), and a suggested command:
    • upgrade: abra app upgrade <domain> <fixed version>
    • refresh: abra app deploy <domain> --force, followed by abra audit to check the result
    • manual: the advisory's notes
  • Refresh check. Before suggesting a redeploy, the audit reads the tag's current image from the registry and applies the advisory to it. Upstreams stop rebuilding patch-pinned tags such as nextcloud:35.0.2-fpm once the next patch is out, so if the current image is not fixed either, the suggestion becomes a recipe upgrade. If the registry cannot be reached, the redeploy suggestion is marked as unchecked.
  • Exit code. 0 when nothing fails, 1 otherwise.
  • Errors. An unreachable server is listed as such and the run continues.

Test. On a test server, deploy one old nextcloud recipe version and one app with a stale image under a current tag. Write an advisory for each. Check that both are reported, that following the suggestions fixes them, and that a second abra audit is clean.

v2: abra audit update for one vulnerability

Turns a single vulnerability into draft advisories, so the maintainer does not have to check every recipe by hand. There are two ways to describe the vulnerability.

By CVE: abra audit update --cve CVE-2026-12345. The CVE is looked up in OSV.dev, which also covers GitHub advisories and package ecosystems such as Packagist, npm, PyPI, Debian and Alpine. This gives the affected packages and their fixed versions. Recipes are then matched in two ways:

  • applications, such as nextcloud, through the +UPSTREAM part of the recipe tags;
  • runtimes and libraries, such as php, through the image config (next paragraph).

By minimum versions: abra audit update --require php=8.5.12,8.4.27,8.3.36,8.2.35. One minimum version per release series. For every image in the catalogue that contains the library:

Image contains Result
A listed series, below its minimum (php 8.4.20) Affected
A listed series, at or above its minimum (php 8.4.27) Fixed
A series that is not listed (php 8.1.x) Unsupported, proposed as a separate end-of-life advisory

Libraries are found through environment variables in the image config, which images inherit from their base:

Library Variable
php PHP_VERSION
node NODE_VERSION
python PYTHON_VERSION
nginx NGINX_VERSION
postgres PG_VERSION
redis REDIS_VERSION

An unknown library is an error that lists the known ones. Finding libraries through SBOMs comes later.

Which recipe versions. Every published version of each recipe that uses the library is checked, so the proposal gets a correct affected range. Image configs are a few KB and are cached by digest in $ABRA_DIR/.cache/audit/images/, so this is cheap after the first run. A digest never changes its content, so this cache never expires.

Strategy. For each affected recipe, the current image under the same tag is checked too:

  • a newer recipe version uses a fixed image → upgrade to that version;
  • the same tag now points to a fixed image → refresh, with the current digest in fixed_digests;
  • nothing is fixed yet → manual.

Output. Draft advisory files in the local checkout, each shown as a diff to accept, edit or skip. v2 does not commit or open pull requests.

Test. Run --require with the versions of a recent php security release, and compare the proposals with a list made by hand. Run --cve with a recent nextcloud CVE and check the recipe range.

v3: arguments and fixes

Makes the audit usable for single apps and for automation, and lets it apply fixes.

  • Subcommands. abra audit app, server, recipe, list and show, plus the aliases abra app audit and abra server audit.
  • Options. --server, --recipe, --severity, --machine, --offline and --advisories.
  • Fixes. --fix, --refresh-only, --no-input and --retries, with verification after each fix and exit code 2 when verification fails.
  • Deploy warnings. abra app deploy and abra app upgrade warn, without refusing, when the target version matches an advisory.
  • Platforms. env conditions are checked per platform on mixed amd64 and arm64 swarms.

Test. Run abra audit --fix against the v1 test setup, with the registry blocked for one attempt (for example with a firewall rule), and check that the retry succeeds and that the exit code is right in each case.

v4: more automatic updates

Moves advisory maintenance from "react to a CVE" to "be told about new problems".

  • abra audit update --sources, run on a schedule in CI, with endoflife.date, OSV, GHSA, CISA KEV and docker-library/repo-info.
  • Deduplication against existing advisories, ignore.yml, and --pr.
  • Digest history and fixed_from, only if images with unusable build times (reproducible builds) turn out to matter.
  • Optional extras: --deep scanning with Trivy or Grype, the lint rule for per-platform digests, a status column in abra app list, and --report.
  • Moving the advisories repo to coop-cloud/advisories.

Decisions

Question Decision
Who maintains the advisories, and how fast? Linus Gasser, manually, in c4dt/advisories, for now
Should --report exist? Not in the first versions
How long to cache the advisory list? No cache: every invocation checks for the latest list
What does abra app deploy do when the target version matches an advisory? It warns, and does not refuse

Open questions

  • Do the registries we depend on keep untagged digests long enough to read their config later? This matters only for the digest history (v4), because the config of a running image can be read from the node. To find out, take old php digests from docker-library/repo-info and try to fetch their manifests from Docker Hub.
  • Which images used by recipes are built reproducibly, so that their Created time is unusable? abra audit update could list them while scanning the catalogue.