Compare commits

..
Author SHA1 Message Date
Linus GasserandClaude Opus 5.5 46cee776f4 docs: add AGENTS.md with the c4dt branch and tag conventions
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 17:26:20 +01:00
Linus GasserandClaude Opus 5.5 59240028d9 audit: add --include-drafts
Apply draft advisories too, so a maintainer can try an advisory on real
servers before publishing it. Draft findings are marked "(draft)";
withdrawn advisories never apply.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 17:17:56 +01:00
Linus GasserandClaude Opus 5.5 ea1dd3e62c docs: describe the C4DT fork in README and README_C4DT.md
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 17:11:16 +01:00
Linus GasserandClaude Opus 5.5 9170b4d23a test(recipe): make the offline unfetched-branch test actually unfetched
SetupWithUpstream creates the feature branch before cloning, so the
clone already has origin/<branch> and EnsureVersionCtx succeeded
offline. Remove the remote-tracking ref first, as the test intends.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 17:06:43 +01:00
Linus Gasser 5f80d5e85c feat(recipe): allow branch names in RECIPE=<name>:<version>
Fetch and resolve a branch name on demand (into refs/remotes/origin/*,
never refs/heads/*, to avoid corrupting default-branch detection),
checking it out detached at its tip as a chaos commit. Resolution order
stays tag -> hash -> branch so existing pins are unaffected. Adds
EnsureVersionCtx so offline mode fails clearly instead of hanging or
erroring opaquely when a branch hasn't been fetched yet.

This supersedes the previous behavior of rejecting branch names with a
clear error.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Y6PrSttbx61PCQTie7HzLg
(cherry picked from commit e30f77caca)
2026-11-20 17:04:56 +01:00
Linus Gasser 836971d66b Ignore .devbox
(cherry picked from commit 33086eaebc)
2026-11-20 17:04:56 +01:00
Linus Gasser 47d9f73267 fix(app): keep recipe version on a single env field and label
Store the deployed version on TYPE (or the legacy RECIPE key when TYPE is
absent) instead of duplicating it across both. For git-URL recipes RECIPE
now stays a bare source pointer, so the two can never drift apart. Drops
the dead dirtyVersion branch and the fragile substring skip in
WriteRecipeVersion.

Also stamp the coop-cloud.<stack>.version label ourselves on upgrade and
rollback rather than trusting the value baked into the recipe's
compose.yml: git-URL/WIP recipes carry a stale or missing label, which
otherwise left the live deployment reporting the wrong version. Stop
getReleaseNotes and the [version] arg validation from fataling when the
deployed version is "unknown".

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
(cherry picked from commit 73801daae8)
2026-11-20 17:04:49 +01:00
Linus Gasser 0c91585eae feat(app): support git-URL recipes in 'abra app new'
Allow `abra app new <git-url>` to use a recipe from outside the
catalogue. On clone, a `.abra-source` sidecar records the canonical
host/path name (the on-disk directory escapes "/" and "." lossily), and
IsClean ignores it. When templating the app's .env, a `RECIPE=<canonical
name>` line is injected so a later `abra app deploy`, possibly on another
machine, re-fetches the recipe from the same git source. `recipe ls` now
shows a source column listing these external recipes alongside catalogue
ones.

(cherry picked from commit 5e8162bc66)
2026-11-20 17:04:49 +01:00
Linus Gasser 0a1578966a feat(recipe): normalize git URLs to canonical host/path names
Add NormalizeRecipeName to unify the various ways a recipe can be
referenced - https/http/ssh URLs, SCP-style git@host:path, already-
canonical host/path, and short catalogue names - to a single stable
"host/path" form, preserving any ":version" suffix. Wire it into
recipe.Get and ValidateRecipe so every entry point accepts git URLs, and
add Recipe.ShortName to recover the bare recipe name from a prefixed one.

(cherry picked from commit 6d9896fc46)
2026-11-20 17:04:49 +01:00
Linus Gasser 73227c9e58 fix(recipe): reject branch names as :version with a clear error
Passing a branch as the ":version" suffix previously failed with an
opaque "unable to resolve" error from go-git. Detect when the requested
revision matches a branch on a configured remote and fail with a message
explaining that ":<version>" only supports tags or commit hashes.

(cherry picked from commit 540842b76a)
2026-11-20 17:04:49 +01:00
Linus Gasser 2f0f3880e6 fix(formatter): guard ShortenID/SmallSHA against short input
ShortenID and SmallSHA sliced their input to a fixed length without
checking it was long enough, panicking on shorter strings. Return the
input unchanged when it is already shorter than the cut. Also replace the
blank Commit placeholder with an explicit "unknown-commit" sentinel.

(cherry picked from commit eeb4ea8b9c)
2026-11-20 17:04:49 +01:00
Linus GasserandClaude Opus 5.5 6fc26afad6 audit: check that fixes exist, short recipe names, list layout
- When a recipe upgrade is needed, search the recipe's newer releases for
  the oldest one whose images are fixed, and complain if there is none.
  Images are read from compose*.yml at each release tag; only tags are
  fetched, the local checkout is never changed.
- For recipe version ranges, check that the advisory's fixed version is
  a released tag.
- Show recipes relative to git.coopcloud.tech/coop-cloud/ (nextcloud,
  c4dt/nextcloud, or the full path). Advisories match recipes the same
  way, so forks are not matched by the canonical recipe's ranges.
- Replace the wide table by a narrow numbered table plus a numbered list
  of fixes wrapped to the terminal width; shorten SSH errors of
  unreachable servers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 16:54:57 +01:00
Linus GasserandClaude Opus 5.5 d691241414 Treat an empty servers directory as nothing to audit
LoadAppFiles("") fails on an empty servers directory; report "no apps
found" and exit 0 instead.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 16:33:30 +01:00
Linus GasserandClaude Opus 5.5 491ea47f50 Check the registry before suggesting a redeploy
A redeploy only fixes an image if its tag now points to a fixed image.
Patch-pinned tags such as nextcloud:35.0.2-fpm are not rebuilt once the
next patch is out, so suggesting a redeploy was wrong for them.

For findings with strategy refresh, read the tag's current image for
the server's platform and apply the advisory again. If it is still not
fixed, suggest a recipe upgrade instead; if the registry cannot be
reached, mark the suggestion as unchecked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 16:32:47 +01:00
Linus GasserandClaude Opus 5.5 165a81de91 Fix abra audit not finding any apps
LoadAppFiles only scans all servers when it gets a single empty string;
called without arguments it scanned none, so audit always reported
"no apps found".

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 16:30:49 +01:00
Linus GasserandClaude Opus 5.5 907a7eaa08 Fix panic on short commit hashes
SmallSHA, ShortenID and the version string sliced fixed lengths, so a
binary built without ldflags (commit = 7 spaces) panicked on every
command. Add formatter.Truncate and use it in all three places.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 16:09:53 +01:00
Linus Gasser d699e51cda First audit version 2026-11-20 16:05:57 +01:00
Linus Gasser 4e88ff9d34 Updating vendor directory 2026-11-20 16:05:35 +01:00
Linus GasserandClaude Opus 5.5 e07454ec46 Add design doc for abra audit
Proposes an `abra audit` command that checks deployed apps against a
list of advisories, covering mutable image tags, stale base images and
vulnerable recipe versions, plus `abra audit update` to draft
advisories from CVEs or minimum library versions.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-20 15:42:42 +01:00
Linus Gasser e5804d224d Allow digests in docker images
For the nextcloud update we need to be able to indicate the
digest of the docker image, as the docker tag is the same
for the vulnerable and the fixed docker image.
This avoids 'abra update' to re-install with a warning an
old image if the connection to docker hub fails.
2026-11-20 14:16:01 +01:00
renovate-bot 9023bb87c3 chore(deps): update nginx docker tag to v1.31.9 2026-11-18 17:01:13 +00:00
ChasquiLabo b15c039e06 chore: translation using Weblate (Spanish)
Currently translated at 91.5% (1056 of 1154 strings)

Translation: Co-op Cloud/abra
Translate-URL: https://translate.coopcloud.tech/projects/co-op-cloud/abra/es/
2026-11-13 02:01:19 +00:00
ChasquiLabo f3536cfaef chore: translation using Weblate (Spanish)
Currently translated at 91.3% (1054 of 1154 strings)

Translation: Co-op Cloud/abra
Translate-URL: https://translate.coopcloud.tech/projects/co-op-cloud/abra/es/
2026-11-13 01:15:12 +00:00
ChasquiLabo 0f4c9f3899 chore: translation using Weblate (Spanish)
Currently translated at 91.3% (1054 of 1154 strings)

Translation: Co-op Cloud/abra
Translate-URL: https://translate.coopcloud.tech/projects/co-op-cloud/abra/es/
2026-11-13 01:05:34 +00:00
ChasquiLabo caddaf8869 chore: translation using Weblate (Spanish)
Currently translated at 91.3% (1054 of 1154 strings)

Translation: Co-op Cloud/abra
Translate-URL: https://translate.coopcloud.tech/projects/co-op-cloud/abra/es/
2026-11-13 01:02:01 +00:00
ChasquiLabo fcd3c10368 chore: translation using Weblate (Spanish)
Currently translated at 91.3% (1054 of 1154 strings)

Translation: Co-op Cloud/abra
Translate-URL: https://translate.coopcloud.tech/projects/co-op-cloud/abra/es/
2026-11-13 00:15:36 +00:00
ChasquiLabo 0d5d2b67a3 chore: translation using Weblate (Spanish)
Currently translated at 91.3% (1054 of 1154 strings)

Translation: Co-op Cloud/abra
Translate-URL: https://translate.coopcloud.tech/projects/co-op-cloud/abra/es/
2026-11-13 00:08:54 +00:00
ChasquiLabo e611b8644b chore: translation using Weblate (Spanish)
Currently translated at 91.3% (1054 of 1154 strings)

Translation: Co-op Cloud/abra
Translate-URL: https://translate.coopcloud.tech/projects/co-op-cloud/abra/es/
2026-11-12 23:49:17 +00:00
ChasquiLabo 172182b9a5 chore: translation using Weblate (Spanish)
Currently translated at 91.3% (1054 of 1154 strings)

Translation: Co-op Cloud/abra
Translate-URL: https://translate.coopcloud.tech/projects/co-op-cloud/abra/es/
2026-11-12 23:45:21 +00:00
ChasquiLabo 99ee157e8b chore: translation using Weblate (Spanish)
Currently translated at 91.3% (1054 of 1154 strings)

Translation: Co-op Cloud/abra
Translate-URL: https://translate.coopcloud.tech/projects/co-op-cloud/abra/es/
2026-11-12 23:39:15 +00:00
ChasquiLabo 1f6d21f69d chore: translation using Weblate (Spanish)
Currently translated at 91.3% (1054 of 1154 strings)

Translation: Co-op Cloud/abra
Translate-URL: https://translate.coopcloud.tech/projects/co-op-cloud/abra/es/
2026-11-12 23:25:51 +00:00
ChasquiLabo 94c710296d chore: translation using Weblate (Spanish)
Currently translated at 91.3% (1054 of 1154 strings)

Translation: Co-op Cloud/abra
Translate-URL: https://translate.coopcloud.tech/projects/co-op-cloud/abra/es/
2026-11-12 23:06:42 +00:00
ChasquiLabo 53df7f876c chore: translation using Weblate (Spanish)
Currently translated at 91.3% (1054 of 1154 strings)

Translation: Co-op Cloud/abra
Translate-URL: https://translate.coopcloud.tech/projects/co-op-cloud/abra/es/
2026-11-12 22:51:26 +00:00
ChasquiLabo eb591f0843 chore: translation using Weblate (Spanish)
Currently translated at 91.1% (1052 of 1154 strings)

Translation: Co-op Cloud/abra
Translate-URL: https://translate.coopcloud.tech/projects/co-op-cloud/abra/es/
2026-11-12 22:06:33 +00:00
ChasquiLabo d9b3fa41cd chore: translation using Weblate (Spanish)
Currently translated at 91.1% (1052 of 1154 strings)

Translation: Co-op Cloud/abra
Translate-URL: https://translate.coopcloud.tech/projects/co-op-cloud/abra/es/
2026-11-12 19:38:22 +00:00
ChasquiLabo 6af3a2354d chore: translation using Weblate (Spanish)
Currently translated at 91.1% (1052 of 1154 strings)

Translation: Co-op Cloud/abra
Translate-URL: https://translate.coopcloud.tech/projects/co-op-cloud/abra/es/
2026-11-12 19:32:36 +00:00
ChasquiLabo 96c5a8406f chore: translation using Weblate (Spanish)
Currently translated at 91.0% (1051 of 1154 strings)

Translation: Co-op Cloud/abra
Translate-URL: https://translate.coopcloud.tech/projects/co-op-cloud/abra/es/
2026-11-12 18:50:28 +00:00
ChasquiLabo 2b374e9b08 chore: translation using Weblate (Spanish)
Currently translated at 91.0% (1051 of 1154 strings)

Translation: Co-op Cloud/abra
Translate-URL: https://translate.coopcloud.tech/projects/co-op-cloud/abra/es/
2026-11-12 18:48:42 +00:00
ChasquiLabo 0caf585969 chore: translation using Weblate (Spanish)
Currently translated at 90.9% (1050 of 1154 strings)

Translation: Co-op Cloud/abra
Translate-URL: https://translate.coopcloud.tech/projects/co-op-cloud/abra/es/
2026-11-12 17:35:19 +00:00
ChasquiLabo df7cc8048a chore: translation using Weblate (Spanish)
Currently translated at 90.9% (1050 of 1154 strings)

Translation: Co-op Cloud/abra
Translate-URL: https://translate.coopcloud.tech/projects/co-op-cloud/abra/es/
2026-11-12 16:49:17 +00:00
ChasquiLabo f134b70d49 chore: translation using Weblate (Spanish)
Currently translated at 90.9% (1050 of 1154 strings)

Translation: Co-op Cloud/abra
Translate-URL: https://translate.coopcloud.tech/projects/co-op-cloud/abra/es/
2026-11-12 14:21:32 +00:00
ChasquiLabo 7517798a21 chore: translation using Weblate (Spanish)
Currently translated at 90.9% (1050 of 1154 strings)

Translation: Co-op Cloud/abra
Translate-URL: https://translate.coopcloud.tech/projects/co-op-cloud/abra/es/
2026-11-11 23:42:02 +00:00
ChasquiLabo 0f06c39a35 chore: translation using Weblate (Spanish)
Currently translated at 90.9% (1050 of 1154 strings)

Translation: Co-op Cloud/abra
Translate-URL: https://translate.coopcloud.tech/projects/co-op-cloud/abra/es/
2026-11-11 23:35:28 +00:00
ChasquiLabo 7c660b299a chore: translation using Weblate (Spanish)
Currently translated at 90.9% (1050 of 1154 strings)

Translation: Co-op Cloud/abra
Translate-URL: https://translate.coopcloud.tech/projects/co-op-cloud/abra/es/
2026-11-11 23:13:05 +00:00
ChasquiLabo d1d9eab103 chore: translation using Weblate (Spanish)
Currently translated at 90.9% (1050 of 1154 strings)

Translation: Co-op Cloud/abra
Translate-URL: https://translate.coopcloud.tech/projects/co-op-cloud/abra/es/
2026-11-11 23:00:37 +00:00
ChasquiLabo 1696aced77 chore: translation using Weblate (Spanish)
Currently translated at 90.9% (1050 of 1154 strings)

Translation: Co-op Cloud/abra
Translate-URL: https://translate.coopcloud.tech/projects/co-op-cloud/abra/es/
2026-11-11 22:36:45 +00:00
ChasquiLabo 5773074dd6 chore: translation using Weblate (Spanish)
Currently translated at 90.9% (1050 of 1154 strings)

Translation: Co-op Cloud/abra
Translate-URL: https://translate.coopcloud.tech/projects/co-op-cloud/abra/es/
2026-11-11 22:32:03 +00:00
ChasquiLabo 533051d63e chore: translation using Weblate (Spanish)
Currently translated at 90.9% (1050 of 1154 strings)

Translation: Co-op Cloud/abra
Translate-URL: https://translate.coopcloud.tech/projects/co-op-cloud/abra/es/
2026-11-11 20:45:44 +00:00
ChasquiLabo b4cb9b3920 chore: translation using Weblate (Spanish)
Currently translated at 90.9% (1050 of 1154 strings)

Translation: Co-op Cloud/abra
Translate-URL: https://translate.coopcloud.tech/projects/co-op-cloud/abra/es/
2026-11-11 20:45:06 +00:00
ChasquiLabo d3bd408fa6 chore: translation using Weblate (Spanish)
Currently translated at 90.6% (1046 of 1154 strings)

Translation: Co-op Cloud/abra
Translate-URL: https://translate.coopcloud.tech/projects/co-op-cloud/abra/es/
2026-11-11 20:14:08 +00:00
ChasquiLabo bdd8831137 chore: translation using Weblate (Spanish)
Currently translated at 90.6% (1046 of 1154 strings)

Translation: Co-op Cloud/abra
Translate-URL: https://translate.coopcloud.tech/projects/co-op-cloud/abra/es/
2026-11-11 19:24:33 +00:00
renovate-bot 6761f1efa5 chore(deps): update module golang.org/x/term to v0.48.0 2026-11-09 19:01:08 +00:00
renovate-bot 5a600ce656 chore(deps): update module golang.org/x/sys to v0.50.0 2026-11-09 13:01:02 +00:00
devydaveanddevydave 550c9aa48f test: introduce local integration testing (#902)
This PR adds a declarative integration testing setup which includes 2 local VMs client and server which can be started and stopped via a command. The VMs are run with QEMU and created by https://github.com/microvm-nix/microvm.nix
The key pair is generated for this use case and is safe to commit, it resolves the problem that the readFile function can only be used with git tracked files.

---------

Co-authored-by: devydave <yannick-dev@tutanota.com>
Reviewed-on: toolshed/abra#902
Reviewed-by: p4u1 <133+p4u1@noreply.git.coopcloud.tech>
Co-authored-by: devydave <bittedanke@tutanota.com>
2026-10-31 08:28:04 +00:00
renovate-bot 1bffd1831d chore(deps): update nginx docker tag to v1.31.8 2026-10-21 23:01:23 +00:00
renovate-bot eb62135e3b chore(deps): update otel/weaver docker tag to v0.28.0 2026-10-15 03:01:18 +00:00
Weblate e8a494d9f4 chore: update translation files
Updated by "Update PO files to match POT (msgmerge)" add-on in Weblate.

Translation: Co-op Cloud/abra
Translate-URL: https://translate.coopcloud.tech/projects/co-op-cloud/abra/
2026-10-15 02:55:54 +00:00
3wordchant 35c29580bb fix: Run make i18n 2026-10-15 02:55:38 +00:00
renovate-bot e458c531f7 chore(deps): update nginx docker tag to v1.31.7 2026-10-14 20:01:20 +00:00
renovate-bot d0d156dc30 chore(deps): update golang docker tag 2026-10-14 16:01:23 +00:00
renovate-bot 14f05407f3 chore(deps): update module github.com/leonelquinteros/gotext to v1.8.0 2026-10-14 15:02:32 +00:00
jade e611a3be38 Sort invalid versions to the bottom rather than panicking 2026-10-14 17:13:01 +11:00
renovate-bot b5f7d26c9d chore(deps): update tonistiigi/xx docker tag to v1.10.0 2026-10-12 11:01:13 +00:00
renovate-bot 5b42b3d306 chore(deps): update golang docker tag 2026-10-09 17:01:20 +00:00
renovate-bot d99e86fb0e chore(deps): update module golang.org/x/term to v0.47.0 2026-10-09 16:01:52 +00:00
renovate-bot ade646abbc chore(deps): update golang docker tag 2026-10-09 14:01:30 +00:00
renovate-bot 80b3b90905 chore(deps): update module golang.org/x/sys to v0.49.0 2026-10-09 13:01:52 +00:00
renovate-bot fab3f0b97b chore(deps): update golang docker tag 2026-10-04 16:01:27 +00:00
renovate-bot 3819ee38f5 chore(deps): update module github.com/go-git/go-git/v5 to v5.19.3 2026-10-04 15:02:06 +00:00
renovate-bot 74261a7a01 chore(deps): update otel/weaver docker tag to v0.27.0 2026-10-04 01:01:15 +00:00
272 changed files with 17560 additions and 6776 deletions
+3
View File
@@ -7,3 +7,6 @@
/bin
dist/
tests/integration/.bats
tests/resources/path_to_repo
advisories
.devbox
+37
View File
@@ -0,0 +1,37 @@
# AGENTS.md
This is the C4DT fork of [`toolshed/abra`](https://git.coopcloud.tech/toolshed/abra), the Co-op Cloud command-line tool. What differs from upstream is described in [README_C4DT.md](README_C4DT.md); keep that file up to date when adding a change.
## Branches
| Branch | Content |
| --- | --- |
| `main` | Mirror of upstream `toolshed/abra` `main`. No C4DT commits. |
| `c4dt` | The fork's release line: upstream plus all C4DT changes. Release tags are made here. |
| feature branches, e.g. `audit_v1` | Work in progress, merged into `c4dt` when ready. |
- Start new work from `c4dt`, in a feature branch.
- To follow upstream, update `main` from `toolshed/abra`, then merge or rebase `c4dt` onto it. Update the base commit and date in `README.md` and `README_C4DT.md`.
- Changes taken from other forks are cherry-picked with `git cherry-pick -x`, so the original commit stays recorded.
## Tags
Releases are annotated tags on `c4dt`, named `<upstream version>-c4dt.<n>`, e.g. `0.13.0-beta-c4dt.1`:
- `<upstream version>` is the latest upstream tag the fork is based on.
- `<n>` counts the C4DT releases on that base, starting at 1, and restarts at 1 when the upstream version changes.
- Tags are never moved or deleted: users pin them, so a tag must always build the same code.
Build a release with the tag as version, so `abra --version` prints `<tag>-<commit>`:
```bash
go build -ldflags "-X main.Version=<tag> -X main.Commit=$(git rev-parse HEAD)" -o abra ./cmd/abra
```
## Development
- Toolchain: `devbox.json`. Use `devbox run build` and `devbox run test`; the test script sets a temporary `ABRA_DIR`, which the tests require.
- Dependencies are vendored: after changing `go.mod`, run `go mod vendor`.
- `abra audit` is designed in [`doc/DesignAudit.md`](doc/DesignAudit.md), which also lists what later versions will add. Keep the design and the code in sync.
- Advisories for `abra audit` live in a separate repository, [`c4dt/advisories`](https://git.coopcloud.tech/c4dt/advisories), under `advisories/`.
- User-facing strings are wrapped in `i18n.G`.
+9
View File
@@ -84,3 +84,12 @@ build-mo:
release:
@goreleaser release --clean
# TEST_INTEGRATION_FILES overrides the default of running all test files
ifdef TEST_INTEGRATION_FILES
test_files := $(TEST_INTEGRATION_FILES);
else
test_files := *;
endif
test-integration: build
@bats -Tp --verbose-run tests/integration/$(test_files)
+4
View File
@@ -5,6 +5,10 @@
[![Go Reference](https://pkg.go.dev/badge/coopcloud.tech/abra.svg)](https://pkg.go.dev/coopcloud.tech/abra)
[![Translation status](https://translate.coopcloud.tech/widget/co-op-cloud/svg-badge.svg)](https://translate.coopcloud.tech/engage/co-op-cloud/)
> [!NOTE]
> This is an LLM-modified version of `abra`, based on upstream [`toolshed/abra`](https://git.coopcloud.tech/toolshed/abra) `main` of 2026-11-18 (`0.13.0-beta` + 163 commits, `9023bb87`).
> See [README_C4DT.md](README_C4DT.md) for the differences.
The Co-op Cloud utility belt 🎩🐇
<a href="https://github.com/egonelbre/gophers"><img align="right" width="150" src="https://github.com/egonelbre/gophers/raw/master/.thumb/sketch/adventure/poking-fire.png"/></a>
+34
View File
@@ -0,0 +1,34 @@
# C4DT changes to `abra`
This fork of [`toolshed/abra`](https://git.coopcloud.tech/toolshed/abra) is maintained by C4DT. The changes were made with the help of an LLM coding assistant.
It is based on upstream `main` at commit `9023bb87` of 2026-11-18, which is 163 commits after the `0.13.0-beta` tag.
Releases are tags on the `c4dt` branch, named `<upstream version>-c4dt.<n>`, e.g. `0.13.0-beta-c4dt.1`. Pin a tag rather than following a branch. See [AGENTS.md](AGENTS.md) for the branch and tag conventions.
## Overview
| Change | Summary |
| --- | --- |
| [Digest pinning](#digest-pinning) | Recipes can pin images to a digest; `abra recipe upgrade` keeps the pin up to date |
| [`abra audit`](#abra-audit) | Checks all deployed apps against reviewed advisories and suggests fixes that are verified to exist |
| [Type remote](#type-remote) | Recipes can come from any git URL, and be pinned to a branch, in `RECIPE=` |
| [Devbox](#devbox) | `devbox.json` with the toolchain, and `build` and `test` scripts |
## Details
### Digest pinning
Some upstream images, such as `nextcloud`, are rebuilt under the same tag, so the tag alone cannot tell a vulnerable image from a fixed one. Recipes can now pin an image to a digest, for example `nextcloud:34.0.5-fpm@sha256:...`. `abra recipe upgrade` keeps the digest in sync: when it moves a pinned image to a new tag, it writes the new tag's digest, and when the current tag was re-published upstream, it warns and offers to update the pin. A pinned digest also makes a failed registry lookup at deploy time visible, instead of Docker silently keeping an old local image. To support this, `pkg/client/registry.go` gained `GetRegistryDigest`, which returns the index digest that Docker resolves `image:tag` to, and now requests OCI image indexes, which many multi-arch images on Docker Hub require. Commit `e5804d22`.
### `abra audit`
`abra audit` checks every deployed app on every server against the reviewed advisories in [`c4dt/advisories`](https://git.coopcloud.tech/c4dt/advisories), which it pulls on every run. It reads the recipe version and the image digests from each service spec, and the image configs (build time, environment such as `PHP_VERSION`, version labels) from the server's Docker daemon, or from the registry for the server's platform. Advisories match by recipe version range, environment version range, image version, build time, and known-good or known-bad digests. Before suggesting a fix, the audit checks that it works: a redeploy is only suggested if the tag's current image is fixed, and a recipe upgrade names the oldest release whose images are fixed. If no release fixes the problem yet, it says so and warns; recipe repositories are only read from their release tags, so local checkouts are never changed. The output is a short numbered table of failing apps followed by a numbered list of fixes, and the exit code is 1 when anything fails. Unlike the rest of abra, the audit needs an explicitly chosen abra directory (`$ABRA_DIR` or `abraDir` in `abra.yaml`) and never falls back to `~/.abra`. The design, including the planned later versions, is in [`doc/DesignAudit.md`](doc/DesignAudit.md). Commits `e07454ec`, `d699e51c`, `165a81de`, `491ea47f`, `d6912414`, `6fc26afa`.
### Type remote
Recipes no longer have to come from the catalogue. `abra app new` accepts a git URL in any common form (`https://`, `ssh://`, `git@host:path`), which is normalized to a canonical `host/path` name, and the app's `.env` gets a `RECIPE=<host/path>` line so that a later deploy, possibly on another machine, fetches the recipe from the same source. The clone records its source in a `.abra-source` file, and `abra recipe ls` shows these external recipes in a source column. The `:<version>` suffix in `RECIPE=` may now also be a branch name: it is fetched on demand into `refs/remotes/origin/*` and checked out at its tip as a chaos commit, after tags and commit hashes, and fails clearly in offline mode if the branch was never fetched. The deployed version is now stored in a single `.env` field, and upgrade and rollback set the version label themselves, because git-URL recipes often carry a stale label in `compose.yml`. Cherry-picked from `ineiti/abra` branch `type_remote` (commits `73227c9e`, `0a157896`, `0c91585e`, `47d9f732`, `5f80d5e8`); `9170b4d2` fixes one of its tests, which did not test the unfetched case it was named for.
### Devbox
A `devbox.json` pins the toolchain (Go 1.27, make, gettext): `devbox run build` builds abra, and `devbox run test` runs the tests with a temporary `ABRA_DIR`, which the existing tests require. Along with it, the `vendor/` directory was regenerated with `go mod vendor`, because it was out of sync with `go.mod` after an upstream dependency update and the default build failed. A binary built with a plain `go build`, without the `-ldflags` that set the commit, no longer panics: commit hashes are now truncated safely, and the default commit is `unknown-commit`. Commits `4e88ff9d`, `d699e51c`, `907a7eaa`, `2f0f3880`, `836971d6`.
+7 -1
View File
@@ -222,7 +222,13 @@ synchronize your local app environment values with what is deployed live.`),
mergedEnv[k] = v
}
if !strings.Contains(recipeEnvVar, ":") {
// The recipe version is always carried by TYPE (RECIPE, when present,
// is a bare source pointer). Stamp the deployed version onto TYPE and
// leave RECIPE untouched so the two never drift apart.
if t, ok := mergedEnv["TYPE"]; ok {
name, _, _ := strings.Cut(t, ":")
mergedEnv["TYPE"] = fmt.Sprintf("%s:%s", name, version)
} else {
mergedEnv[recipeKey] = fmt.Sprintf("%s:%s", mergedEnv[recipeKey], version)
}
+17 -1
View File
@@ -32,6 +32,18 @@ deploy <domain>" to do so.
You can see what recipes are available (i.e. values for the [recipe] argument)
by running "abra recipe ls".
In addition to short catalogue names, [recipe] also accepts arbitrary git
URLs to use a recipe from outside the catalogue (e.g. a fork or work in
progress). Any of these forms is accepted:
abra app new git.example.com/user/recipe
abra app new https://git.example.com/user/recipe
abra app new git@git.example.com:user/recipe
In that case a RECIPE=<canonical name> line is written to the app's .env
file so a subsequent "abra app deploy" (on this or another machine) will
re-fetch the recipe from the same git source.
Recipe commit hashes are supported values for "[version]".
Passing the "--secrets/-S" flag will automatically generate secrets for your
@@ -295,7 +307,7 @@ func ensureDomainFlag(recipe recipePkg.Recipe, server string) error {
if appDomain == "" && !internal.NoInput {
prompt := &survey.Input{
Message: i18n.G("Specify app domain"),
Default: fmt.Sprintf("%s.%s", recipe.Name, server),
Default: fmt.Sprintf("%s.%s", recipe.ShortName(), server),
}
if err := survey.AskOne(prompt, &appDomain); err != nil {
return err
@@ -306,6 +318,10 @@ func ensureDomainFlag(recipe recipePkg.Recipe, server string) error {
return errors.New(i18n.G("no domain provided"))
}
if strings.ContainsAny(appDomain, "/\\") {
return errors.New(i18n.G("invalid domain '%s': must not contain '/' or '\\'", appDomain))
}
return nil
}
+6
View File
@@ -188,6 +188,12 @@ beforehand. See "abra app backup" for more.`),
appPkg.SetChaosVersionLabel(compose, stackName, chosenDowngrade)
}
// NOTE: stamp the deployed version label ourselves rather than relying
// on the value baked into the recipe's compose.yml. Git-URL/WIP recipes
// often carry a stale or missing label, which would otherwise leave the
// live deployment reporting the wrong version after a rollback.
appPkg.SetVersionLabel(compose, stackName, chosenDowngrade)
// Gather secrets
secretInfo, err := deploy.GatherSecretsForDeploy(cl, app, internal.ShowUnchanged)
if err != nil {
+24 -4
View File
@@ -200,6 +200,12 @@ beforehand. See "abra app backup" for more.`),
appPkg.SetChaosVersionLabel(compose, stackName, chosenUpgrade)
}
// NOTE: stamp the deployed version label ourselves rather than relying
// on the value baked into the recipe's compose.yml. Git-URL/WIP recipes
// often carry a stale or missing label, which would otherwise leave the
// live deployment reporting the wrong version after an upgrade.
appPkg.SetVersionLabel(compose, stackName, chosenUpgrade)
envVars, err := appPkg.CheckEnv(app)
if err != nil {
log.Fatal(err)
@@ -346,9 +352,17 @@ func getReleaseNotes(
return errors.New(i18n.G("parsing chosen upgrade version failed: %s", err))
}
parsedDeployedVersion, err := tagcmp.Parse(deployMeta.Version)
if err != nil {
return errors.New(i18n.G("parsing deployment version failed: %s", err))
// The deployed version can be "unknown" (e.g. deployed without a version
// label, as is common for git-URL/work-in-progress recipes). In that case
// there is no lower bound to filter by, so we gather notes for every
// version below the chosen one instead of failing.
var parsedDeployedVersion tagcmp.Tag
haveDeployedVersion := deployMeta.Version != config.UNKNOWN_DEFAULT
if haveDeployedVersion {
parsedDeployedVersion, err = tagcmp.Parse(deployMeta.Version)
if err != nil {
return errors.New(i18n.G("parsing deployment version failed: %s", err))
}
}
for _, version := range internal.SortVersionsDesc(versions) {
@@ -357,7 +371,7 @@ func getReleaseNotes(
return errors.New(i18n.G("parsing recipe version failed: %s", err))
}
if parsedVersion.IsGreaterThan(parsedDeployedVersion) &&
if (!haveDeployedVersion || parsedVersion.IsGreaterThan(parsedDeployedVersion)) &&
parsedVersion.IsLessThan(parsedChosenUpgrade) {
note, err := app.Recipe.GetReleaseNotes(version, app.Domain)
if err != nil {
@@ -420,6 +434,12 @@ func validateUpgradeVersionArg(
return errors.New(i18n.G("'%s' is not a known version for %s", specificVersion, app.Recipe.Name))
}
// Without a known deployed version we cannot tell whether the requested
// version is an upgrade; trust the explicit choice rather than failing.
if deployMeta.Version == config.UNKNOWN_DEFAULT {
return nil
}
parsedDeployedVersion, err := tagcmp.Parse(deployMeta.Version)
if err != nil {
return errors.New(i18n.G("'%s' is not a known version", deployMeta.Version))
+174
View File
@@ -0,0 +1,174 @@
// Package audit implements the `abra audit` command.
package audit
import (
"context"
"fmt"
"os"
"path/filepath"
"sort"
"strings"
"coopcloud.tech/abra/cli/internal"
appPkg "coopcloud.tech/abra/pkg/app"
auditPkg "coopcloud.tech/abra/pkg/audit"
"coopcloud.tech/abra/pkg/config"
"coopcloud.tech/abra/pkg/i18n"
"coopcloud.tech/abra/pkg/log"
"coopcloud.tech/abra/pkg/recipe"
"github.com/spf13/cobra"
)
// AuditCommand audits all deployed apps on all servers.
var AuditCommand = &cobra.Command{
// translators: `abra audit` command
Use: i18n.G("audit [flags]"),
// translators: Short description for `abra audit` command
Short: i18n.G("Audit deployed apps against known advisories"),
Long: i18n.G(`Check every deployed app on every server against a curated list of
advisories, and suggest how to fix what fails.
The advisories are pulled from %s into $ABRA_DIR/advisories
on every run. If the repository cannot be reached, the local copy is used.
Only published advisories are applied; use --include-drafts to also apply
drafts, e.g. to try a new advisory before publishing it.
The audit needs an explicitly chosen abra directory: set $ABRA_DIR, or set
abraDir in an abra.yaml file in the current directory or one of its parents.
Only failing apps are listed. The exit code is 0 when nothing fails, and 1
otherwise.`, auditPkg.DefaultAdvisoriesURL),
Example: i18n.G(` # audit all apps on all servers
ABRA_DIR=~/.abra abra audit
# also apply advisories that are still drafts
abra audit --include-drafts`),
Args: cobra.NoArgs,
Run: func(cmd *cobra.Command, args []string) {
os.Exit(run())
},
}
func run() int {
abraDir := config.ABRA_DIR_EXPLICIT
if abraDir == "" {
log.Fatal(i18n.G("abra audit needs an explicitly chosen abra directory: set $ABRA_DIR, or abraDir in an abra.yaml file"))
}
advDir := filepath.Join(abraDir, "advisories")
if err := auditPkg.SyncAdvisories(advDir, auditPkg.DefaultAdvisoriesURL, internal.Offline); err != nil {
log.Fatal(err)
}
advisories, warnings, err := auditPkg.LoadAdvisories(filepath.Join(advDir, "advisories"))
if err != nil {
log.Fatal(err)
}
for _, w := range warnings {
log.Warn(w)
}
evalOpts := auditPkg.EvaluateOptions{IncludeDrafts: includeDrafts}
var used, drafts int
for _, a := range advisories {
if evalOpts.Applies(a) {
used++
if a.Status == auditPkg.StatusDraft {
drafts++
}
}
}
log.Debug(i18n.G("loaded %d advisories, using %d", len(advisories), used))
if includeDrafts {
log.Info(i18n.G("including %d draft advisories, which have not been reviewed yet", drafts))
}
if entries, err := os.ReadDir(config.SERVERS_DIR); err == nil && len(entries) == 0 {
log.Info(i18n.G("no apps found in %s", config.SERVERS_DIR))
return 0
}
appFiles, err := appPkg.LoadAppFiles("") // "" means all servers
if err != nil {
log.Fatal(err)
}
apps, err := appPkg.GetApps(appFiles, "")
if err != nil {
log.Fatal(err)
}
if len(apps) == 0 {
log.Info(i18n.G("no apps found in %s", config.SERVERS_DIR))
return 0
}
states, serverErrs := auditPkg.CollectState(apps, auditPkg.CollectOptions{Offline: internal.Offline})
findings := auditPkg.Evaluate(states, advisories, evalOpts)
if !internal.Offline {
var recipes []recipe.Recipe
for _, app := range apps {
recipes = append(recipes, app.Recipe)
}
checker := auditPkg.FixChecker{
Fetch: auditPkg.RegistryFetch,
Recipes: auditPkg.NewGitRecipes(recipes, internal.Offline),
}
checker.CheckFixes(context.Background(), findings)
}
for _, f := range findings {
for _, r := range f.Reasons {
log.Debug(fmt.Sprintf("%s %s: %s", f.App, f.Advisory.ID, r))
}
}
sort.SliceStable(findings, func(i, j int) bool {
if findings[i].Server != findings[j].Server {
return findings[i].Server < findings[j].Server
}
return findings[i].App < findings[j].App
})
if len(findings) > 0 || len(serverErrs) > 0 {
if err := printReport(findings, serverErrs); err != nil {
log.Fatal(err)
}
}
servers := make(map[string]bool)
for _, st := range states {
servers[st.Server] = true
}
log.Info(i18n.G(
"audited %d deployed apps on %d servers against %d advisories: %d findings, %d unreachable servers",
len(states), len(servers), used, len(findings), len(serverErrs),
))
var noFix []string
for _, f := range findings {
if f.NoFix {
noFix = append(noFix, fmt.Sprintf("%s (%s)", f.App, auditPkg.ShortRecipeName(f.Recipe)))
}
}
if len(noFix) > 0 {
log.Warn(i18n.G("no released fix yet for: %s", strings.Join(noFix, ", ")))
}
if len(findings) > 0 || len(serverErrs) > 0 {
return 1
}
return 0
}
var includeDrafts bool
func init() {
AuditCommand.Flags().BoolVar(
&includeDrafts,
i18n.G("include-drafts"),
false,
i18n.G("also apply draft advisories, to try them before publishing"),
)
}
+123
View File
@@ -0,0 +1,123 @@
package audit
import (
"fmt"
"os"
"strconv"
"strings"
auditPkg "coopcloud.tech/abra/pkg/audit"
"coopcloud.tech/abra/pkg/formatter"
"coopcloud.tech/abra/pkg/i18n"
"golang.org/x/term"
)
// printReport prints a narrow overview table, followed by a numbered list
// with the fix for each row. Long texts do not fit in a table, so they go in
// the list, wrapped to the terminal width.
func printReport(findings []auditPkg.Finding, serverErrs []auditPkg.ServerError) error {
servers := make(map[string]bool)
for _, f := range findings {
servers[f.Server] = true
}
showServer := len(servers) > 1
if len(findings) > 0 {
table, err := formatter.CreateTable()
if err != nil {
return err
}
headers := []string{"#"}
if showServer {
headers = append(headers, i18n.G("SERVER"))
}
headers = append(headers, i18n.G("APP"), i18n.G("RECIPE"), i18n.G("ADVISORY"), i18n.G("SEVERITY"), i18n.G("STATUS"))
table.Headers(headers...)
for i, f := range findings {
row := []string{strconv.Itoa(i + 1)}
if showServer {
row = append(row, f.Server)
}
advisory := f.Advisory.ID
if f.Advisory.Status == auditPkg.StatusDraft {
advisory += " " + i18n.G("(draft)")
}
row = append(row, f.App, auditPkg.ShortRecipeName(f.Recipe), advisory, f.Advisory.Severity, f.Status)
table.Row(row...)
}
if err := formatter.PrintTable(table); err != nil {
return err
}
fmt.Println()
width := termWidth()
for i, f := range findings {
prefix := fmt.Sprintf("%d. ", i+1)
fmt.Println(wrap(prefix+f.App+": "+f.Fix, width, strings.Repeat(" ", len(prefix))))
}
}
if len(serverErrs) > 0 {
if len(findings) > 0 {
fmt.Println()
}
width := termWidth()
for _, e := range serverErrs {
fmt.Println(wrap(i18n.G("unreachable: %s: %s", e.Server, shortError(e.Err)), width, " "))
}
}
return nil
}
// termWidth returns the terminal width, or 0 (no wrapping) when stdout is not
// a terminal or ABRA_CI is set.
func termWidth() int {
if os.Getenv("ABRA_CI") == "1" {
return 0
}
w, _, err := term.GetSize(int(os.Stdout.Fd()))
if err != nil || w < 40 {
return 0
}
return w - 1
}
// wrap breaks text into lines of at most width characters at spaces,
// indenting continuation lines. A width of 0 disables wrapping.
func wrap(text string, width int, indent string) string {
if width <= 0 || len(text) <= width {
return text
}
var lines []string
line := ""
for _, word := range strings.Fields(text) {
switch {
case line == "":
line = word
case len(line)+1+len(word) > width:
lines = append(lines, line)
line = indent + word
default:
line += " " + word
}
}
if line != "" {
lines = append(lines, line)
}
return strings.Join(lines, "\n")
}
// shortError keeps the useful part of docker/SSH connection errors, which
// otherwise contain several lines of boilerplate.
func shortError(err error) string {
msg := err.Error()
if i := strings.LastIndex(msg, "stderr="); i >= 0 {
msg = msg[i+len("stderr="):]
}
return strings.TrimSpace(msg)
}
+21
View File
@@ -0,0 +1,21 @@
package audit
import (
"errors"
"testing"
"github.com/stretchr/testify/assert"
)
func TestWrap(t *testing.T) {
assert.Equal(t, "short", wrap("short", 40, " "))
assert.Equal(t, "1. one two\n three\n four five", wrap("1. one two three four five", 12, " "))
assert.Equal(t, "no wrapping at all here", wrap("no wrapping at all here", 0, " "))
}
func TestShortError(t *testing.T) {
err := errors.New(`error during connect: Get "http://docker.example.com/v1.51/info": command [ssh -- host docker system dial-stdio] has exited with exit status 255, please make sure the URL is valid, and Docker 18.09 or later is installed on the remote host: stderr=ssh: connect to host srv port 22: No route to host
`)
assert.Equal(t, "ssh: connect to host srv port 22: No route to host", shortError(err))
assert.Equal(t, "plain", shortError(errors.New("plain")))
}
+9 -3
View File
@@ -59,9 +59,15 @@ func ValidateRecipe(args []string, cmdName string) recipe.Recipe {
log.Fatal(i18n.G("no recipe name provided"))
}
if _, ok := knownRecipes[recipeName]; !ok {
if !strings.Contains(recipeName, "/") {
log.Fatal(i18n.G("no recipe '%s' exists?", recipeName))
recipeName = recipe.NormalizeRecipeName(recipeName)
lookupName := recipeName
if i := strings.LastIndex(lookupName, ":"); i >= 0 {
lookupName = lookupName[:i]
}
if _, ok := knownRecipes[lookupName]; !ok {
if !strings.Contains(lookupName, "/") {
log.Fatal(i18n.G("no recipe '%s' exists? pass a git URL (e.g. https://git.example.com/user/recipe) to use a recipe outside the catalogue", lookupName))
}
}
+45
View File
@@ -2,11 +2,13 @@ package recipe
import (
"fmt"
"path"
"sort"
"strconv"
"strings"
"coopcloud.tech/abra/cli/internal"
"coopcloud.tech/abra/pkg/config"
"coopcloud.tech/abra/pkg/formatter"
"coopcloud.tech/abra/pkg/i18n"
"coopcloud.tech/abra/pkg/log"
@@ -41,6 +43,7 @@ var RecipeListCommand = &cobra.Command{
headers := []string{
i18n.G("name"),
i18n.G("source"),
i18n.G("category"),
i18n.G("status"),
i18n.G("healthcheck"),
@@ -56,6 +59,7 @@ var RecipeListCommand = &cobra.Command{
for _, recipe := range recipes {
row := []string{
recipe.Name,
i18n.G("catalogue"),
recipe.Category,
strconv.Itoa(recipe.Features.Status),
recipe.Features.Healthcheck,
@@ -76,6 +80,23 @@ var RecipeListCommand = &cobra.Command{
}
}
externals := externalRecipes(catl)
sort.Strings(externals)
for _, name := range externals {
row := []string{
name,
i18n.G("external"),
"-", "-", "-", "-", "-", "-", "-",
}
if pattern != "" {
if !strings.Contains(name, pattern) {
continue
}
}
table.Row(row...)
rows = append(rows, row)
}
if len(rows) > 0 {
if internal.MachineReadable {
out, err := formatter.ToJSON(headers, rows)
@@ -93,6 +114,30 @@ var RecipeListCommand = &cobra.Command{
},
}
// externalRecipes returns canonical names of locally-cloned recipes that
// were sourced from an arbitrary git URL (i.e. they carry a .abra-source
// sidecar) and are not already listed in the catalogue.
func externalRecipes(catl recipe.RecipeCatalogue) []string {
dirs, err := recipe.GetRecipesLocal()
if err != nil {
log.Debug(i18n.G("can't read local recipes: %s", err))
return nil
}
var names []string
for _, dir := range dirs {
canonical := recipe.ReadRecipeSource(path.Join(config.RECIPES_DIR, dir))
if canonical == "" {
continue
}
if _, inCatalogue := catl[canonical]; inCatalogue {
continue
}
names = append(names, canonical)
}
return names
}
var (
pattern string
)
+51
View File
@@ -129,6 +129,9 @@ interface.`),
log.Fatal(err)
}
// digest pinned images shared between services are only checked once
digestChecked := make(map[string]bool)
for _, service := range config.Services {
img, err := reference.ParseNormalizedNamed(service.Image)
if err != nil {
@@ -178,6 +181,13 @@ interface.`),
sort.Sort(tagcmp.ByTagDesc(compatible))
if canonical, ok := img.(reference.Canonical); ok && !digestChecked[service.Image] {
digestChecked[service.Image] = true
if err := refreshDigest(recipe, img, image, canonical.Digest().String()); err != nil {
log.Fatal(err)
}
}
if len(compatible) == 0 && !allTags {
log.Info(i18n.G("no new versions available for %s, assuming %s is the latest (use -a/--all-tags to see all anyway)", image, tag))
continue // skip on to the next tag and don't update any compose files
@@ -364,6 +374,47 @@ interface.`),
},
}
// refreshDigest checks whether the current tag of a digest pinned image was
// re-published upstream with a new digest and offers to update the pin.
func refreshDigest(recipe recipePkg.Recipe, img reference.Named, image, digest string) error {
tag := img.(reference.NamedTagged).Tag()
registryDigest, err := client.GetRegistryDigest(img, tag)
if err != nil {
return err
}
if registryDigest == digest {
log.Debug(i18n.G("digest of %s:%s is up to date", image, tag))
return nil
}
log.Warn(i18n.G("%s:%s was re-published upstream (digest %s → %s)", image, tag, digest, registryDigest))
update := false
if !internal.NoInput {
prompt := &survey.Confirm{
Message: i18n.G("update digest of %s:%s?", image, tag),
Default: true,
}
if err := survey.AskOne(prompt, &update); err != nil {
return err
}
}
if !update {
return nil
}
if _, err := recipe.UpdateTag(image, tag); err != nil {
return err
}
log.Info(i18n.G("digest updated for %s:%s", image, tag))
return nil
}
var (
allTags bool
createCommit bool
+5 -1
View File
@@ -7,6 +7,7 @@ import (
"strings"
"coopcloud.tech/abra/cli/app"
"coopcloud.tech/abra/cli/audit"
"coopcloud.tech/abra/cli/catalogue"
"coopcloud.tech/abra/cli/internal"
"coopcloud.tech/abra/cli/recipe"
@@ -85,10 +86,12 @@ func Run(version, commit string) {
Config:
$ABRA_DIR: %s`, config.ABRA_DIR),
Version: fmt.Sprintf("%s-%s", version, commit[:7]),
Version: fmt.Sprintf("%s-%s", version, formatter.Truncate(commit, 7)),
ValidArgs: []string{
// translators: `abra app` command for autocompletion
i18n.G("app"),
// translators: `abra audit` command for autocompletion
i18n.G("audit"),
// translators: `abra autocomplete` command for autocompletion
i18n.G("autocomplete"),
// translators: `abra catalogue` command for autocompletion
@@ -254,6 +257,7 @@ Config:
AutocompleteCommand,
manCommand,
app.AppCommand,
audit.AuditCommand,
catalogue.CatalogueCommand,
server.ServerCommand,
recipe.RecipeCommand,
+1 -1
View File
@@ -16,7 +16,7 @@ func main() {
Version = "dev"
}
if Commit == "" {
Commit = " "
Commit = "unknown-commit"
}
cli.Run(Version, Commit)
+17
View File
@@ -0,0 +1,17 @@
{
"$schema": "https://raw.githubusercontent.com/jetify-com/devbox/main/.schema/devbox.schema.json",
"packages": [
"go@1.27",
"gnumake@latest",
"gettext@latest",
"gnused@latest"
],
"shell": {
"scripts": {
"build": "make build",
"test": "export ABRA_DIR=\"$(mktemp -d)/abra\"; rc=0; make test || rc=$?; rm -rf \"$(dirname \"$ABRA_DIR\")\"; exit $rc",
"update-pot": "make update-pot",
"i18n": "make i18n"
}
}
}
+305
View File
@@ -0,0 +1,305 @@
{
"lockfile_version": "1",
"packages": {
"gettext@latest": {
"last_modified": "2026-10-20T14:23:57Z",
"resolved": "github:NixOS/nixpkgs/5adc50831c259f87faeb09ddf930d43597197473#gettext",
"source": "devbox-search",
"version": "1.0",
"systems": {
"aarch64-darwin": {
"outputs": [
{
"name": "out",
"path": "/nix/store/36c168mc93w4mlmqknam03ffyi6qg0w6-gettext-1.0",
"default": true
},
{
"name": "man",
"path": "/nix/store/38mvyy6a3ga3xqbzzbzfhpmmd4dll8hl-gettext-1.0-man",
"default": true
},
{
"name": "doc",
"path": "/nix/store/4i2r8pmdirlpig74ymxmi03sgl0inyd4-gettext-1.0-doc"
},
{
"name": "info",
"path": "/nix/store/ml58bdjablvqamxwc28ws5y9222hdvpx-gettext-1.0-info"
}
],
"store_path": "/nix/store/36c168mc93w4mlmqknam03ffyi6qg0w6-gettext-1.0"
},
"aarch64-linux": {
"outputs": [
{
"name": "out",
"path": "/nix/store/qh35hki623iqrj2mqfxnzjqawxn93gk7-gettext-1.0",
"default": true
},
{
"name": "man",
"path": "/nix/store/v0arqmc3cx4ls1vwg4c62kz7apcn76ml-gettext-1.0-man",
"default": true
},
{
"name": "doc",
"path": "/nix/store/4ficj1yfkd8bn17mlv81jsib04hfyg8c-gettext-1.0-doc"
},
{
"name": "info",
"path": "/nix/store/w0slr4m7ywbkmlm8dbksjp22adcab935-gettext-1.0-info"
}
],
"store_path": "/nix/store/qh35hki623iqrj2mqfxnzjqawxn93gk7-gettext-1.0"
},
"x86_64-darwin": {
"outputs": [
{
"name": "out",
"path": "/nix/store/9mxzw5vwn99vx2b74ipjzk8x4b784fjy-gettext-1.0",
"default": true
},
{
"name": "man",
"path": "/nix/store/r87hlf9pn8q3a8rnplzhy9lbgaadqcr8-gettext-1.0-man",
"default": true
},
{
"name": "doc",
"path": "/nix/store/xmssbi5i2w3fv67hrv6c29wdiv8gxywi-gettext-1.0-doc"
},
{
"name": "info",
"path": "/nix/store/j4lz4zqicd1mw2l5c272ialxhl0nkc0b-gettext-1.0-info"
}
],
"store_path": "/nix/store/9mxzw5vwn99vx2b74ipjzk8x4b784fjy-gettext-1.0"
},
"x86_64-linux": {
"outputs": [
{
"name": "out",
"path": "/nix/store/8qdw9shgr7dd3315jim8sfva2dnhq86c-gettext-1.0",
"default": true
},
{
"name": "man",
"path": "/nix/store/4acc3iagl3xlljb1aqn8j25vx5n400bc-gettext-1.0-man",
"default": true
},
{
"name": "doc",
"path": "/nix/store/dnz19fkr7spnpc7affh8ivcmk32g9xda-gettext-1.0-doc"
},
{
"name": "info",
"path": "/nix/store/v3z399ilsdjn461y0l2gi8h0z7jzsy6a-gettext-1.0-info"
}
],
"store_path": "/nix/store/8qdw9shgr7dd3315jim8sfva2dnhq86c-gettext-1.0"
}
}
},
"github:NixOS/nixpkgs/nixpkgs-unstable": {
"last_modified": "2026-10-20T14:23:57Z",
"resolved": "github:NixOS/nixpkgs/5adc50831c259f87faeb09ddf930d43597197473?lastModified=1792506237&narHash=sha256-Q%2F8KLB6jNfpaUN1HAedE%2Fu%2FEt9feE%2FzYSsutBLKrCFk%3D"
},
"gnumake@latest": {
"last_modified": "2026-10-20T14:23:57Z",
"resolved": "github:NixOS/nixpkgs/5adc50831c259f87faeb09ddf930d43597197473#gnumake",
"source": "devbox-search",
"version": "4.4.1",
"systems": {
"aarch64-darwin": {
"outputs": [
{
"name": "out",
"path": "/nix/store/306s97mvw5slpgg7hnb26xy37dhy4xk0-gnumake-4.4.1",
"default": true
},
{
"name": "man",
"path": "/nix/store/g0hyk9rpf10di0l5x6gxjahh5rip30kn-gnumake-4.4.1-man",
"default": true
},
{
"name": "doc",
"path": "/nix/store/xl1nz3znzjb27wrig0vrf5fj3z7jqh5s-gnumake-4.4.1-doc"
},
{
"name": "info",
"path": "/nix/store/99bqn610vynhvhwg8srd9g2l990cfa7a-gnumake-4.4.1-info"
}
],
"store_path": "/nix/store/306s97mvw5slpgg7hnb26xy37dhy4xk0-gnumake-4.4.1"
},
"aarch64-linux": {
"outputs": [
{
"name": "out",
"path": "/nix/store/nk0i2hc4x8qpshj326jzn36hkbgi7j7z-gnumake-4.4.1",
"default": true
},
{
"name": "man",
"path": "/nix/store/nacivbyfzp1w0xydskbxrkapp2gn552s-gnumake-4.4.1-man",
"default": true
},
{
"name": "debug",
"path": "/nix/store/f7yag335k3y4ymn9hb5q1yzqc1njf1hi-gnumake-4.4.1-debug"
},
{
"name": "doc",
"path": "/nix/store/782lakmsm9nxf4hpnn8whmq541cphv6r-gnumake-4.4.1-doc"
},
{
"name": "info",
"path": "/nix/store/51cb4z2jrvqdnab9cc8hjf478c8f55hw-gnumake-4.4.1-info"
}
],
"store_path": "/nix/store/nk0i2hc4x8qpshj326jzn36hkbgi7j7z-gnumake-4.4.1"
},
"x86_64-darwin": {
"outputs": [
{
"name": "out",
"path": "/nix/store/0bg5b97b9hbhz8907bfcrch1i116xppa-gnumake-4.4.1",
"default": true
},
{
"name": "man",
"path": "/nix/store/wb4yn3d7vmi1hr4f8pz38rjvcl7f70ma-gnumake-4.4.1-man",
"default": true
},
{
"name": "doc",
"path": "/nix/store/3b9cjf16jx4ncpp3b4k6m8q01nxsrw1n-gnumake-4.4.1-doc"
},
{
"name": "info",
"path": "/nix/store/rriz5p4q8a6sca7sw3p3js1m5dl09ia8-gnumake-4.4.1-info"
}
],
"store_path": "/nix/store/0bg5b97b9hbhz8907bfcrch1i116xppa-gnumake-4.4.1"
},
"x86_64-linux": {
"outputs": [
{
"name": "out",
"path": "/nix/store/vs2sa1msvkjxbb70h47819zq8a9srj21-gnumake-4.4.1",
"default": true
},
{
"name": "man",
"path": "/nix/store/ncw4l9vr02fh2prknq5dhmdycb00rdmr-gnumake-4.4.1-man",
"default": true
},
{
"name": "debug",
"path": "/nix/store/k7lz9zpj52al794kkw0d80hwj2v7pn9h-gnumake-4.4.1-debug"
},
{
"name": "doc",
"path": "/nix/store/7c3i6ivjqgc7hz21bchima12lxk8c1s4-gnumake-4.4.1-doc"
},
{
"name": "info",
"path": "/nix/store/9lbfzcvwx1c859fy6k0l9cbkp6fcxd9l-gnumake-4.4.1-info"
}
],
"store_path": "/nix/store/vs2sa1msvkjxbb70h47819zq8a9srj21-gnumake-4.4.1"
}
}
},
"gnused@latest": {
"last_modified": "2026-10-20T14:23:57Z",
"resolved": "github:NixOS/nixpkgs/5adc50831c259f87faeb09ddf930d43597197473#gnused",
"source": "devbox-search",
"version": "4.10",
"systems": {
"aarch64-darwin": {
"outputs": [
{
"name": "out",
"path": "/nix/store/8fyr2rna4vqibkqsbzyfd4z6fna5c09p-gnused-4.10",
"default": true
},
{
"name": "info",
"path": "/nix/store/6c8gqzdibqfzgnp8876lp4zh1z9f13g1-gnused-4.10-info"
}
],
"store_path": "/nix/store/8fyr2rna4vqibkqsbzyfd4z6fna5c09p-gnused-4.10"
},
"aarch64-linux": {
"outputs": [
{
"name": "out",
"path": "/nix/store/03rgyjxclvhxnif2klf13nkr92rpxqqg-gnused-4.10",
"default": true
},
{
"name": "info",
"path": "/nix/store/sgxd46ij300v6jwrxj5cfjz7a8029jxy-gnused-4.10-info"
}
],
"store_path": "/nix/store/03rgyjxclvhxnif2klf13nkr92rpxqqg-gnused-4.10"
},
"x86_64-linux": {
"outputs": [
{
"name": "out",
"path": "/nix/store/9wc73n5izv0m7ppjbibi4hbr68rba1lj-gnused-4.10",
"default": true
},
{
"name": "info",
"path": "/nix/store/4gfax7m36hw70wn2cqy0dh3dr1r3rlr3-gnused-4.10-info"
}
],
"store_path": "/nix/store/9wc73n5izv0m7ppjbibi4hbr68rba1lj-gnused-4.10"
}
}
},
"go@1.27": {
"last_modified": "2026-10-20T14:23:57Z",
"resolved": "github:NixOS/nixpkgs/5adc50831c259f87faeb09ddf930d43597197473#go_1_27",
"source": "devbox-search",
"version": "1.27.3",
"systems": {
"aarch64-darwin": {
"outputs": [
{
"name": "out",
"path": "/nix/store/s5m10yz53f8778k4mi1irdqzy8iaqri1-go-1.27.3",
"default": true
}
],
"store_path": "/nix/store/s5m10yz53f8778k4mi1irdqzy8iaqri1-go-1.27.3"
},
"aarch64-linux": {
"outputs": [
{
"name": "out",
"path": "/nix/store/0940szmkaskjshd3z0gp2m2kfxcnc424-go-1.27.3",
"default": true
}
],
"store_path": "/nix/store/0940szmkaskjshd3z0gp2m2kfxcnc424-go-1.27.3"
},
"x86_64-linux": {
"outputs": [
{
"name": "out",
"path": "/nix/store/1fng2iqlx5y8rqvhjp9203mymzyny6cz-go-1.27.3",
"default": true
}
],
"store_path": "/nix/store/1fng2iqlx5y8rqvhjp9203mymzyny6cz-go-1.27.3"
}
}
}
}
}
+436
View File
@@ -0,0 +1,436 @@
# abra audit — design
2026-11-20 · Linus Gasser
## Summary
Add a top-level `abra audit` command. It checks every deployed app on every server against a curated list of advisories, and offers to fix what fails. A maintainer-side `abra audit update` turns upstream vulnerability sources into proposed advisories.
## Problem
Operators of Co-op Cloud servers cannot easily tell whether their apps are safe:
- **Mutable tags.** Images such as `php` or `nextcloud` are often rebuilt under the same tag. A server can run a vulnerable image while its compose file looks current.
- **Stale base images.** An image like `nextcloud` is built on `php`. When `php` is fixed, `nextcloud` is not fixed until upstream rebuilds it.
- **Old recipe versions.** Some recipe versions ship known-vulnerable upstream versions, and nothing tells the operator.
- **Unreliable redeploys.** A redeploy should pull the new image. If the registry is overloaded at deploy time, Docker falls back to the bare tag and each node may keep its old local image, with only a warning.
## Goals and non-goals
Goals:
- One command reports, for every deployed app, whether it matches a known advisory.
- Detect vulnerable images even when the tag has not changed, including vulnerable base images.
- Propose a fix for each finding and, on request, apply it and verify the result.
- Help advisory maintainers find new problems from upstream sources, without missing images that were fixed before anyone looked.
- Work on mixed amd64 and arm64 swarms.
Non-goals:
- A full CVE scanner. Deep scanning is delegated to Trivy or Grype as an option.
- Rebuilding upstream images. abra can detect a stale base image but not patch it.
- Publishing advisories automatically. A human reviews every advisory before operators see it.
## Commands
`abra audit` is a top-level command, because an audit spans servers, apps and recipes. `abra app check` already exists (it checks `.env` variables), so that name is not available.
| Command | Who | What it does |
| --- | --- | --- |
| `abra audit` | operator | Audit all deployed apps on all servers |
| `abra audit app <domain>` | operator | Audit one deployed app |
| `abra audit server <server>` | operator | Audit all apps on one server |
| `abra audit recipe <recipe>` | maintainer, CI | Static check of recipe versions and pinned images; nothing deployed needed |
| `abra audit list` | anyone | List advisories, filtered by `--recipe` or `--severity` |
| `abra audit show <id>` | anyone | Details and fix notes for one advisory |
| `abra audit update` | advisory maintainer | Propose new or changed advisories from a CVE, from minimum versions, or from upstream sources |
Two aliases are kept, because operators look under `app` and `server` first:
- `abra app audit <domain>` is the same as `abra audit app <domain>`.
- `abra server audit <server>` is the same as `abra audit server <server>`.
**Storage.** The audit needs an explicitly chosen abra directory. It is taken from, in order:
1. `$ABRA_DIR`;
2. the `abraDir` setting of an `abra.yaml` file in the current directory or one of its parents, as the rest of abra already supports.
If neither exists, all `abra audit` commands fail with a clear error. They do not fall back to `~/.abra`, unlike the rest of abra. Below, `$ABRA_DIR` means this directory. Everything the audit stores lives below it:
| Path | Contents |
| --- | --- |
| `$ABRA_DIR/servers` | Server and app `.env` files, as for the rest of abra |
| `$ABRA_DIR/advisories` | Checkout of the advisories repo: shared, reviewed data |
| `$ABRA_DIR/.cache/audit/images` | Local cache of image configs, by digest; from v2 on, safe to delete |
**Fetching.** There is no cache lifetime: every invocation checks whether the advisory list has changed, and pulls it if so. The list is a git repo, so this is a `git fetch` of a few KB when nothing changed. When the repo cannot be reached, the audit warns, says how old the local copy is, and uses it.
### Options for `audit`, `app` and `server`
```
-s, --server <name> only apps on this server
-r, --recipe <name> only apps of this recipe
-a, --advisories <url|path> advisory source (default: c4dt/advisories)
--severity <level> minimum severity: low, medium, high, critical
--include-drafts also apply draft advisories (marked as such)
-f, --fix propose fixes, apply after confirmation
-n, --no-input apply fixes without asking
--refresh-only only re-pull images, never upgrade the recipe
--retries <n> re-checks after a fix, with backoff (default 3)
-o, --offline no registry or advisory fetch; recipe checks only
--deep also scan running digests with Trivy or Grype
--report send unknown digests to the advisories repo (opt-in)
-m, --machine JSON output for cron and monitoring
```
`abra audit recipe` takes only `--machine`, so CI can fail on findings.
### Exit codes
The exit code describes the state **after** the run, so a finding that was fixed and verified no longer counts.
| Code | Meaning |
| --- | --- |
| 0 | No findings remain: none were found, or all were fixed and verified |
| 1 | Findings remain that were not fixed: no `--fix`, fix declined, or strategy `manual` |
| 2 | A fix was tried but could not be verified, for example because the registry was unreachable |
With several apps, the highest code wins. The report lists every finding with its final status (`fixed`, `affected`, `unknown`, `unverified`), so fixed findings stay visible in the output and in `--machine` JSON.
## Advisory format
Advisories are YAML files, one per advisory, in a separate git repo under `advisories/`. They are kept apart from the recipe catalogue because they change on a different schedule and need their own review. The repo starts as `c4dt/advisories` and can move to `coop-cloud/advisories` once the format has settled.
```yaml
id: CCA-2026-0012
title: PHP 8.3 CGI argument injection
severity: critical # low | medium | high | critical
status: published # draft | published | withdrawn
refs: [CVE-2026-12345, https://example.org/advisory]
published: 2026-10-20
# Recipe versions: semver range on the recipe part of the tag
recipes:
nextcloud:
affected: "<7.0.3+30.0.1"
fixed: "7.0.3+30.0.1"
# Images: an image is fixed if ANY condition holds
images:
- match: "nextcloud:30*-apache"
fixed_from: sha256:2222... # this digest and all first seen after it
affected_digests: [sha256:1111...]
- match: "*" # any image...
base: "php:8.3*" # ...built on php 8.3
env: { PHP_VERSION: "<8.3.14" } # read from the image config
fix:
strategy: refresh # upgrade | refresh | manual
notes: "Upstream images rebuilt on 2026-10-18; a redeploy is enough."
```
### Image conditions
Digests are hashes and have no order, so "newer than digest X" cannot be computed from the digest itself. Instead, the format uses what the image says about itself, in order of preference:
| Condition | Field | Reliability | Notes |
| --- | --- | --- | --- |
| Version in the image | `env`, `version` | Best | Tests the fix itself, per platform, without pulling layers |
| Build time | `created_after` | Good, if the image records its real build time | Covers rebuilds where no version changes; see below |
| Base ancestry | `base`, `base_digest_from` | Good | Docker Official Images record their base in OCI annotations |
| Allowlist | `fixed_digests` | Exact but closed | Later rebuilds need the list updated |
| Denylist | `affected_digests` | Exact but incomplete | For one known-bad digest |
| Digest history | `fixed_from` | Good | Fallback for images whose build time is unusable; later version |
An image that matches `match` but satisfies no condition is reported as **unknown**, never as OK.
How the conditions combine, for an image that matches `match`:
1. A digest in `affected_digests` is affected; a digest in `fixed_digests` is fixed.
2. `env` both selects and tests. If the image lacks one of the variables, the entry does not apply, because the image is not built on that runtime. Otherwise the ranges are **affected** ranges: if every variable is in its range, `env` says affected, else fixed. Write `">=8.3.0 <8.3.14"` to limit the entry to one release series.
3. `version` is an affected range on `org.opencontainers.image.version`.
4. `created_after` says fixed for images built at or after that time, and affected for older ones, unless the build time is unusable.
5. If any condition says fixed, the image is fixed. Otherwise, if one says affected, it is affected. Otherwise it is unknown.
`match` is a glob where `*` matches any characters. A pattern with a `:` is matched against `name:tag`, otherwise against the name only, using Docker's short names (`nextcloud`, not `docker.io/library/nextcloud`). Version ranges are space- or comma-separated constraints (`<`, `<=`, `>`, `>=`, `=`, `!=`) that must all hold, with `||` between alternatives. A recipe range without an upstream part (`<7.0.3`) only compares the recipe part of the tag.
### Ordering images by version and build time
Most fixes change a version that the image shows, so no digest bookkeeping is needed:
- **Application version.** `NEXTCLOUD_VERSION` in the config, or the annotation `org.opencontainers.image.version` (for example `35.0.3-apache`).
- **Inherited runtime version.** Images inherit environment variables from their base, so `nextcloud:apache` also carries `PHP_VERSION=8.5.13`. A nextcloud rebuilt on a fixed php shows the new `PHP_VERSION`, even though `NEXTCLOUD_VERSION` is unchanged.
That leaves rebuilds where every visible version stays the same, typically a Debian package such as openssl patched in the base image. For these, the build time decides: `created_after` is the `Created` time of the first fixed image, and any image of that tag built later counts as fixed.
**Is the build time monotonic?** Per tag, in practice yes: every rebuild is a new build run with a later timestamp. On Docker Hub on 2026-11-20, `php:8.3-apache` recorded its real build time (2026-11-19 14:40 UTC), and `nextcloud:apache` was built 33 minutes later on top of it. Both also carry the annotations `org.opencontainers.image.base.name` and `base.digest`. The exceptions are known:
| Case | Effect | Handling |
| --- | --- | --- |
| Reproducible builds (`SOURCE_DATE_EPOCH`, ko, Nix, Bazel) set `Created` to a commit date or to 1970 | A base-only rebuild keeps the same `Created`, so old and new cannot be told apart | Detect it, then report unknown |
| Upstream moves a tag back to an older image | The older `Created` is reported as affected | Correct as is |
| Clock skew between build machines | Minutes | Irrelevant next to fix dates days apart |
The audit treats `Created` as unusable, and reports unknown, when:
- it is before 2000, which catches the 1970 epoch;
- it is earlier than the `Created` of the base image named in `base.digest`, because an image cannot be built before its base;
- two different digests of the same tag have the same `Created`.
The advisory can then fall back to `fixed_digests`, or later to the digest history.
### Fix strategies
| Strategy | Meaning | What `--fix` does |
| --- | --- | --- |
| `upgrade` | A newer recipe version fixes it | `abra app upgrade <domain> <fixed version>` |
| `refresh` | The same tag now points to a fixed image | Redeploy the same version, then verify the running digest |
| `manual` | No fixed image exists yet | Report only, with the notes and options such as pinning a digest |
## Digest history (fallback, later version)
The digest history is only needed for images whose build time is unusable, as listed above. It is deferred to v4, and built only if such images turn out to matter.
The advisories repo would keep an append-only history per image tag, recording when each digest was first seen:
```yaml
# digests/nextcloud/30-apache.yml (append-only)
- digest: sha256:1111...
first_seen: 2026-10-02T08:00:00Z
config: { created: 2026-10-01, NEXTCLOUD_VERSION: 30.0.3, PHP_VERSION: 8.3.12 }
- digest: sha256:2222...
first_seen: 2026-10-18T14:00:00Z
config: { created: 2026-10-18, NEXTCLOUD_VERSION: 30.0.3, PHP_VERSION: 8.3.14 }
```
It would be filled from three sources:
1. **`abra audit update`** runs regularly from CI and appends every new digest it sees, with its image config. The config is stored because registries may delete old digests once they are untagged.
2. **Backfill from `docker-library/repo-info`**, which keeps years of per-tag digests for official images in git.
3. **Running servers.** With `--report`, `abra audit` sends digests it finds that are not in the history. This is opt-in.
## Images fixed before anyone looked
Suppose a CVE is published and upstream pushes a fixed image under the same tag before `update` runs. `update` never sees the vulnerable image, but old installations still run it. This is handled because advisories are driven by the CVE, not by the current image:
- The CVE creates the advisory, whether or not today's image is clean.
- A clean current image only sets the strategy to `refresh`. Its versions become the `env` condition, or its `Created` time becomes `created_after`.
- On a server, an image with older versions or an earlier build time is reported as affected. Both lead to a refresh and a verification.
`update` only needs to see the fixed image, and that is the one available when it runs.
## Audit and fix
`abra audit` checks each app in the same way, and a fix counts only after the running digest has been verified.
1. **Read state.** From the service spec, read the recipe version label and each image reference `image:tag@sha256:...`. For per-platform checks, read the node's platform and fetch that image config from the registry.
2. **Match advisories.** First recipe ranges, then image conditions.
3. **Classify** each finding as affected or unknown. No finding means clean.
4. **Fix** (with `--fix`) according to the advisory's strategy, then verify.
```mermaid
flowchart TD
read[Read service spec] --> match[Match advisories]
match --> finding{Finding?}
finding -- no --> clean[Clean, exit 0]
finding -- affected or unknown --> strategy[Pick fix strategy]
strategy -- manual --> report[Report, exit 1]
strategy -- upgrade or refresh --> apply[Upgrade / refresh]
apply --> verified{Digest fixed?}
verified -- yes --> fixed[Fixed, exit 0]
verified -- no --> retry[Retry, backoff]
retry -- within --retries --> apply
retry -- retries used up --> unverified[Unverified, exit 2]
```
Verification waits for the service to converge, then reads the spec again. It fails if the spec holds a bare tag (the registry lookup failed) or a digest that still matches the advisory. Retries use backoff, for example 30 s, 2 min and 10 min, before giving up with exit code 2.
## `abra audit update`
`abra audit update` is a maintainer tool. It compares the recipes in the catalogue with a vulnerability, and proposes new or changed advisories for a human to review. It has three modes; the first two come first (see [Versions](#versions)).
```
abra audit update --cve <CVE-ID> # one known vulnerability
abra audit update --require <lib>=<v>[,<v>...] # minimum patched versions
abra audit update --sources [recipe...] # scan upstream sources (later)
--repo <path> local checkout of the advisories repo (default $ABRA_DIR/advisories)
--source <list> with --sources: endoflife,osv,ghsa,kev,repo-info,trivy (default: all but trivy)
--since <date> with --sources: only upstream changes after this date (default: last run)
-n, --no-input write all proposals as drafts without asking
--pr commit accepted proposals on a branch and open a PR
```
### Sources
| Source | Gives | Becomes |
| --- | --- | --- |
| [endoflife.date](https://endoflife.date) | End-of-life dates for runtimes such as php and postgres | `medium` advisory, strategy `upgrade` |
| [OSV.dev](https://osv.dev) and [GitHub Advisory Database](https://github.com/advisories) | CVEs with affected and fixed upstream versions | Recipe ranges, mapped through the `+UPSTREAM` part of recipe tags |
| [CISA KEV](https://www.cisa.gov/known-exploited-vulnerabilities-catalog) | CVEs exploited in the wild | Severity raised to `critical` |
| [docker-library/repo-info](https://github.com/docker-library/repo-info) | Digest and package history of official images | Digest history; `fixed_from` for rebuilt images |
| Trivy (opt-in, slow) | CVEs found in current catalogue images | Proposals the other sources missed |
### Steps
1. **Inventory.** List every recipe version in the catalogue and its images. Fetch each image config for runtime versions (`PHP_VERSION`, `NODE_VERSION`) and find its base image.
2. **Record digests.** Append new digests and their configs to the digest history.
3. **Query sources** for the upstream apps and runtimes found in step 1.
4. **Classify.** For each CVE, write a condition that separates affected from fixed images. Look at the current image only to choose the strategy: `refresh` if the current image is clean, `manual` if it is not.
5. **Deduplicate.** Match on CVE ID and affected recipe or image. If an advisory already covers the finding, propose a change (wider range, new fixed digest, higher severity) instead of a new entry.
6. **Review.** Show each proposal as a YAML diff: accept, edit, skip, or ignore permanently. Ignored findings go to `ignore.yml` with a reason, so they do not return.
7. **Write** accepted proposals with `status: draft`, optionally on a branch with a PR. Only a human sets `status: published`.
A human is needed because the sources say what is vulnerable upstream. Only a maintainer can judge whether it matters for the recipe's configuration and which fix strategy applies.
## Multiple platforms
abra already works on mixed amd64 and arm64 swarms, because it leaves the platform choice to Docker. The audit must compare digests at the right level.
**Current behaviour.** abra has no platform-specific code. `deploy`, `upgrade` and `rollback` use `ResolveImageAlways` (`cli/app/deploy.go:157`), so Docker resolves `image:tag` to the index digest that covers all platforms. Each node then pulls its own variant. `GetRegistryDigest` in `pkg/client/registry.go` also returns the index digest, so pinned `image:tag@sha256:...` references work on every platform.
| Question | Compare | Where to read it |
| --- | --- | --- |
| Has the tag moved? | Index digest | Service spec, and a registry HEAD request |
| Which PHP version is inside? | Per-platform image config | Pick the node's platform from the index (`docker node inspect` gives `Description.Platform`) |
| Is it built on the fixed base? | Per-platform layers | Same per-platform manifest |
Pitfalls:
- **A new index digest does not prove the fix for your platform.** Any platform's rebuild changes the index digest. Some upstreams rebuild arm64 later or not at all, so important fixes are checked per platform.
- **Copied per-platform digests.** The Docker Hub web page shows per-platform digests most prominently. A maintainer who pins one breaks other platforms with `no matching manifest`. `abra recipe lint` should warn when a pinned digest is not an index but the image is multi-arch.
- **Containers show the image ID.** `docker inspect` on a container gives the config digest, not the manifest digest. Use the service spec for comparisons.
## What Docker allows
abra can detect almost any stale or vulnerable layer, but it can only fix an image that upstream has already rebuilt.
| Possible | How |
| --- | --- |
| Know exactly what is running | Swarm stores `image:tag@sha256:...` in the service spec |
| Tell whether a tag has moved | Registry HEAD request for the tag's manifest digest |
| Read versions without pulling | Fetch the image config (a few KB): `Created` and `Env`, which often inherits `PHP_VERSION` or `NGINX_VERSION` from the base |
| Find the base image | OCI annotations `org.opencontainers.image.base.name` and `base.digest` where set; otherwise layer-prefix matching against the base's layers |
| Go deeper, to libc or openssl | SBOM and provenance attestations, which Docker Official Images publish |
| Not possible | Consequence |
| --- | --- |
| Patch a base layer of someone else's image | If upstream has not rebuilt, the options are waiting, pinning a known-good older digest, or having the recipe build its own image |
| Trust a redeploy without checking | On registry failure Docker deploys the bare tag; every refresh must be verified |
**Undeploy is not needed.** `undeploy` then `deploy` has the same registry problem as a plain redeploy, and adds downtime. A redeploy of the same version with `ResolveImageAlways` already asks the registry again.
**Pinned digests make failures visible.** With `image: nextcloud:30-apache@sha256:...` in the recipe, a failed registry lookup fails the deploy instead of running a stale image. The `allow_digest` branch adds this support to `abra recipe upgrade`.
## Versions
The feature is built in four versions. Each one is usable on its own, and is tried on real servers before the next one starts.
| Version | Adds | Changes servers? |
| --- | --- | --- |
| v1 | `abra audit`: report failing apps and suggest commands | No |
| v2 | `abra audit update --cve` and `--require`: draft advisories for one vulnerability | No |
| v3 | Subcommands, options, `--fix` with verification, deploy warnings | Yes, with `--fix` |
| v4 | Automatic advisory proposals from upstream sources, digest history | No |
### v1: basic audit
Answers "which of my apps need attention, and what do I run?" using only a hand-written advisory list.
- **Command.** `abra audit`, without arguments. It goes through every server in `$ABRA_DIR/servers` and every deployed app on it, using the same data as `abra app ls -S`.
- **Advisories.** Hand-written YAML in `c4dt/advisories`, pulled on every run. The first advisory is the nextcloud same-tag rebuild. Only `published` advisories apply; `--include-drafts` also applies `draft` ones, marked "(draft)" in the output, so a maintainer can try an advisory on real servers before publishing it. `withdrawn` advisories never apply.
- **Conditions.** Only recipe ranges (`affected`, `fixed`), `env`, `version`, `created_after` (with the checks for unusable build times), `fixed_digests` and `affected_digests`. Fields v1 does not know are skipped with a warning, so newer advisories do not break older abra versions.
- **Reads only.** Service specs (version label and image digests) and image configs. On a single-node server, the common case, the image config is read from the node itself with `docker image inspect`, so no registry is needed.
- **Output.** Only failing apps. A narrow numbered table lists app, recipe, advisory, severity and status (`affected` or `unknown`), with a server column only when several servers are audited. Below it, a numbered list gives the suggested fix for each row, wrapped to the terminal width, because fixes are too long for a table column. Unreachable servers follow, with only the useful part of the SSH error. The suggested fixes are:
- upgrade: `abra app upgrade <domain> <fixed version>`
- refresh: `abra app deploy <domain> --force`, followed by `abra audit` to check the result
- manual: the advisory's notes
- **Fix check.** Every suggestion is checked before it is shown:
- Before suggesting a redeploy, the audit reads the tag's current image from the registry and applies the advisory to it. Upstreams stop rebuilding patch-pinned tags such as `nextcloud:35.0.2-fpm` once the next patch is out, so if the current image is not fixed either, the suggestion becomes a recipe upgrade.
- For a recipe upgrade, the newer releases of the recipe are searched for the oldest one whose images are fixed. Images are read from the `compose*.yml` files of each release tag, without changing the local checkout. If no release is fixed, the audit says so and warns: a new recipe release is needed first.
- For a recipe version range, the advisory's `fixed` version must be a released tag.
- If the registry or the recipe cannot be read, the suggestion is kept and marked as unchecked.
- **Recipe names.** Recipes are shown relative to `git.coopcloud.tech/coop-cloud/`: `nextcloud` for the canonical recipe, `c4dt/nextcloud` for a fork on the same server, and the full path elsewhere. Advisories name recipes the same way, so `nextcloud` in an advisory does not match the fork `c4dt/nextcloud`, which has its own versions.
- **Exit code.** 0 when nothing fails, 1 otherwise.
- **Errors.** An unreachable server is listed as such and the run continues.
**Test.** On a test server, deploy one old nextcloud recipe version and one app with a stale image under a current tag. Write an advisory for each. Check that both are reported, that following the suggestions fixes them, and that a second `abra audit` is clean.
### v2: `abra audit update` for one vulnerability
Turns a single vulnerability into draft advisories, so the maintainer does not have to check every recipe by hand. There are two ways to describe the vulnerability.
**By CVE: `abra audit update --cve CVE-2026-12345`.** The CVE is looked up in [OSV.dev](https://osv.dev), which also covers GitHub advisories and package ecosystems such as Packagist, npm, PyPI, Debian and Alpine. This gives the affected packages and their fixed versions. Recipes are then matched in two ways:
- applications, such as nextcloud, through the `+UPSTREAM` part of the recipe tags;
- runtimes and libraries, such as php, through the image config (next paragraph).
**By minimum versions: `abra audit update --require php=8.5.12,8.4.27,8.3.36,8.2.35`.** One minimum version per release series. For every image in the catalogue that contains the library:
| Image contains | Result |
| --- | --- |
| A listed series, below its minimum (php 8.4.20) | Affected |
| A listed series, at or above its minimum (php 8.4.27) | Fixed |
| A series that is not listed (php 8.1.x) | Unsupported, proposed as a separate end-of-life advisory |
Libraries are found through environment variables in the image config, which images inherit from their base:
| Library | Variable |
| --- | --- |
| php | `PHP_VERSION` |
| node | `NODE_VERSION` |
| python | `PYTHON_VERSION` |
| nginx | `NGINX_VERSION` |
| postgres | `PG_VERSION` |
| redis | `REDIS_VERSION` |
An unknown library is an error that lists the known ones. Finding libraries through SBOMs comes later.
**Which recipe versions.** Every published version of each recipe that uses the library is checked, so the proposal gets a correct `affected` range. Image configs are a few KB and are cached by digest in `$ABRA_DIR/.cache/audit/images/`, so this is cheap after the first run. A digest never changes its content, so this cache never expires.
**Strategy.** For each affected recipe, the current image under the same tag is checked too:
- a newer recipe version uses a fixed image → `upgrade` to that version;
- the same tag now points to a fixed image → `refresh`, with the current digest in `fixed_digests`;
- nothing is fixed yet → `manual`.
**Output.** Draft advisory files in the local checkout, each shown as a diff to accept, edit or skip. v2 does not commit or open pull requests.
**Test.** Run `--require` with the versions of a recent php security release, and compare the proposals with a list made by hand. Run `--cve` with a recent nextcloud CVE and check the recipe range.
### v3: arguments and fixes
Makes the audit usable for single apps and for automation, and lets it apply fixes.
- **Subcommands.** `abra audit app`, `server`, `recipe`, `list` and `show`, plus the aliases `abra app audit` and `abra server audit`.
- **Options.** `--server`, `--recipe`, `--severity`, `--machine`, `--offline` and `--advisories`.
- **Fixes.** `--fix`, `--refresh-only`, `--no-input` and `--retries`, with verification after each fix and exit code 2 when verification fails.
- **Deploy warnings.** `abra app deploy` and `abra app upgrade` warn, without refusing, when the target version matches an advisory.
- **Platforms.** `env` conditions are checked per platform on mixed amd64 and arm64 swarms.
**Test.** Run `abra audit --fix` against the v1 test setup, with the registry blocked for one attempt (for example with a firewall rule), and check that the retry succeeds and that the exit code is right in each case.
### v4: more automatic updates
Moves advisory maintenance from "react to a CVE" to "be told about new problems".
- `abra audit update --sources`, run on a schedule in CI, with endoflife.date, OSV, GHSA, CISA KEV and docker-library/repo-info.
- Deduplication against existing advisories, `ignore.yml`, and `--pr`.
- Digest history and `fixed_from`, only if images with unusable build times (reproducible builds) turn out to matter.
- Optional extras: `--deep` scanning with Trivy or Grype, the lint rule for per-platform digests, a status column in `abra app list`, and `--report`.
- Moving the advisories repo to `coop-cloud/advisories`.
## Decisions
| Question | Decision |
| --- | --- |
| Who maintains the advisories, and how fast? | Linus Gasser, manually, in `c4dt/advisories`, for now |
| Should `--report` exist? | Not in the first versions |
| How long to cache the advisory list? | No cache: every invocation checks for the latest list |
| What does `abra app deploy` do when the target version matches an advisory? | It warns, and does not refuse |
## Open questions
- [ ] Do the registries we depend on keep untagged digests long enough to read their config later? This matters only for the digest history (v4), because the config of a running image can be read from the node. To find out, take old `php` digests from docker-library/repo-info and try to fetch their manifests from Docker Hub.
- [ ] Which images used by recipes are built reproducibly, so that their `Created` time is unusable? `abra audit update` could list them while scanning the catalogue.
Generated
+38
View File
@@ -18,6 +18,27 @@
"type": "github"
}
},
"microvm": {
"inputs": {
"nixpkgs": [
"nixpkgs"
],
"spectrum": "spectrum"
},
"locked": {
"lastModified": 1784666190,
"narHash": "sha256-xgfS6slV7J3baMooNN1UuBi51RIgg9y0DbCxfSA0668=",
"owner": "astro",
"repo": "microvm.nix",
"rev": "fa5340ac684cdce8a22b6d4a0bcebb0cc999275e",
"type": "github"
},
"original": {
"owner": "astro",
"repo": "microvm.nix",
"type": "github"
}
},
"nixpkgs": {
"locked": {
"lastModified": 1778443072,
@@ -37,9 +58,26 @@
"root": {
"inputs": {
"flake-utils": "flake-utils",
"microvm": "microvm",
"nixpkgs": "nixpkgs"
}
},
"spectrum": {
"flake": false,
"locked": {
"lastModified": 1783694892,
"narHash": "sha256-xO8f7Qng+18FK2UlB9vcrkxCaQMCt5WjCH24aW/11eg=",
"ref": "refs/heads/main",
"rev": "24c4346e30fdea8d8e80f34aec3554a15a667d24",
"revCount": 1410,
"type": "git",
"url": "https://spectrum-os.org/git/spectrum"
},
"original": {
"type": "git",
"url": "https://spectrum-os.org/git/spectrum"
}
},
"systems": {
"locked": {
"lastModified": 1681028828,
+87 -18
View File
@@ -4,34 +4,103 @@
inputs = {
nixpkgs.url = "github:nixos/nixpkgs?ref=nixos-unstable";
flake-utils.url = "github:numtide/flake-utils";
microvm = {
url = "github:astro/microvm.nix";
inputs.nixpkgs.follows = "nixpkgs";
};
};
outputs =
{
self,
nixpkgs,
microvm,
flake-utils,
}:
flake-utils.lib.eachDefaultSystem (
system:
let
pkgs = nixpkgs.legacyPackages.${system};
in
{
packages = rec {
abra = pkgs.callPackage ./package.nix { };
default = abra;
let
system = "x86_64-linux";
pkgs = nixpkgs.legacyPackages.${system};
publicKey = builtins.readFile ./tests/resources/local_integration_ssh.pub;
privateKey = builtins.readFile ./tests/resources/local_integration_ssh;
pathToRepo = builtins.readFile ./tests/resources/path_to_repo;
# local DNS aliases for the server host
extraHosts = builtins.readFile ./tests/resources/extra_hosts;
username = "abra";
password = "abra";
defaultDNS = [
# Quad9.net
"9.9.9.9"
"149.112.112.112"
"2620:fe::fe"
"2620:fe::9"
];
in
{
nixosConfigurations = {
abra-client = nixpkgs.lib.nixosSystem {
system = "x86_64-linux";
modules = [
microvm.nixosModules.microvm
./nix/hosts/client/configuration.nix
];
specialArgs = {
inherit
publicKey
privateKey
username
password
defaultDNS
extraHosts
pathToRepo
;
};
};
apps = rec {
abra = flake-utils.lib.mkApp { drv = self.packages.${system}.abra; };
default = abra;
abra-server = nixpkgs.lib.nixosSystem {
system = "x86_64-linux";
modules = [
microvm.nixosModules.microvm
./nix/hosts/server/configuration.nix
];
specialArgs = {
inherit
publicKey
privateKey
username
password
defaultDNS
;
};
};
devShells.default = pkgs.mkShell {
packages = with pkgs; [
go_1_26
gnumake
};
nixosModules = rec {
host = {
imports = [
microvm.nixosModules.host
./nix/modules/host.nix
];
};
}
);
default = host;
};
packages = rec {
abra = pkgs.callPackage ./nix/package.nix { };
default = abra;
};
apps = rec {
abra = flake-utils.lib.mkApp { drv = self.packages.${system}.abra; };
default = abra;
};
devShells.${system}.default = pkgs.mkShell {
# testing env variables
BATS_LIB_PATH = "~/.local/share/bats/";
TEST_SERVER = "abra.local";
ABRA_DIR = "$HOME/.abra_test";
packages = with pkgs; [
go_1_26
gnumake
gopls
];
};
};
}
+28 -32
View File
@@ -1,6 +1,6 @@
module coopcloud.tech/abra
go 1.26.0
go 1.27
require (
coopcloud.tech/tagcmp v0.0.0-20260515102403-c26951b55977
@@ -9,18 +9,26 @@ require (
github.com/charmbracelet/bubbletea v1.3.10
github.com/charmbracelet/lipgloss v1.1.0
github.com/charmbracelet/log v1.0.0
github.com/containers/image v3.0.2+incompatible
github.com/decentral1se/passgen v1.0.1
github.com/distribution/reference v0.6.0
github.com/docker/cli v28.4.0+incompatible
github.com/docker/docker v28.5.2+incompatible
github.com/docker/go-units v0.5.0
github.com/go-git/go-git/v5 v5.19.2
github.com/go-git/go-git/v5 v5.19.3
github.com/google/go-cmp v0.7.0
github.com/leonelquinteros/gotext v1.7.2
github.com/hashicorp/go-retryablehttp v0.7.8
github.com/leonelquinteros/gotext v1.8.0
github.com/moby/sys/signal v0.7.1
github.com/moby/term v0.5.2
github.com/opencontainers/go-digest v1.0.0
github.com/opencontainers/image-spec v1.1.1
github.com/pkg/errors v0.9.1
github.com/schollz/progressbar/v3 v3.19.1
golang.org/x/term v0.46.0
github.com/spf13/cobra v1.10.1
github.com/stretchr/testify v1.12.1
golang.org/x/sys v0.50.0
golang.org/x/term v0.48.0
gopkg.in/yaml.v3 v3.0.1
gotest.tools/v3 v3.5.2
)
@@ -48,24 +56,29 @@ require (
github.com/containerd/errdefs/pkg v0.3.0 // indirect
github.com/containerd/log v0.1.0 // indirect
github.com/containerd/platforms v0.2.1 // indirect
github.com/containers/storage v1.38.2 // indirect
github.com/cpuguy83/go-md2man/v2 v2.0.7 // indirect
github.com/cyphar/filepath-securejoin v0.6.1 // indirect
github.com/docker/distribution v2.8.3+incompatible // indirect
github.com/docker/docker-credential-helpers v0.9.5 // indirect
github.com/docker/go-connections v0.6.0 // indirect
github.com/docker/go-metrics v0.0.1 // indirect
github.com/docker/libtrust v0.0.0-20160708172513-aabc10ec26b7 // indirect
github.com/emirpasic/gods v1.18.1 // indirect
github.com/erikgeiser/coninput v0.0.0-20211004153227-1c3628e74d0f // indirect
github.com/felixge/httpsnoop v1.0.4 // indirect
github.com/fvbommel/sortorder v1.1.0 // indirect
github.com/ghodss/yaml v1.0.0 // indirect
github.com/go-git/gcfg v1.5.1-0.20230307220236-3a3c6141e376 // indirect
github.com/go-git/go-billy/v5 v5.9.0 // indirect
github.com/go-git/go-billy/v5 v5.9.2 // indirect
github.com/go-logfmt/logfmt v0.6.1 // indirect
github.com/go-logr/logr v1.4.3 // indirect
github.com/go-logr/stdr v1.2.2 // indirect
github.com/go-viper/mapstructure/v2 v2.5.0 // indirect
github.com/golang/groupcache v0.0.0-20241129210726-2c02b8208cf8 // indirect
github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/gorilla/mux v1.8.1 // indirect
github.com/grpc-ecosystem/grpc-gateway/v2 v2.28.0 // indirect
github.com/hashicorp/go-cleanhttp v0.5.2 // indirect
github.com/inconshreveable/mousetrap v1.1.0 // indirect
@@ -73,7 +86,6 @@ require (
github.com/kballard/go-shellquote v0.0.0-20180428030007-95032a82bc51 // indirect
github.com/kevinburke/ssh_config v1.6.0 // indirect
github.com/klauspost/compress v1.18.5 // indirect
github.com/klauspost/cpuid/v2 v2.3.0 // indirect
github.com/lucasb-eyer/go-colorful v1.4.0 // indirect
github.com/mattn/go-colorable v0.1.14 // indirect
github.com/mattn/go-isatty v0.0.22 // indirect
@@ -84,7 +96,9 @@ require (
github.com/mitchellh/colorstring v0.0.0-20190213212951-d06e56a500db // indirect
github.com/moby/docker-image-spec v1.3.1 // indirect
github.com/moby/go-archive v0.1.0 // indirect
github.com/moby/patternmatcher v0.6.1 // indirect
github.com/moby/sys/atomicwriter v0.1.0 // indirect
github.com/moby/sys/sequential v0.6.0 // indirect
github.com/moby/sys/user v0.4.0 // indirect
github.com/moby/sys/userns v0.1.0 // indirect
github.com/morikuni/aec v1.1.0 // indirect
@@ -92,19 +106,22 @@ require (
github.com/muesli/cancelreader v0.2.2 // indirect
github.com/muesli/termenv v0.16.0 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/opencontainers/go-digest v1.0.0 // indirect
github.com/opencontainers/runc v1.1.13 // indirect
github.com/opencontainers/runtime-spec v1.1.0 // indirect
github.com/pjbgf/sha1cd v0.6.0 // indirect
github.com/pjbgf/sha1cd v0.7.0 // indirect
github.com/prometheus/client_golang v1.23.2 // indirect
github.com/prometheus/client_model v0.6.2 // indirect
github.com/prometheus/common v0.67.5 // indirect
github.com/prometheus/procfs v0.20.1 // indirect
github.com/rivo/uniseg v0.4.7 // indirect
github.com/russross/blackfriday/v2 v2.1.0 // indirect
github.com/sergi/go-diff v1.4.0 // indirect
github.com/sirupsen/logrus v1.9.4 // indirect
github.com/skeema/knownhosts v1.3.2 // indirect
github.com/spf13/pflag v1.0.10 // indirect
github.com/theupdateframework/notary v0.7.0 // indirect
github.com/xanzy/ssh-agent v0.3.3 // indirect
github.com/xeipuuv/gojsonpointer v0.0.0-20190905194746-02993c407bfb // indirect
github.com/xeipuuv/gojsonreference v0.0.0-20180127040603-bd5ef7bd5415 // indirect
github.com/xeipuuv/gojsonschema v1.2.0 // indirect
github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e // indirect
@@ -122,10 +139,10 @@ require (
go.opentelemetry.io/proto/otlp v1.10.0 // indirect
go.yaml.in/yaml/v2 v2.4.4 // indirect
go.yaml.in/yaml/v3 v3.0.5 // indirect
golang.org/x/crypto v0.53.0 // indirect
golang.org/x/crypto v0.56.0 // indirect
golang.org/x/exp v0.0.0-20260410095643-746e56fc9e2f // indirect
golang.org/x/net v0.56.0 // indirect
golang.org/x/text v0.39.0 // indirect
golang.org/x/net v0.57.0 // indirect
golang.org/x/text v0.41.0 // indirect
golang.org/x/time v0.15.0 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20260401024825-9d38bb4040a9 // indirect
google.golang.org/genproto/googleapis/rpc v0.0.0-20260401024825-9d38bb4040a9 // indirect
@@ -135,27 +152,6 @@ require (
gopkg.in/yaml.v2 v2.4.0 // indirect
)
require (
github.com/containers/image v3.0.2+incompatible
github.com/containers/storage v1.38.2 // indirect
github.com/decentral1se/passgen v1.0.1
github.com/docker/docker-credential-helpers v0.9.5 // indirect
github.com/fvbommel/sortorder v1.1.0 // indirect
github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 // indirect
github.com/gorilla/mux v1.8.1 // indirect
github.com/hashicorp/go-retryablehttp v0.7.8
github.com/moby/patternmatcher v0.6.1 // indirect
github.com/moby/sys/sequential v0.6.0 // indirect
github.com/opencontainers/image-spec v1.1.1 // indirect
github.com/prometheus/client_golang v1.23.2 // indirect
github.com/sergi/go-diff v1.4.0 // indirect
github.com/spf13/cobra v1.10.1
github.com/stretchr/testify v1.12.1
github.com/theupdateframework/notary v0.7.0 // indirect
github.com/xeipuuv/gojsonpointer v0.0.0-20190905194746-02993c407bfb // indirect
golang.org/x/sys v0.48.0
)
replace github.com/docker/cli v28.4.0+incompatible => git.coopcloud.tech/toolshed/docker-cli v28.5.3-0.20260202112816-30df2d0b3a00+incompatible
replace github.com/spf13/cobra => github.com/decentral1se/cobra v1.10.2
+22 -20
View File
@@ -388,12 +388,12 @@ github.com/gliderlabs/ssh v0.3.8 h1:a4YXD1V7xMF9g5nTkdfnja3Sxy1PVDCj1Zg4Wb8vY6c=
github.com/gliderlabs/ssh v0.3.8/go.mod h1:xYoytBv1sV0aL3CavoDuJIQNURXkkfPA/wxQ1pL1fAU=
github.com/go-git/gcfg v1.5.1-0.20230307220236-3a3c6141e376 h1:+zs/tPmkDkHx3U66DAb0lQFJrpS6731Oaa12ikc+DiI=
github.com/go-git/gcfg v1.5.1-0.20230307220236-3a3c6141e376/go.mod h1:an3vInlBmSxCcxctByoQdvwPiA7DTK7jaaFDBTtu0ic=
github.com/go-git/go-billy/v5 v5.9.0 h1:jItGXszUDRtR/AlferWPTMN4j38BQ88XnXKbilmmBPA=
github.com/go-git/go-billy/v5 v5.9.0/go.mod h1:jCnQMLj9eUgGU7+ludSTYoZL/GGmii14RxKFj7ROgHw=
github.com/go-git/go-billy/v5 v5.9.2 h1:OXFSRyz4g20upsGDJgQG9Bak1l/ZEv8GHVYB52O71sE=
github.com/go-git/go-billy/v5 v5.9.2/go.mod h1:ExsU+jcGwXTBOnyilvAnEM1wug1IxHr4yP2ZXsNRtV0=
github.com/go-git/go-git-fixtures/v4 v4.3.2-0.20231010084843-55a94097c399 h1:eMje31YglSBqCdIqdhKBW8lokaMrL3uTkpGYlE2OOT4=
github.com/go-git/go-git-fixtures/v4 v4.3.2-0.20231010084843-55a94097c399/go.mod h1:1OCfN199q1Jm3HZlxleg+Dw/mwps2Wbk9frAWm+4FII=
github.com/go-git/go-git/v5 v5.19.2 h1:wkfn7vOlUBu8ivAWKBWisTiwJK4jYHzTF8Ndv1LyGqY=
github.com/go-git/go-git/v5 v5.19.2/go.mod h1:QqCBE1EFN5ddFmrliLQ3/ntRCUjZU3EJuwuB/jWEHjk=
github.com/go-git/go-git/v5 v5.19.3 h1:qHttbZ+Am7wEjdTpR1XMQ4rl42FK9SIXzZ4o9x7s6M4=
github.com/go-git/go-git/v5 v5.19.3/go.mod h1:Ye4C8dVigkqrv9UsB4NMdSLV4yAIkLiIhW61gcOyhl4=
github.com/go-gl/glfw v0.0.0-20190409004039-e6da0acd62b1/go.mod h1:vR7hzQXu2zJy9AVAgeJqvqgH9Q5CA+iKCZ2gyEVpxRU=
github.com/go-gl/glfw/v3.3/glfw v0.0.0-20191125211704-12ad95a8df72/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
github.com/go-gl/glfw/v3.3/glfw v0.0.0-20200222043503-6f7a984d4dc4/go.mod h1:tQ2UAYgL5IevRw8kRxooKSPJfGvJ9fJQFa0TUsXzTg8=
@@ -585,8 +585,6 @@ github.com/klauspost/compress v1.11.13/go.mod h1:aoV0uJVorq1K+umq18yTdKaF57EivdY
github.com/klauspost/compress v1.14.2/go.mod h1:/3/Vjq9QcHkK5uEr5lBEmyoZ1iFhe47etQ6QUkpK6sk=
github.com/klauspost/compress v1.18.5 h1:/h1gH5Ce+VWNLSWqPzOVn6XBO+vJbCNGvjoaGBFW2IE=
github.com/klauspost/compress v1.18.5/go.mod h1:cwPg85FWrGar70rWktvGQj8/hthj3wpl0PGDogxkrSQ=
github.com/klauspost/cpuid/v2 v2.3.0 h1:S4CRMLnYUhGeDFDqkGriYKdfoFlDnMtqTiI/sFzhA9Y=
github.com/klauspost/cpuid/v2 v2.3.0/go.mod h1:hqwkgyIinND0mEev00jJYCxPNVRVXFQeu1XKlok6oO0=
github.com/klauspost/pgzip v1.2.5/go.mod h1:Ch1tH69qFZu15pkjo5kYi6mth2Zzwzt50oCQKQE9RUs=
github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
github.com/konsorten/go-windows-terminal-sequences v1.0.2/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
@@ -604,8 +602,8 @@ github.com/kr/text v0.2.0 h1:5Nx0Ya0ZqY2ygV366QzturHI13Jq95ApcVaJBhpS+AY=
github.com/kr/text v0.2.0/go.mod h1:eLer722TekiGuMkidMxC/pM04lWEeraHUUmBw8l2grE=
github.com/kylelemons/godebug v1.1.0 h1:RPNrshWIDI6G2gRW9EHilWtl7Z6Sb1BR0xunSBf0SNc=
github.com/kylelemons/godebug v1.1.0/go.mod h1:9/0rRGxNHcop5bhtWyNeEfOS8JIWk580+fNqagV/RAw=
github.com/leonelquinteros/gotext v1.7.2 h1:bDPndU8nt+/kRo1m4l/1OXiiy2v7Z7dfPQ9+YP7G1Mc=
github.com/leonelquinteros/gotext v1.7.2/go.mod h1:9/haCkm5P7Jay1sxKDGJ5WIg4zkz8oZKw4ekNpALob8=
github.com/leonelquinteros/gotext v1.8.0 h1:wwpxINWmWq3JN3WFI9BoSupBFrvAqxZTtf3qppijYkU=
github.com/leonelquinteros/gotext v1.8.0/go.mod h1:Wqke380+8unZq2F4oTmmqmbUY9zWQMzwImdC34XOAo8=
github.com/lib/pq v0.0.0-20150723085316-0dad96c0b94f/go.mod h1:5WUZQaWbwv1U+lTReE5YruASi9Al49XbQIvNi/34Woo=
github.com/linuxkit/virtsock v0.0.0-20201010232012-f8cee7dfc7a3/go.mod h1:3r6x7q95whyfWQpmGZTu3gk3v2YkMi05HEzl7Tf7YEo=
github.com/lucasb-eyer/go-colorful v1.4.0 h1:UtrWVfLdarDgc44HcS7pYloGHJUjHV/4FwW4TvVgFr4=
@@ -752,8 +750,8 @@ github.com/opencontainers/selinux v1.10.0/go.mod h1:2i0OySw99QjzBBQByd1Gr9gSjvuh
github.com/opentracing/opentracing-go v1.1.0/go.mod h1:UkNAQd3GIcIGf0SeVgPpRdFStlNbqXla1AfSYxPUl2o=
github.com/pelletier/go-toml v1.8.1/go.mod h1:T2/BmBdy8dvIRq1a/8aqjN41wvWlN4lrapLU/GW4pbc=
github.com/peterbourgon/diskv v2.0.1+incompatible/go.mod h1:uqqh8zWWbv1HBMNONnaR/tNboyR3/BZd58JJSHlUSCU=
github.com/pjbgf/sha1cd v0.6.0 h1:3WJ8Wz8gvDz29quX1OcEmkAlUg9diU4GxJHqs0/XiwU=
github.com/pjbgf/sha1cd v0.6.0/go.mod h1:lhpGlyHLpQZoxMv8HcgXvZEhcGs0PG/vsZnEJ7H0iCM=
github.com/pjbgf/sha1cd v0.7.0 h1:ZRNPKHj+gfkLBf0KJv/p1Hmz+7bqCT5o311YX0nq+DA=
github.com/pjbgf/sha1cd v0.7.0/go.mod h1:pKR5Li+qTCo+ebqITZfwPlJGoCFCvSO00qxjbNtTUFQ=
github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pkg/errors v0.8.1-0.20171018195549-f15c970de5b7/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
github.com/pkg/errors v0.8.1/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
@@ -965,8 +963,8 @@ golang.org/x/crypto v0.0.0-20201117144127-c1f2f97bffc9/go.mod h1:jdWPYTVW3xRLrWP
golang.org/x/crypto v0.0.0-20210322153248-0c34fe9e7dc2/go.mod h1:T9bdIzuCu7OtxOm1hfPfRQxPLYneinmdGuTeoZ9dtd4=
golang.org/x/crypto v0.0.0-20210921155107-089bfa567519/go.mod h1:GvvjBRRGRdwPK5ydBHafDWAxML/pGHZbMvKqRZ5+Abc=
golang.org/x/crypto v0.0.0-20220622213112-05595931fe9d/go.mod h1:IxCIyHEi3zRg3s0A5j5BB6A9Jmi73HwBIUl50j+osU4=
golang.org/x/crypto v0.53.0 h1:QZ4Muo8THX6CizN2vPPd5fBGHyogrdK9fG4wLPFUsto=
golang.org/x/crypto v0.53.0/go.mod h1:DNLU434OwVakk9PzuwV8w62mAJpRJL3vsgcfp4Qnsio=
golang.org/x/crypto v0.56.0 h1:GUh5Ii4J5jtcseSMiRqr1jXCNHoxjeV9Fmekc2oLy6Y=
golang.org/x/crypto v0.56.0/go.mod h1:OMW5y6CY9l38uPLmxU6l6pwcXp1obtLo3e6gT7gQR2I=
golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
golang.org/x/exp v0.0.0-20190510132918-efd6b22b2522/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8=
@@ -1042,8 +1040,8 @@ golang.org/x/net v0.0.0-20210405180319-a5a99cb37ef4/go.mod h1:p54w0d4576C0XHj96b
golang.org/x/net v0.0.0-20210825183410-e898025ed96a/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20211112202133-69e39bad7dc2/go.mod h1:9nx3DQGgdP8bBQD5qxJ1jj9UTztislL4KSBs9R2vV5Y=
golang.org/x/net v0.0.0-20220722155237-a158d28d115b/go.mod h1:XRhObCWvk6IyKnWLug+ECip1KBveYUHfp+8e9klMJ9c=
golang.org/x/net v0.56.0 h1:Rw8j/hFzGvJUZwNBXnAtf5sVDVt+65SK2C7IxCxZt5o=
golang.org/x/net v0.56.0/go.mod h1:D3Ku6r+V6JROoZK144D2XfMHFcMq/0zSfLelVTCFKec=
golang.org/x/net v0.57.0 h1:K5+3DljvIuDG9/Jv9rvyMywYNFCQ9RSUY6OOTTkT+tE=
golang.org/x/net v0.57.0/go.mod h1:KpXc8iv+r3XplLAG/f7Jsf9RPszJzdR0f58q9vGOuEU=
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
golang.org/x/oauth2 v0.0.0-20190226205417-e64efc72b421/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
golang.org/x/oauth2 v0.0.0-20190604053449-0f29369cfe45/go.mod h1:gOpvHmFTYa4IltrdGE7lF6nIHvwfUNPOp7c8zoXwtLw=
@@ -1138,13 +1136,17 @@ golang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e/go.mod h1:oPkhp1MJrh7nUepCBc
golang.org/x/sys v0.0.0-20220520151302-bc2c85ada10a/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.48.0 h1:bbX/i/6MgT9BVLM9RT1thmxL04yeTAhbEz4SyadbXoo=
golang.org/x/sys v0.48.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
golang.org/x/sys v0.49.0 h1:XbzkgYJHdqh/8m2Uu0W/dQv8nktxx4BFHp1M0gROTXA=
golang.org/x/sys v0.49.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
golang.org/x/sys v0.50.0 h1:dkKZEFTEKaLr1sjhs0WiKqoi8Jj5nHbS3Njmx1UoMJs=
golang.org/x/sys v0.50.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
golang.org/x/term v0.0.0-20201117132131-f5c789dd3221/go.mod h1:Nr5EML6q2oocZ2LXRh80K7BxOlk5/8JxuGnuhpl+muw=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
golang.org/x/term v0.46.0 h1:3+OXuTbaKDgwk8jTi3aSLHRlmWqHEUDUtxnbFigO4YE=
golang.org/x/term v0.46.0/go.mod h1:+K02xbkittuwc0Am4abfA3Fc+XRGXkvBXNO88NCXPoc=
golang.org/x/term v0.47.0 h1:cpyOwQkr2Nzxe2p6KnrTG7Ei2wtzjsuSjhpXXepZ/BM=
golang.org/x/term v0.47.0/go.mod h1:efAfAuG6RKABvg1ArGebRkvX6LXul14SM67o2e4mUCY=
golang.org/x/term v0.48.0 h1:G6UU1eiSB8nl3DzeUGt7Mg+2SctI8tPS6Z5/mBt6cOk=
golang.org/x/term v0.48.0/go.mod h1:BRDLHCZr2SUjIeBHHpRyQeC9xzSAVbzY2rvKgafnHf4=
golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
@@ -1154,8 +1156,8 @@ golang.org/x/text v0.3.4/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
golang.org/x/text v0.4.0/go.mod h1:mrYo+phRRbMaCq/xk9113O4dZlRixOauAjOtrjsXDZ8=
golang.org/x/text v0.39.0 h1:UbZz4pLOvn600D6Oh6GGEI6VAmndrEBLv8/6BEXzyus=
golang.org/x/text v0.39.0/go.mod h1:3UwRclnC2g0TU9x8PZiyfOajCd1zaUNHF9cvqcQZ+ZM=
golang.org/x/text v0.41.0 h1:vz/seA0lnX87Othu2f/0L24RcgrXD9/YFTSuGjj3rH8=
golang.org/x/text v0.41.0/go.mod h1:jvf1O8ajNzZqhSrQBPbutR/EB83Cc0CFrezNQIwbb5M=
golang.org/x/time v0.0.0-20180412165947-fbb02b2291d2/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
+43
View File
@@ -0,0 +1,43 @@
GIT_ROOT := $$(git rev-parse --show-toplevel)
test-integration:
@$(GIT_ROOT)/nix/test_integration.sh
test-integration-hosts:
$(GIT_ROOT)/scripts/tests/extra_hosts
CLIENT_VM := abra-client
vm-client-status:
sudo systemctl status microvm@$(CLIENT_VM).service
vm-client-create:test-integration-hosts
cd $(GIT_ROOT) && sudo microvm -f git+file://$(GIT_ROOT) -c $(CLIENT_VM)
vm-client-start:
sudo systemctl start microvm@$(CLIENT_VM).service
vm-client-update:test-integration-hosts
cd $(GIT_ROOT) && sudo microvm -R -f git+file://$(GIT_ROOT) -u $(CLIENT_VM)
vm-client-run:
sudo microvm -f git+file://$(GIT_ROOT) -r $(CLIENT_VM)
vm-client-stop:
sudo systemctl stop microvm@$(CLIENT_VM)
vm-client-delete: vm-client-stop
sudo rm -rf /var/lib/microvms/$(CLIENT_VM)
vm-client-connect:
ssh abra@10.0.0.2 -i $(GIT_ROOT)/tests/resources/local_integration_ssh
SERVER_VM := abra-server
vm-server-status:
sudo systemctl status microvm@$(SERVER_VM).service
vm-server-create:
cd $(GIT_ROOT) && sudo microvm -f git+file://$(GIT_ROOT) -c $(SERVER_VM)
vm-server-start:
sudo systemctl start microvm@$(SERVER_VM).service
vm-server-update:
cd $(GIT_ROOT) && sudo microvm -R -f git+file://$(GIT_ROOT) -u $(SERVER_VM)
vm-server-run:
cd $(GIT_ROOT) && sudo microvm -f git+file://$(GIT_ROOT) -r $(SERVER_VM)
vm-server-stop:
sudo systemctl stop microvm@$(SERVER_VM)
vm-server-delete: vm-server-stop
sudo rm -rf /var/lib/microvms/$(SERVER_VM)
vm-server-connect:
ssh abra@10.0.0.3 -i $(GIT_ROOT)/tests/resources/local_integration_ssh
+48
View File
@@ -0,0 +1,48 @@
# Hosts
These hosts are VMs used for running the integration tests on your local machine.
The hosts contain a client where abra with the tests will be run and a server
to simulate a remote machine.
## Prepare the host
For this setup to work you need a machine with NixOS and flakes enabled.
1. Import the abra flake
The install example is based on the [using nix flakes wiki page](https://nixos.wiki/wiki/flakes#Using_nix_flakes_with_NixOS).
```nix
inputs = {
abra = {
url = "git+https://git.coopcloud.tech/toolshed/abra.git";
};
};
```
2. Add the host module to your configuration
Now add the host module to your configuration. At the toplevel of a flake it could look like this:
```nix
desktop = inputs.nixpkgs.lib.nixosSystem {
system = "x86_64-linux";
modules = [
abra.nixosModules.host
./configuration.nix
];
specialArgs = inputs;
};
```
3. Adjust the required config
```nix
abra.testing = {
enable = true; # loads the required config, when set true
externalInterface = "eth1"; # adjust this to your network interface that has access to the internet
};
```
You can look up the interface with `ip link`. After that rebuild your system and you are ready to go.
## Get started
After preparing the host run:
```
make test-integration
```
This runs the automatic setup of the VMs and network. You will be prompted if you want to run all
or specific files.
+145
View File
@@ -0,0 +1,145 @@
{
lib,
pkgs,
publicKey,
username,
pathToRepo,
defaultDNS,
extraHosts,
...
}:
let
index = 2;
mac = "00:00:00:00:00:01";
serverIp = "10.0.0.3";
in
{
imports = [
../../modules/ssh.nix
../../modules/user.nix
../../modules/docker.nix
];
microvm = {
vcpu = 4;
mem = 2049; # see issue related in microvm repo with 2048, so add 1MB
interfaces = [
{
id = "vm${toString index}";
type = "tap";
inherit mac;
}
];
shares = [
{
proto = "virtiofs";
tag = "repo";
# Source path can be absolute or relative
# to /var/lib/microvms/$hostName
source = "${lib.trim pathToRepo}";
mountPoint = "/home/${username}/abra";
}
];
};
boot.tmp = {
useTmpfs = true;
tmpfsSize = "2G";
};
networking = {
hostName = "abra-client";
useNetworkd = true;
};
systemd.network.networks."10-eth" = {
matchConfig.MACAddress = mac;
# Static IP configuration
address = [
"10.0.0.${toString index}/32"
"fec0::${lib.toHexString index}/128"
];
routes = [
{
# A route to the host
Destination = "10.0.0.0/32";
GatewayOnLink = true;
}
{
# Route to server
Destination = "${serverIp}/32";
Gateway = "10.0.0.0";
GatewayOnLink = true;
}
{
# Default route
Destination = "0.0.0.0/0";
Gateway = "10.0.0.0";
GatewayOnLink = true;
}
{
# Default route
Destination = "::/0";
Gateway = "fec0::";
GatewayOnLink = true;
}
];
networkConfig = {
# DNS servers no longer come from DHCP nor Router
# Advertisements. Perhaps you want to change the defaults:
DNS = defaultDNS;
};
};
# open the ports to allow ssh access from the host
networking.firewall.allowedTCPPorts = [ 22 ];
networking.firewall.allowedUDPPorts = [ 22 ];
networking.extraHosts = extraHosts;
environment.variables = {
CGO_ENABLED = 0;
TEST_SERVER = "abra.local";
ABRA_DIR = "$HOME/.abra_test";
};
programs.ssh = {
startAgent = true;
knownHostsFiles = [
(pkgs.writeText "local.keys" ''
abra.local ${publicKey}
'')
];
extraConfig = ''
Host abra.local
HostName abra.local
Port 22
User ${username}
'';
};
programs.git = {
enable = true;
# many recipe commands need a preset git user
config = [
{
user = {
name = "abra dev";
email = "helo@coopcloud.tech";
};
}
];
};
environment.systemPackages = with pkgs; [
# build dependencies, copied from flake.nix
go_1_26
gnumake
# testing dependencies
(bats.withLibraries (p: [
p.bats-assert
p.bats-file
p.bats-support
]))
jq
wget
];
}
+42
View File
@@ -0,0 +1,42 @@
{
serverIp,
username,
...
}:
{
# Home Manager needs a bit of information about you and the
# paths it should manage.
home.username = username;
home.homeDirectory = "/home/${username}";
# This value determines the Home Manager release that your
# configuration is compatible with. This helps avoid breakage
# when a new Home Manager release introduces backwards
# incompatible changes.
#
# You can update Home Manager without changing this value. See
# the Home Manager release notes for a list of state version
# changes in each release.
home.stateVersion = "26.05";
# Let Home Manager install and manage itself.
programs.home-manager.enable = true;
programs.ssh = {
enable = true;
enableDefaultConfig = false;
settings = {
"abra.local" = {
HostName = serverIp;
User = username;
Port = 22;
IdentityFile = "/home/${username}/abra/tests/resources/local_integration_ssh";
};
"*.abra.local" = {
HostName = serverIp;
User = username;
Port = 22;
IdentityFile = "/home/${username}/abra/tests/resources/local_integration_sshn";
};
};
};
}
+91
View File
@@ -0,0 +1,91 @@
{
lib,
defaultDNS,
...
}:
let
index = 3;
mac = "00:00:00:00:00:02";
clientIp = "10.0.0.2";
in
{
imports = [
../../modules/ssh.nix
../../modules/user.nix
../../modules/docker.nix
];
microvm = {
vcpu = 4;
mem = 8097; # see issue related in microvm repo with 2048, so add 1MB
interfaces = [
{
id = "vm${toString index}";
type = "tap";
inherit mac;
}
];
};
boot.tmp = {
useTmpfs = true;
tmpfsSize = "8G";
};
networking = {
hostName = "abra-server";
useNetworkd = true;
};
systemd.network.networks."11-eth" = {
matchConfig.MACAddress = mac;
# Static IP configuration
address = [
"10.0.0.${toString index}/32"
"fec0::${lib.toHexString index}/128"
];
routes = [
{
# A route to the host
Destination = "10.0.0.0/32";
GatewayOnLink = true;
}
{
# Route to server
Destination = "${clientIp}/32";
Gateway = "10.0.0.0";
GatewayOnLink = true;
}
{
# Default route
Destination = "0.0.0.0/0";
Gateway = "10.0.0.0";
GatewayOnLink = true;
}
{
# Default route
Destination = "::/0";
Gateway = "fec0::";
GatewayOnLink = true;
}
];
networkConfig = {
# DNS servers no longer come from DHCP nor Router
# Advertisements. Perhaps you want to change the defaults:
DNS = defaultDNS;
};
};
# open the ports to allow ssh access from the host
networking.firewall.allowedTCPPorts = [
22
80
443
1312 # deploy with udp and tcp on same port test
];
networking.firewall.allowedUDPPorts = [
22
80
443
1312 # deploy with udp and tcp on same port test
];
}
+7
View File
@@ -0,0 +1,7 @@
{ ... }:
{
virtualisation.docker = {
enable = true;
liveRestore = false;
};
}
+59
View File
@@ -0,0 +1,59 @@
{
lib,
config,
...
}:
let
maxVMs = 3;
in
{
options = {
abra.testing.enable = lib.mkOption {
type = lib.types.bool;
default = false;
description = "Enable the abra integration testing framework";
};
abra.testing.externalInterface = lib.mkOption {
type = lib.types.str;
default = null;
description = "Change this to the interface with upstream Internet access";
};
};
config = lib.mkIf config.abra.testing.enable {
networking.useNetworkd = true;
systemd.network.wait-online.enable = false;
systemd.network.networks = builtins.listToAttrs (
map (index: {
name = "30-vm${toString index}";
value = {
matchConfig.Name = "vm${toString index}";
# Host's addresses
address = [
"10.0.0.0/32"
"fec0::/128"
];
# Setup routes to the VM
routes = [
{
Destination = "10.0.0.${toString index}/32";
}
{
Destination = "fec0::${lib.toHexString index}/128";
}
];
# Enable routing
networkConfig = {
IPv4Forwarding = true;
IPv6Forwarding = true;
};
};
}) (lib.genList (i: i + 1) maxVMs)
);
networking.nat = {
enable = true;
internalIPs = [ "10.0.0.0/24" ];
externalInterface = config.abra.testing.externalInterface;
};
};
}
+33
View File
@@ -0,0 +1,33 @@
{
privateKey,
publicKey,
username,
...
}:
{
# default host key location
environment.etc."ssh/ssh_host_ed25519_key" = {
text = privateKey;
mode = "0600";
};
environment.etc."ssh/ssh_host_ed25519_key.pub" = {
text = publicKey;
mode = "0644";
};
services.openssh = {
enable = true;
settings = {
PasswordAuthentication = false;
PermitRootLogin = "no";
AllowUsers = [ "${username}" ];
};
generateHostKeys = false;
hostKeys = [
{
path = "/etc/ssh/ssh_host_ed25519_key";
type = "ed25519";
}
];
};
}
+22
View File
@@ -0,0 +1,22 @@
{
username,
password,
publicKey,
...
}:
{
users.users."${username}" = {
isNormalUser = true;
password = password;
home = "/home/${username}";
description = "Abra user";
extraGroups = [
"wheel"
"networkmanager"
"docker"
];
openssh.authorizedKeys.keys = [
publicKey
];
};
}
View File
+40
View File
@@ -0,0 +1,40 @@
#!/usr/bin/env bash
GIT_ROOT=$(git rev-parse --show-toplevel)
# write path to repo
echo "$GIT_ROOT" > "$GIT_ROOT/tests/resources/path_to_repo"
# create or update a VM, start it and wait for successful ping
start_vm () {
if [ -d "/var/lib/microvms/abra-$1" ]; then
make "vm-$1-update"
else
make "vm-$1-create"
fi
make "vm-$1-start"
host_available=0
while [ $host_available == 0 ]; do
if ping -c 1 "$2" &> /dev/null
then
host_available=1
else
echo "could not ping $1, waiting for 5 seconds..."
sleep 5
fi
done
}
start_vm "client" "10.0.0.2"
start_vm "server" "10.0.0.3"
# revert dynamic content of file to prevent commiting changes
git restore -- "$GIT_ROOT/tests/resources/path_to_repo"
read -rp "Press enter to run all tests or specify test files: " TEST_INTEGRATION_FILES
# -t option is needed for stdin output that is checked in integration tests
# sleep is needed because adding ssh key sometimes does not find agent when adding without timeout
ssh -t abra@10.0.0.2 -i "$GIT_ROOT/tests/resources/local_integration_ssh" "cd abra; sleep 5; chmod 600 ./tests/resources/local_integration_ssh; ssh-add ./tests/resources/local_integration_ssh; make test-integration TEST_INTEGRATION_FILES=$TEST_INTEGRATION_FILES"
+111 -51
View File
@@ -260,12 +260,30 @@ func ReadAppEnvFile(appFile AppFile, name AppName) (App, error) {
func NewApp(env envfile.AppEnv, name string, appFile AppFile) (App, error) {
domain := env["DOMAIN"]
recipeName, exists := env["RECIPE"]
if !exists {
recipeName, exists = env["TYPE"]
if !exists {
return App{}, errors.New(i18n.G("%s is missing the TYPE env var?", name))
}
typeVar, hasType := env["TYPE"]
recipeVar, hasRecipe := env["RECIPE"]
if !hasType && !hasRecipe {
return App{}, errors.New(i18n.G("%s is missing the TYPE env var?", name))
}
// The recipe identity is taken from RECIPE when present (it records the
// canonical git source for externally-sourced recipes), otherwise from
// TYPE. The deployed version is always carried by TYPE when a TYPE line
// exists; only legacy apps that set RECIPE alone carry it there. RECIPE
// itself must stay a bare source pointer, never a versioned value.
identity := typeVar
if hasRecipe {
identity = recipeVar
}
versionSource := identity
if hasType {
versionSource = typeVar
}
recipeName, _, _ := strings.Cut(identity, ":")
if _, version, found := strings.Cut(versionSource, ":"); found && version != "" {
recipeName = fmt.Sprintf("%s:%s", recipeName, version)
}
return App{
@@ -392,11 +410,15 @@ func TemplateAppEnvSample(r recipe.Recipe, appName, server, domain string) error
newContents := strings.Replace(
string(read),
fmt.Sprintf("%s.example.com", r.Name),
fmt.Sprintf("%s.example.com", r.ShortName()),
domain,
-1,
)
if strings.Contains(r.Name, "/") {
newContents = injectRecipeLine(newContents, r.Name)
}
err = os.WriteFile(appEnvPath, []byte(newContents), 0)
if err != nil {
return err
@@ -407,6 +429,37 @@ func TemplateAppEnvSample(r recipe.Recipe, appName, server, domain string) error
return nil
}
// injectRecipeLine ensures the env file carries a RECIPE=<canonical name>
// line so a downstream `abra app deploy` (potentially on another machine)
// can re-fetch the recipe from its original git source. If a RECIPE= line
// already exists in the copied .env.sample it is replaced; otherwise a new
// line is inserted immediately after the TYPE= line, or appended at the
// end if no TYPE= line is present.
func injectRecipeLine(contents, canonicalName string) string {
lines := strings.Split(contents, "\n")
for i, line := range lines {
trimmed := strings.TrimLeft(line, " \t")
if strings.HasPrefix(trimmed, "RECIPE=") {
lines[i] = "RECIPE=" + canonicalName
return strings.Join(lines, "\n")
}
}
for i, line := range lines {
trimmed := strings.TrimLeft(line, " \t")
if strings.HasPrefix(trimmed, "TYPE=") {
out := make([]string, 0, len(lines)+1)
out = append(out, lines[:i+1]...)
out = append(out, "RECIPE="+canonicalName)
out = append(out, lines[i+1:]...)
return strings.Join(out, "\n")
}
}
if contents != "" && !strings.HasSuffix(contents, "\n") {
contents += "\n"
}
return contents + "RECIPE=" + canonicalName + "\n"
}
// SanitiseAppName makes a app name usable with Docker by replacing illegal
// characters.
func SanitiseAppName(name string) string {
@@ -514,10 +567,22 @@ func ExposeAllEnv(
_, exists := service.Environment[k]
if !exists {
value := v
if k == "TYPE" || k == "RECIPE" {
switch k {
case "TYPE":
// NOTE(d1): don't use the wrong version from the app env
// since we are deploying a new version
value = toDeployVersion
// since we are deploying a new version. Keep the
// recipe name already recorded in TYPE and only
// swap in the version being deployed.
name, _, _ := strings.Cut(v, ":")
if _, version, found := strings.Cut(toDeployVersion, ":"); found {
value = fmt.Sprintf("%s:%s", name, version)
} else {
value = name
}
case "RECIPE":
// RECIPE records the canonical git source only; the
// version is carried by TYPE, so keep it bare here.
value, _, _ = strings.Cut(toDeployVersion, ":")
}
service.Environment[k] = &value
log.Debug(i18n.G("%s: %s: %s", stackName, k, value))
@@ -642,60 +707,55 @@ func (a App) WriteRecipeVersion(version string, dryRun bool) error {
if err != nil {
return err
}
defer file.Close()
var (
dirtyVersion string
skipped bool
lines []string
scanner = bufio.NewScanner(file)
lines []string
hasType bool
scanner = bufio.NewScanner(file)
)
for scanner.Scan() {
line := scanner.Text()
if !strings.HasPrefix(line, "RECIPE=") && !strings.HasPrefix(line, "TYPE=") {
lines = append(lines, line)
continue
}
if strings.HasPrefix(line, "#") {
lines = append(lines, line)
continue
}
if strings.Contains(line, version) && !a.Recipe.Dirty && !strings.HasSuffix(line, config.DIRTY_DEFAULT) {
skipped = true
lines = append(lines, line)
continue
}
splitted := strings.Split(line, ":")
line = fmt.Sprintf("%s:%s", splitted[0], version)
lines = append(lines, line)
if strings.HasPrefix(line, "TYPE=") {
hasType = true
}
}
if err := scanner.Err(); err != nil {
file.Close()
log.Fatal(err)
}
file.Close()
// The recipe version is carried by a single field: TYPE when present (the
// standard recipe key), otherwise the legacy RECIPE key. When both exist
// (git-URL recipes) RECIPE stays a bare source pointer and only TYPE is
// versioned, so the two can never drift apart.
targetPrefix := "RECIPE="
if hasType {
targetPrefix = "TYPE="
}
for i, line := range lines {
if !strings.HasPrefix(line, targetPrefix) || strings.HasPrefix(line, "#") {
continue
}
name, _, _ := strings.Cut(line, ":")
lines[i] = fmt.Sprintf("%s:%s", name, version)
}
if dryRun {
log.Debug(i18n.G("skipping writing version %s because dry run", version))
return nil
}
if err := os.WriteFile(a.Path, []byte(strings.Join(lines, "\n")), os.ModePerm); err != nil {
log.Fatal(err)
}
if a.Recipe.Dirty && dirtyVersion != "" {
version = dirtyVersion
}
if !dryRun {
if err := os.WriteFile(a.Path, []byte(strings.Join(lines, "\n")), os.ModePerm); err != nil {
log.Fatal(err)
}
} else {
log.Debug(i18n.G("skipping writing version %s because dry run", version))
}
if !skipped {
log.Debug(i18n.G("version %s saved to %s.env", version, a.Domain))
} else {
log.Debug(i18n.G("skipping version %s write as already exists in %s.env", version, a.Domain))
}
log.Debug(i18n.G("version %s saved to %s.env", version, a.Domain))
return nil
}
+56
View File
@@ -3,6 +3,8 @@ package app_test
import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"reflect"
"testing"
@@ -271,6 +273,60 @@ func TestWriteRecipeVersionOverwrite(t *testing.T) {
assert.Equal(t, "foo", app.Recipe.EnvVersion)
}
// TestWriteRecipeVersionSingleField ensures the version is written to exactly
// one field: TYPE when present (even alongside a RECIPE source pointer, as with
// git-URL recipes), otherwise the legacy RECIPE field. RECIPE must stay bare
// when a TYPE line exists so the two can never drift apart.
func TestWriteRecipeVersionSingleField(t *testing.T) {
cases := []struct {
name string
input string
want string
}{
{
name: "git recipe: TYPE and RECIPE both present",
input: "TYPE=myrecipe\nRECIPE=git.example.com/user/myrecipe\nDOMAIN=x\n",
want: "TYPE=myrecipe:1.2.3+4.5.6\nRECIPE=git.example.com/user/myrecipe\nDOMAIN=x",
},
{
name: "git recipe: re-versioning replaces only TYPE",
input: "TYPE=myrecipe:0.1.0+1.0.0\nRECIPE=git.example.com/user/myrecipe\nDOMAIN=x\n",
want: "TYPE=myrecipe:1.2.3+4.5.6\nRECIPE=git.example.com/user/myrecipe\nDOMAIN=x",
},
{
name: "legacy recipe: only RECIPE present",
input: "RECIPE=test_recipe\nDOMAIN=x\n",
want: "RECIPE=test_recipe:1.2.3+4.5.6\nDOMAIN=x",
},
{
name: "catalogue recipe: only TYPE present",
input: "TYPE=nextcloud\nDOMAIN=x\n",
want: "TYPE=nextcloud:1.2.3+4.5.6\nDOMAIN=x",
},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
p := filepath.Join(t.TempDir(), "app.env")
if err := os.WriteFile(p, []byte(tc.input), 0o644); err != nil {
t.Fatal(err)
}
app := appPkg.App{Path: p, Domain: "x"}
if err := app.WriteRecipeVersion("1.2.3+4.5.6", false); err != nil {
t.Fatal(err)
}
got, err := os.ReadFile(p)
if err != nil {
t.Fatal(err)
}
assert.Equal(t, tc.want, string(got))
})
}
}
func TestWriteRecipeVersionUnknown(t *testing.T) {
test.Setup()
t.Cleanup(func() { test.Teardown() })
+9 -51
View File
@@ -10,58 +10,16 @@ import (
composetypes "github.com/docker/cli/cli/compose/types"
)
// setLabel writes a label to both the service object and the task template of
// every service of the stack.
//
// The task template labels end up on the containers themselves and are therefore
// visible to container-level tooling (cAdvisor, log shippers, docker events),
// which cannot read service labels at all.
//
// The service object is written as well because it is free: changing
// Spec.Labels updates the service without recreating any task and because
// `docker service ls --filter label=...` matches service labels only.
func setLabel(compose *composetypes.Config, key string, value string) {
for i := range compose.Services {
if compose.Services[i].Deploy.Labels == nil {
compose.Services[i].Deploy.Labels = composetypes.Labels{}
}
if compose.Services[i].Labels == nil {
compose.Services[i].Labels = composetypes.Labels{}
}
compose.Services[i].Deploy.Labels[key] = value
compose.Services[i].Labels[key] = value
}
}
// setAppLabel writes a label to the service object of the app service only.
func setAppLabel(compose *composetypes.Config, key string, value string) {
for i := range compose.Services {
if compose.Services[i].Name != "app" {
continue
}
if compose.Services[i].Deploy.Labels == nil {
compose.Services[i].Deploy.Labels = composetypes.Labels{}
}
compose.Services[i].Deploy.Labels[key] = value
}
}
// SetRecipeLabel adds two distinct labels to signal which recipe is connected
// to the deployed app:
// - 'coop-cloud.${STACK_NAME}.recipe=${RECIPE}' on the app service object,
// unchanged, as read back by GetLabel
// - 'coop-cloud.recipe=${RECIPE}' on every service of the stack, so that
// tooling can attribute any container to a recipe without knowing the
// stack name up front
// SetRecipeLabel adds the label 'coop-cloud.${STACK_NAME}.recipe=${RECIPE}' to the app container
// to signal which recipe is connected to the deployed app
func SetRecipeLabel(compose *composetypes.Config, stackName string, recipe string) {
log.Debug(i18n.G("set recipe labels 'coop-cloud.%s.recipe' and 'coop-cloud.recipe' to %s for %s", stackName, recipe, stackName))
setAppLabel(compose, fmt.Sprintf("coop-cloud.%s.recipe", stackName), recipe)
setLabel(compose, "coop-cloud.recipe", recipe)
for _, service := range compose.Services {
if service.Name == "app" {
log.Debug(i18n.G("set recipe label 'coop-cloud.%s.recipe' to %s for %s", stackName, recipe, stackName))
labelKey := fmt.Sprintf("coop-cloud.%s.recipe", stackName)
service.Deploy.Labels[labelKey] = recipe
}
}
}
// SetChaosLabel adds the label 'coop-cloud.${STACK_NAME}.chaos=true/false' to the app container
-41
View File
@@ -8,7 +8,6 @@ import (
testPkg "coopcloud.tech/abra/pkg/test"
stack "coopcloud.tech/abra/pkg/upstream/stack"
composetypes "github.com/docker/cli/cli/compose/types"
"github.com/stretchr/testify/assert"
)
@@ -62,43 +61,3 @@ func TestGetTimeoutFromLabel(t *testing.T) {
assert.Equal(t, timeout, test.expectedTimeout)
}
}
func TestSetRecipeLabel(t *testing.T) {
// the app service brings labels along on both levels, the db service
// has none at all, so the pre-existing and the nil map case are covered
compose := &composetypes.Config{
Services: []composetypes.ServiceConfig{
{
Name: "app",
Labels: composetypes.Labels{"example.container.label": "keep me"},
Deploy: composetypes.DeployConfig{
Labels: composetypes.Labels{"coop-cloud.backupbot.enabled": "true"},
},
},
{Name: "db"},
},
}
appPkg.SetRecipeLabel(compose, "test_example_com", "test-recipe")
services := make(map[string]composetypes.ServiceConfig)
for _, service := range compose.Services {
services[service.Name] = service
}
// the stack scoped label stays on the app service object only
assert.Equal(t, "test-recipe",
services["app"].Deploy.Labels["coop-cloud.test_example_com.recipe"])
assert.NotContains(t, services["db"].Deploy.Labels,
"coop-cloud.test_example_com.recipe")
// the static label reaches every container of the stack
for name, service := range services {
assert.Equal(t, "test-recipe", service.Labels["coop-cloud.recipe"], name)
assert.Equal(t, "test-recipe", service.Deploy.Labels["coop-cloud.recipe"], name)
}
// labels the recipe brought along are left alone
assert.Equal(t, "keep me", services["app"].Labels["example.container.label"])
assert.Equal(t, "true", services["app"].Deploy.Labels["coop-cloud.backupbot.enabled"])
}
+30
View File
@@ -0,0 +1,30 @@
package app
import "testing"
func TestInjectRecipeLineAfterType(t *testing.T) {
in := "DOMAIN=example.com\nTYPE=foo\nVERSION=1.0\n"
want := "DOMAIN=example.com\nTYPE=foo\nRECIPE=org/foo\nVERSION=1.0\n"
got := injectRecipeLine(in, "org/foo")
if got != want {
t.Errorf("injectRecipeLine inserted RECIPE in wrong position\nwant:\n%q\ngot:\n%q", want, got)
}
}
func TestInjectRecipeLineReplacesExisting(t *testing.T) {
in := "TYPE=foo\nRECIPE=old\nDOMAIN=example.com\n"
want := "TYPE=foo\nRECIPE=org/foo\nDOMAIN=example.com\n"
got := injectRecipeLine(in, "org/foo")
if got != want {
t.Errorf("injectRecipeLine should replace existing RECIPE line\nwant:\n%q\ngot:\n%q", want, got)
}
}
func TestInjectRecipeLineNoTypeAppends(t *testing.T) {
in := "DOMAIN=example.com\n"
want := "DOMAIN=example.com\nRECIPE=org/foo\n"
got := injectRecipeLine(in, "org/foo")
if got != want {
t.Errorf("injectRecipeLine should append when TYPE is missing\nwant:\n%q\ngot:\n%q", want, got)
}
}
+310
View File
@@ -0,0 +1,310 @@
// Package audit checks deployed apps against a curated list of advisories.
package audit
import (
"errors"
"fmt"
"io/fs"
"os"
"path/filepath"
"slices"
"sort"
"strings"
"time"
"coopcloud.tech/abra/pkg/i18n"
"gopkg.in/yaml.v3"
)
// Advisory statuses.
const (
StatusDraft = "draft"
StatusPublished = "published"
StatusWithdrawn = "withdrawn"
)
// Fix strategies.
const (
StrategyUpgrade = "upgrade"
StrategyRefresh = "refresh"
StrategyManual = "manual"
)
var severities = []string{"low", "medium", "high", "critical"}
// Advisory describes one known problem and which recipe versions and images
// it affects.
type Advisory struct {
ID string `yaml:"id"`
Title string `yaml:"title"`
Severity string `yaml:"severity"`
Status string `yaml:"status"`
Refs []string `yaml:"refs"`
Published string `yaml:"published"`
Recipes map[string]RecipeRange `yaml:"recipes"`
Images []ImageRule `yaml:"images"`
Fix Fix `yaml:"fix"`
// Path is the file the advisory was read from.
Path string `yaml:"-"`
}
// RecipeRange gives the affected recipe versions of one recipe.
type RecipeRange struct {
Affected string `yaml:"affected"`
Fixed string `yaml:"fixed"`
affected Range
}
// ImageRule describes which images are affected. An image that matches
// Match is fixed if ANY of the fixing conditions holds. See Evaluate.
type ImageRule struct {
Match string `yaml:"match"`
Env map[string]string `yaml:"env"`
Version string `yaml:"version"`
CreatedAfter string `yaml:"created_after"`
FixedDigests []string `yaml:"fixed_digests"`
AffectedDigests []string `yaml:"affected_digests"`
// Skipped lists the fields this version of abra does not understand.
Skipped []string `yaml:"-"`
env map[string]Range
version *Range
createdAfter *time.Time
}
// Fix describes how to fix the advisory.
type Fix struct {
Strategy string `yaml:"strategy"`
Notes string `yaml:"notes"`
}
var (
knownTopLevel = []string{"id", "title", "severity", "status", "refs", "published", "recipes", "images", "fix"}
knownRecipe = []string{"affected", "fixed"}
knownImage = []string{"match", "env", "version", "created_after", "fixed_digests", "affected_digests"}
knownFix = []string{"strategy", "notes"}
)
// LoadAdvisories reads every *.yml / *.yaml file below dir. Advisories that
// cannot be parsed are skipped and reported as warnings, so that one broken
// file does not stop the audit. Fields that this version of abra does not
// know are skipped with a warning, so newer advisories do not break older
// abra versions.
func LoadAdvisories(dir string) ([]Advisory, []string, error) {
var advisories []Advisory
var warnings []string
if _, err := os.Stat(dir); err != nil {
return nil, nil, errors.New(i18n.G("no advisories found: %s", err))
}
err := filepath.WalkDir(dir, func(path string, d fs.DirEntry, err error) error {
if err != nil {
return err
}
if d.IsDir() {
return nil
}
ext := filepath.Ext(path)
if ext != ".yml" && ext != ".yaml" {
return nil
}
data, err := os.ReadFile(path)
if err != nil {
return err
}
adv, warns, err := ParseAdvisory(data)
rel, _ := filepath.Rel(dir, path)
for _, w := range warns {
warnings = append(warnings, fmt.Sprintf("%s: %s", rel, w))
}
if err != nil {
warnings = append(warnings, i18n.G("%s: skipping advisory: %s", rel, err))
return nil
}
adv.Path = path
advisories = append(advisories, adv)
return nil
})
if err != nil {
return nil, warnings, err
}
sort.Slice(advisories, func(i, j int) bool { return advisories[i].ID < advisories[j].ID })
return advisories, warnings, nil
}
// ParseAdvisory parses and validates one advisory.
func ParseAdvisory(data []byte) (Advisory, []string, error) {
var adv Advisory
var warnings []string
var root yaml.Node
if err := yaml.Unmarshal(data, &root); err != nil {
return adv, nil, err
}
if len(root.Content) == 0 {
return adv, nil, errors.New(i18n.G("empty file"))
}
doc := root.Content[0]
if doc.Kind != yaml.MappingNode {
return adv, nil, errors.New(i18n.G("advisory is not a mapping"))
}
unknown := func(n *yaml.Node, known []string, where string) []string {
var skipped []string
if n == nil || n.Kind != yaml.MappingNode {
return nil
}
for i := 0; i+1 < len(n.Content); i += 2 {
key := n.Content[i].Value
if !slices.Contains(known, key) {
skipped = append(skipped, key)
warnings = append(warnings, i18n.G("unknown field %s%s skipped", where, key))
}
}
return skipped
}
unknown(doc, knownTopLevel, "")
if n := mapValue(doc, "recipes"); n != nil && n.Kind == yaml.MappingNode {
for i := 0; i+1 < len(n.Content); i += 2 {
unknown(n.Content[i+1], knownRecipe, fmt.Sprintf("recipes.%s.", n.Content[i].Value))
}
}
if n := mapValue(doc, "fix"); n != nil {
unknown(n, knownFix, "fix.")
}
if err := doc.Decode(&adv); err != nil {
return adv, warnings, err
}
if n := mapValue(doc, "images"); n != nil && n.Kind == yaml.SequenceNode {
for i, img := range n.Content {
if i < len(adv.Images) {
adv.Images[i].Skipped = unknown(img, knownImage, fmt.Sprintf("images[%d].", i))
}
}
}
if err := adv.validate(); err != nil {
return adv, warnings, err
}
return adv, warnings, nil
}
func mapValue(n *yaml.Node, key string) *yaml.Node {
if n == nil || n.Kind != yaml.MappingNode {
return nil
}
for i := 0; i+1 < len(n.Content); i += 2 {
if n.Content[i].Value == key {
return n.Content[i+1]
}
}
return nil
}
func (a *Advisory) validate() error {
if a.ID == "" {
return errors.New(i18n.G("missing id"))
}
a.Severity = strings.ToLower(a.Severity)
if a.Severity != "" && !slices.Contains(severities, a.Severity) {
return errors.New(i18n.G("%s: unknown severity %s", a.ID, a.Severity))
}
a.Status = strings.ToLower(a.Status)
if a.Status == "" {
a.Status = StatusDraft
}
if !slices.Contains([]string{StatusDraft, StatusPublished, StatusWithdrawn}, a.Status) {
return errors.New(i18n.G("%s: unknown status %s", a.ID, a.Status))
}
a.Fix.Strategy = strings.ToLower(a.Fix.Strategy)
if a.Fix.Strategy != "" && !slices.Contains([]string{StrategyUpgrade, StrategyRefresh, StrategyManual}, a.Fix.Strategy) {
return errors.New(i18n.G("%s: unknown fix strategy %s", a.ID, a.Fix.Strategy))
}
if len(a.Recipes) == 0 && len(a.Images) == 0 {
return errors.New(i18n.G("%s: neither recipes nor images given", a.ID))
}
for name, rr := range a.Recipes {
raw := rr.Affected
if raw == "" {
if rr.Fixed == "" {
return errors.New(i18n.G("%s: recipe %s needs affected or fixed", a.ID, name))
}
raw = "<" + rr.Fixed
}
r, err := ParseRange(raw)
if err != nil {
return errors.New(i18n.G("%s: recipe %s: %s", a.ID, name, err))
}
rr.affected = r
a.Recipes[name] = rr
}
for i := range a.Images {
img := &a.Images[i]
if img.Match == "" {
return errors.New(i18n.G("%s: images[%d] has no match", a.ID, i))
}
if len(img.Env) > 0 {
img.env = make(map[string]Range)
for k, v := range img.Env {
r, err := ParseRange(v)
if err != nil {
return errors.New(i18n.G("%s: images[%d].env.%s: %s", a.ID, i, k, err))
}
img.env[k] = r
}
}
if img.Version != "" {
r, err := ParseRange(img.Version)
if err != nil {
return errors.New(i18n.G("%s: images[%d].version: %s", a.ID, i, err))
}
img.version = &r
}
if img.CreatedAfter != "" {
t, err := parseTime(img.CreatedAfter)
if err != nil {
return errors.New(i18n.G("%s: images[%d].created_after: %s", a.ID, i, err))
}
img.createdAfter = &t
}
for _, d := range append(slices.Clone(img.FixedDigests), img.AffectedDigests...) {
if !strings.HasPrefix(d, "sha256:") {
return errors.New(i18n.G("%s: images[%d]: digest %s must start with sha256:", a.ID, i, d))
}
}
}
return nil
}
func parseTime(s string) (time.Time, error) {
layouts := []string{time.RFC3339Nano, "2006-01-02T15:04:05", "2006-01-02 15:04:05", "2006-01-02"}
for _, l := range layouts {
if t, err := time.Parse(l, s); err == nil {
return t, nil
}
}
return time.Time{}, errors.New(i18n.G("unable to parse time %s", s))
}
+282
View File
@@ -0,0 +1,282 @@
package audit
import (
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestRange(t *testing.T) {
tests := []struct {
rng string
version string
want bool
}{
{"<7.0.3+30.0.1", "7.0.2+30.0.1", true},
{"<7.0.3+30.0.1", "7.0.3+30.0.1", false},
{"<7.0.3+30.0.1", "7.0.3+30.0.0", true},
{"<7.0.3", "7.0.2+31.0.0", true},
{"<7.0.3", "7.0.3+29.0.0", false},
{"<8.3.14", "8.3.12", true},
{">=8.3.0 <8.3.14", "8.2.20", false},
{">=8.3.0, <8.3.14", "8.3.13", true},
{">=8.3.0 <8.3.14 || >=8.4.0 <8.4.2", "8.4.1", true},
{">=8.3.0 <8.3.14 || >=8.4.0 <8.4.2", "8.4.2", false},
{"< 35.0.4", "35.0.3-apache", true},
{"8.3", "8.3.0", true},
{"!=1.2.3", "1.2.3", false},
{"<=v1.2", "1.2.0", true},
}
for _, tc := range tests {
r, err := ParseRange(tc.rng)
require.NoError(t, err, tc.rng)
v, err := ParseVersion(tc.version)
require.NoError(t, err, tc.version)
assert.Equal(t, tc.want, r.Contains(v), "%s in %s", tc.version, tc.rng)
}
for _, bad := range []string{"", "<", "<foo", "1.2 || "} {
_, err := ParseRange(bad)
assert.Error(t, err, bad)
}
}
const sampleAdvisory = `
id: CCA-2026-0012
title: PHP 8.3 CGI argument injection
severity: critical
status: published
refs: [CVE-2026-12345]
published: 2026-10-20
future_field: whatever
recipes:
nextcloud:
affected: "<7.0.3+30.0.1"
fixed: "7.0.3+30.0.1"
images:
- match: "nextcloud:30*-apache"
created_after: 2026-10-18
affected_digests: [sha256:1111111111111111111111111111111111111111111111111111111111111111]
fixed_digests: [sha256:2222222222222222222222222222222222222222222222222222222222222222]
- match: "*"
base: "php:8.3*"
env: { PHP_VERSION: ">=8.3.0 <8.3.14" }
fix:
strategy: refresh
notes: "Upstream images rebuilt on 2026-10-18; a redeploy is enough."
`
func TestParseAdvisory(t *testing.T) {
adv, warns, err := ParseAdvisory([]byte(sampleAdvisory))
require.NoError(t, err)
assert.Equal(t, "CCA-2026-0012", adv.ID)
assert.Equal(t, StatusPublished, adv.Status)
assert.Equal(t, "2026-10-20", adv.Published)
assert.Equal(t, "7.0.3+30.0.1", adv.Recipes["nextcloud"].Fixed)
require.Len(t, adv.Images, 2)
require.NotNil(t, adv.Images[0].createdAfter)
assert.Equal(t, time.Date(2026, 10, 18, 0, 0, 0, 0, time.UTC), *adv.Images[0].createdAfter)
assert.Equal(t, []string{"base"}, adv.Images[1].Skipped)
joined := strings.Join(warns, "\n")
assert.Contains(t, joined, "future_field")
assert.Contains(t, joined, "images[1].base")
}
func TestParseAdvisoryInvalid(t *testing.T) {
for name, doc := range map[string]string{
"no id": "status: published\nrecipes: {a: {fixed: 1.0.0}}",
"bad severity": "id: X\nseverity: extreme\nrecipes: {a: {fixed: 1.0.0}}",
"no conditions": "id: X",
"bad range": "id: X\nrecipes: {a: {affected: '<foo'}}",
"bad digest": "id: X\nimages: [{match: a, fixed_digests: [abc]}]",
"bad strategy": "id: X\nrecipes: {a: {fixed: 1.0.0}}\nfix: {strategy: pray}",
"image no match": "id: X\nimages: [{env: {A: '<1'}}]",
} {
_, _, err := ParseAdvisory([]byte(doc))
assert.Error(t, err, name)
}
}
func TestLoadAdvisories(t *testing.T) {
dir := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(dir, "2026"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(dir, "2026", "good.yml"), []byte(sampleAdvisory), 0o644))
require.NoError(t, os.WriteFile(filepath.Join(dir, "broken.yaml"), []byte("id: [\n"), 0o644))
require.NoError(t, os.WriteFile(filepath.Join(dir, "README.md"), []byte("# hi"), 0o644))
advs, warns, err := LoadAdvisories(dir)
require.NoError(t, err)
require.Len(t, advs, 1)
assert.Contains(t, strings.Join(warns, "\n"), "broken.yaml")
}
func mustImage(t *testing.T, service, ref string, info *ImageInfo) ImageRef {
t.Helper()
img, err := ParseImageRef(service, ref)
require.NoError(t, err)
if info != nil {
cp := *info
img.Info = &cp
}
return img
}
func at(s string) *time.Time {
tm, err := time.Parse(time.RFC3339, s)
if err != nil {
panic(err)
}
return &tm
}
func TestEvaluate(t *testing.T) {
adv, _, err := ParseAdvisory([]byte(sampleAdvisory))
require.NoError(t, err)
advs := []Advisory{adv}
newImg := &ImageInfo{Created: at("2026-10-19T10:00:00Z"), Env: map[string]string{"PHP_VERSION": "8.3.14"}}
oldImg := &ImageInfo{Created: at("2026-10-01T10:00:00Z"), Env: map[string]string{"PHP_VERSION": "8.3.12"}}
noPHP := &ImageInfo{Created: at("2026-10-01T10:00:00Z"), Env: map[string]string{}}
states := []AppState{
{
// clean: fixed recipe, rebuilt image
Server: "s1", Name: "clean.example.com", Recipe: "nextcloud", Version: "7.0.3+30.0.1",
Images: []ImageRef{
mustImage(t, "app", "nextcloud:30.0.1-apache@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", newImg),
mustImage(t, "db", "postgres:15@sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", noPHP),
},
},
{
// old recipe version
Server: "s1", Name: "old.example.com", Recipe: "nextcloud", Version: "7.0.2+30.0.0",
Images: []ImageRef{mustImage(t, "app", "nextcloud:30.0.0-apache@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", newImg)},
},
{
// current recipe, stale image under the same tag
Server: "s1", Name: "stale.example.com", Recipe: "nextcloud", Version: "7.0.3+30.0.1",
Images: []ImageRef{mustImage(t, "app", "nextcloud:30.0.1-apache@sha256:cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc", oldImg)},
},
{
// some other recipe built on old php
Server: "s2", Name: "wp.example.com", Recipe: "wordpress", Version: "1.0.0+6.0.0",
Images: []ImageRef{mustImage(t, "app", "wordpress:6@sha256:dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd", oldImg)},
},
{
// known bad digest
Server: "s2", Name: "bad.example.com", Recipe: "nextcloud", Version: "7.0.3+30.0.1",
Images: []ImageRef{mustImage(t, "app", "nextcloud:30.0.1-apache@sha256:1111111111111111111111111111111111111111111111111111111111111111",
&ImageInfo{Created: at("2026-10-20T10:00:00Z"), Env: map[string]string{"PHP_VERSION": "8.3.14"}})},
},
{
// no image config and no digest: unknown
Server: "s2", Name: "nocfg.example.com", Recipe: "nextcloud", Version: "7.0.3+30.0.1",
Images: []ImageRef{mustImage(t, "app", "nextcloud:30.0.1-apache", nil)},
},
{
// reproducible build: unusable build time, php fixed by env
Server: "s2", Name: "repro.example.com", Recipe: "nextcloud", Version: "7.0.3+30.0.1",
Images: []ImageRef{mustImage(t, "app", "nextcloud:30.0.1-apache@sha256:eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee",
&ImageInfo{Created: at("1970-01-01T00:00:00Z"), Env: map[string]string{"PHP_VERSION": "8.3.13"}})},
},
}
markUnusableCreated(states)
findings := Evaluate(states, advs, EvaluateOptions{})
got := map[string]Finding{}
for _, f := range findings {
got[f.App] = f
}
assert.NotContains(t, got, "clean.example.com")
require.Contains(t, got, "old.example.com")
assert.Equal(t, FindingAffected, got["old.example.com"].Status)
assert.Equal(t, "abra app upgrade old.example.com 7.0.3+30.0.1", got["old.example.com"].Fix)
require.Contains(t, got, "stale.example.com")
assert.Equal(t, FindingAffected, got["stale.example.com"].Status)
assert.Equal(t, "abra app deploy stale.example.com --force && abra audit", got["stale.example.com"].Fix)
require.Contains(t, got, "wp.example.com")
assert.Equal(t, FindingAffected, got["wp.example.com"].Status)
require.Contains(t, got, "bad.example.com")
assert.Equal(t, FindingAffected, got["bad.example.com"].Status)
require.Contains(t, got, "nocfg.example.com")
assert.Equal(t, FindingUnknown, got["nocfg.example.com"].Status)
require.Contains(t, got, "repro.example.com")
assert.Equal(t, FindingAffected, got["repro.example.com"].Status, "env still says affected")
assert.Contains(t, strings.Join(got["repro.example.com"].Reasons, " "), "PHP_VERSION=8.3.13")
// drafts are ignored, unless asked for
advs[0].Status = StatusDraft
assert.Empty(t, Evaluate(states, advs, EvaluateOptions{}))
assert.Len(t, Evaluate(states, advs, EvaluateOptions{IncludeDrafts: true}), len(findings))
// withdrawn advisories are always ignored
advs[0].Status = StatusWithdrawn
assert.Empty(t, Evaluate(states, advs, EvaluateOptions{IncludeDrafts: true}))
}
func TestCreatedUnusable(t *testing.T) {
adv, _, err := ParseAdvisory([]byte(`
id: X
status: published
images:
- match: "app:1"
created_after: 2026-10-18
fix: {strategy: refresh}
`))
require.NoError(t, err)
same := at("2026-10-01T00:00:00Z")
states := []AppState{
{Server: "a", Name: "a1", Images: []ImageRef{mustImage(t, "app", "app:1@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", &ImageInfo{Created: same})}},
{Server: "b", Name: "b1", Images: []ImageRef{mustImage(t, "app", "app:1@sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", &ImageInfo{Created: same})}},
{Server: "c", Name: "c1", Images: []ImageRef{mustImage(t, "app", "app:1@sha256:cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc",
&ImageInfo{Created: at("2026-10-01T00:00:00Z"), BaseCreated: at("2026-10-05T00:00:00Z")})}},
{Server: "d", Name: "d1", Images: []ImageRef{mustImage(t, "app", "app:1@sha256:dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd", &ImageInfo{Created: at("2026-10-02T00:00:00Z")})}},
{Server: "e", Name: "e1", Images: []ImageRef{mustImage(t, "app", "app:1@sha256:ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff", &ImageInfo{Created: at("2026-10-20T00:00:00Z")})}},
}
markUnusableCreated(states)
got := map[string]string{}
for _, f := range Evaluate(states, []Advisory{adv}, EvaluateOptions{}) {
got[f.App] = f.Status
}
assert.Equal(t, map[string]string{
"a1": FindingUnknown, // shares build time with b1
"b1": FindingUnknown, // shares build time with a1
"c1": FindingUnknown, // built before its base
"d1": FindingAffected, // usable and old
}, got)
}
func TestGlobMatch(t *testing.T) {
img, err := ParseImageRef("app", "git.coopcloud.tech/coop-cloud/foo:1.2-apache@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa")
require.NoError(t, err)
assert.True(t, globMatch("*", img))
assert.True(t, globMatch("git.coopcloud.tech/coop-cloud/foo", img))
assert.True(t, globMatch("*/foo:1.*-apache", img))
assert.False(t, globMatch("foo:*", img))
img, err = ParseImageRef("app", "docker.io/library/nextcloud:30.0.1-apache")
require.NoError(t, err)
assert.Equal(t, "nextcloud", img.Name)
assert.True(t, globMatch("nextcloud:30*-apache", img))
}
+329
View File
@@ -0,0 +1,329 @@
package audit
import (
"fmt"
"regexp"
"slices"
"strings"
"coopcloud.tech/abra/pkg/i18n"
)
// Finding statuses.
const (
FindingAffected = "affected"
FindingUnknown = "unknown"
)
// Finding is one advisory that applies to one deployed app.
type Finding struct {
Server string `json:"server"`
App string `json:"app"`
Recipe string `json:"recipe"`
Version string `json:"version"`
Advisory *Advisory `json:"-"`
Status string `json:"status"`
// Reasons explains, per recipe or image, why the finding was raised.
Reasons []string `json:"reasons"`
// Fix is a suggested command, or the advisory's notes for manual fixes.
Fix string `json:"fix"`
// NoFix is set when no recipe release fixes the finding yet.
NoFix bool `json:"noFix"`
recipeAffected bool
// images that are affected or unknown, for VerifyRefresh
images []ImageRef
os, arch string
}
// verdict of one image rule for one image.
type verdict int
const (
notApplicable verdict = iota
fixed
affected
unknown
)
// EvaluateOptions configures Evaluate.
type EvaluateOptions struct {
// IncludeDrafts also applies advisories with status draft, so that a
// maintainer can try an advisory before publishing it.
IncludeDrafts bool
}
// Applies reports whether an advisory is used: published advisories always,
// drafts only on request, withdrawn advisories never.
func (o EvaluateOptions) Applies(adv Advisory) bool {
switch adv.Status {
case StatusPublished:
return true
case StatusDraft:
return o.IncludeDrafts
}
return false
}
// Evaluate matches every app against every advisory that applies.
func Evaluate(states []AppState, advisories []Advisory, opts EvaluateOptions) []Finding {
var findings []Finding
for _, st := range states {
for i := range advisories {
adv := &advisories[i]
if !opts.Applies(*adv) {
continue
}
if f, ok := evaluateApp(st, adv); ok {
findings = append(findings, f)
}
}
}
return findings
}
func evaluateApp(st AppState, adv *Advisory) (Finding, bool) {
f := Finding{
Server: st.Server,
App: st.Name,
Recipe: st.Recipe,
Version: st.Version,
Advisory: adv,
}
var gotAffected, gotUnknown bool
// 1. recipe ranges
if rr, ok := recipeRange(adv, st.Recipe); ok && st.Version != "" {
v, err := ParseVersion(st.Version)
if err == nil && rr.affected.Contains(v) {
gotAffected = true
f.recipeAffected = true
f.Reasons = append(f.Reasons, i18n.G("recipe version %s is in %s", st.Version, rr.affected.Raw))
}
}
// 2. image conditions
for _, img := range st.Images {
for _, rule := range adv.Images {
if !globMatch(rule.Match, img) {
continue
}
v, reason := evaluateImage(rule, img)
reason = fmt.Sprintf("%s (%s): %s", img.Service, img.NameTag(), reason)
switch v {
case affected:
gotAffected = true
case unknown:
gotUnknown = true
default:
continue
}
if !slices.Contains(f.Reasons, reason) {
f.Reasons = append(f.Reasons, reason)
}
if !slices.ContainsFunc(f.images, func(i ImageRef) bool { return i.Ref == img.Ref }) {
f.images = append(f.images, img)
}
}
}
switch {
case gotAffected:
f.Status = FindingAffected
case gotUnknown:
f.Status = FindingUnknown
default:
return f, false
}
f.Fix = suggestFix(f, st)
f.os, f.arch = st.OS, st.Arch
return f, true
}
// evaluateImage applies one rule to one image that matches it:
//
// 1. a digest in affected_digests is affected;
// 2. a digest in fixed_digests is fixed;
// 3. the env block selects images: if a variable is missing, the rule does
// not apply (the image is not built on that runtime). If every variable
// is in its range, env votes affected, otherwise fixed;
// 4. version is an affected range on the image version, and votes the same way;
// 5. created_after votes fixed for images built at or after that time, and
// affected for older images, but only when the build time is usable.
//
// The image is fixed if any condition votes fixed, affected if one votes
// affected, and unknown otherwise: an image that matches but satisfies no
// condition is never reported as OK.
func evaluateImage(rule ImageRule, img ImageRef) (verdict, string) {
if img.Digest != "" && slices.Contains(rule.AffectedDigests, img.Digest) {
return affected, i18n.G("digest %s is known to be affected", shortDigest(img.Digest))
}
if img.Digest != "" && slices.Contains(rule.FixedDigests, img.Digest) {
return fixed, ""
}
needsConfig := len(rule.env) > 0 || rule.version != nil || rule.createdAfter != nil
info := img.Info
if needsConfig && info == nil {
if img.InfoErr != nil {
return unknown, i18n.G("image config unavailable: %s", img.InfoErr)
}
return unknown, i18n.G("image config unavailable")
}
var votedAffected []string
var notes []string
if len(rule.env) > 0 {
for k := range rule.env {
if _, ok := info.Env[k]; !ok {
return notApplicable, ""
}
}
allIn := true
var vals []string
for k, r := range rule.env {
val := info.Env[k]
vals = append(vals, fmt.Sprintf("%s=%s", k, val))
v, err := ParseVersion(val)
if err != nil {
allIn = false
notes = append(notes, i18n.G("unable to parse %s=%s", k, val))
continue
}
if !r.Contains(v) {
return fixed, ""
}
}
if allIn {
slices.Sort(vals)
votedAffected = append(votedAffected, strings.Join(vals, ", "))
}
}
if rule.version != nil {
raw := info.Version()
if raw == "" {
notes = append(notes, i18n.G("image has no version"))
} else if v, err := ParseVersion(raw); err != nil {
notes = append(notes, i18n.G("unable to parse image version %s", raw))
} else if rule.version.Contains(v) {
votedAffected = append(votedAffected, i18n.G("version %s is in %s", raw, rule.version.Raw))
} else {
return fixed, ""
}
}
if rule.createdAfter != nil {
switch {
case info.Created == nil:
notes = append(notes, i18n.G("image has no build time"))
case info.createdUnusable != "":
notes = append(notes, info.createdUnusable)
case !info.Created.Before(*rule.createdAfter):
return fixed, ""
default:
votedAffected = append(votedAffected, i18n.G("built %s, before %s", info.Created.UTC().Format("2006-01-02 15:04"), rule.createdAfter.UTC().Format("2006-01-02 15:04")))
}
}
if len(votedAffected) > 0 {
return affected, strings.Join(votedAffected, "; ")
}
if img.Digest == "" {
notes = append(notes, i18n.G("service spec has no digest (registry lookup failed at deploy?)"))
}
if len(rule.Skipped) > 0 {
notes = append(notes, i18n.G("conditions not supported by this abra: %s", strings.Join(rule.Skipped, ", ")))
}
if len(notes) == 0 {
notes = append(notes, i18n.G("no condition proves digest %s fixed", shortDigest(img.Digest)))
}
return unknown, strings.Join(notes, "; ")
}
// recipeRange looks up the advisory's range for a recipe. Advisories name
// recipes in the short form of ShortRecipeName, so "nextcloud" covers
// git.coopcloud.tech/coop-cloud/nextcloud but not a fork such as
// git.coopcloud.tech/c4dt/nextcloud, which has its own versions.
func recipeRange(adv *Advisory, recipe string) (RecipeRange, bool) {
if rr, ok := adv.Recipes[recipe]; ok {
return rr, true
}
rr, ok := adv.Recipes[ShortRecipeName(recipe)]
return rr, ok
}
func upgradeCommand(adv *Advisory, app, recipe string) string {
if rr, ok := recipeRange(adv, recipe); ok && rr.Fixed != "" {
return fmt.Sprintf("abra app upgrade %s %s", app, rr.Fixed)
}
return fmt.Sprintf("abra app upgrade %s", app)
}
func suggestFix(f Finding, st AppState) string {
adv := f.Advisory
upgrade := func() string { return upgradeCommand(adv, st.Name, st.Recipe) }
// a recipe version in the affected range is only fixed by upgrading
if f.recipeAffected {
return upgrade()
}
switch adv.Fix.Strategy {
case StrategyUpgrade:
return upgrade()
case StrategyRefresh:
return fmt.Sprintf("abra app deploy %s --force && abra audit", st.Name)
default:
if adv.Fix.Notes != "" {
return strings.TrimSpace(adv.Fix.Notes)
}
return i18n.G("no fix available yet")
}
}
// globMatch matches an advisory pattern against an image. "*" matches any
// sequence of characters, "?" a single one. Patterns with a ":" are matched
// against "name:tag", others against the name only.
func globMatch(pattern string, img ImageRef) bool {
target := img.Name
if strings.Contains(pattern, ":") {
target = img.NameTag()
}
return globToRegexp(pattern).MatchString(target)
}
func globToRegexp(pattern string) *regexp.Regexp {
var b strings.Builder
b.WriteString("^")
for _, r := range pattern {
switch r {
case '*':
b.WriteString(".*")
case '?':
b.WriteString(".")
default:
b.WriteString(regexp.QuoteMeta(string(r)))
}
}
b.WriteString("$")
return regexp.MustCompile(b.String())
}
func shortDigest(d string) string {
if d == "" {
return i18n.G("(none)")
}
d = strings.TrimPrefix(d, "sha256:")
if len(d) > 12 {
d = d[:12]
}
return "sha256:" + d
}
+290
View File
@@ -0,0 +1,290 @@
package audit
import (
"context"
"fmt"
"strings"
"time"
"coopcloud.tech/abra/pkg/i18n"
"coopcloud.tech/abra/pkg/log"
)
// FetchFunc reads the config of an image for a platform from a registry.
type FetchFunc func(ctx context.Context, img ImageRef, os, arch string) (*ImageInfo, error)
// RegistryFetch is the default FetchFunc.
func RegistryFetch(ctx context.Context, img ImageRef, os, arch string) (*ImageInfo, error) {
return FetchFromRegistry(ctx, img, os, arch)
}
// FixChecker checks that the suggested fixes actually exist.
type FixChecker struct {
Fetch FetchFunc
Recipes RecipeSource
cache map[string]fetchResult
}
type fetchResult struct {
img *ImageRef
err error
}
// CheckFixes verifies the suggested fix of each finding:
//
// - a redeploy only helps if the tag now points to a fixed image. Upstreams
// stop rebuilding patch-pinned tags such as nextcloud:35.0.2-fpm once the
// next patch is out, so the tag's current image is read from the
// registry and checked against the advisory. If it is not fixed, a
// recipe upgrade is needed instead;
// - for a recipe upgrade, the newer releases of the recipe are searched for
// the first one whose images are fixed. If there is none, the finding
// says so: a new recipe release is needed first;
// - for a recipe version range, the advisory's fixed version must be a
// released tag of the recipe.
//
// When the registry or the recipe cannot be read, the suggestion is kept and
// marked as unchecked.
func (c *FixChecker) CheckFixes(ctx context.Context, findings []Finding) {
if c.cache == nil {
c.cache = make(map[string]fetchResult)
}
for i := range findings {
f := &findings[i]
switch {
case f.recipeAffected:
c.checkRecipeRange(f)
case len(f.images) == 0:
case f.Advisory.Fix.Strategy == StrategyRefresh:
c.checkRefresh(ctx, f)
case f.Advisory.Fix.Strategy == StrategyUpgrade:
c.checkUpgrade(ctx, f, "")
}
}
}
func (c *FixChecker) checkRecipeRange(f *Finding) {
rr, ok := recipeRange(f.Advisory, f.Recipe)
if !ok || rr.Fixed == "" || c.Recipes == nil {
return
}
versions, err := c.Recipes.Versions(f.Recipe)
if err != nil {
log.Debug(i18n.G("%s: unable to list versions of %s: %s", f.App, f.Recipe, err))
f.Fix += " " + i18n.G("(unchecked: %s)", err)
return
}
want, _ := ParseVersion(rr.Fixed)
for _, v := range versions {
if pv, err := ParseVersion(v); err == nil && pv.Compare(want) >= 0 {
return
}
}
f.Fix = i18n.G("no fixed release of %s yet: the advisory names %s, but the latest release is %s",
ShortRecipeName(f.Recipe), rr.Fixed, lastOr(versions, i18n.G("none")))
f.NoFix = true
}
func (c *FixChecker) checkRefresh(ctx context.Context, f *Finding) {
if f.arch == "" {
f.Fix += " " + i18n.G("(unchecked: platform unknown)")
return
}
var stale []string
var fetchErr error
for _, img := range f.images {
if img.Tag == "" {
stale = appendUnique(stale, img.Name)
continue
}
cur, err := c.current(ctx, img, f.os, f.arch, false)
if err != nil {
fetchErr = err
log.Debug(i18n.G("%s: unable to read current %s from registry: %s", f.App, img.NameTag(), err))
continue
}
if cur.Digest == img.Digest || !fixedByAdvisory(f.Advisory, *cur) {
log.Debug(i18n.G("%s: current %s (%s) is not fixed", f.App, img.NameTag(), shortDigest(cur.Digest)))
stale = appendUnique(stale, img.NameTag())
}
}
switch {
case len(stale) > 0:
f.Reasons = append(f.Reasons, i18n.G("redeploy would not help: the current image of %s is not fixed either", strings.Join(stale, ", ")))
c.checkUpgrade(ctx, f, i18n.G("%s not rebuilt with a fix", strings.Join(stale, ", ")))
case fetchErr != nil:
f.Fix += " " + i18n.G("(unchecked: registry unreachable)")
}
}
// checkUpgrade looks for the oldest recipe release newer than the deployed
// one whose images are all fixed.
func (c *FixChecker) checkUpgrade(ctx context.Context, f *Finding, prefix string) {
join := func(msg string) string {
if prefix == "" {
return msg
}
return prefix + "; " + msg
}
generic := upgradeCommand(f.Advisory, f.App, f.Recipe)
if c.Recipes == nil || f.arch == "" {
f.Fix = join(generic + " " + i18n.G("(unchecked: no fixed release looked up)"))
return
}
versions, err := c.Recipes.Versions(f.Recipe)
if err != nil {
log.Debug(i18n.G("%s: unable to list versions of %s: %s", f.App, f.Recipe, err))
f.Fix = join(generic + " " + i18n.G("(unchecked: %s)", err))
return
}
current, curErr := ParseVersion(f.Version)
var newer []string
for _, v := range versions {
pv, _ := ParseVersion(v)
if curErr != nil || pv.Compare(current) > 0 {
newer = append(newer, v)
}
}
names := make(map[string]bool)
for _, img := range f.images {
names[img.Name] = true
}
short := ShortRecipeName(f.Recipe)
if len(newer) == 0 {
f.Fix = join(i18n.G("no fixed release of %s yet: %s is the latest; a new release with a fixed image is needed",
short, lastOr(versions, f.Version)))
f.NoFix = true
return
}
// Check the latest release first: if it is not fixed, no release is.
fixedAt := func(v string) (bool, error) { return c.versionFixed(ctx, f, v, names) }
latest := newer[len(newer)-1]
ok, err := fixedAt(latest)
if err != nil {
f.Fix = join(generic + " " + i18n.G("(unchecked: %s)", err))
return
}
if !ok {
var imgs []string
if all, err := c.Recipes.Images(f.Recipe, latest); err == nil {
for _, ref := range all {
if img, err := ParseImageRef("", ref); err == nil && names[img.Name] {
imgs = append(imgs, img.NameTag())
}
}
}
f.Fix = join(i18n.G("no fixed release of %s yet: the latest, %s, still uses %s; a new release with a fixed image is needed",
short, latest, strings.Join(imgs, ", ")))
f.NoFix = true
return
}
best := latest
for _, v := range newer[:len(newer)-1] {
if ok, err := fixedAt(v); err == nil && ok {
best = v
break
}
}
f.Fix = join(fmt.Sprintf("abra app upgrade %s %s", f.App, best))
}
// versionFixed reports whether every image of the recipe version that has
// one of the given names is fixed. An image that is no longer used counts as
// fixed.
func (c *FixChecker) versionFixed(ctx context.Context, f *Finding, version string, names map[string]bool) (bool, error) {
refs, err := c.Recipes.Images(f.Recipe, version)
if err != nil {
return false, err
}
for _, ref := range refs {
img, err := ParseImageRef("", ref)
if err != nil || !names[img.Name] {
continue
}
cur, err := c.current(ctx, img, f.os, f.arch, true)
if err != nil {
return false, err
}
if !fixedByAdvisory(f.Advisory, *cur) {
log.Debug(i18n.G("%s %s: %s is not fixed", ShortRecipeName(f.Recipe), version, img.NameTag()))
return false, nil
}
}
return true, nil
}
// current reads the image a reference points to now: by digest if byDigest
// is set and the reference has one, by tag otherwise.
func (c *FixChecker) current(ctx context.Context, img ImageRef, os, arch string, byDigest bool) (*ImageRef, error) {
ref := img
if !byDigest {
ref.Digest = ""
}
key := strings.Join([]string{ref.NameTag(), ref.Digest, os, arch}, "|")
if r, ok := c.cache[key]; ok {
return r.img, r.err
}
ref.Info, ref.InfoErr = nil, nil
fctx, cancel := context.WithTimeout(ctx, 60*time.Second)
defer cancel()
info, err := c.Fetch(fctx, ref, os, arch)
if err != nil {
c.cache[key] = fetchResult{err: err}
return nil, err
}
info.createdUnusable = createdUnusable(info)
if info.Digest != "" {
ref.Digest = info.Digest
}
ref.Info = info
c.cache[key] = fetchResult{img: &ref}
return &ref, nil
}
// fixedByAdvisory reports whether no image rule of the advisory flags img.
func fixedByAdvisory(adv *Advisory, img ImageRef) bool {
for _, rule := range adv.Images {
if !globMatch(rule.Match, img) {
continue
}
if v, _ := evaluateImage(rule, img); v == affected || v == unknown {
return false
}
}
return true
}
func appendUnique(list []string, s string) []string {
if containsString(list, s) {
return list
}
return append(list, s)
}
func lastOr(list []string, def string) string {
if len(list) == 0 {
return def
}
return list[len(list)-1]
}
+218
View File
@@ -0,0 +1,218 @@
package audit
import (
"context"
"errors"
"os"
"path/filepath"
"strings"
"testing"
"time"
"coopcloud.tech/abra/pkg/recipe"
"github.com/go-git/go-git/v5"
"github.com/go-git/go-git/v5/plumbing/object"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
const phpAdvisory = `
id: PHP
status: published
recipes:
ranged: {fixed: 2.0.0+1.0.0}
images:
- match: "*"
env: { PHP_VERSION: ">=8.5.0 <8.5.13" }
fix: {strategy: refresh}
`
// fakeRecipes maps recipe -> version -> images.
type fakeRecipes map[string]map[string][]string
func (f fakeRecipes) Versions(name string) ([]string, error) {
vs, ok := f[name]
if !ok {
return nil, errors.New("no such recipe")
}
var out []string
for v := range vs {
out = append(out, v)
}
sortVersions(out)
return out, nil
}
func (f fakeRecipes) Images(name, version string) ([]string, error) {
return f[name][version], nil
}
func digestOf(c string) string { return "sha256:" + strings.Repeat(c, 64) }
func TestCheckFixes(t *testing.T) {
adv, _, err := ParseAdvisory([]byte(phpAdvisory))
require.NoError(t, err)
old := &ImageInfo{Env: map[string]string{"PHP_VERSION": "8.5.12"}}
app := func(name, recipe, version, image string, platform bool) AppState {
st := AppState{Server: "s", Name: name, Recipe: recipe, Version: version,
Images: []ImageRef{mustImage(t, "app", image, old)}}
if platform {
st.OS, st.Arch = "linux", "amd64"
}
return st
}
states := []AppState{
app("floating", "nc", "1.0.0+35.0.2", "nextcloud:35-fpm@"+digestOf("a"), true),
app("pinned", "nc", "1.0.0+35.0.2", "nextcloud:35.0.2-fpm@"+digestOf("b"), true),
app("fork", "git.coopcloud.tech/c4dt/nextcloud", "16.2.2+35.0.2-fpm", "nextcloud:35.0.2-fpm@"+digestOf("b"), true),
app("stale", "stale", "1.0.0+35.0.1", "nextcloud:35.0.1-fpm@"+digestOf("e"), true),
app("offline", "nc", "1.0.0+35.0.2", "other:1@"+digestOf("c"), true),
app("noplatform", "nc", "1.0.0+35.0.2", "nextcloud:35-fpm@"+digestOf("a"), false),
app("ranged", "ranged", "1.0.0+1.0.0", "postgres:15@"+digestOf("f"), true),
}
recipes := fakeRecipes{
"nc": {
"1.0.0+35.0.2": {"nextcloud:35.0.2-fpm", "postgres:15"},
"1.0.1+35.0.3": {"nextcloud:35.0.3-fpm", "postgres:15"},
"1.1.0+35.0.4": {"nextcloud:35.0.4-fpm", "postgres:15"},
},
"git.coopcloud.tech/c4dt/nextcloud": {
"16.2.1+35.0.2-fpm": {"nextcloud:35.0.2-fpm"},
"16.2.2+35.0.2-fpm": {"nextcloud:35.0.2-fpm"},
},
"stale": {
"1.0.0+35.0.1": {"nextcloud:35.0.1-fpm"},
"1.0.1+35.0.2": {"nextcloud:35.0.2-fpm"},
},
"ranged": {"1.0.0+1.0.0": {"postgres:15"}},
}
var fetched []string
fetch := func(_ context.Context, img ImageRef, os, arch string) (*ImageInfo, error) {
fetched = append(fetched, img.NameTag())
php := map[string]string{
"nextcloud:35-fpm": "8.5.13",
"nextcloud:35.0.2-fpm": "8.5.12",
"nextcloud:35.0.1-fpm": "8.5.12",
"nextcloud:35.0.3-fpm": "8.5.13",
"nextcloud:35.0.4-fpm": "8.5.13",
}
v, ok := php[img.NameTag()]
if !ok {
return nil, errors.New("registry down")
}
d := digestOf("d")
if img.NameTag() == "nextcloud:35.0.2-fpm" {
d = digestOf("b")
}
return &ImageInfo{Digest: d, Env: map[string]string{"PHP_VERSION": v}}, nil
}
findings := Evaluate(states, []Advisory{adv}, EvaluateOptions{})
require.Len(t, findings, 7)
checker := FixChecker{Fetch: fetch, Recipes: recipes}
checker.CheckFixes(context.Background(), findings)
got := map[string]Finding{}
for _, f := range findings {
got[f.App] = f
}
assert.Equal(t, "abra app deploy floating --force && abra audit", got["floating"].Fix)
assert.Equal(t, "nextcloud:35.0.2-fpm not rebuilt with a fix; abra app upgrade pinned 1.0.1+35.0.3", got["pinned"].Fix)
assert.False(t, got["pinned"].NoFix)
assert.Equal(t, "nextcloud:35.0.2-fpm not rebuilt with a fix; no fixed release of c4dt/nextcloud yet: 16.2.2+35.0.2-fpm is the latest; a new release with a fixed image is needed", got["fork"].Fix)
assert.True(t, got["fork"].NoFix)
assert.Equal(t, "nextcloud:35.0.1-fpm not rebuilt with a fix; no fixed release of stale yet: the latest, 1.0.1+35.0.2, still uses nextcloud:35.0.2-fpm; a new release with a fixed image is needed", got["stale"].Fix)
assert.True(t, got["stale"].NoFix)
assert.Equal(t, "abra app deploy offline --force && abra audit (unchecked: registry unreachable)", got["offline"].Fix)
assert.Equal(t, "abra app deploy noplatform --force && abra audit (unchecked: platform unknown)", got["noplatform"].Fix)
assert.Equal(t, "no fixed release of ranged yet: the advisory names 2.0.0+1.0.0, but the latest release is 1.0.0+1.0.0", got["ranged"].Fix)
assert.True(t, got["ranged"].NoFix)
// each image is fetched once
seen := map[string]int{}
for _, f := range fetched {
seen[f]++
}
for img, n := range seen {
if img != "other:1" {
assert.Equal(t, 1, n, img)
}
}
}
func TestShortRecipeName(t *testing.T) {
for in, want := range map[string]string{
"nextcloud": "nextcloud",
"git.coopcloud.tech/coop-cloud/nextcloud": "nextcloud",
"git.coopcloud.tech/c4dt/nextcloud": "c4dt/nextcloud",
"https://git.coopcloud.tech/c4dt/nextcloud.git": "c4dt/nextcloud",
"github.com/foo/nextcloud": "github.com/foo/nextcloud",
} {
assert.Equal(t, want, ShortRecipeName(in), in)
}
}
func TestRecipeRangeForks(t *testing.T) {
adv, _, err := ParseAdvisory([]byte("id: X\nstatus: published\nrecipes: {nextcloud: {fixed: 7.0.0}}\n"))
require.NoError(t, err)
_, ok := recipeRange(&adv, "nextcloud")
assert.True(t, ok)
_, ok = recipeRange(&adv, "git.coopcloud.tech/coop-cloud/nextcloud")
assert.True(t, ok)
_, ok = recipeRange(&adv, "git.coopcloud.tech/c4dt/nextcloud")
assert.False(t, ok, "a fork has its own versions")
}
func TestComposeImages(t *testing.T) {
assert.Equal(t, []string{"nextcloud:35.0.2-fpm", "nginx:1.29", "postgres:15"}, composeImages(`
services:
app: {image: nextcloud:35.0.2-fpm}
web: {image: "${WEB_IMAGE:-nginx:1.29}"}
db: {image: postgres:15}
custom: {image: "${CUSTOM_IMAGE}"}
build: {build: .}
`))
}
func TestGitRecipes(t *testing.T) {
dir := t.TempDir()
repo, err := git.PlainInit(dir, false)
require.NoError(t, err)
sig := &object.Signature{Name: "t", Email: "t@example.com", When: time.Now()}
release := func(version, image string) {
commitFile(t, repo, dir, "compose.yml", "services:\n app:\n image: "+image+"\n")
head, err := repo.Head()
require.NoError(t, err)
_, err = repo.CreateTag(version, head.Hash(), &git.CreateTagOptions{Tagger: sig, Message: version})
require.NoError(t, err)
}
release("1.0.0+35.0.2-fpm", "nextcloud:35.0.2-fpm")
release("1.0.1+35.0.3-fpm", "nextcloud:35.0.3-fpm")
// a local edit must survive
require.NoError(t, os.WriteFile(filepath.Join(dir, "compose.yml"), []byte("local edit\n"), 0o644))
g := NewGitRecipes([]recipe.Recipe{{Name: "nc", Dir: dir}}, true)
versions, err := g.Versions("nc")
require.NoError(t, err)
assert.Equal(t, []string{"1.0.0+35.0.2-fpm", "1.0.1+35.0.3-fpm"}, versions)
images, err := g.Images("nc", "1.0.0+35.0.2-fpm")
require.NoError(t, err)
assert.Equal(t, []string{"nextcloud:35.0.2-fpm"}, images)
data, err := os.ReadFile(filepath.Join(dir, "compose.yml"))
require.NoError(t, err)
assert.Equal(t, "local edit\n", string(data))
}
+314
View File
@@ -0,0 +1,314 @@
package audit
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"strings"
"time"
"coopcloud.tech/abra/pkg/i18n"
"github.com/containers/image/docker"
"github.com/containers/image/manifest"
"github.com/containers/image/pkg/blobinfocache/none"
"github.com/containers/image/types"
"github.com/distribution/reference"
dockerClient "github.com/docker/docker/client"
"github.com/opencontainers/go-digest"
imgspecv1 "github.com/opencontainers/image-spec/specs-go/v1"
)
// OCI annotations and labels read by the audit.
const (
AnnotationVersion = "org.opencontainers.image.version"
AnnotationBaseName = "org.opencontainers.image.base.name"
AnnotationBaseDigest = "org.opencontainers.image.base.digest"
)
// ImageRef is one image as referenced in a service spec.
type ImageRef struct {
Service string `json:"service"`
// Ref is the reference as stored in the service spec, typically
// image:tag@sha256:...
Ref string `json:"ref"`
// Name is the familiar image name, e.g. "nextcloud" or "git.coopcloud.tech/coop-cloud/foo".
Name string `json:"name"`
Tag string `json:"tag"`
Digest string `json:"digest"`
Info *ImageInfo `json:"-"`
InfoErr error `json:"-"`
named reference.Named
}
// ImageInfo holds what the image says about itself.
type ImageInfo struct {
// Digest is the manifest (index) digest, set when read from a registry.
Digest string
Created *time.Time
Env map[string]string
Labels map[string]string
Annotations map[string]string
// BaseCreated is the build time of the base image named in the
// base.digest annotation, if it could be read.
BaseCreated *time.Time
// Source is "node" or "registry".
Source string
// createdUnusable is set when the build time cannot be trusted.
createdUnusable string
}
// ParseImageRef parses an image reference from a service spec.
func ParseImageRef(service, ref string) (ImageRef, error) {
img := ImageRef{Service: service, Ref: ref}
named, err := reference.ParseNormalizedNamed(ref)
if err != nil {
return img, err
}
img.named = named
img.Name = reference.FamiliarName(named)
if t, ok := named.(reference.Tagged); ok {
img.Tag = t.Tag()
}
if d, ok := named.(reference.Digested); ok {
img.Digest = d.Digest().String()
}
return img, nil
}
// NameTag returns "name:tag", the string matched against advisory patterns.
func (i ImageRef) NameTag() string {
if i.Tag == "" {
return i.Name
}
return i.Name + ":" + i.Tag
}
// Version returns the image's version from its label or annotation.
func (i *ImageInfo) Version() string {
if v := i.Labels[AnnotationVersion]; v != "" {
return v
}
return i.Annotations[AnnotationVersion]
}
// InspectOnNode reads the image config from the docker daemon of the server.
// This needs no registry, and works on single-node servers where the running
// image is always present.
func InspectOnNode(ctx context.Context, cl *dockerClient.Client, img ImageRef) (*ImageInfo, error) {
ref := img.Ref
if img.Digest != "" && img.named != nil {
ref = fmt.Sprintf("%s@%s", reference.TrimNamed(img.named).String(), img.Digest)
}
resp, err := cl.ImageInspect(ctx, ref)
if err != nil {
return nil, err
}
info := &ImageInfo{Source: "node", Env: map[string]string{}, Labels: map[string]string{}}
if resp.Created != "" {
if t, err := time.Parse(time.RFC3339Nano, resp.Created); err == nil {
info.Created = &t
}
}
if resp.Config != nil {
info.Env = parseEnv(resp.Config.Env)
if resp.Config.Labels != nil {
info.Labels = resp.Config.Labels
}
}
return info, nil
}
// manifestDoc covers the fields needed from image indexes, Docker manifest
// lists and image manifests.
type manifestDoc struct {
MediaType string `json:"mediaType"`
Annotations map[string]string `json:"annotations"`
Config *struct {
Digest string `json:"digest"`
MediaType string `json:"mediaType"`
Size int64 `json:"size"`
} `json:"config"`
Manifests []struct {
Digest string `json:"digest"`
Annotations map[string]string `json:"annotations"`
Platform *struct {
Architecture string `json:"architecture"`
OS string `json:"os"`
Variant string `json:"variant"`
} `json:"platform"`
} `json:"manifests"`
}
// FetchFromRegistry reads the image config for the given platform from the
// registry. Only a few KB are transferred; no layers are pulled. When the
// image records its base in the base.digest annotation, the base image's
// build time is read as well.
func FetchFromRegistry(ctx context.Context, img ImageRef, os, arch string) (*ImageInfo, error) {
info, err := fetchConfig(ctx, img, os, arch)
if err != nil {
return nil, err
}
baseName := info.Annotations[AnnotationBaseName]
baseDigest := info.Annotations[AnnotationBaseDigest]
if baseName != "" && baseDigest != "" {
if base, err := ParseImageRef("", fmt.Sprintf("%s@%s", stripTagDigest(baseName), baseDigest)); err == nil {
if bi, err := fetchConfig(ctx, base, os, arch); err == nil {
info.BaseCreated = bi.Created
}
}
}
return info, nil
}
func stripTagDigest(name string) string {
named, err := reference.ParseNormalizedNamed(name)
if err != nil {
return name
}
return reference.TrimNamed(named).String()
}
func fetchConfig(ctx context.Context, img ImageRef, os, arch string) (*ImageInfo, error) {
if img.named == nil {
return nil, errors.New(i18n.G("invalid image reference %s", img.Ref))
}
// NOTE: containers/image refuses references with both tag and digest
refStr := reference.TrimNamed(img.named).String()
if img.Digest != "" {
refStr += "@" + img.Digest
} else if img.Tag != "" {
refStr += ":" + img.Tag
}
ref, err := docker.ParseReference("//" + refStr)
if err != nil {
return nil, err
}
src, err := ref.NewImageSource(ctx, &types.SystemContext{})
if err != nil {
return nil, err
}
defer src.Close()
raw, _, err := src.GetManifest(ctx, nil)
if err != nil {
return nil, err
}
topDigest, err := manifest.Digest(raw)
if err != nil {
return nil, err
}
var doc manifestDoc
if err := json.Unmarshal(raw, &doc); err != nil {
return nil, err
}
annotations := map[string]string{}
for k, v := range doc.Annotations {
annotations[k] = v
}
if len(doc.Manifests) > 0 {
var chosen string
for _, m := range doc.Manifests {
if m.Platform != nil && m.Platform.OS == os && m.Platform.Architecture == arch {
chosen = m.Digest
for k, v := range m.Annotations {
annotations[k] = v
}
break
}
}
if chosen == "" {
return nil, errors.New(i18n.G("%s has no image for %s/%s", img.Ref, os, arch))
}
d := digest.Digest(chosen)
raw, _, err = src.GetManifest(ctx, &d)
if err != nil {
return nil, err
}
doc = manifestDoc{}
if err := json.Unmarshal(raw, &doc); err != nil {
return nil, err
}
for k, v := range doc.Annotations {
annotations[k] = v
}
}
if doc.Config == nil || doc.Config.Digest == "" {
return nil, errors.New(i18n.G("%s: unsupported manifest type %s", img.Ref, doc.MediaType))
}
blob, _, err := src.GetBlob(ctx, types.BlobInfo{Digest: digest.Digest(doc.Config.Digest), Size: doc.Config.Size}, none.NoCache)
if err != nil {
return nil, err
}
defer blob.Close()
cfgBytes, err := io.ReadAll(io.LimitReader(blob, 4<<20))
if err != nil {
return nil, err
}
var cfg imgspecv1.Image
if err := json.Unmarshal(cfgBytes, &cfg); err != nil {
return nil, err
}
info := &ImageInfo{
Source: "registry",
Digest: topDigest.String(),
Created: cfg.Created,
Env: parseEnv(cfg.Config.Env),
Labels: cfg.Config.Labels,
Annotations: annotations,
}
if info.Labels == nil {
info.Labels = map[string]string{}
}
return info, nil
}
func parseEnv(env []string) map[string]string {
out := make(map[string]string, len(env))
for _, e := range env {
k, v, _ := strings.Cut(e, "=")
out[k] = v
}
return out
}
// NormaliseArch maps the architecture names reported by the docker daemon
// (uname -m style) to OCI platform names.
func NormaliseArch(arch string) string {
switch arch {
case "x86_64", "amd64":
return "amd64"
case "aarch64", "arm64":
return "arm64"
case "armv7l", "armv7", "armhf", "arm":
return "arm"
case "i386", "i686", "386":
return "386"
}
return arch
}
+229
View File
@@ -0,0 +1,229 @@
package audit
import (
"errors"
"path"
"regexp"
"sort"
"strings"
"coopcloud.tech/abra/pkg/i18n"
"coopcloud.tech/abra/pkg/log"
"coopcloud.tech/abra/pkg/recipe"
"github.com/go-git/go-git/v5"
"github.com/go-git/go-git/v5/plumbing"
"github.com/go-git/go-git/v5/plumbing/object"
"gopkg.in/yaml.v3"
)
const canonicalRecipePrefix = "git.coopcloud.tech/coop-cloud/"
// ShortRecipeName returns the shortest unambiguous form of a recipe name,
// relative to the canonical git.coopcloud.tech/coop-cloud/ location:
//
// nextcloud -> nextcloud
// git.coopcloud.tech/coop-cloud/nextcloud -> nextcloud
// git.coopcloud.tech/c4dt/nextcloud -> c4dt/nextcloud
// github.com/foo/nextcloud -> github.com/foo/nextcloud
func ShortRecipeName(name string) string {
name = strings.TrimPrefix(name, "https://")
name = strings.TrimSuffix(name, ".git")
if strings.HasPrefix(name, canonicalRecipePrefix) {
return strings.TrimPrefix(name, canonicalRecipePrefix)
}
if strings.HasPrefix(name, "git.coopcloud.tech/") {
return strings.TrimPrefix(name, "git.coopcloud.tech/")
}
return name
}
// RecipeSource lists the released versions of a recipe and the images each
// version uses.
type RecipeSource interface {
Versions(recipeName string) ([]string, error)
Images(recipeName, version string) ([]string, error)
}
// GitRecipes reads recipe versions from the local recipe checkouts. It never
// changes the working tree: tags are fetched into the repository, and compose
// files are read from the tag's tree.
type GitRecipes struct {
Offline bool
recipes map[string]recipe.Recipe
repos map[string]*git.Repository
errs map[string]error
}
// NewGitRecipes prepares access to the recipes of the given recipes.
func NewGitRecipes(recipes []recipe.Recipe, offline bool) *GitRecipes {
g := &GitRecipes{
Offline: offline,
recipes: make(map[string]recipe.Recipe),
repos: make(map[string]*git.Repository),
errs: make(map[string]error),
}
for _, r := range recipes {
g.recipes[r.Name] = r
}
return g
}
func (g *GitRecipes) open(name string) (*git.Repository, error) {
if repo, ok := g.repos[name]; ok {
return repo, nil
}
if err, ok := g.errs[name]; ok {
return nil, err
}
repo, err := g.openUncached(name)
if err != nil {
g.errs[name] = err
return nil, err
}
g.repos[name] = repo
return repo, nil
}
func (g *GitRecipes) openUncached(name string) (*git.Repository, error) {
r, ok := g.recipes[name]
if !ok {
return nil, errors.New(i18n.G("unknown recipe %s", name))
}
repo, err := git.PlainOpen(r.Dir)
if err != nil {
if g.Offline {
return nil, err
}
if err := r.EnsureExists(); err != nil {
return nil, err
}
if repo, err = git.PlainOpen(r.Dir); err != nil {
return nil, err
}
}
if !g.Offline {
err := repo.Fetch(&git.FetchOptions{RemoteName: "origin", Tags: git.AllTags})
if err != nil && !errors.Is(err, git.NoErrAlreadyUpToDate) {
log.Debug(i18n.G("%s: unable to fetch tags, using local tags: %s", name, err))
}
}
return repo, nil
}
// Versions returns the parseable release tags, oldest first.
func (g *GitRecipes) Versions(name string) ([]string, error) {
repo, err := g.open(name)
if err != nil {
return nil, err
}
iter, err := repo.Tags()
if err != nil {
return nil, err
}
var tags []string
_ = iter.ForEach(func(ref *plumbing.Reference) error {
tag := ref.Name().Short()
if _, err := ParseVersion(tag); err == nil {
tags = append(tags, tag)
}
return nil
})
sortVersions(tags)
return tags, nil
}
// Images returns the images used by a recipe version, read from the
// compose.yml and compose.*.yml files of that tag.
func (g *GitRecipes) Images(name, version string) ([]string, error) {
repo, err := g.open(name)
if err != nil {
return nil, err
}
hash, err := repo.ResolveRevision(plumbing.Revision("refs/tags/" + version))
if err != nil {
return nil, err
}
commit, err := repo.CommitObject(*hash)
if err != nil {
return nil, err
}
tree, err := commit.Tree()
if err != nil {
return nil, err
}
var images []string
err = tree.Files().ForEach(func(f *object.File) error {
if strings.Contains(f.Name, "/") {
return nil
}
if ok, _ := path.Match("compose*yml", f.Name); !ok {
return nil
}
content, err := f.Contents()
if err != nil {
return err
}
for _, img := range composeImages(content) {
if !containsString(images, img) {
images = append(images, img)
}
}
return nil
})
return images, err
}
var envDefault = regexp.MustCompile(`\$\{[A-Za-z0-9_]+:?-([^}]*)\}`)
// composeImages extracts services.*.image from a compose file. Variables with
// a default (${FOO:-bar}) are replaced by the default; images with other
// variables are skipped.
func composeImages(content string) []string {
var doc struct {
Services map[string]struct {
Image string `yaml:"image"`
} `yaml:"services"`
}
if err := yaml.Unmarshal([]byte(content), &doc); err != nil {
return nil
}
var images []string
for _, svc := range doc.Services {
img := envDefault.ReplaceAllString(svc.Image, "$1")
if img == "" || strings.Contains(img, "$") {
continue
}
images = append(images, img)
}
sort.Strings(images)
return images
}
func sortVersions(tags []string) {
sort.SliceStable(tags, func(i, j int) bool {
a, _ := ParseVersion(tags[i])
b, _ := ParseVersion(tags[j])
return a.Compare(b) < 0
})
}
func containsString(list []string, s string) bool {
for _, l := range list {
if l == s {
return true
}
}
return false
}
+33
View File
@@ -0,0 +1,33 @@
package audit
import (
"context"
"os"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// TestFetchFromRegistry talks to Docker Hub, so it only runs with
// ABRA_TEST_NETWORK=1.
func TestFetchFromRegistry(t *testing.T) {
if os.Getenv("ABRA_TEST_NETWORK") != "1" {
t.Skip("set ABRA_TEST_NETWORK=1 to run")
}
for _, arch := range []string{"amd64", "arm64"} {
img, err := ParseImageRef("app", "nextcloud:apache")
require.NoError(t, err)
info, err := FetchFromRegistry(context.Background(), img, "linux", arch)
require.NoError(t, err)
assert.NotEmpty(t, info.Env["PHP_VERSION"], arch)
assert.NotEmpty(t, info.Env["NEXTCLOUD_VERSION"], arch)
require.NotNil(t, info.Created, arch)
t.Logf("%s: created=%s php=%s nextcloud=%s base=%s baseCreated=%v",
arch, info.Created, info.Env["PHP_VERSION"], info.Env["NEXTCLOUD_VERSION"],
info.Annotations[AnnotationBaseName], info.BaseCreated)
}
}
+114
View File
@@ -0,0 +1,114 @@
package audit
import (
"errors"
"os"
"path/filepath"
"time"
gitPkg "coopcloud.tech/abra/pkg/git"
"coopcloud.tech/abra/pkg/i18n"
"coopcloud.tech/abra/pkg/log"
"github.com/go-git/go-git/v5"
)
// DefaultAdvisoriesURL is the git repository holding the reviewed advisories.
const DefaultAdvisoriesURL = "https://git.coopcloud.tech/c4dt/advisories.git"
// fetchMarker is touched after each successful fetch, so the age of the local
// copy can be reported when the remote cannot be reached.
const fetchMarker = "abra-audit-last-fetch"
// SyncAdvisories makes sure dir holds an up-to-date checkout of the
// advisories repository. There is no cache lifetime: every call checks the
// remote. When the remote cannot be reached, a warning with the age of the
// local copy is logged and the local copy is used.
//
// A directory that is not a git checkout is used as is, which is handy for
// testing advisories locally.
func SyncAdvisories(dir, url string, offline bool) error {
if _, err := os.Stat(dir); os.IsNotExist(err) {
if offline {
return errors.New(i18n.G("no local advisories in %s and offline mode is on", dir))
}
log.Info(i18n.G("fetching advisories from %s", url))
if err := gitPkg.Clone(dir, url); err != nil {
_ = os.RemoveAll(dir)
return errors.New(i18n.G("unable to fetch advisories from %s: %s", url, err))
}
if _, err := git.PlainOpen(dir); err != nil {
_ = os.RemoveAll(dir)
return errors.New(i18n.G("unable to fetch advisories from %s: %s", url, err))
}
touchMarker(dir)
return nil
}
repo, err := git.PlainOpen(dir)
if err != nil {
log.Warn(i18n.G("%s is not a git checkout, using it as is", dir))
return nil
}
if offline {
log.Info(i18n.G("offline mode: using local advisories (%s)", localAge(dir, repo)))
return nil
}
wt, err := repo.Worktree()
if err != nil {
return err
}
err = wt.Pull(&git.PullOptions{RemoteName: "origin"})
if err != nil && !errors.Is(err, git.NoErrAlreadyUpToDate) {
log.Warn(i18n.G("unable to update advisories: %s", err))
log.Warn(i18n.G("using local advisories (%s)", localAge(dir, repo)))
return nil
}
if err == nil {
log.Debug(i18n.G("advisories updated"))
} else {
log.Debug(i18n.G("advisories already up to date"))
}
touchMarker(dir)
return nil
}
func touchMarker(dir string) {
p := filepath.Join(dir, ".git", fetchMarker)
now := time.Now()
if err := os.WriteFile(p, []byte(now.UTC().Format(time.RFC3339)+"\n"), 0o644); err != nil {
log.Debug(i18n.G("unable to write %s: %s", p, err))
}
}
// localAge describes how old the local copy is: the time of the last
// successful fetch, or of the last commit if abra never fetched it.
func localAge(dir string, repo *git.Repository) string {
if st, err := os.Stat(filepath.Join(dir, ".git", fetchMarker)); err == nil {
return i18n.G("last fetched %s ago", humanAge(time.Since(st.ModTime())))
}
if head, err := repo.Head(); err == nil {
if c, err := repo.CommitObject(head.Hash()); err == nil {
return i18n.G("last commit %s ago", humanAge(time.Since(c.Committer.When)))
}
}
return i18n.G("age unknown")
}
func humanAge(d time.Duration) string {
switch {
case d < time.Hour:
return i18n.G("%d minutes", int(d.Minutes()))
case d < 48*time.Hour:
return i18n.G("%d hours", int(d.Hours()))
default:
return i18n.G("%d days", int(d.Hours()/24))
}
}
+59
View File
@@ -0,0 +1,59 @@
package audit
import (
"os"
"path/filepath"
"testing"
"time"
"github.com/go-git/go-git/v5"
"github.com/go-git/go-git/v5/plumbing/object"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func commitFile(t *testing.T, repo *git.Repository, dir, name, content string) {
t.Helper()
require.NoError(t, os.MkdirAll(filepath.Dir(filepath.Join(dir, name)), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(dir, name), []byte(content), 0o644))
wt, err := repo.Worktree()
require.NoError(t, err)
_, err = wt.Add(name)
require.NoError(t, err)
_, err = wt.Commit("add "+name, &git.CommitOptions{
Author: &object.Signature{Name: "test", Email: "test@example.com", When: time.Now()},
})
require.NoError(t, err)
}
func TestSyncAdvisories(t *testing.T) {
upstream := t.TempDir()
repo, err := git.PlainInit(upstream, false)
require.NoError(t, err)
commitFile(t, repo, upstream, "advisories/a.yml", "id: A\nrecipes: {x: {fixed: 1.0.0}}\n")
local := filepath.Join(t.TempDir(), "advisories")
// offline without a local copy fails
require.Error(t, SyncAdvisories(local, upstream, true))
// first run clones
require.NoError(t, SyncAdvisories(local, upstream, false))
assert.FileExists(t, filepath.Join(local, "advisories", "a.yml"))
assert.FileExists(t, filepath.Join(local, ".git", fetchMarker))
// next run pulls new advisories
commitFile(t, repo, upstream, "advisories/b.yml", "id: B\nrecipes: {x: {fixed: 1.0.0}}\n")
require.NoError(t, SyncAdvisories(local, upstream, false))
assert.FileExists(t, filepath.Join(local, "advisories", "b.yml"))
// already up to date
require.NoError(t, SyncAdvisories(local, upstream, false))
// unreachable remote: warn and use the local copy
require.NoError(t, os.RemoveAll(upstream))
require.NoError(t, SyncAdvisories(local, upstream, false))
advs, _, err := LoadAdvisories(filepath.Join(local, "advisories"))
require.NoError(t, err)
assert.Len(t, advs, 2)
}
+267
View File
@@ -0,0 +1,267 @@
package audit
import (
"context"
"fmt"
"sort"
"sync"
"time"
appPkg "coopcloud.tech/abra/pkg/app"
"coopcloud.tech/abra/pkg/client"
"coopcloud.tech/abra/pkg/i18n"
"coopcloud.tech/abra/pkg/log"
"coopcloud.tech/abra/pkg/upstream/convert"
"coopcloud.tech/abra/pkg/upstream/stack"
dockerClient "github.com/docker/docker/client"
)
// AppState is what is deployed for one app.
type AppState struct {
Server string `json:"server"`
Name string `json:"name"`
Recipe string `json:"recipe"`
Version string `json:"version"`
Chaos bool `json:"chaos"`
Images []ImageRef `json:"images"`
// OS and Arch of the server, used to read per-platform image configs.
// Empty when unknown or offline.
OS string `json:"-"`
Arch string `json:"-"`
}
// ServerError records a server that could not be audited.
type ServerError struct {
Server string
Err error
}
// CollectOptions configures CollectState.
type CollectOptions struct {
// Offline disables registry lookups; image configs are then only read
// from the server's docker daemon.
Offline bool
}
// CollectState reads the deployed state of all given apps. Servers are
// queried in parallel. Apps that are not deployed are left out. A server that
// cannot be reached is returned as a ServerError, and the other servers are
// still audited.
func CollectState(apps []appPkg.App, opts CollectOptions) ([]AppState, []ServerError) {
byServer := make(map[string][]appPkg.App)
for _, app := range apps {
byServer[app.Server] = append(byServer[app.Server], app)
}
var (
mu sync.Mutex
wg sync.WaitGroup
states []AppState
errs []ServerError
)
for server, serverApps := range byServer {
wg.Add(1)
go func(server string, serverApps []appPkg.App) {
defer wg.Done()
s, err := collectServer(server, serverApps, opts)
mu.Lock()
defer mu.Unlock()
if err != nil {
errs = append(errs, ServerError{Server: server, Err: err})
return
}
states = append(states, s...)
}(server, serverApps)
}
wg.Wait()
sort.Slice(states, func(i, j int) bool {
if states[i].Server != states[j].Server {
return states[i].Server < states[j].Server
}
return states[i].Name < states[j].Name
})
sort.Slice(errs, func(i, j int) bool { return errs[i].Server < errs[j].Server })
markUnusableCreated(states)
return states, errs
}
func collectServer(server string, apps []appPkg.App, opts CollectOptions) ([]AppState, error) {
cl, err := client.New(server, client.WithTimeout(10))
if err != nil {
return nil, err
}
status := stack.GetAllDeployedServices(cl, server)
if status.Err != nil {
return nil, status.Err
}
ctx := context.Background()
var platformOS, platformArch string
if !opts.Offline {
if info, err := cl.Info(ctx); err == nil {
platformOS = info.OSType
platformArch = NormaliseArch(info.Architecture)
} else {
log.Debug(i18n.G("%s: unable to read platform: %s", server, err))
}
}
byStack := make(map[string]*AppState)
var order []string
for _, app := range apps {
byStack[app.StackName()] = &AppState{
Server: server,
Name: app.Name,
Recipe: app.Recipe.Name,
OS: platformOS,
Arch: platformArch,
}
}
deployed := make(map[string]bool)
cache := make(map[string]imageResult)
for _, svc := range status.Services {
stackName := svc.Spec.Labels[convert.LabelNamespace]
st, ok := byStack[stackName]
if !ok {
continue
}
if !deployed[stackName] {
deployed[stackName] = true
order = append(order, stackName)
}
if v, ok := svc.Spec.Labels[fmt.Sprintf("coop-cloud.%s.version", stackName)]; ok && v != "" {
st.Version = v
}
if c := svc.Spec.Labels[fmt.Sprintf("coop-cloud.%s.chaos", stackName)]; c == "true" {
st.Chaos = true
}
if svc.Spec.TaskTemplate.ContainerSpec == nil {
continue
}
ref := svc.Spec.TaskTemplate.ContainerSpec.Image
serviceName := svc.Spec.Name
if ns := stackName + "_"; len(serviceName) > len(ns) && serviceName[:len(ns)] == ns {
serviceName = serviceName[len(ns):]
}
img, err := ParseImageRef(serviceName, ref)
if err != nil {
img.InfoErr = err
st.Images = append(st.Images, img)
continue
}
res, ok := cache[ref]
if !ok {
res = inspectImage(ctx, cl, img, platformOS, platformArch, opts)
cache[ref] = res
}
img.Info, img.InfoErr = res.info, res.err
st.Images = append(st.Images, img)
}
var states []AppState
for _, stackName := range order {
st := byStack[stackName]
sort.Slice(st.Images, func(i, j int) bool { return st.Images[i].Service < st.Images[j].Service })
states = append(states, *st)
}
return states, nil
}
type imageResult struct {
info *ImageInfo
err error
}
func inspectImage(ctx context.Context, cl *dockerClient.Client, img ImageRef, platformOS, platformArch string, opts CollectOptions) imageResult {
nodeCtx, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()
info, nodeErr := InspectOnNode(nodeCtx, cl, img)
if nodeErr == nil {
return imageResult{info: info}
}
log.Debug(i18n.G("%s: not found on node: %s", img.Ref, nodeErr))
if opts.Offline || platformArch == "" {
return imageResult{err: nodeErr}
}
regCtx, cancel := context.WithTimeout(ctx, 60*time.Second)
defer cancel()
info, err := FetchFromRegistry(regCtx, img, platformOS, platformArch)
if err != nil {
return imageResult{err: fmt.Errorf("%s", i18n.G("node: %s; registry: %s", nodeErr, err))}
}
return imageResult{info: info}
}
// markUnusableCreated flags build times that cannot be trusted:
// - before 2000, which catches reproducible builds using the 1970 epoch;
// - earlier than the build time of the base image;
// - shared by two different digests of the same tag.
func markUnusableCreated(states []AppState) {
// tag -> created -> digests
seen := make(map[string]map[time.Time]map[string]bool)
for _, st := range states {
for _, img := range st.Images {
if img.Info == nil || img.Info.Created == nil || img.Digest == "" {
continue
}
c := img.Info.Created.UTC()
if seen[img.NameTag()] == nil {
seen[img.NameTag()] = make(map[time.Time]map[string]bool)
}
if seen[img.NameTag()][c] == nil {
seen[img.NameTag()][c] = make(map[string]bool)
}
seen[img.NameTag()][c][img.Digest] = true
}
}
for _, st := range states {
for _, img := range st.Images {
info := img.Info
if info == nil || info.Created == nil {
continue
}
c := info.Created.UTC()
switch {
case createdUnusable(info) != "":
info.createdUnusable = createdUnusable(info)
case len(seen[img.NameTag()][c]) > 1:
info.createdUnusable = i18n.G("several digests of %s share the build time %s", img.NameTag(), c.Format(time.RFC3339))
}
}
}
}
// createdUnusable runs the checks that need only the image itself: a build
// time before 2000, or earlier than the base image's.
func createdUnusable(info *ImageInfo) string {
if info.Created == nil {
return ""
}
c := info.Created.UTC()
switch {
case c.Before(time.Date(2000, 1, 1, 0, 0, 0, 0, time.UTC)):
return i18n.G("build time %s is before 2000 (reproducible build?)", c.Format(time.RFC3339))
case info.BaseCreated != nil && c.Before(*info.BaseCreated):
return i18n.G("build time %s is earlier than its base image", c.Format(time.RFC3339))
}
return ""
}
+169
View File
@@ -0,0 +1,169 @@
package audit
import (
"errors"
"regexp"
"strconv"
"strings"
"coopcloud.tech/abra/pkg/i18n"
)
// Version is a loosely parsed version. Recipe tags have the form
// RECIPE+UPSTREAM (e.g. 7.0.3+30.0.1), so a version is a list of groups
// separated by "+", each group a list of numbers. Anything after the leading
// numeric part of a group (e.g. "-apache" in "35.0.3-apache") is ignored.
type Version struct {
Raw string
Groups [][]int
}
var numericPrefix = regexp.MustCompile(`^[vV]?(\d+(?:\.\d+)*)`)
// ParseVersion parses a version such as "8.3.14", "v1.2", "35.0.3-apache" or
// "7.0.3+30.0.1".
func ParseVersion(raw string) (Version, error) {
v := Version{Raw: raw}
raw = strings.TrimSpace(raw)
if raw == "" {
return v, errors.New(i18n.G("empty version"))
}
for _, group := range strings.Split(raw, "+") {
m := numericPrefix.FindStringSubmatch(group)
if m == nil {
return v, errors.New(i18n.G("unable to parse version %s", raw))
}
var nums []int
for _, part := range strings.Split(m[1], ".") {
n, err := strconv.Atoi(part)
if err != nil {
return v, errors.New(i18n.G("unable to parse version %s", raw))
}
nums = append(nums, n)
}
v.Groups = append(v.Groups, nums)
}
return v, nil
}
// Compare returns -1, 0 or 1. Only the groups present in both versions are
// compared, so "7.0.3" equals "7.0.3+30.0.1": a constraint without an
// upstream part only constrains the recipe part. Missing numbers inside a
// group count as 0.
func (v Version) Compare(o Version) int {
groups := min(len(v.Groups), len(o.Groups))
for g := 0; g < groups; g++ {
a, b := v.Groups[g], o.Groups[g]
for i := 0; i < max(len(a), len(b)); i++ {
var x, y int
if i < len(a) {
x = a[i]
}
if i < len(b) {
y = b[i]
}
if x < y {
return -1
}
if x > y {
return 1
}
}
}
return 0
}
type constraint struct {
op string
version Version
}
// Range is a set of version constraints. Constraints separated by spaces or
// commas must all hold; alternatives are separated by "||". For example
// ">=8.3.0 <8.3.14 || >=8.4.0 <8.4.2".
type Range struct {
Raw string
alts [][]constraint
}
var constraintRe = regexp.MustCompile(`^(<=|>=|==|!=|<|>|=)?\s*(.+)$`)
// ParseRange parses a version range.
func ParseRange(raw string) (Range, error) {
r := Range{Raw: raw}
if strings.TrimSpace(raw) == "" {
return r, errors.New(i18n.G("empty version range"))
}
for _, alt := range strings.Split(raw, "||") {
// NOTE: allow "< 1.2" by gluing a lone operator to the next token
fields := strings.FieldsFunc(alt, func(r rune) bool { return r == ' ' || r == ',' || r == '\t' })
var tokens []string
for i := 0; i < len(fields); i++ {
if constraintRe.MatchString(fields[i]) && strings.Trim(fields[i], "<>=!") == "" && i+1 < len(fields) {
tokens = append(tokens, fields[i]+fields[i+1])
i++
continue
}
tokens = append(tokens, fields[i])
}
if len(tokens) == 0 {
return r, errors.New(i18n.G("empty alternative in version range %s", raw))
}
var cs []constraint
for _, tok := range tokens {
m := constraintRe.FindStringSubmatch(tok)
if m == nil {
return r, errors.New(i18n.G("unable to parse version range %s", raw))
}
op := m[1]
if op == "" || op == "==" {
op = "="
}
v, err := ParseVersion(m[2])
if err != nil {
return r, errors.New(i18n.G("unable to parse version range %s: %s", raw, err))
}
cs = append(cs, constraint{op: op, version: v})
}
r.alts = append(r.alts, cs)
}
return r, nil
}
// Contains reports whether the version satisfies the range.
func (r Range) Contains(v Version) bool {
for _, alt := range r.alts {
ok := true
for _, c := range alt {
cmp := v.Compare(c.version)
switch c.op {
case "<":
ok = cmp < 0
case "<=":
ok = cmp <= 0
case ">":
ok = cmp > 0
case ">=":
ok = cmp >= 0
case "=":
ok = cmp == 0
case "!=":
ok = cmp != 0
}
if !ok {
break
}
}
if ok {
return true
}
}
return false
}
+52 -1
View File
@@ -4,18 +4,35 @@ import (
"context"
"errors"
"fmt"
"slices"
"coopcloud.tech/abra/pkg/i18n"
"github.com/containers/image/docker"
"github.com/containers/image/manifest"
"github.com/containers/image/types"
"github.com/distribution/reference"
imgspecv1 "github.com/opencontainers/image-spec/specs-go/v1"
)
func init() {
// NOTE: containers/image does not request OCI image indexes by default.
// Many multi-arch images (e.g. on Docker Hub) are published as OCI indexes
// and the registry refuses to serve them without the matching Accept header.
if !slices.Contains(manifest.DefaultRequestedManifestMIMETypes, imgspecv1.MediaTypeImageIndex) {
manifest.DefaultRequestedManifestMIMETypes = append(
manifest.DefaultRequestedManifestMIMETypes,
imgspecv1.MediaTypeImageIndex,
)
}
}
// GetRegistryTags retrieves all tags of an image from a container registry.
func GetRegistryTags(img reference.Named) ([]string, error) {
var tags []string
ref, err := docker.ParseReference(fmt.Sprintf("//%s", img))
// NOTE: strip tag and digest, containers/image refuses references
// which have both (e.g. nextcloud:34.0.5-fpm@sha256:...)
ref, err := docker.ParseReference(fmt.Sprintf("//%s", reference.TrimNamed(img)))
if err != nil {
return tags, errors.New(i18n.G("failed to parse image %s, saw: %s", img, err.Error()))
}
@@ -28,3 +45,37 @@ func GetRegistryTags(img reference.Named) ([]string, error) {
return tags, nil
}
// GetRegistryDigest retrieves the manifest digest of an image tag from a
// container registry. For multi-arch images, this is the digest of the image
// index, which is what Docker resolves "image:tag" to.
func GetRegistryDigest(img reference.Named, tag string) (string, error) {
tagged, err := reference.WithTag(reference.TrimNamed(img), tag)
if err != nil {
return "", errors.New(i18n.G("failed to parse image %s, saw: %s", img, err.Error()))
}
ref, err := docker.ParseReference(fmt.Sprintf("//%s", tagged))
if err != nil {
return "", errors.New(i18n.G("failed to parse image %s, saw: %s", tagged, err.Error()))
}
ctx := context.Background()
src, err := ref.NewImageSource(ctx, &types.SystemContext{})
if err != nil {
return "", err
}
defer src.Close()
manifestBytes, _, err := src.GetManifest(ctx, nil)
if err != nil {
return "", err
}
digest, err := manifest.Digest(manifestBytes)
if err != nil {
return "", err
}
return digest.String(), nil
}
+13 -2
View File
@@ -73,6 +73,17 @@ type Abra struct {
// 2. check if abraDir was set in a config file
// 3. use $HOME/.abra when above two options failed
func (a Abra) GetAbraDir() string {
if dir := a.GetExplicitAbraDir(); dir != "" {
return dir
}
log.Debug(i18n.G("using default abra dir"))
return os.ExpandEnv("$HOME/.abra")
}
// GetExplicitAbraDir returns the abra dir only when it was explicitly chosen,
// either through $ABRA_DIR or through the abraDir setting of a config file.
// It returns an empty string otherwise, and never falls back to $HOME/.abra.
func (a Abra) GetExplicitAbraDir() string {
if dir, exists := os.LookupEnv("ABRA_DIR"); exists && dir != "" {
log.Debug(i18n.G("read abra dir from $ABRA_DIR"))
return dir
@@ -85,8 +96,7 @@ func (a Abra) GetAbraDir() string {
// Make the path absolute
return path.Join(a.configPath, a.AbraDir)
}
log.Debug(i18n.G("using default abra dir"))
return os.ExpandEnv("$HOME/.abra")
return ""
}
func (a Abra) GetServersDir() string { return path.Join(a.GetAbraDir(), "servers") }
@@ -98,6 +108,7 @@ var config = LoadAbraConfig()
var (
ABRA_DIR = config.GetAbraDir()
ABRA_DIR_EXPLICIT = config.GetExplicitAbraDir()
SERVERS_DIR = config.GetServersDir()
RECIPES_DIR = config.GetRecipesDir()
LOGS_DIR = config.GetLogsDir()
+14 -2
View File
@@ -26,12 +26,24 @@ var BoldUnderlineStyle = lipgloss.NewStyle().
Bold(true).
Underline(true)
// ShortenID returns the first 12 characters of an ID, or the whole ID if it
// is shorter.
func ShortenID(str string) string {
return str[:12]
return Truncate(str, 12)
}
// SmallSHA returns the first 8 characters of a hash, or the whole hash if it
// is shorter (e.g. the placeholder commit of a build without ldflags).
func SmallSHA(hash string) string {
return hash[:8]
return Truncate(hash, 8)
}
// Truncate returns the first n characters of str, or str if it is shorter.
func Truncate(str string, n int) string {
if len(str) <= n {
return str
}
return str[:n]
}
// RemoveSha remove image sha from a string that are added in some docker outputs
+16
View File
@@ -9,3 +9,19 @@ import (
func TestBoldDirtyDefault(t *testing.T) {
assert.Equal(t, "foo", BoldDirtyDefault("foo"))
}
func TestSmallSHA(t *testing.T) {
assert.Equal(t, "d699e51c", SmallSHA("d699e51c4c2a8f0e"))
assert.Equal(t, " ", SmallSHA(" "))
assert.Equal(t, "", SmallSHA(""))
}
func TestShortenID(t *testing.T) {
assert.Equal(t, "0123456789ab", ShortenID("0123456789abcdef"))
assert.Equal(t, "abc", ShortenID("abc"))
}
func TestTruncate(t *testing.T) {
assert.Equal(t, "abc", Truncate("abcdef", 3))
assert.Equal(t, "ab", Truncate("ab", 7))
}
+5
View File
@@ -36,6 +36,11 @@ func IsClean(repoPath string) (bool, error) {
return false, err
}
// Ignore the abra-managed sidecar file that records the canonical
// source URL for externally-cloned recipes; it lives alongside the
// recipe but is not part of the upstream tree.
patterns = append(patterns, gitignore.ParsePattern(".abra-source", nil))
if len(patterns) > 0 {
worktree.Excludes = append(patterns, worktree.Excludes...)
}
+36
View File
@@ -2,6 +2,8 @@ package git
import (
"errors"
"os"
"path/filepath"
"testing"
"github.com/go-git/go-git/v5"
@@ -13,3 +15,37 @@ func TestIsClean(t *testing.T) {
assert.Equal(t, isClean, false)
assert.True(t, errors.Is(err, git.ErrRepositoryNotExists))
}
// TestIsCleanIgnoresAbraSource confirms that the .abra-source sidecar
// file written by abra next to externally-cloned recipes does not cause
// IsClean to report the worktree as dirty.
func TestIsCleanIgnoresAbraSource(t *testing.T) {
dir := t.TempDir()
if _, err := git.PlainInit(dir, false); err != nil {
t.Fatalf("git init failed: %s", err)
}
sidecar := filepath.Join(dir, ".abra-source")
if err := os.WriteFile(sidecar, []byte("git.example.com/u/recipe\n"), 0o644); err != nil {
t.Fatalf("writing sidecar failed: %s", err)
}
isClean, err := IsClean(dir)
if err != nil {
t.Fatalf("IsClean returned error: %s", err)
}
assert.True(t, isClean, "expected worktree with only .abra-source to be reported clean")
// Sanity check: an unrelated untracked file should still mark it dirty.
other := filepath.Join(dir, "random.txt")
if err := os.WriteFile(other, []byte("hello"), 0o644); err != nil {
t.Fatalf("writing extra file failed: %s", err)
}
isClean, err = IsClean(dir)
if err != nil {
t.Fatalf("IsClean returned error: %s", err)
}
assert.False(t, isClean, "expected worktree with unrelated untracked file to be reported dirty")
}
+97 -72
View File
@@ -7,13 +7,13 @@
msgid ""
msgstr "Project-Id-Version: \n"
"Report-Msgid-Bugs-To: EMAIL\n"
"POT-Creation-Date: 2026-08-31 19:21+0000\n"
"POT-Creation-Date: 2026-10-22 21:25+0200\n"
"PO-Revision-Date: YEAR-MO-DA HO:MI+ZONE\n"
"Last-Translator: FULL NAME <EMAIL@ADDRESS>\n"
"Language-Team: LANGUAGE <LL@li.org>\n"
"Language: \n"
"MIME-Version: 1.0\n"
"Content-Type: text/plain; charset=UTF-8\n"
"Content-Type: text/plain; charset=CHARSET\n"
"Content-Transfer-Encoding: 8bit\n"
#: ./cli/app/cp.go:38
@@ -219,7 +219,7 @@ msgstr ""
msgid "%s created (version: %s)"
msgstr ""
#: ./cli/recipe/upgrade.go:335
#: ./cli/recipe/upgrade.go:345
#, c-format
msgid "%s currently has these unstaged changes 👇"
msgstr ""
@@ -304,7 +304,7 @@ msgstr ""
msgid "%s has no secrets to generate, skipping..."
msgstr ""
#: ./pkg/recipe/compose.go:107
#: ./pkg/recipe/compose.go:109
#, c-format
msgid "%s has no version label? try running \"abra recipe sync %s\" first?"
msgstr ""
@@ -324,12 +324,12 @@ msgstr ""
msgid "%s is already fetched"
msgstr ""
#: ./pkg/recipe/compose.go:233
#: ./pkg/recipe/compose.go:277
#, c-format
msgid "%s is already set, nothing to do?"
msgstr ""
#: ./pkg/recipe/compose.go:72
#: ./pkg/recipe/compose.go:74
#, c-format
msgid "%s is missing a compose.yml or compose.*.yml file?"
msgstr ""
@@ -379,7 +379,7 @@ msgstr ""
msgid "%s must first be deployed on %s before moving"
msgstr ""
#: ./cli/recipe/upgrade.go:150
#: ./cli/recipe/upgrade.go:153
#, c-format
msgid "%s not considered semver-like"
msgstr ""
@@ -549,6 +549,11 @@ msgstr ""
msgid "%s: waiting %d seconds before next retry"
msgstr ""
#: ./cli/recipe/upgrade.go:392
#, c-format
msgid "%s:%s was re-published upstream (digest %s → %s)"
msgstr ""
#: ./cli/app/upgrade.go:425
#, c-format
msgid "'%s' is not a known version"
@@ -638,17 +643,17 @@ msgstr ""
msgid "CHAOS DEPLOY"
msgstr ""
#: ./pkg/recipe/compose.go:47
#: ./pkg/recipe/compose.go:49
#, c-format
msgid "COMPOSE_FILE (=\"%s\") parsing failed?"
msgstr ""
#: ./pkg/recipe/compose.go:58
#: ./pkg/recipe/compose.go:60
#, c-format
msgid "COMPOSE_FILE detected (%s), loading %s"
msgstr ""
#: ./pkg/recipe/compose.go:38
#: ./pkg/recipe/compose.go:40
#, c-format
msgid "COMPOSE_FILE detected, loading %s"
msgstr ""
@@ -1687,7 +1692,7 @@ msgctxt "recipe fetch"
msgid "a"
msgstr ""
#: ./cli/recipe/upgrade.go:408
#: ./cli/recipe/upgrade.go:459
msgctxt "recipe upgrade"
msgid "a"
msgstr ""
@@ -1737,7 +1742,7 @@ msgstr ""
msgid "accepts at most 2 args with --local/-l"
msgstr ""
#: ./pkg/recipe/compose.go:249
#: ./pkg/recipe/compose.go:293
#, c-format
msgid "add '- \"%s\"' manually to the 'app' service in %s"
msgstr ""
@@ -1780,7 +1785,7 @@ msgstr ""
msgid "all-services"
msgstr ""
#: ./cli/recipe/upgrade.go:407
#: ./cli/recipe/upgrade.go:458
msgid "all-tags"
msgstr ""
@@ -1919,7 +1924,7 @@ msgstr ""
msgid "c"
msgstr ""
#: ./cli/recipe/upgrade.go:416
#: ./cli/recipe/upgrade.go:467
msgctxt "recipe upgrade"
msgid "c"
msgstr ""
@@ -1928,7 +1933,7 @@ msgstr ""
msgid "can not insert from file and read from stdin"
msgstr ""
#: ./cli/recipe/upgrade.go:323
#: ./cli/recipe/upgrade.go:333
#, c-format
msgid "can upgrade service: %s, image: %s, tag: %s ::"
msgstr ""
@@ -2108,7 +2113,7 @@ msgstr ""
msgid "chore: publish new catalogue release changes"
msgstr ""
#: ./cli/recipe/upgrade.go:352
#: ./cli/recipe/upgrade.go:362
msgid "chore: update image tags"
msgstr ""
@@ -2170,19 +2175,19 @@ msgstr ""
msgid "commandconn: starting %s with %v"
msgstr ""
#: ./cli/recipe/upgrade.go:415
#: ./cli/recipe/upgrade.go:466
msgid "commit"
msgstr ""
#: ./cli/recipe/upgrade.go:418
#: ./cli/recipe/upgrade.go:469
msgid "commit changes"
msgstr ""
#: ./cli/recipe/upgrade.go:342
#: ./cli/recipe/upgrade.go:352
msgid "commit changes?"
msgstr ""
#: ./cli/recipe/upgrade.go:356
#: ./cli/recipe/upgrade.go:366
#, c-format
msgid "committed changes as '%s'"
msgstr ""
@@ -2230,12 +2235,12 @@ msgstr ""
msgid "connection timed out for %s"
msgstr ""
#: ./pkg/recipe/compose.go:191
#: ./pkg/recipe/compose.go:235
#, c-format
msgid "considering %s config(s) for label update"
msgstr ""
#: ./pkg/recipe/compose.go:123
#: ./pkg/recipe/compose.go:132
#, c-format
msgid "considering %s config(s) for tag update"
msgstr ""
@@ -2258,7 +2263,7 @@ msgstr ""
msgid "context lacks Docker endpoint"
msgstr ""
#: ./pkg/recipe/compose.go:229
#: ./pkg/recipe/compose.go:273
#, c-format
msgid "coop-cloud.${STACK_NAME}.version=%s"
msgstr ""
@@ -2465,12 +2470,12 @@ msgstr ""
msgid "detected ABRA_CI=1"
msgstr ""
#: ./cli/recipe/upgrade.go:204
#: ./cli/recipe/upgrade.go:214
#, c-format
msgid "detected compatible upgradable tags %s for %s"
msgstr ""
#: ./cli/recipe/upgrade.go:177
#: ./cli/recipe/upgrade.go:180
#, c-format
msgid "detected potential upgradable tags %s for %s"
msgstr ""
@@ -2509,6 +2514,16 @@ msgstr ""
msgid "different versions for secret '%s', '%s' and %s'"
msgstr ""
#: ./cli/recipe/upgrade.go:388
#, c-format
msgid "digest of %s:%s is up to date"
msgstr ""
#: ./cli/recipe/upgrade.go:413
#, c-format
msgid "digest updated for %s:%s"
msgstr ""
#: ./pkg/recipe/recipe.go:202
#, c-format
msgid "dir: %s, "
@@ -2658,7 +2673,7 @@ msgstr ""
msgid "ensuring env version %s"
msgstr ""
#: ./cli/recipe/upgrade.go:285
#: ./cli/recipe/upgrade.go:295
msgid "enter / return to confirm, choose 'skip' to not upgrade this tag, vim mode is enabled"
msgstr ""
@@ -2839,7 +2854,7 @@ msgstr ""
msgid "failed to migrate app config: %s"
msgstr ""
#: ./pkg/client/registry.go:20
#: ./pkg/client/registry.go:37 ./pkg/client/registry.go:55 ./pkg/client/registry.go:60
#, c-format
msgid "failed to parse image %s, saw: %s"
msgstr ""
@@ -3047,7 +3062,7 @@ msgstr ""
msgid "generated secrets %s shown again, please take note of them %s"
msgstr ""
#: ./pkg/app/compose.go:105
#: ./pkg/app/compose.go:63
#, c-format
msgid "get label '%s'"
msgstr ""
@@ -3291,15 +3306,15 @@ msgstr ""
msgid "including VOLUMES=%v in backupbot exec invocation"
msgstr ""
#: ./cli/recipe/release.go:634 ./cli/recipe/upgrade.go:378
#: ./cli/recipe/release.go:634 ./cli/recipe/upgrade.go:429
msgid "increase the major part of the version"
msgstr ""
#: ./cli/recipe/release.go:642 ./cli/recipe/upgrade.go:386
#: ./cli/recipe/release.go:642 ./cli/recipe/upgrade.go:437
msgid "increase the minor part of the version"
msgstr ""
#: ./cli/recipe/release.go:650 ./cli/recipe/upgrade.go:394
#: ./cli/recipe/release.go:650 ./cli/recipe/upgrade.go:445
msgid "increase the patch part of the version"
msgstr ""
@@ -3464,7 +3479,7 @@ msgstr ""
msgid "list [flags]"
msgstr ""
#: ./cli/recipe/upgrade.go:410
#: ./cli/recipe/upgrade.go:461
msgid "list all tags, not just upgrades"
msgstr ""
@@ -3534,11 +3549,11 @@ msgstr ""
#. with no spaces in between
#. translators: `abra man` aliases. use a comma separated list of aliases
#. with no spaces in between
#: ./cli/app/list.go:319 ./cli/app/move.go:34 ./cli/app/ps.go:205 ./cli/app/secret.go:553 ./cli/app/secret.go:649 ./cli/recipe/list.go:104 ./cli/recipe/upgrade.go:400 ./cli/recipe/version.go:139 ./cli/run.go:152 ./cli/server/list.go:106
#: ./cli/app/list.go:319 ./cli/app/move.go:34 ./cli/app/ps.go:205 ./cli/app/secret.go:553 ./cli/app/secret.go:649 ./cli/recipe/list.go:104 ./cli/recipe/upgrade.go:451 ./cli/recipe/version.go:139 ./cli/run.go:152 ./cli/server/list.go:106
msgid "m"
msgstr ""
#: ./cli/app/list.go:318 ./cli/app/ps.go:204 ./cli/app/secret.go:552 ./cli/app/secret.go:648 ./cli/recipe/list.go:103 ./cli/recipe/upgrade.go:399 ./cli/recipe/version.go:138 ./cli/server/list.go:105
#: ./cli/app/list.go:318 ./cli/app/ps.go:204 ./cli/app/secret.go:552 ./cli/app/secret.go:648 ./cli/recipe/list.go:103 ./cli/recipe/upgrade.go:450 ./cli/recipe/version.go:138 ./cli/server/list.go:105
msgid "machine"
msgstr ""
@@ -3547,7 +3562,7 @@ msgstr ""
msgid "main app service version for %s is empty?"
msgstr ""
#: ./cli/internal/recipe.go:48 ./cli/internal/recipe.go:66 ./cli/internal/recipe.go:80 ./cli/recipe/release.go:631 ./cli/recipe/upgrade.go:375
#: ./cli/internal/recipe.go:48 ./cli/internal/recipe.go:66 ./cli/internal/recipe.go:80 ./cli/recipe/release.go:631 ./cli/recipe/upgrade.go:426
msgid "major"
msgstr ""
@@ -3576,7 +3591,7 @@ msgstr ""
msgid "migrating app config from %s to %s"
msgstr ""
#: ./cli/internal/recipe.go:48 ./cli/internal/recipe.go:68 ./cli/internal/recipe.go:82 ./cli/recipe/release.go:639 ./cli/recipe/upgrade.go:383
#: ./cli/internal/recipe.go:48 ./cli/internal/recipe.go:68 ./cli/internal/recipe.go:82 ./cli/recipe/release.go:639 ./cli/recipe/upgrade.go:434
msgid "minor"
msgstr ""
@@ -3695,7 +3710,7 @@ msgstr ""
msgid "no %s exists, skipping reading gitignore paths"
msgstr ""
#: ./pkg/app/compose.go:111
#: ./pkg/app/compose.go:69
#, c-format
msgid "no %s label found for %s"
msgstr ""
@@ -3705,7 +3720,7 @@ msgstr ""
msgid "no .git directory in %s?"
msgstr ""
#: ./pkg/recipe/compose.go:29
#: ./pkg/recipe/compose.go:31
#, c-format
msgid "no COMPOSE_FILE detected, loading default: %s"
msgstr ""
@@ -3731,7 +3746,7 @@ msgstr ""
msgid "no changes discovered in %s, nothing to publish?"
msgstr ""
#: ./cli/recipe/upgrade.go:361
#: ./cli/recipe/upgrade.go:371
msgid "no changes, skip creating commit"
msgstr ""
@@ -3756,7 +3771,7 @@ msgstr ""
msgid "no domain provided"
msgstr ""
#: ./pkg/recipe/compose.go:248
#: ./pkg/recipe/compose.go:292
msgid "no existing label found, automagic insertion not supported yet"
msgstr ""
@@ -3765,7 +3780,7 @@ msgstr ""
msgid "no git tags found for %s"
msgstr ""
#: ./cli/recipe/upgrade.go:182
#: ./cli/recipe/upgrade.go:192
#, c-format
msgid "no new versions available for %s, assuming %s is the latest (use -a/--all-tags to see all anyway)"
msgstr ""
@@ -3897,12 +3912,12 @@ msgstr ""
msgid "not requesting a remote TTY"
msgstr ""
#: ./cli/recipe/upgrade.go:305
#: ./cli/recipe/upgrade.go:315
#, c-format
msgid "not upgrading %s, skipping as requested"
msgstr ""
#: ./cli/recipe/upgrade.go:244
#: ./cli/recipe/upgrade.go:254
#, c-format
msgid "not upgrading from %s to %s for %s, because the upgrade type is more serious than what user wants"
msgstr ""
@@ -3956,12 +3971,12 @@ msgstr ""
msgid "p"
msgstr ""
#: ./cli/recipe/upgrade.go:163
#: ./cli/recipe/upgrade.go:166
#, c-format
msgid "parsed %s for %s"
msgstr ""
#: ./pkg/recipe/compose.go:159
#: ./pkg/recipe/compose.go:178
#, c-format
msgid "parsed %s from %s"
msgstr ""
@@ -3999,7 +4014,7 @@ msgstr ""
msgid "pass command not found on $PATH, is it installed?"
msgstr ""
#: ./cli/internal/recipe.go:48 ./cli/internal/recipe.go:70 ./cli/internal/recipe.go:84 ./cli/recipe/release.go:647 ./cli/recipe/upgrade.go:391
#: ./cli/internal/recipe.go:48 ./cli/internal/recipe.go:70 ./cli/internal/recipe.go:84 ./cli/recipe/release.go:647 ./cli/recipe/upgrade.go:442
msgid "patch"
msgstr ""
@@ -4053,7 +4068,7 @@ msgstr ""
msgid "prefer offline & filesystem access"
msgstr ""
#: ./cli/app/list.go:321 ./cli/app/ps.go:207 ./cli/app/secret.go:555 ./cli/app/secret.go:651 ./cli/recipe/list.go:106 ./cli/recipe/upgrade.go:402 ./cli/recipe/version.go:141 ./cli/server/list.go:108
#: ./cli/app/list.go:321 ./cli/app/ps.go:207 ./cli/app/secret.go:555 ./cli/app/secret.go:651 ./cli/recipe/list.go:106 ./cli/recipe/upgrade.go:453 ./cli/recipe/version.go:141 ./cli/server/list.go:108
msgid "print machine-readable output"
msgstr ""
@@ -4436,7 +4451,7 @@ msgstr ""
msgid "retrieved %s as target container on %s"
msgstr ""
#: ./pkg/recipe/compose.go:59
#: ./pkg/recipe/compose.go:61
#, c-format
msgid "retrieved %s configs for %s"
msgstr ""
@@ -4446,7 +4461,7 @@ msgstr ""
msgid "retrieved %s for %s"
msgstr ""
#: ./cli/recipe/upgrade.go:144
#: ./cli/recipe/upgrade.go:147
#, c-format
msgid "retrieved %s from remote registry for %s"
msgstr ""
@@ -4475,7 +4490,7 @@ msgstr ""
msgid "retrieving docker auth token: failed create docker cli: %s"
msgstr ""
#: ./pkg/recipe/recipe.go:659
#: ./pkg/recipe/recipe.go:656
msgid "retrieving recipes"
msgstr ""
@@ -4723,12 +4738,12 @@ msgstr ""
msgid "serverAdd: cleanUp: unable to list files in %s: %s"
msgstr ""
#: ./cli/recipe/upgrade.go:227
#: ./cli/recipe/upgrade.go:237
#, c-format
msgid "service %s is at version %s, but pinned to %s, please correct your compose.yml file manually!"
msgstr ""
#: ./cli/recipe/upgrade.go:223
#: ./cli/recipe/upgrade.go:233
#, c-format
msgid "service %s, image %s pinned to %s, no compatible upgrade found"
msgstr ""
@@ -4746,24 +4761,24 @@ msgstr ""
msgid "set 'main' as the default branch"
msgstr ""
#: ./pkg/app/compose.go:72
#: ./pkg/app/compose.go:30
#, c-format
msgid "set label 'coop-cloud.%s.chaos' to %v for %s"
msgstr ""
#: ./pkg/app/compose.go:83
#: ./pkg/app/compose.go:41
#, c-format
msgid "set label 'coop-cloud.%s.chaos-version' to %v for %s"
msgstr ""
#: ./pkg/app/compose.go:93
#: ./pkg/app/compose.go:51
#, c-format
msgid "set label 'coop-cloud.%s.version' to %v for %s"
msgstr ""
#: ./pkg/app/compose.go:61
#: ./pkg/app/compose.go:18
#, c-format
msgid "set recipe labels 'coop-cloud.%s.recipe' and 'coop-cloud.recipe' to %s for %s"
msgid "set recipe label 'coop-cloud.%s.recipe' to %s for %s"
msgstr ""
#: ./pkg/git/init.go:60
@@ -5007,7 +5022,7 @@ msgstr ""
msgid "sync [flags]"
msgstr ""
#: ./pkg/recipe/compose.go:243
#: ./pkg/recipe/compose.go:287
#, c-format
msgid "synced label %s to service %s"
msgstr ""
@@ -5025,7 +5040,7 @@ msgstr ""
msgid "tag at commit %s is unannotated or otherwise broken"
msgstr ""
#: ./cli/recipe/upgrade.go:301
#: ./cli/recipe/upgrade.go:311
#, c-format
msgid "tag upgraded from %s to %s for %s"
msgstr ""
@@ -5064,7 +5079,7 @@ msgstr ""
msgid "timed out on undeploy (timeout=%v sec)"
msgstr ""
#: ./pkg/app/compose.go:122
#: ./pkg/app/compose.go:80
#, c-format
msgid "timeout label: %s"
msgstr ""
@@ -5172,7 +5187,7 @@ msgstr ""
msgid "unable to continue, input required for initial version"
msgstr ""
#: ./pkg/app/compose.go:127
#: ./pkg/app/compose.go:85
#, c-format
msgid "unable to convert timeout label %s to int: %s"
msgstr ""
@@ -5212,7 +5227,7 @@ msgstr ""
msgid "unable to determine server of app %s, please pass --server/-s"
msgstr ""
#: ./cli/recipe/upgrade.go:252
#: ./cli/recipe/upgrade.go:262
#, c-format
msgid "unable to determine versioning semantics of %s, listing all tags"
msgstr ""
@@ -5286,12 +5301,12 @@ msgstr ""
msgid "unable to open worktree of %s: %s"
msgstr ""
#: ./cli/recipe/upgrade.go:159
#: ./cli/recipe/upgrade.go:162
#, c-format
msgid "unable to parse %s, error was: %s, skipping upgrade for %s"
msgstr ""
#: ./pkg/recipe/compose.go:153
#: ./pkg/recipe/compose.go:167
#, c-format
msgid "unable to parse %s, skipping"
msgstr ""
@@ -5331,7 +5346,7 @@ msgstr ""
msgid "unable to read remotes in %s: %s"
msgstr ""
#: ./cli/recipe/upgrade.go:153
#: ./cli/recipe/upgrade.go:156
#, c-format
msgid "unable to read tag for image %s, is it missing? skipping upgrade for %s"
msgstr ""
@@ -5381,6 +5396,11 @@ msgstr ""
msgid "unable to retrieve container for %s: %s"
msgstr ""
#: ./pkg/recipe/compose.go:193
#, c-format
msgid "unable to retrieve digest for %s: %s"
msgstr ""
#: ./cli/app/list.go:157
#, c-format
msgid "unable to retrieve tags for %s: %s"
@@ -5513,12 +5533,17 @@ msgstr ""
msgid "up"
msgstr ""
#: ./cli/recipe/upgrade.go:397
#, c-format
msgid "update digest of %s:%s?"
msgstr ""
#: ./pkg/upstream/stack/stack.go:477
#, c-format
msgid "updating %s"
msgstr ""
#: ./pkg/recipe/compose.go:171 ./pkg/recipe/compose.go:237
#: ./pkg/recipe/compose.go:213 ./pkg/recipe/compose.go:281
#, c-format
msgid "updating %s to %s in %s"
msgstr ""
@@ -5543,17 +5568,17 @@ msgstr ""
msgid "upgrade [flags]"
msgstr ""
#: ./cli/recipe/upgrade.go:248
#: ./cli/recipe/upgrade.go:258
#, c-format
msgid "upgrade to which tag? (service: %s, image: %s, tag: %s)"
msgstr ""
#: ./cli/recipe/upgrade.go:254
#: ./cli/recipe/upgrade.go:264
#, c-format
msgid "upgrade to which tag? (service: %s, tag: %s)"
msgstr ""
#: ./cli/recipe/upgrade.go:221
#: ./cli/recipe/upgrade.go:231
#, c-format
msgid "upgrading service %s from %s to %s (pinned tag: %s)"
msgstr ""
@@ -5806,11 +5831,11 @@ msgstr ""
msgid "writing recipe version failed: %s"
msgstr ""
#: ./cli/recipe/release.go:632 ./cli/recipe/upgrade.go:376
#: ./cli/recipe/release.go:632 ./cli/recipe/upgrade.go:427
msgid "x"
msgstr ""
#: ./cli/recipe/release.go:640 ./cli/recipe/upgrade.go:384
#: ./cli/recipe/release.go:640 ./cli/recipe/upgrade.go:435
msgid "y"
msgstr ""
@@ -5822,7 +5847,7 @@ msgstr ""
msgid "you can only use one version flag: --major, --minor or --patch"
msgstr ""
#: ./cli/recipe/release.go:648 ./cli/recipe/upgrade.go:392
#: ./cli/recipe/release.go:648 ./cli/recipe/upgrade.go:443
msgid "z"
msgstr ""
Binary file not shown.
+156 -709
View File
File diff suppressed because one or more lines are too long
+49 -5
View File
@@ -6,8 +6,10 @@ import (
"io/ioutil"
"os"
"path/filepath"
"regexp"
"strings"
"coopcloud.tech/abra/pkg/client"
"coopcloud.tech/abra/pkg/formatter"
"coopcloud.tech/abra/pkg/i18n"
"coopcloud.tech/abra/pkg/log"
@@ -110,7 +112,14 @@ func (r Recipe) GetVersionLabelLocal() (string, error) {
return label, nil
}
// getRegistryDigest is a variable so that it can be stubbed in tests.
var getRegistryDigest = client.GetRegistryDigest
// UpdateTag updates an image tag in-place on file system local compose files.
// Images which are pinned to a digest (e.g. "nextcloud:34.0.5-fpm@sha256:...")
// have their digest updated to the one of the new tag, as retrieved from the
// container registry. Passing the current tag of a digest pinned image
// refreshes the digest, e.g. when upstream re-published the same tag.
func (r Recipe) UpdateTag(image, tag string) (bool, error) {
fullPattern := fmt.Sprintf("%s/compose**yml", r.Dir)
image = formatter.StripTagMeta(image)
@@ -122,6 +131,11 @@ func (r Recipe) UpdateTag(image, tag string) (bool, error) {
log.Debug(i18n.G("considering %s config(s) for tag update", strings.Join(composeFiles, ", ")))
var (
updated bool
newDigest string
)
for _, composeFile := range composeFiles {
opts := stack.Deploy{Composefiles: []string{composeFile}}
@@ -140,7 +154,7 @@ func (r Recipe) UpdateTag(image, tag string) (bool, error) {
continue // may be a compose.$optional.yml file
}
img, _ := reference.ParseNormalizedNamed(service.Image)
img, err := reference.ParseNormalizedNamed(service.Image)
if err != nil {
return false, err
}
@@ -154,30 +168,60 @@ func (r Recipe) UpdateTag(image, tag string) (bool, error) {
continue
}
var composeDigest string
if canonical, ok := img.(reference.Canonical); ok {
composeDigest = canonical.Digest().String()
}
composeImage := formatter.StripTagMeta(reference.Path(img))
log.Debug(i18n.G("parsed %s from %s", composeTag, service.Image))
if image == composeImage {
bytes, err := ioutil.ReadFile(composeFile)
bytes, err := os.ReadFile(composeFile)
if err != nil {
return false, err
}
old := fmt.Sprintf("%s:%s", composeImage, composeTag)
new := fmt.Sprintf("%s:%s", composeImage, tag)
replacedBytes := strings.Replace(string(bytes), old, new, -1)
if composeDigest != "" {
if newDigest == "" {
newDigest, err = getRegistryDigest(img, tag)
if err != nil {
return false, errors.New(i18n.G("unable to retrieve digest for %s: %s", new, err))
}
}
old = fmt.Sprintf("%s@%s", old, composeDigest)
new = fmt.Sprintf("%s@%s", new, newDigest)
}
if old == new {
continue
}
// NOTE: only match the full image reference, so that "foo:1.2" does
// not match "foo:1.2.3" or the "foo:1.2" part of "foo:1.2@sha256:..."
pattern := regexp.MustCompile(regexp.QuoteMeta(old) + `([^@\w.\-]|$)`)
replaced := pattern.ReplaceAllString(string(bytes), new+"${1}")
if replaced == string(bytes) {
continue
}
log.Debug(i18n.G("updating %s to %s in %s", old, new, compose.Filename))
if err := os.WriteFile(compose.Filename, []byte(replacedBytes), 0o764); err != nil {
if err := os.WriteFile(compose.Filename, []byte(replaced), 0o764); err != nil {
return false, err
}
updated = true
}
}
}
return false, nil
return updated, nil
}
// UpdateLabel updates a label in-place on file system local compose files.
+127
View File
@@ -0,0 +1,127 @@
package recipe
import (
"os"
"path"
"testing"
"github.com/distribution/reference"
"github.com/stretchr/testify/assert"
)
const (
oldDigest = "sha256:1111111111111111111111111111111111111111111111111111111111111111"
newDigest = "sha256:2222222222222222222222222222222222222222222222222222222222222222"
)
func setupUpdateTagRecipe(t *testing.T, compose string) (Recipe, string) {
t.Helper()
dir := t.TempDir()
composePath := path.Join(dir, "compose.yml")
envPath := path.Join(dir, ".env.sample")
assert.NoError(t, os.WriteFile(composePath, []byte(compose), 0o644))
assert.NoError(t, os.WriteFile(envPath, []byte("TYPE=foo\n"), 0o644))
return Recipe{
Name: "foo",
Dir: dir,
ComposePath: composePath,
SampleEnvPath: envPath,
}, composePath
}
func stubRegistryDigest(t *testing.T, digest string) *[]string {
t.Helper()
var calls []string
orig := getRegistryDigest
getRegistryDigest = func(img reference.Named, tag string) (string, error) {
calls = append(calls, reference.Path(img)+":"+tag)
return digest, nil
}
t.Cleanup(func() { getRegistryDigest = orig })
return &calls
}
func TestUpdateTag(t *testing.T) {
testcases := []struct {
name string
compose string
image string
tag string
expected string
updated bool
lookups int
}{
{
name: "plain tag",
compose: "version: \"3.8\"\nservices:\n app:\n image: nginx:1.31.7\n",
image: "nginx",
tag: "1.31.8",
expected: "version: \"3.8\"\nservices:\n app:\n image: nginx:1.31.8\n",
updated: true,
},
{
name: "shorter tag does not mangle longer tag",
compose: "version: \"3.8\"\nservices:\n app:\n image: nginx:1.31\n web:\n image: nginx:1.31.7\n",
image: "nginx",
tag: "1.32",
expected: "version: \"3.8\"\nservices:\n app:\n image: nginx:1.32\n web:\n image: nginx:1.32\n",
updated: true,
},
{
name: "digest is updated with tag",
compose: "version: \"3.8\"\nservices:\n app:\n image: nextcloud:34.0.5-fpm@" + oldDigest + "\n",
image: "nextcloud",
tag: "34.0.6-fpm",
expected: "version: \"3.8\"\nservices:\n app:\n image: nextcloud:34.0.6-fpm@" + newDigest + "\n",
updated: true,
lookups: 1,
},
{
name: "digest is refreshed for same tag",
compose: "version: \"3.8\"\nservices:\n app:\n image: \"nextcloud:34.0.5-fpm@" + oldDigest + "\"\n",
image: "nextcloud",
tag: "34.0.5-fpm",
expected: "version: \"3.8\"\nservices:\n app:\n image: \"nextcloud:34.0.5-fpm@" + newDigest + "\"\n",
updated: true,
lookups: 1,
},
{
name: "same tag without digest is a no-op",
compose: "version: \"3.8\"\nservices:\n app:\n image: nextcloud:34.0.5-fpm\n",
image: "nextcloud",
tag: "34.0.5-fpm",
expected: "version: \"3.8\"\nservices:\n app:\n image: nextcloud:34.0.5-fpm\n",
updated: false,
},
{
name: "pinned and unpinned services sharing an image",
compose: "version: \"3.8\"\nservices:\n app:\n image: nextcloud:34.0.5-fpm@" + oldDigest + "\n cron:\n image: nextcloud:34.0.5-fpm\n",
image: "nextcloud",
tag: "34.0.6-fpm",
expected: "version: \"3.8\"\nservices:\n app:\n image: nextcloud:34.0.6-fpm@" + newDigest + "\n cron:\n image: nextcloud:34.0.6-fpm\n",
updated: true,
lookups: 1,
},
}
for _, tc := range testcases {
t.Run(tc.name, func(t *testing.T) {
r, composePath := setupUpdateTagRecipe(t, tc.compose)
calls := stubRegistryDigest(t, newDigest)
updated, err := r.UpdateTag(tc.image, tc.tag)
assert.NoError(t, err)
assert.Equal(t, tc.updated, updated)
assert.Len(t, *calls, tc.lookups)
content, err := os.ReadFile(composePath)
assert.NoError(t, err)
assert.Equal(t, tc.expected, string(content))
})
}
}
+102 -10
View File
@@ -4,6 +4,7 @@ import (
"errors"
"fmt"
"os"
"path"
"slices"
"sort"
"strings"
@@ -20,6 +21,11 @@ import (
"github.com/go-git/go-git/v5/plumbing"
)
// SourceFile is the sidecar file written next to an externally-cloned
// recipe so the canonical "host/path" name can be recovered later (the
// on-disk directory name escapes "/" and "." to "_", which is lossy).
const SourceFile = ".abra-source"
type EnsureContext struct {
Chaos bool
Offline bool
@@ -59,7 +65,7 @@ func (r Recipe) Ensure(ctx EnsureContext) error {
return errors.New(i18n.G(`cannot redeploy previous chaos version (%s), did you mean to use "--chaos"?`))
}
if _, err := r.EnsureVersion(r.EnvVersion); err != nil {
if _, err := r.EnsureVersionCtx(ctx, r.EnvVersion); err != nil {
return err
}
@@ -79,6 +85,12 @@ func (r Recipe) EnsureExists() error {
if err := gitPkg.Clone(r.Dir, r.GitURL); err != nil {
return err
}
if strings.Contains(r.Name, "/") {
sidecar := path.Join(r.Dir, SourceFile)
if err := os.WriteFile(sidecar, []byte(r.Name+"\n"), 0o644); err != nil {
log.Debug(i18n.G("failed to write recipe source sidecar %s: %s", sidecar, err))
}
}
}
if err := gitPkg.EnsureGitRepo(r.Dir); err != nil {
@@ -88,6 +100,18 @@ func (r Recipe) EnsureExists() error {
return nil
}
// ReadRecipeSource returns the canonical name recorded in the .abra-source
// sidecar inside the given recipe directory, or the empty string if no
// sidecar exists. This lets callers recover the unescaped "host/path"
// form for externally-cloned recipes.
func ReadRecipeSource(recipeDir string) string {
data, err := os.ReadFile(path.Join(recipeDir, SourceFile))
if err != nil {
return ""
}
return strings.TrimSpace(string(data))
}
// IsChaosCommit determines if a version sttring is a chaos commit or not.
func (r Recipe) IsChaosCommit(version string) (bool, error) {
isChaosCommit := false
@@ -125,6 +149,13 @@ func (r Recipe) IsChaosCommit(version string) (bool, error) {
// EnsureVersion checks whether a specific version exists for a recipe.
func (r Recipe) EnsureVersion(version string) (bool, error) {
return r.EnsureVersionCtx(EnsureContext{}, version)
}
// EnsureVersionCtx is like EnsureVersion but is aware of the surrounding
// EnsureContext, so that e.g. offline mode can skip network fetches when
// resolving a branch name that isn't already available locally.
func (r Recipe) EnsureVersionCtx(ctx EnsureContext, version string) (bool, error) {
isChaosCommit := false
if err := gitPkg.EnsureGitRepo(r.Dir); err != nil {
@@ -159,30 +190,46 @@ func (r Recipe) EnsureVersion(version string) (bool, error) {
}
var opts *git.CheckoutOptions
var checkedOutRef string
if tagRef.String() == "" {
log.Debug(i18n.G("attempting to checkout '%s' as chaos commit", version))
hash, err := repo.ResolveRevision(plumbing.Revision(version))
if err != nil {
log.Fatal(i18n.G("unable to resolve '%s': %s", version, err))
}
hash, hashErr := repo.ResolveRevision(plumbing.Revision(version))
if hashErr != nil {
branchHash, found, branchErr := resolveBranch(repo, version, ctx.Offline)
if errors.Is(branchErr, errOfflineBranchUnresolved) {
return isChaosCommit, errors.New(i18n.G("'%s' is not available locally and cannot be fetched in offline mode; retry without --offline", version))
}
if branchErr != nil {
return isChaosCommit, branchErr
}
if !found {
return isChaosCommit, errors.New(i18n.G("unable to resolve '%s': %s", version, hashErr))
}
opts = &git.CheckoutOptions{Hash: *hash, Create: false, Force: true}
isChaosCommit = true
opts = &git.CheckoutOptions{Hash: branchHash, Create: false, Force: true}
isChaosCommit = true
checkedOutRef = version
} else {
opts = &git.CheckoutOptions{Hash: *hash, Create: false, Force: true}
isChaosCommit = true
checkedOutRef = hash.String()
}
} else {
opts = &git.CheckoutOptions{Branch: tagRef, Create: false, Force: true}
checkedOutRef = tagRef.Short()
}
worktree, err := repo.Worktree()
if err != nil {
return isChaosCommit, nil
return isChaosCommit, err
}
if err := worktree.Checkout(opts); err != nil {
return isChaosCommit, nil
return isChaosCommit, err
}
log.Debug(i18n.G("successfully checked %s out to %s in %s", r.Name, tagRef.Short(), r.Dir))
log.Debug(i18n.G("successfully checked %s out to %s in %s", r.Name, checkedOutRef, r.Dir))
return isChaosCommit, nil
}
@@ -485,6 +532,51 @@ func (r Recipe) GetRecipeVersions() (RecipeVersions, []string, error) {
return versions, uniqueWarnings, nil
}
// resolveBranch resolves name as a branch on the "origin" remote, fetching
// it on demand into a remote-tracking ref (never into refs/heads/*, which
// would corrupt GetDefaultBranch's local-branch-name heuristic). It returns
// (hash, true, nil) if name is a branch, (_, false, nil) if name is
// confirmed not to be a branch, and a non-nil error for any other failure
// (e.g. network/auth issues) that should not be silently treated as "not a
// branch". When offline is true, no fetch is attempted and only an
// already-fetched remote-tracking ref is consulted.
func resolveBranch(repo *git.Repository, name string, offline bool) (plumbing.Hash, bool, error) {
remoteTrackingRef := plumbing.NewRemoteReferenceName("origin", name)
if !offline {
refSpec := gitCfg.RefSpec(fmt.Sprintf("+refs/heads/%s:%s", name, remoteTrackingRef))
err := repo.Fetch(&git.FetchOptions{
RemoteName: "origin",
RefSpecs: []gitCfg.RefSpec{refSpec},
Force: true,
})
if err != nil {
var noMatch git.NoMatchingRefSpecError
switch {
case errors.Is(err, git.NoErrAlreadyUpToDate):
case errors.As(err, &noMatch):
return plumbing.ZeroHash, false, nil
default:
return plumbing.ZeroHash, false, errors.New(i18n.G("unable to fetch branch '%s': %s", name, err))
}
}
}
ref, err := repo.Reference(remoteTrackingRef, true)
if err != nil {
if offline {
return plumbing.ZeroHash, false, errOfflineBranchUnresolved
}
return plumbing.ZeroHash, false, errors.New(i18n.G("unable to resolve fetched branch '%s': %s", name, err))
}
return ref.Hash(), true, nil
}
// errOfflineBranchUnresolved is returned by resolveBranch when offline mode
// prevents fetching a branch that isn't already available locally.
var errOfflineBranchUnresolved = errors.New(i18n.G("not fetched and cannot resolve offline"))
// Head retrieves latest HEAD metadata.
func (r Recipe) Head() (*plumbing.Reference, error) {
repo, err := git.PlainOpen(r.Dir)
+131
View File
@@ -6,6 +6,8 @@ import (
"testing"
"coopcloud.tech/abra/pkg/test"
"github.com/go-git/go-git/v5"
"github.com/go-git/go-git/v5/plumbing"
"github.com/stretchr/testify/assert"
)
@@ -36,3 +38,132 @@ func TestIsDirty(t *testing.T) {
assert.True(t, dirty)
}
func TestEnsureVersionBranch(t *testing.T) {
test.SetupWithUpstream()
t.Cleanup(func() { test.Teardown() })
upstream, err := git.PlainOpen(test.UpstreamRecipeDir)
if err != nil {
t.Fatal(err)
}
featureRef, err := upstream.Reference(plumbing.NewBranchReferenceName(test.FeatureBranch), true)
if err != nil {
t.Fatal(err)
}
r := Get(test.RecipeName)
isChaosCommit, err := r.EnsureVersion(test.FeatureBranch)
if err != nil {
t.Fatal(err)
}
assert.True(t, isChaosCommit)
head, err := r.Head()
if err != nil {
t.Fatal(err)
}
assert.Equal(t, featureRef.Hash(), head.Hash())
}
func TestEnsureVersionBranchRefetchesOnRedeploy(t *testing.T) {
test.SetupWithUpstream()
t.Cleanup(func() { test.Teardown() })
r := Get(test.RecipeName)
if _, err := r.EnsureVersion(test.FeatureBranch); err != nil {
t.Fatal(err)
}
if err := test.CreateBranchWithCommit(test.UpstreamRecipeDir, test.FeatureBranch, "feature2.txt", "more feature work"); err != nil {
t.Fatal(err)
}
upstream, err := git.PlainOpen(test.UpstreamRecipeDir)
if err != nil {
t.Fatal(err)
}
featureRef, err := upstream.Reference(plumbing.NewBranchReferenceName(test.FeatureBranch), true)
if err != nil {
t.Fatal(err)
}
if _, err := r.EnsureVersion(test.FeatureBranch); err != nil {
t.Fatal(err)
}
head, err := r.Head()
if err != nil {
t.Fatal(err)
}
assert.Equal(t, featureRef.Hash(), head.Hash())
}
func TestEnsureVersionOfflineUnfetchedBranch(t *testing.T) {
test.SetupWithUpstream()
t.Cleanup(func() { test.Teardown() })
r := Get(test.RecipeName)
// SetupWithUpstream creates the branch before cloning, so the clone
// already has it: remove the remote-tracking ref to make it unfetched.
repo, err := git.PlainOpen(r.Dir)
if err != nil {
t.Fatal(err)
}
if err := repo.Storer.RemoveReference(plumbing.NewRemoteReferenceName("origin", test.FeatureBranch)); err != nil {
t.Fatal(err)
}
headBefore, err := r.Head()
if err != nil {
t.Fatal(err)
}
_, err = r.EnsureVersionCtx(EnsureContext{Offline: true}, test.FeatureBranch)
assert.ErrorContains(t, err, "offline")
headAfter, err := r.Head()
if err != nil {
t.Fatal(err)
}
assert.Equal(t, headBefore.Hash(), headAfter.Hash())
}
func TestEnsureVersionUnknownNameErrors(t *testing.T) {
test.SetupWithUpstream()
t.Cleanup(func() { test.Teardown() })
r := Get(test.RecipeName)
_, err := r.EnsureVersion("does-not-exist-anywhere")
assert.Error(t, err)
}
func TestEnsureBranchPinnedRecipe(t *testing.T) {
test.SetupWithUpstream()
t.Cleanup(func() { test.Teardown() })
upstream, err := git.PlainOpen(test.UpstreamRecipeDir)
if err != nil {
t.Fatal(err)
}
featureRef, err := upstream.Reference(plumbing.NewBranchReferenceName(test.FeatureBranch), true)
if err != nil {
t.Fatal(err)
}
r := Get(test.RecipeName + ":" + test.FeatureBranch)
if err := r.Ensure(EnsureContext{}); err != nil {
t.Fatal(err)
}
head, err := r.Head()
if err != nil {
t.Fatal(err)
}
assert.Equal(t, featureRef.Hash(), head.Hash())
}
+103 -18
View File
@@ -8,6 +8,7 @@ import (
"net/url"
"os"
"path"
"regexp"
"sort"
"strconv"
"strings"
@@ -121,7 +122,84 @@ type Features struct {
SSO string `json:"sso"`
}
// scpURLPattern matches SCP-style git URLs like git@host:path or git@host:port/path.
// Captures: 1=host(:port)?, 2=path.
var scpURLPattern = regexp.MustCompile(`^[\w.-]+@([\w.-]+(?::\d+)?):(.+)$`)
// NormalizeRecipeName canonicalizes a recipe identifier to a stable
// "host/path" form (or returns short catalog names unchanged). Accepts:
//
// - https://host/path[.git][:version]
// - http://host/path[.git][:version]
// - ssh://git@host[:port]/path[.git][:version]
// - git@host:path[.git][:version] (SCP-style)
// - host/path[.git][:version] (already canonical)
// - short-name[:version] (catalog recipe, pass through)
//
// The optional trailing :version suffix is preserved verbatim. The .git
// suffix and trailing slashes on the path are stripped so the four URL
// forms of the same repository collapse to one canonical value.
func NormalizeRecipeName(input string) string {
input = strings.TrimSpace(input)
// Split off the version suffix first, but only if it's not part of a
// scheme (https://) or SCP-style git@host: prefix. The simplest way is
// to detect those prefixes and treat the rest as the path-with-version.
var (
body = input
version string
)
switch {
case strings.HasPrefix(input, "https://") || strings.HasPrefix(input, "http://") || strings.HasPrefix(input, "ssh://"):
u, err := url.Parse(input)
if err != nil {
return input
}
host := u.Hostname() // strip port to keep canonical form colon-free
p := strings.TrimPrefix(u.Path, "/")
p, version = splitVersion(p)
body = host + "/" + p
case scpURLPattern.MatchString(input):
m := scpURLPattern.FindStringSubmatch(input)
host := m[1]
if i := strings.Index(host, ":"); i >= 0 {
host = host[:i] // strip port
}
p, v := splitVersion(m[2])
body = host + "/" + p
version = v
default:
body, version = splitVersion(input)
}
body = strings.TrimSuffix(body, "/")
body = strings.TrimSuffix(body, ".git")
if version != "" {
return body + ":" + version
}
return body
}
// splitVersion separates a trailing ":version" suffix from a recipe path.
// It only treats the final colon-separated segment as a version when it is
// non-empty and contains no slash (a slash means the colon belonged to the
// path, e.g. a host:port or scheme separator).
func splitVersion(s string) (string, string) {
idx := strings.LastIndex(s, ":")
if idx < 0 {
return s, ""
}
candidate := s[idx+1:]
if candidate == "" || strings.Contains(candidate, "/") {
return s, ""
}
return s[:idx], candidate
}
func Get(name string) Recipe {
name = NormalizeRecipeName(name)
version := ""
versionRaw := ""
if strings.Contains(name, ":") {
@@ -209,6 +287,16 @@ func (r Recipe) String() string {
return out
}
// ShortName returns the final path segment of the recipe name, i.e. the
// bare recipe name without any "host/org/" prefix carried by externally
// sourced (git URL) recipes. For catalogue recipes it returns Name unchanged.
func (r Recipe) ShortName() string {
if i := strings.LastIndex(r.Name, "/"); i >= 0 {
return r.Name[i+1:]
}
return r.Name
}
func escapeRecipeName(recipeName string) string {
recipeName = strings.ReplaceAll(recipeName, "/", "_")
recipeName = strings.ReplaceAll(recipeName, ".", "_")
@@ -592,15 +680,7 @@ func ReadReposMetadata(debug bool) (RepoCatalogue, error) {
// sortRecipeVersions sorts the recipe semver versions
func sortRecipeVersions(versions RecipeVersions) {
sort.Slice(versions, func(i, j int) bool {
version1, err := tagcmp.Parse(getVersionString(versions[i]))
if err != nil {
panic(err)
}
version2, err := tagcmp.Parse(getVersionString(versions[j]))
if err != nil {
panic(err)
}
return version1.IsLessThan(version2)
return safeCompareVersions(getVersionString(versions[i]), getVersionString(versions[j]))
})
}
@@ -616,18 +696,23 @@ func getVersionString(versionMap map[string]map[string]ServiceMeta) string {
// sortVersionStrings sorts a list of semver version strings
func sortVersionStrings(versions []string) {
sort.Slice(versions, func(i, j int) bool {
version1, err := tagcmp.Parse(versions[i])
if err != nil {
panic(err)
}
version2, err := tagcmp.Parse(versions[j])
if err != nil {
panic(err)
}
return version1.IsLessThan(version2)
return safeCompareVersions(versions[i], versions[j])
})
}
// Invalid versions are sorted as less than valid ones.
func safeCompareVersions(v1, v2 string) bool {
version1, err1 := tagcmp.Parse(v1)
if err1 != nil {
return true
}
version2, err2 := tagcmp.Parse(v2)
if err2 != nil {
return false
}
return version1.IsLessThan(version2)
}
// GetRecipeCatalogueVersions list the recipe versions listed in the recipe catalogue.
func GetRecipeCatalogueVersions(recipeName string, catl RecipeCatalogue) ([]string, error) {
var versions []string
+163
View File
@@ -88,6 +88,88 @@ func TestGet(t *testing.T) {
AbraShPath: path.Join(cfg.GetAbraDir(), "recipes/mygit_org_myorg_cool-recipe/abra.sh"),
},
},
{
name: "https://mygit.org/myorg/cool-recipe",
recipe: Recipe{
Name: "mygit.org/myorg/cool-recipe",
Dir: path.Join(cfg.GetAbraDir(), "/recipes/mygit_org_myorg_cool-recipe"),
GitURL: "https://mygit.org/myorg/cool-recipe.git",
SSHURL: "ssh://git@mygit.org/myorg/cool-recipe.git",
ComposePath: path.Join(cfg.GetAbraDir(), "recipes/mygit_org_myorg_cool-recipe/compose.yml"),
ReadmePath: path.Join(cfg.GetAbraDir(), "recipes/mygit_org_myorg_cool-recipe/README.md"),
SampleEnvPath: path.Join(cfg.GetAbraDir(), "recipes/mygit_org_myorg_cool-recipe/.env.sample"),
AbraShPath: path.Join(cfg.GetAbraDir(), "recipes/mygit_org_myorg_cool-recipe/abra.sh"),
},
},
{
name: "https://mygit.org/myorg/cool-recipe.git",
recipe: Recipe{
Name: "mygit.org/myorg/cool-recipe",
Dir: path.Join(cfg.GetAbraDir(), "/recipes/mygit_org_myorg_cool-recipe"),
GitURL: "https://mygit.org/myorg/cool-recipe.git",
SSHURL: "ssh://git@mygit.org/myorg/cool-recipe.git",
ComposePath: path.Join(cfg.GetAbraDir(), "recipes/mygit_org_myorg_cool-recipe/compose.yml"),
ReadmePath: path.Join(cfg.GetAbraDir(), "recipes/mygit_org_myorg_cool-recipe/README.md"),
SampleEnvPath: path.Join(cfg.GetAbraDir(), "recipes/mygit_org_myorg_cool-recipe/.env.sample"),
AbraShPath: path.Join(cfg.GetAbraDir(), "recipes/mygit_org_myorg_cool-recipe/abra.sh"),
},
},
{
name: "https://mygit.org/myorg/cool-recipe.git:1.2.4",
recipe: Recipe{
Name: "mygit.org/myorg/cool-recipe",
EnvVersion: "1.2.4",
EnvVersionRaw: "1.2.4",
Dir: path.Join(cfg.GetAbraDir(), "/recipes/mygit_org_myorg_cool-recipe"),
GitURL: "https://mygit.org/myorg/cool-recipe.git",
SSHURL: "ssh://git@mygit.org/myorg/cool-recipe.git",
ComposePath: path.Join(cfg.GetAbraDir(), "recipes/mygit_org_myorg_cool-recipe/compose.yml"),
ReadmePath: path.Join(cfg.GetAbraDir(), "recipes/mygit_org_myorg_cool-recipe/README.md"),
SampleEnvPath: path.Join(cfg.GetAbraDir(), "recipes/mygit_org_myorg_cool-recipe/.env.sample"),
AbraShPath: path.Join(cfg.GetAbraDir(), "recipes/mygit_org_myorg_cool-recipe/abra.sh"),
},
},
{
name: "ssh://git@mygit.org/myorg/cool-recipe.git",
recipe: Recipe{
Name: "mygit.org/myorg/cool-recipe",
Dir: path.Join(cfg.GetAbraDir(), "/recipes/mygit_org_myorg_cool-recipe"),
GitURL: "https://mygit.org/myorg/cool-recipe.git",
SSHURL: "ssh://git@mygit.org/myorg/cool-recipe.git",
ComposePath: path.Join(cfg.GetAbraDir(), "recipes/mygit_org_myorg_cool-recipe/compose.yml"),
ReadmePath: path.Join(cfg.GetAbraDir(), "recipes/mygit_org_myorg_cool-recipe/README.md"),
SampleEnvPath: path.Join(cfg.GetAbraDir(), "recipes/mygit_org_myorg_cool-recipe/.env.sample"),
AbraShPath: path.Join(cfg.GetAbraDir(), "recipes/mygit_org_myorg_cool-recipe/abra.sh"),
},
},
{
name: "git@mygit.org:myorg/cool-recipe.git",
recipe: Recipe{
Name: "mygit.org/myorg/cool-recipe",
Dir: path.Join(cfg.GetAbraDir(), "/recipes/mygit_org_myorg_cool-recipe"),
GitURL: "https://mygit.org/myorg/cool-recipe.git",
SSHURL: "ssh://git@mygit.org/myorg/cool-recipe.git",
ComposePath: path.Join(cfg.GetAbraDir(), "recipes/mygit_org_myorg_cool-recipe/compose.yml"),
ReadmePath: path.Join(cfg.GetAbraDir(), "recipes/mygit_org_myorg_cool-recipe/README.md"),
SampleEnvPath: path.Join(cfg.GetAbraDir(), "recipes/mygit_org_myorg_cool-recipe/.env.sample"),
AbraShPath: path.Join(cfg.GetAbraDir(), "recipes/mygit_org_myorg_cool-recipe/abra.sh"),
},
},
{
name: "git@mygit.org:myorg/cool-recipe:1.2.4",
recipe: Recipe{
Name: "mygit.org/myorg/cool-recipe",
EnvVersion: "1.2.4",
EnvVersionRaw: "1.2.4",
Dir: path.Join(cfg.GetAbraDir(), "/recipes/mygit_org_myorg_cool-recipe"),
GitURL: "https://mygit.org/myorg/cool-recipe.git",
SSHURL: "ssh://git@mygit.org/myorg/cool-recipe.git",
ComposePath: path.Join(cfg.GetAbraDir(), "recipes/mygit_org_myorg_cool-recipe/compose.yml"),
ReadmePath: path.Join(cfg.GetAbraDir(), "recipes/mygit_org_myorg_cool-recipe/README.md"),
SampleEnvPath: path.Join(cfg.GetAbraDir(), "recipes/mygit_org_myorg_cool-recipe/.env.sample"),
AbraShPath: path.Join(cfg.GetAbraDir(), "recipes/mygit_org_myorg_cool-recipe/abra.sh"),
},
},
}
for _, tc := range testcases {
@@ -101,6 +183,49 @@ func TestGet(t *testing.T) {
}
}
func TestNormalizeRecipeName(t *testing.T) {
cases := []struct {
in, want string
}{
// catalog short names pass through
{"foo", "foo"},
{"foo:1.2.3", "foo:1.2.3"},
// bare host/path form
{"mygit.org/myorg/cool-recipe", "mygit.org/myorg/cool-recipe"},
{"mygit.org/myorg/cool-recipe.git", "mygit.org/myorg/cool-recipe"},
{"mygit.org/myorg/cool-recipe.git:1.2.3", "mygit.org/myorg/cool-recipe:1.2.3"},
{"mygit.org/myorg/cool-recipe/", "mygit.org/myorg/cool-recipe"},
// https://
{"https://mygit.org/myorg/cool-recipe", "mygit.org/myorg/cool-recipe"},
{"https://mygit.org/myorg/cool-recipe.git", "mygit.org/myorg/cool-recipe"},
{"https://mygit.org/myorg/cool-recipe.git:1.2.3", "mygit.org/myorg/cool-recipe:1.2.3"},
{"http://mygit.org/myorg/cool-recipe", "mygit.org/myorg/cool-recipe"},
// ssh://, with and without port
{"ssh://git@mygit.org/myorg/cool-recipe.git", "mygit.org/myorg/cool-recipe"},
{"ssh://git@mygit.org:2222/myorg/cool-recipe.git", "mygit.org/myorg/cool-recipe"},
// SCP-style git@host:path
{"git@mygit.org:myorg/cool-recipe", "mygit.org/myorg/cool-recipe"},
{"git@mygit.org:myorg/cool-recipe.git", "mygit.org/myorg/cool-recipe"},
{"git@mygit.org:myorg/cool-recipe:1.2.3", "mygit.org/myorg/cool-recipe:1.2.3"},
// whitespace
{" https://mygit.org/myorg/cool-recipe ", "mygit.org/myorg/cool-recipe"},
}
for _, tc := range cases {
t.Run(tc.in, func(t *testing.T) {
got := NormalizeRecipeName(tc.in)
if got != tc.want {
t.Errorf("NormalizeRecipeName(%q) = %q, want %q", tc.in, got, tc.want)
}
})
}
}
func TestGetVersionLabelLocalDoesNotUseTimeoutLabel(t *testing.T) {
test.Setup()
t.Cleanup(func() { test.Teardown() })
@@ -123,3 +248,41 @@ func TestGetVersionLabelLocalDoesNotUseTimeoutLabel(t *testing.T) {
assert.NotEqual(t, label, timeout)
}
}
// This is not a full test of comparison, as that occurs in the external
// tagcmp package. This is a sunny-day integration test
func TestSortRecipeVersionsSortsTwoValidStrings(t *testing.T) {
versions := RecipeVersions{
{"2.0.0": {}}, // Index 0
{"1.0.0": {}}, // Index 1
}
sortRecipeVersions(versions)
assert.Equal(t, "1.0.0", getVersionString(versions[0]), "Version 1.0.0 should be first")
assert.Equal(t, "2.0.0", getVersionString(versions[1]), "Version 2.0.0 should be second")
}
func TestSortRecipeVersionsSortsInvalidVersionStringToTheBottom(t *testing.T) {
versions := RecipeVersions{
{"2.0.0": {}}, // Index 0
{"invalid": {}}, // Index 1
}
sortRecipeVersions(versions)
assert.Equal(t, "invalid", getVersionString(versions[0]), "The invalid version should be first")
assert.Equal(t, "2.0.0", getVersionString(versions[1]), "Version 2.0.0 should be second")
}
func TestSortRecipeVersionsSortsInvalidVersionStringToTheBottomInReverse(t *testing.T) {
versions := RecipeVersions{
{"invalid": {}}, // Index 0
{"2.0.0": {}}, // Index 1
}
sortRecipeVersions(versions)
assert.Equal(t, "invalid", getVersionString(versions[0]), "The invalid version should be first")
assert.Equal(t, "2.0.0", getVersionString(versions[1]), "Version 2.0.0 should be second")
}
+1 -1
View File
@@ -3,7 +3,7 @@ version: "3.8"
services:
app:
image: nginx:1.31.6
image: nginx:1.31.9
secrets:
- test_pass_one
- test_pass_two
+123
View File
@@ -8,6 +8,9 @@ import (
gitPkg "coopcloud.tech/abra/pkg/git"
"git.coopcloud.tech/toolshed/godotenv"
"github.com/go-git/go-git/v5"
"github.com/go-git/go-git/v5/plumbing"
"github.com/go-git/go-git/v5/plumbing/object"
)
var (
@@ -76,6 +79,126 @@ func Setup() {
}
}
// UpstreamRecipeDir is the "remote" repo used by SetupWithUpstream, kept
// outside recipes/ so Teardown's blanket removal of $ABRA_DIR still cleans
// it up.
var UpstreamRecipeDir = os.ExpandEnv("$ABRA_DIR/_upstream/test_recipe")
// FeatureBranch is the non-default branch created by SetupWithUpstream.
const FeatureBranch = "feature"
// SetupWithUpstream is like Setup, but the recipe under $ABRA_DIR/recipes
// is a real clone of a separate "upstream" repo (UpstreamRecipeDir) rather
// than a git-inited copy of the fixture. The upstream repo additionally has
// a FeatureBranch with one extra commit not present on main, so tests can
// exercise resolving/fetching a non-default branch as a recipe :version.
func SetupWithUpstream() {
Teardown()
if err := os.Mkdir(os.ExpandEnv("$ABRA_DIR"), 0764); err != nil {
if !os.IsExist(err) {
log.Fatal(err)
}
}
if err := os.Mkdir(os.ExpandEnv("$ABRA_DIR/servers"), 0700); err != nil {
if !os.IsExist(err) {
log.Fatal(err)
}
}
if err := os.Mkdir(os.ExpandEnv("$ABRA_DIR/recipes"), 0764); err != nil {
if !os.IsExist(err) {
log.Fatal(err)
}
}
serverSrcDir := os.ExpandEnv("$PWD/../../tests/resources/test_server")
serverDestDir := os.ExpandEnv("$ABRA_DIR/servers/test_server")
if err := os.CopyFS(serverDestDir, os.DirFS(serverSrcDir)); err != nil {
log.Fatal(err)
}
if err := os.MkdirAll(path.Dir(UpstreamRecipeDir), 0764); err != nil {
log.Fatal(err)
}
recipeSrcDir := os.ExpandEnv("$PWD/../../tests/resources/test_recipe")
if err := os.CopyFS(UpstreamRecipeDir, os.DirFS(recipeSrcDir)); err != nil {
log.Fatal(err)
}
if err := os.WriteFile(path.Join(UpstreamRecipeDir, ".env.sample"), []byte("TYPE=test_recipe\n"), 0o644); err != nil {
log.Fatal(err)
}
if err := gitPkg.Init(UpstreamRecipeDir, true, "tester", "helo@coopcloud.tech"); err != nil {
log.Fatal(err)
}
if err := CreateBranchWithCommit(UpstreamRecipeDir, FeatureBranch, "feature.txt", "feature work"); err != nil {
log.Fatal(err)
}
recipeDestDir := os.ExpandEnv("$ABRA_DIR/recipes/test_recipe")
if err := gitPkg.Clone(recipeDestDir, UpstreamRecipeDir); err != nil {
log.Fatal(err)
}
}
// CreateBranchWithCommit checks out branchName in repoPath (creating it off
// the current HEAD if it doesn't exist yet, or extending it if it does),
// writes fileName with contents, commits it, then returns to the repo's
// default branch (leaving the new commit reachable only via the branch's
// ref, matching how a contributor would push a feature branch upstream
// without switching the repo's HEAD away from main).
func CreateBranchWithCommit(repoPath, branchName, fileName, contents string) error {
repo, err := git.PlainOpen(repoPath)
if err != nil {
return err
}
head, err := repo.Head()
if err != nil {
return err
}
worktree, err := repo.Worktree()
if err != nil {
return err
}
branchRef := plumbing.NewBranchReferenceName(branchName)
create := true
if _, err := repo.Reference(branchRef, true); err == nil {
create = false
}
checkoutOpts := &git.CheckoutOptions{Branch: branchRef, Create: create}
if create {
checkoutOpts.Hash = head.Hash()
}
if err := worktree.Checkout(checkoutOpts); err != nil {
return err
}
if err := os.WriteFile(path.Join(repoPath, fileName), []byte(contents), 0o644); err != nil {
return err
}
if _, err := worktree.Add(fileName); err != nil {
return err
}
if _, err := worktree.Commit(fileName, &git.CommitOptions{
Author: &object.Signature{Name: "tester", Email: "helo@coopcloud.tech"},
}); err != nil {
return err
}
return worktree.Checkout(&git.CheckoutOptions{Branch: head.Name()})
}
func AddEnv(envKey, envValue string) error {
filePath := os.ExpandEnv(fmt.Sprintf("$ABRA_DIR/servers/%s/%s.env", ServerName, AppName))
+31
View File
@@ -0,0 +1,31 @@
#!/usr/bin/env bash
# This file generates local DNS aliases for the client VM for integration testing
GIT_ROOT=$(git rev-parse --show-toplevel)
echo "generating hosts file..."
hosts_file_path="$GIT_ROOT/tests/resources/extra_hosts"
echo "removing old file..."
rm -f "$hosts_file_path"
# the server domain
server_ip="10.0.0.3"
domain="abra.local"
echo "$server_ip $domain" >> "$hosts_file_path"
# static subdomains used in integration tests
subdomains=("gitea" "zammad" "custom-html" "foo" "foobar")
for subdomain in "${subdomains[@]}"
do
echo "$server_ip $subdomain.$domain" >> "$hosts_file_path"
done
search_dir="$GIT_ROOT/tests/integration"
for entry in "$search_dir"/*.bats
do
file_name=$(basename "${entry}")
# converting file names to testing subdomains
subdomain=$(echo "$file_name" | tr . _)
echo "$server_ip $subdomain.$domain" >> "$hosts_file_path"
done
echo "writing extra hosts success"
+2 -1
View File
@@ -7,4 +7,5 @@
* Integration tests are in `./tests/integration`. Please see [these
docs](https://docs.coopcloud.tech/abra/hack/#integration-tests) for
instructions and tips on how to run them.
instructions and tips on how to run them. If you want to run it locally via nix,
read the [nix testing docs](../nix/hosts/README.md).
+9 -5
View File
@@ -598,8 +598,7 @@ teardown(){
run $ABRA app deploy "$TEST_APP_DOMAIN" --no-input
assert_success
run docker inspect --format='{{range .Config.Env}}{{println .}}{{end}}' \
$(docker ps -f name="$TEST_APP_DOMAIN_$TEST_SERVER" -q)
_inspect_env_test_app
assert_success
assert_output --partial "WITH_COMMENT=foo"
@@ -610,8 +609,7 @@ teardown(){
run $ABRA app deploy "$TEST_APP_DOMAIN" --no-input --force
assert_success
run docker inspect --format='{{range .Config.Env}}{{println .}}{{end}}' \
$(docker ps -f name="$TEST_APP_DOMAIN_$TEST_SERVER" -q)
_inspect_env_test_app
assert_success
refute_output --partial "WITH_COMMENT=foo"
assert_output --partial "WITH_COMMENT=bar"
@@ -629,8 +627,14 @@ teardown(){
run $ABRA app deploy "$TEST_APP_DOMAIN" --no-input
assert_success
run docker service inspect --format '{{ range .Endpoint.Ports }}{{ .Protocol }}={{ .PublishedPort }}{{ end }}' \
if _is_local;
then
run docker service inspect --format "{{ range .Endpoint.Ports }}{{ .Protocol }}={{ .PublishedPort }}{{ end }}" \
"${TEST_APP_DOMAIN//./_}_app"
else
run ssh "$TEST_SERVER" "docker service inspect --format '{{ range .Endpoint.Ports }}{{ .Protocol }}={{ .PublishedPort }}{{ end }}' ${TEST_APP_DOMAIN//./_}_app"
fi
assert_success
assert_output --partial "tcp=1312"
assert_output --partial "udp=1312"
@@ -133,8 +133,7 @@ teardown(){
run $ABRA app deploy "$TEST_APP_DOMAIN" "0.1.0+1.20.0" --no-input
assert_success
run docker inspect --format='{{range .Config.Env}}{{println .}}{{end}}' \
$(docker ps -f name="$TEST_APP_DOMAIN_$TEST_SERVER" -q)
_inspect_env_test_app
assert_success
assert_output --partial "$TEST_RECIIPE:0.1.0+1.20.0"
}
+11 -4
View File
@@ -140,15 +140,22 @@ teardown(){
_undeploy_app
sanitisedDomainName="${TEST_APP_DOMAIN//./_}"
run docker config create "${sanitisedDomainName}_test_conf_v99" "$ABRA_DIR/recipes/abra-test-recipe/abra.sh"
assert_success
remote_ssh_command=""
if _is_local;
then
run docker config create "${sanitisedDomainName}_test_conf_v99" "$ABRA_DIR/recipes/abra-test-recipe/abra.sh"
else
remote_ssh_command="ssh '$TEST_SERVER'"
run cat "$ABRA_DIR/recipes/abra-test-recipe/abra.sh" | ssh "$TEST_SERVER" docker config create "${sanitisedDomainName}_test_conf_v99 -"
assert_success
fi
assert bash -c "docker config ls | grep -q test_conf_v99"
assert bash -c "$remote_ssh_command docker config ls | grep -q test_conf_v99"
run $ABRA app rm "$TEST_APP_DOMAIN" --no-input
assert_success
refute bash -c "docker config ls | grep -q test_conf_v99"
refute bash -c "$remote_ssh_command docker config ls | grep -q test_conf_v99"
}
@test "remove .env file" {
@@ -54,8 +54,7 @@ teardown(){
--no-input
assert_success
run docker inspect --format='{{range .Config.Env}}{{println .}}{{end}}' \
$(docker ps -f name="$TEST_APP_DOMAIN_$TEST_SERVER" -q)
_inspect_env_test_app
assert_success
assert_output --partial "$TEST_RECIIPE:0.1.0+1.20.0"
}
@@ -53,8 +53,7 @@ teardown(){
run $ABRA app upgrade "$TEST_APP_DOMAIN" "0.2.0+1.21.0" --no-input
assert_success
run docker inspect --format='{{range .Config.Env}}{{println .}}{{end}}' \
$(docker ps -f name="$TEST_APP_DOMAIN_$TEST_SERVER" -q)
_inspect_env_test_app
assert_success
assert_output --partial "$TEST_RECIIPE:0.2.0+1.21.0"
assert_output --partial "$TEST_RECIPE:0.2.0+1.21.0"
}
+11 -2
View File
@@ -31,9 +31,18 @@ _ensure_ssh_agent() {
exit 1
fi
export SSH_AUTH_SOCK="$HOME/.ssh/ssh_auth_sock"
if [ ! -S ~/.ssh/ssh_auth_sock ]; then
if [[ ! -v SSH_AUTH_SOCK ]]; then
export SSH_AUTH_SOCK="$HOME/.ssh/ssh_auth_sock"
eval `ssh-agent`
ln -sf "$SSH_AUTH_SOCK" ~/.ssh/ssh_auth_sock
fi
}
_is_local() {
if [[ "$TEST_SERVER" == "default" ]];
then
return 0
else
return 1
fi
}
+30 -6
View File
@@ -1,13 +1,37 @@
#!/usr/bin/env bash
_ensure_swarm() {
if [ "$(docker info | grep Swarm | sed 's/Swarm: //g' | tr -d ' ')" == "inactive" ]; then
run docker swarm init --advertise-addr 127.0.0.1:2377
assert_success
if [ "$(docker info | grep Swarm | sed 's/Swarm: //g' | tr -d ' ')" == "inactive" ]; then
run docker swarm init --advertise-addr 127.0.0.1:2377
assert_success
fi
if ! $(docker network ls | grep -q 'proxy'); then
run docker network create -d overlay proxy
assert_success
if ! docker network ls | grep -q 'proxy'; then
run docker network create -d overlay proxy
assert_success
fi
if ! _is_local;
then
if [ "$(ssh "$TEST_SERVER" docker info | grep Swarm | sed 's/Swarm: //g' | tr -d ' ')" == "inactive" ]; then
run ssh "$TEST_SERVER" docker swarm init --advertise-addr 127.0.0.1:2377
assert_success
fi
if ! ssh "$TEST_SERVER" docker network ls | grep -q 'proxy'; then
run ssh "$TEST_SERVER" docker network create -d overlay proxy
assert_success
fi
fi
}
_inspect_env_test_app() {
if _is_local;
then
containerId="$(docker ps -f name="$TEST_APP_DOMAIN_$TEST_SERVER" -q)"
run docker inspect --format='{{range .Config.Env}}{{println .}}{{end}}' "$containerId"
else
containerId="$(ssh "$TEST_SERVER" docker ps -f name="$TEST_APP_DOMAIN" -q)"
run ssh "$TEST_SERVER" "docker inspect --format='{{range .Config.Env}}{{println .}}{{end}}' '$containerId'"
fi
}
+5 -7
View File
@@ -30,17 +30,15 @@ teardown(){
}
@test "retrieve recipe if missing" {
if [[ -d "$ABRA_DIR/recipe/custom-html" ]]; then
run rm -rf "$ABRA_DIR/recipes/custom-html"
if [[ -d "$ABRA_DIR/recipe/traefik" ]]; then
run rm -rf "$ABRA_DIR/recipes/traefik"
assert_success
assert_not_exists "$ABRA_DIR/recipes/custom-html"
assert_not_exists "$ABRA_DIR/recipes/traefik"
fi
run $ABRA recipe upgrade "custom-html" --no-input
run $ABRA recipe upgrade "traefik" --no-input
assert_success
assert_output --partial 'can upgrade service: app'
assert_exists "$ABRA_DIR/recipes/custom-html"
assert_exists "$ABRA_DIR/recipes/traefik"
}
@test "error if unstaged changes" {
+3 -3
View File
@@ -27,7 +27,7 @@ teardown(){
assert bash -c "docker context ls | grep -q $TEST_SERVER"
server_dir_perms=$(stat -c "%a" "$ABRA_DIR/servers/$TEST_SERVER")
assert_equal $server_dir_perms "700"
assert_equal "$server_dir_perms" "700"
}
@test "error if using name and --local together" {
@@ -43,8 +43,8 @@ teardown(){
assert bash -c "docker context ls | grep -q default"
assert_output --partial 'local server successfully added'
server_dir_perms=$(stat -c "%a" "$ABRA_DIR/servers/$TEST_SERVER")
assert_equal $server_dir_perms "700"
server_dir_perms=$(stat -c "%a" "$ABRA_DIR/servers/default")
assert_equal "$server_dir_perms" "700"
}
@test "create local server fails when no docker swarm" {
+57
View File
@@ -0,0 +1,57 @@
10.0.0.3 abra.local
10.0.0.3 gitea.abra.local
10.0.0.3 zammad.abra.local
10.0.0.3 custom-html.abra.local
10.0.0.3 foo.abra.local
10.0.0.3 foobar.abra.local
10.0.0.3 app_check_bats.abra.local
10.0.0.3 app_cmd_bats.abra.local
10.0.0.3 app_config_bats.abra.local
10.0.0.3 app_cp_bats.abra.local
10.0.0.3 app_deploy_bats.abra.local
10.0.0.3 app_deploy_env_version_bats.abra.local
10.0.0.3 app_deploy_overview_bats.abra.local
10.0.0.3 app_deploy_remote_recipes_bats.abra.local
10.0.0.3 app_env_bats.abra.local
10.0.0.3 app_env_version_bats.abra.local
10.0.0.3 app_labels_bats.abra.local
10.0.0.3 app_list_bats.abra.local
10.0.0.3 app_logs_bats.abra.local
10.0.0.3 app_move_bats.abra.local
10.0.0.3 app_new_bats.abra.local
10.0.0.3 app_ps_bats.abra.local
10.0.0.3 app_remove_bats.abra.local
10.0.0.3 app_restart_bats.abra.local
10.0.0.3 app_rollback_bats.abra.local
10.0.0.3 app_rollback_env_version_bats.abra.local
10.0.0.3 app_rollback_overview_bats.abra.local
10.0.0.3 app_run_bats.abra.local
10.0.0.3 app_secret_bats.abra.local
10.0.0.3 app_secret_env_version_bats.abra.local
10.0.0.3 app_services_bats.abra.local
10.0.0.3 app_undeploy_bats.abra.local
10.0.0.3 app_undeploy_env_version_bats.abra.local
10.0.0.3 app_undeploy_overview_bats.abra.local
10.0.0.3 app_upgrade_bats.abra.local
10.0.0.3 app_upgrade_env_version_bats.abra.local
10.0.0.3 app_upgrade_overview_bats.abra.local
10.0.0.3 app_volume_bats.abra.local
10.0.0.3 autocomplete_bats.abra.local
10.0.0.3 catalogue_bats.abra.local
10.0.0.3 dirs_bats.abra.local
10.0.0.3 install_bats.abra.local
10.0.0.3 recipe_diff_bats.abra.local
10.0.0.3 recipe_fetch_bats.abra.local
10.0.0.3 recipe_lint_bats.abra.local
10.0.0.3 recipe_list_bats.abra.local
10.0.0.3 recipe_new_bats.abra.local
10.0.0.3 recipe_release_bats.abra.local
10.0.0.3 recipe_reset_bats.abra.local
10.0.0.3 recipe_upgrade_bats.abra.local
10.0.0.3 recipe_version_bats.abra.local
10.0.0.3 server_add_bats.abra.local
10.0.0.3 server_list_bats.abra.local
10.0.0.3 server_prune_bats.abra.local
10.0.0.3 server_remove_bats.abra.local
10.0.0.3 upgrade_bats.abra.local
10.0.0.3 version_bats.abra.local
+7
View File
@@ -0,0 +1,7 @@
-----BEGIN OPENSSH PRIVATE KEY-----
b3BlbnNzaC1rZXktdjEAAAAABG5vbmUAAAAEbm9uZQAAAAAAAAABAAAAMwAAAAtzc2gtZW
QyNTUxOQAAACBQ967UhxmFkq71WXvVRkEmtehCGsEnBVppAXxgtHHm7wAAAJhFGwTnRRsE
5wAAAAtzc2gtZWQyNTUxOQAAACBQ967UhxmFkq71WXvVRkEmtehCGsEnBVppAXxgtHHm7w
AAAEAnHEZcf2NeRQEcJC/aVgUWsdOz+vQgEG9ZY+3ErCeaKFD3rtSHGYWSrvVZe9VGQSa1
6EIawScFWmkBfGC0cebvAAAAFWFicmEgaW50ZWdyYXRpb24gdGVzdA==
-----END OPENSSH PRIVATE KEY-----
@@ -0,0 +1 @@
ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIFD3rtSHGYWSrvVZe9VGQSa16EIawScFWmkBfGC0cebv abra integration test
+1
View File
@@ -0,0 +1 @@
/path/to/this/repo
+1 -1
View File
@@ -3,7 +3,7 @@ version: "3.8"
services:
app:
image: nginx:1.31.6
image: nginx:1.31.9
networks:
- proxy
deploy:
+3 -3
View File
@@ -3,16 +3,16 @@ kind: pipeline
name: coopcloud.tech/tagcmp
steps:
- name: gofmt
image: golang:1.27
image: golang:1.26
commands:
- test -z "$(gofmt -l .)"
- name: go build
image: golang:1.27
image: golang:1.26
commands:
- go build -v .
- name: go test
image: golang:1.27
image: golang:1.26
commands:
- go test . -cover
+4
View File
@@ -329,6 +329,10 @@ func (c *Config) unmarshalCore() {
if parsed := parseConfigBool(s.Options.Get(protectHFSKey)); parsed.IsSet() {
c.Core.ProtectHFS = parsed
}
if s.Options.Get(repositoryFormatVersionKey) == string(format.Version_1) {
c.Core.RepositoryFormatVersion = format.Version_1
}
}
func (c *Config) unmarshalUser() {
@@ -139,8 +139,12 @@ func (dw *deltaSelector) fixAndBreakChains(objectsToPack []*ObjectToPack) error
m[otp.Hash()] = otp
}
// visiting holds the objects on the current resolution path, so that a
// delta chain looping back on itself can be detected and broken.
visiting := make(map[plumbing.Hash]bool)
for _, otp := range objectsToPack {
if err := dw.fixAndBreakChainsOne(m, otp); err != nil {
if err := dw.fixAndBreakChainsOne(m, otp, visiting); err != nil {
return err
}
}
@@ -148,7 +152,11 @@ func (dw *deltaSelector) fixAndBreakChains(objectsToPack []*ObjectToPack) error
return nil
}
func (dw *deltaSelector) fixAndBreakChainsOne(objectsToPack map[plumbing.Hash]*ObjectToPack, otp *ObjectToPack) error {
func (dw *deltaSelector) fixAndBreakChainsOne(
objectsToPack map[plumbing.Hash]*ObjectToPack,
otp *ObjectToPack,
visiting map[plumbing.Hash]bool,
) error {
if !otp.Object.Type().IsDelta() {
return nil
}
@@ -174,7 +182,21 @@ func (dw *deltaSelector) fixAndBreakChainsOne(objectsToPack map[plumbing.Hash]*O
return dw.undeltify(otp)
}
if err := dw.fixAndBreakChainsOne(objectsToPack, base); err != nil {
// Mark this object as being resolved before looking at its base, so that
// a delta based on itself is caught by the check below.
h := otp.Hash()
visiting[h] = true
defer delete(visiting, h)
// A delta chain that loops back onto an object we are already resolving
// cannot be written: every delta needs its base written first. Break the
// chain here instead of following the cycle, which would recurse until
// the goroutine stack is exhausted.
if visiting[do.BaseHash()] {
return dw.undeltify(otp)
}
if err := dw.fixAndBreakChainsOne(objectsToPack, base, visiting); err != nil {
return err
}
+34 -2
View File
@@ -474,6 +474,14 @@ type TreeWalker struct {
recursive bool
seen map[plumbing.Hash]bool
// skipPathValidation disables the pathutil.ValidTreePath check in Next.
// It is set by inspection-only callers (e.g. the revlist object walk)
// that never funnel entry names into the filesystem and must enumerate
// trees faithfully, including entries with names upstream Git accepts
// but that are unsafe to materialise (control characters, `.git`-shaped
// names).
skipPathValidation bool
s storer.EncodedObjectStorer
t *Tree
}
@@ -496,10 +504,32 @@ func NewTreeWalker(t *Tree, recursive bool, seen map[plumbing.Hash]bool) *TreeWa
}
}
// SkipPathValidation disables the pathutil.ValidTreePath check performed by
// Next, and must be called before the first Next.
//
// It is for inspection-only walks that never funnel an entry name into the
// filesystem — enumerating which objects exist, rather than materialising
// them — and that must therefore see the tree faithfully, including entries
// whose names upstream Git accepts but that are unsafe to check out. Callers
// that hand the returned name to filesystem or archive output must not use
// it; path safety for those is enforced at the materialisation boundaries
// (FindEntry, TreeEntryFile, archive, FileIter).
func (w *TreeWalker) SkipPathValidation() {
w.skipPathValidation = true
}
// Next returns the next object from the tree. Objects are returned in order
// and subtrees are included. After the last object has been returned further
// calls to Next() will return io.EOF.
//
// Each entry's name is validated against pathutil.ValidTreePath as it
// surfaces, so callers that funnel the returned name into filesystem
// or archive output can trust it is free of `.git`-shaped components,
// HFS+/NTFS variants, Windows reserved names, and traversal sequences.
// A malformed entry stops the walk with the validator's error;
// inspection-only callers that need to enumerate raw, unvalidated
// names can read Tree.Entries directly or call SkipPathValidation.
//
// In the current implementation any objects which cannot be found in the
// underlying repository will be skipped automatically. It is possible that this
// may change in future versions.
@@ -536,8 +566,10 @@ func (w *TreeWalker) Next() (name string, entry TreeEntry, err error) {
continue
}
if err := pathutil.ValidTreePath(entry.Name); err != nil {
return name, entry, err
if !w.skipPathValidation {
if err := pathutil.ValidTreePath(entry.Name); err != nil {
return name, entry, err
}
}
if entry.Mode == filemode.Dir {
+6
View File
@@ -106,6 +106,12 @@ func transformChildren(t *Tree) ([]noder.Noder, error) {
ret := make([]noder.Noder, 0, len(t.Entries))
walker := NewTreeWalker(t, false, nil) // don't recurse
// The diff walk is read-only and never materialises entry names into the
// filesystem, so it must enumerate the tree faithfully — including entries
// with names that are unsafe to check out but valid per upstream Git (e.g.
// control characters). Path safety is enforced at materialisation
// boundaries (FindEntry, TreeEntryFile, archive, FileIter), not here.
walker.SkipPathValidation()
// don't defer walker.Close() for efficiency reasons.
for {
_, e, err = walker.Next()
+7
View File
@@ -187,6 +187,13 @@ func iterateCommitTrees(
cb(tree.Hash)
treeWalker := object.NewTreeWalker(tree, true, seen)
// This walk only enumerates which objects are reachable, to decide what
// to send; it never materialises an entry name into the filesystem. It
// must therefore enumerate the tree faithfully, including entries with
// names that are unsafe to check out but valid per upstream Git (e.g.
// control characters). Path safety is enforced at materialisation
// boundaries (FindEntry, TreeEntryFile, archive, FileIter), not here.
treeWalker.SkipPathValidation()
for {
_, e, err := treeWalker.Next()
+379 -22
View File
@@ -6,8 +6,12 @@ import (
"context"
"crypto/tls"
"crypto/x509"
"errors"
"fmt"
"io"
"net"
"net/http"
"net/netip"
"net/url"
"reflect"
"strconv"
@@ -73,7 +77,7 @@ func advertisedReferences(ctx context.Context, s *session, serviceName string) (
s.endpoint.String(), infoRefsPath, serviceName,
)
req, err := http.NewRequest(http.MethodGet, url, nil)
req, err := newRequest(http.MethodGet, url, nil)
if err != nil {
return nil, err
}
@@ -177,6 +181,24 @@ func NewClient(c *http.Client) transport.Transport {
// and other custom options specific to the client.
// If the net/http client is nil or empty, it will use a net/http client configured
// with http.DefaultTransport.
//
// Credentials this client adds where the transport cannot see them are not
// subject to the redirect stripping described on AuthMethod: a RoundTripper
// injects after the hop is decided, and Client.Jar is consulted after
// CheckRedirect, so a domain cookie still follows a redirect to a subdomain
// the transport counts as another origin. Apply them in an AuthMethod instead
// if that is not wanted. A CheckRedirect hook set on this client runs
// alongside the transport's own, but any header it adds when a redirect
// leaves the repository's origin is discarded the same way.
//
// A RoundTripper is therefore also how to authenticate to a new origin a
// redirect has moved the repository to: match on the request URL and inject
// the credential only for that origin, so it is not sent anywhere else. The
// transport keeps its own CheckRedirect on the copy it makes of this client,
// so the policy and the origin checks still apply.
//
// None of this applies to a RoundTripper that follows redirects itself:
// CheckRedirect is not consulted then, so no stripping happens at all.
func NewClientWithOptions(c *http.Client, opts *ClientOptions) transport.Transport {
if c == nil {
c = &http.Client{
@@ -370,11 +392,20 @@ func (s *session) ModifyEndpointIfRedirect(res *http.Response) error {
if !strings.HasSuffix(r.URL.Path, infoRefsPath) {
return fmt.Errorf("http redirect: target %q does not end with %s", r.URL.Path, infoRefsPath)
}
if r.URL.Scheme != "http" && r.URL.Scheme != "https" {
// A scheme is case-insensitive per RFC 3986, and checkRedirect folds case
// when it reads the same hop, so fold here too rather than reject a
// spelling that check let through. url.Parse and transport.NewEndpoint
// both lowercase what they parse, so only a hand-built URL or Endpoint
// arrives uppercased. The folded form is what gets stored below, so every
// later request built from the endpoint carries the canonical spelling.
scheme := strings.ToLower(r.URL.Scheme)
if scheme != "http" && scheme != "https" {
return fmt.Errorf("http redirect: unsupported scheme %q", r.URL.Scheme)
}
if r.URL.Scheme != s.endpoint.Protocol &&
!(s.endpoint.Protocol == "http" && r.URL.Scheme == "https") {
// schemeUpgrade rather than an inline comparison, so the one cross-scheme
// change go-git permits has a single definition shared with
// credentialsMayFollow.
if !strings.EqualFold(scheme, s.endpoint.Protocol) && !schemeUpgrade(s.endpoint.Protocol, scheme) {
return fmt.Errorf("http redirect: changes scheme from %q to %q", s.endpoint.Protocol, r.URL.Scheme)
}
@@ -384,7 +415,20 @@ func (s *session) ModifyEndpointIfRedirect(res *http.Response) error {
return err
}
if host != s.endpoint.Host || effectivePort(r.URL.Scheme, port) != effectivePort(s.endpoint.Protocol, s.endpoint.Port) {
// The session stores the endpoint and re-applies its credentials on every
// later request, so clear them once the redirect has left the origin they
// were issued for. This uses the same predicate as stripCredentials, so
// both halves share one definition of an origin.
//
// The two are deliberately asymmetric in one respect: stripCredentials is
// sticky over the whole chain, so an origin -> evil -> origin redirect
// leaves the discovery GET's later hops unauthenticated even though the
// chain returned home. This compares the endpoint only against the final
// URL, so the same round trip leaves the session authenticated. That is
// not a leak - the final URL's origin is the original one - but it means
// such a chain can make the discovery GET anonymous while the session's
// POSTs are authenticated, which can surface as a confusing 401.
if !credentialsMayFollow(endpointURL(s.endpoint), r.URL) {
s.endpoint.User = ""
s.endpoint.Password = ""
s.auth = nil
@@ -393,7 +437,7 @@ func (s *session) ModifyEndpointIfRedirect(res *http.Response) error {
s.endpoint.Host = host
s.endpoint.Port = port
s.endpoint.Protocol = r.URL.Scheme
s.endpoint.Protocol = scheme
s.endpoint.Path = r.URL.Path[:len(r.URL.Path)-len(infoRefsPath)]
return nil
}
@@ -419,19 +463,132 @@ func endpointPort(port string) (int, error) {
return parsed, nil
}
func effectivePort(scheme string, port int) int {
if port != 0 {
return port
}
// schemeUpgrade reports whether the scheme transition from one URL to another
// is the one cross-scheme change go-git permits: a plain-http origin upgrading
// to https. It strictly improves confidentiality and is how servers steer
// clients off cleartext.
//
// Permitting it at all is a deliberate deviation: curl, git and the Fetch
// standard all count scheme as part of host identity and drop credentials on
// the upgrade. Auth is sent pre-emptively here, so an http origin has already
// spent its credential in cleartext on the first request and refusing the
// upgrade would break the clone without unspending it. The host is unchanged,
// where an on-path attacker needs a valid certificate to receive anything.
func schemeUpgrade(from, to string) bool {
return strings.EqualFold(from, "http") && strings.EqualFold(to, "https")
}
switch strings.ToLower(scheme) {
case "http":
return 80
case "https":
return 443
default:
return 0
// canonicalHost returns u's hostname in the form origins are compared in.
//
// An address literal is normalised by netip, so the many spellings of one
// address are one origin. Two literals are the same origin exactly when netip
// parses them to the same Addr, which is also how the WHATWG URL Standard
// compares hosts. An IPv4-mapped literal is deliberately not unmapped onto
// the IPv4 it dials: reaching the same endpoint is not the same authority,
// since net/http sends the literal as written in Host and a server may route
// the two spellings to different virtual hosts.
//
// netip also keeps a scope zone verbatim, which is what origin comparison
// needs: net resolves a zone to an interface by exact name, so folding %eth0
// onto %ETH0 would call two hosts the same origin that net dials down
// different interfaces.
//
// A registered name is ASCII-lowercased. That fold is the only liberty taken;
// every other difference in spelling is a different origin.
//
// A trailing root dot is one such difference and is kept, for the same reason
// as the IPv4-mapped literal: curl and the WHATWG URL Standard both hold
// "example.com." and "example.com" to be distinct hosts, and although
// crypto/tls and crypto/x509 fold the dot when they authenticate the peer,
// net/http sends the name as written in Host.
//
// The fold is deliberately ASCII-only. strings.ToLower and strings.EqualFold
// apply Unicode case mapping, which folds U+03C2 onto U+03C3 and so would
// call two hosts the same origin when they resolve to different servers. An
// ASCII-only fold cannot merge two names DNS keeps apart.
//
// No IDNA mapping is applied either, so a unicode hostname is a different
// origin from the punycode encoding of it, and from another Unicode case of
// itself, even though all three reach the same server. Mapping through
// golang.org/x/net/idna would join them, but it can only widen this equality,
// never narrow it, so leaving it out can cost a credential across such a
// redirect and cannot forward one. Against that cost, go-git pins x/net while
// net/http uses the copy vendored into the toolchain: the two are versioned
// separately, so a release that moves the Unicode tables under one and not
// the other would have this merge origins net/http still dials apart. That is
// the failure this comparison exists to prevent, and comparing bytes has no
// such mode.
func canonicalHost(u *url.URL) string {
host := u.Hostname()
if addr, err := netip.ParseAddr(host); err == nil {
return addr.String()
}
b := []byte(host)
for i := range b {
if b[i] >= 'A' && b[i] <= 'Z' {
b[i] += 'a' - 'A'
}
}
return string(b)
}
// effectivePort returns u's port as the connection will use it: the scheme's
// well-known port when the URL does not spell one out, and without leading
// zeroes, so "https://x", "https://x:443" and "https://x:0443" all agree.
func effectivePort(u *url.URL) string {
port := u.Port()
if port == "" {
switch strings.ToLower(u.Scheme) {
case "http":
return "80"
case "https":
return "443"
default:
return ""
}
}
if trimmed := strings.TrimLeft(port, "0"); trimmed != "" {
return trimmed
}
return "0"
}
// credentialsMayFollow reports whether credentials issued for one URL may be
// sent to another.
//
// The relation is deliberately asymmetric: scheme, host and effective port
// must all match, except that a plain http origin may upgrade to https on the
// same host (see schemeUpgrade). That exception is confined to the two
// default ports: 80 to 443 is the upgrade servers actually steer clients
// through, whereas a non-default port carries no such convention, so
// http://host:8080 to https://host:8443 is a move to another origin like any
// other port change.
//
// Host matching is exact. Unlike Go's http.Client, which forwards credentials
// from a host to any subdomain of it, a subdomain is a different origin here —
// matching canonical git and libcurl.
func credentialsMayFollow(from, to *url.URL) bool {
if canonicalHost(from) != canonicalHost(to) {
return false
}
if strings.EqualFold(from.Scheme, to.Scheme) {
return effectivePort(from) == effectivePort(to)
}
return schemeUpgrade(from.Scheme, to.Scheme) &&
effectivePort(from) == "80" && effectivePort(to) == "443"
}
// endpointURL renders an Endpoint's origin as a URL, so that the session's
// credential clearing and the per-hop stripping share one definition of an
// origin and cannot drift apart.
func endpointURL(ep *transport.Endpoint) *url.URL {
host := strings.Trim(ep.Host, "[]")
if ep.Port != 0 {
host = net.JoinHostPort(host, strconv.Itoa(ep.Port))
} else if strings.Contains(host, ":") {
host = "[" + host + "]"
}
return &url.URL{Scheme: ep.Protocol, Host: host}
}
func (c *client) cloneHTTPClient(transport http.RoundTripper) *http.Client {
@@ -448,26 +605,215 @@ func wrapCheckRedirect(policy RedirectPolicy, next func(*http.Request, []*http.R
if err := checkRedirect(req, via, policy); err != nil {
return err
}
// Strip before the caller's hook so it observes what will actually
// be sent, and again afterwards so a hook of the common "preserve
// my headers across redirects" shape - which copies from via[0],
// the original unsanitized request - cannot reinstate them.
// Carrying credentials across an origin boundary is deliberately
// unsupported.
stripCredentials(req, via)
if next != nil {
return next(req, via)
if err := next(req, via); err != nil {
return err
}
}
stripCredentials(req, via)
return nil
}
}
// safeHeaders lists the headers go-git sets itself, none of which can carry a
// caller credential. stripCredentials keeps only these when a redirect leaves
// the credential's origin. Adding a name here makes it forwardable across an
// origin boundary - do not add anything a caller can put a secret in.
//
// This narrows rather than eliminates the exposure: an AuthMethod that writes
// a credential into one of these names directly - for example
// Header.Set("User-Agent", "token "+secret) - still survives a cross-origin
// redirect. Such a value is also sent to the origin and to any proxy in path,
// so it should not be placed there whether or not a redirect follows.
var safeHeaders = map[string]struct{}{
"User-Agent": {},
"Host": {},
"Accept": {},
"Content-Type": {},
"Content-Length": {},
}
func filterHeaders(h http.Header) http.Header {
filtered := make(http.Header)
for key, values := range h {
if _, ok := safeHeaders[http.CanonicalHeaderKey(key)]; ok {
filtered[key] = values
}
}
return filtered
}
// stripCredentials removes credentials from req once the redirect chain has
// left the origin of the original, credential-bearing request.
//
// CheckRedirect is the only hook that runs while a redirected request's
// headers are still mutable: http.Client.Do performs the entire chain
// internally, so anything the transport does after Do returns - including
// ModifyEndpointIfRedirect - is too late for the hops themselves.
//
// Two subtleties:
//
// - net/http rebuilds every redirect request from the original request's
// headers before calling this, so a header removed at one hop reappears
// at the next. The decision is therefore recomputed per hop.
// - The decision is sticky: once the chain has left the origin, credentials
// stay gone even if a later hop returns to it. Stickiness is derived from
// via rather than stored, because this closure is shared across a
// session's requests.
//
// Stripping keeps only the headers go-git sets itself (safeHeaders). An
// allowlist is used rather than a list of credential header names because
// caller credentials arrive under names that cannot be enumerated -
// PRIVATE-TOKEN, X-Api-Key, gateway headers - which is exactly what
// net/http's fixed list of sensitive header names gets wrong. It is also
// immune to header-name canonicalisation: an AuthMethod that writes a raw map
// key is still removed.
func stripCredentials(req *http.Request, via []*http.Request) {
if len(via) == 0 {
return
}
// net/http sets a URL on every request it builds, and req.URL is non-nil
// by construction: checkRedirect dereferences req.URL.Scheme on each path
// that returns nil, so it runs first or not at all. This nil check and
// the two in crossedOrigin are defensive, against a synthetic caller.
// Each treats a URL it cannot read as an origin crossing; removing one
// panics in canonicalHost rather than leaking.
if origin := via[0].URL; origin != nil && !crossedOrigin(origin, req, via) {
return
}
req.Header = filterHeaders(req.Header)
if req.URL != nil {
req.URL.User = nil
}
}
// crossedOrigin reports whether any hop so far, including the pending one, has
// left origin.
func crossedOrigin(origin *url.URL, req *http.Request, via []*http.Request) bool {
if req.URL == nil || !credentialsMayFollow(origin, req.URL) {
return true
}
for _, prev := range via[1:] {
if prev.URL == nil || !credentialsMayFollow(origin, prev.URL) {
return true
}
}
return false
}
// redactedURL returns the string form of u with the userinfo password
// replaced, for use in error messages. (*url.URL).String() renders the
// password verbatim, and request URLs are built from the endpoint, which
// carries whatever credentials the caller put in the clone URL.
func redactedURL(u *url.URL) string {
if u == nil {
return ""
}
if u.User == nil {
return u.String()
}
if _, hasPassword := u.User.Password(); !hasPassword {
return u.String()
}
redacted := *u
redacted.User = url.UserPassword(u.User.Username(), "REDACTED")
return redacted.String()
}
// redactedRawURL is redactedURL for a string that may not parse. Request URLs
// are assembled from Endpoint.String(), which re-emits Endpoint.Path raw, so a
// path holding a stray percent produces a string url.Parse rejects. url.Parse
// reports the input verbatim and applies no redaction of its own; only
// http.Client strips a password, and only from errors it raises itself.
func redactedRawURL(raw string) string {
i := strings.Index(raw, "://")
if i < 0 {
return raw
}
authority := raw[i+3:]
if end := strings.IndexByte(authority, '/'); end >= 0 {
authority = authority[:end]
}
at := strings.LastIndexByte(authority, '@')
if at < 0 {
return raw
}
colon := strings.IndexByte(authority[:at], ':')
if colon < 0 {
// Username only, left alone, as redactedURL leaves it.
return raw
}
return raw[:i+3+colon+1] + "REDACTED" + raw[i+3+at:]
}
// newRequest wraps http.NewRequest so that a URL it cannot parse does not
// reach the caller with the endpoint's credentials still in it.
func newRequest(method, rawURL string, body io.Reader) (*http.Request, error) {
req, err := http.NewRequest(method, rawURL, body)
if err != nil {
var uerr *url.Error
if errors.As(err, &uerr) {
uerr.URL = redactedRawURL(uerr.URL)
}
return nil, err
}
return req, nil
}
func checkRedirect(req *http.Request, via []*http.Request, policy RedirectPolicy) error {
// CheckRedirect is the only hook that runs before the next hop leaves
// the client. ModifyEndpointIfRedirect inspects the chain after
// client.Do has followed all of it, so a hop rejected there has already
// carried the request headers to its server.
//
// The wording matches the message ModifyEndpointIfRedirect produces for
// the same hop, which this check reaches first.
if len(via) != 0 {
// A hop whose scheme cannot be read cannot be shown not to have been
// https, so it is assumed to have been, and a cleartext target is
// rejected. Skipping the comparison instead would let an
// undeterminable hop turn the check off, which is the wrong default
// for a credential control; crossedOrigin fails closed the same way.
// An empty scheme is as unreadable as a nil URL, so both take the
// assumed-https default.
//
// The comparisons fold case because a scheme is case-insensitive per
// RFC 3986. net/url lowercases what it parses, so only a hand-built
// URL reaches here uppercased - the same synthetic caller the nil
// checks guard against - and for that caller "HTTPS" to "http" is
// still a downgrade.
prevScheme := "https"
if prev := via[len(via)-1]; prev.URL != nil && prev.URL.Scheme != "" {
prevScheme = prev.URL.Scheme
}
if strings.EqualFold(prevScheme, "https") && strings.EqualFold(req.URL.Scheme, "http") {
return fmt.Errorf("http redirect: changes scheme from %q to %q: %s",
prevScheme, req.URL.Scheme, redactedURL(req.URL))
}
}
switch policy {
case FollowRedirects:
case NoFollowRedirects:
return fmt.Errorf("http redirect: redirects disabled to %s", req.URL)
return fmt.Errorf("http redirect: redirects disabled to %s", redactedURL(req.URL))
case "", FollowInitialRedirects:
if !isInitialRequest(req) {
return fmt.Errorf("http redirect: redirect on non-initial request to %s", req.URL)
return fmt.Errorf("http redirect: redirect on non-initial request to %s", redactedURL(req.URL))
}
default:
return fmt.Errorf("http redirect: invalid redirect policy %q", policy)
}
if req.URL.Scheme != "http" && req.URL.Scheme != "https" {
// Folded for the same reason as the guard above: a scheme is
// case-insensitive per RFC 3986, so a spelling the downgrade check read
// as https must not be rejected here as a scheme go-git cannot speak.
if !strings.EqualFold(req.URL.Scheme, "http") && !strings.EqualFold(req.URL.Scheme, "https") {
return fmt.Errorf("http redirect: unsupported scheme %q", req.URL.Scheme)
}
if len(via) >= 10 {
@@ -481,6 +827,17 @@ func (*session) Close() error {
}
// AuthMethod is concrete implementation of common.AuthMethod for HTTP services
//
// Headers SetAuth adds are dropped when a redirect leaves the repository's
// origin: only the headers the transport sets itself survive that boundary.
// This applies to non-credential headers too, so an implementation that adds a
// trace or tenant header loses it on such a hop.
//
// That filter matches header names, not values. An implementation that writes
// a credential into a name the transport also uses - User-Agent, Host, Accept,
// Content-Type, Content-Length - has that value carried across the boundary
// with the name. Such a credential is also sent to the origin and to any proxy
// in path, so it should not be placed there whether or not a redirect follows.
type AuthMethod interface {
transport.AuthMethod
SetAuth(r *http.Request)
@@ -598,6 +955,6 @@ func (e *Err) StatusCode() int {
func (e *Err) Error() string {
return fmt.Sprintf("unexpected requesting %q status code: %d",
e.Response.Request.URL, e.Response.StatusCode,
redactedURL(e.Response.Request.URL), e.Response.StatusCode,
)
}
@@ -88,7 +88,7 @@ func (s *rpSession) doRequest(
body = content
}
req, err := http.NewRequest(method, url, body)
req, err := newRequest(method, url, body)
if err != nil {
return nil, plumbing.NewPermanentError(err)
}
+1 -1
View File
@@ -86,7 +86,7 @@ func (s *upSession) doRequest(
body = content
}
req, err := http.NewRequest(method, url, body)
req, err := newRequest(method, url, body)
if err != nil {
return nil, plumbing.NewPermanentError(err)
}
+1
View File
@@ -433,6 +433,7 @@ func dotGitCommonDirectory(fs billy.Filesystem) (commonDir billy.Filesystem, err
if err != nil {
return nil, err
}
defer ioutil.CheckClose(f, &err)
b, err := io.ReadAll(f)
if err != nil {
-24
View File
@@ -1,24 +0,0 @@
# Compiled Object files, Static and Dynamic libs (Shared Objects)
*.o
*.a
*.so
# Folders
_obj
_test
# Architecture specific extensions/prefixes
*.[568vq]
[568vq].out
*.cgo1.go
*.cgo2.c
_cgo_defun.c
_cgo_gotypes.go
_cgo_export.*
_testmain.go
*.exe
*.test
*.prof
-57
View File
@@ -1,57 +0,0 @@
version: 2
builds:
-
id: "cpuid"
binary: cpuid
main: ./cmd/cpuid/main.go
env:
- CGO_ENABLED=0
flags:
- -ldflags=-s -w
goos:
- aix
- linux
- freebsd
- netbsd
- windows
- darwin
goarch:
- 386
- amd64
- arm64
goarm:
- 7
archives:
-
id: cpuid
name_template: "cpuid-{{ .Os }}_{{ .Arch }}{{ if .Arm }}v{{ .Arm }}{{ end }}"
format_overrides:
- goos: windows
format: zip
files:
- LICENSE
checksum:
name_template: 'checksums.txt'
changelog:
sort: asc
filters:
exclude:
- '^doc:'
- '^docs:'
- '^test:'
- '^tests:'
- '^Update\sREADME.md'
nfpms:
-
file_name_template: "cpuid_package_{{ .Os }}_{{ .Arch }}{{ if .Arm }}v{{ .Arm }}{{ end }}"
vendor: Klaus Post
homepage: https://github.com/klauspost/cpuid
maintainer: Klaus Post <klauspost@gmail.com>
description: CPUID Tool
license: BSD 3-Clause
formats:
- deb
- rpm
-35
View File
@@ -1,35 +0,0 @@
Developer Certificate of Origin
Version 1.1
Copyright (C) 2015- Klaus Post & Contributors.
Email: klauspost@gmail.com
Everyone is permitted to copy and distribute verbatim copies of this
license document, but changing it is not allowed.
Developer's Certificate of Origin 1.1
By making a contribution to this project, I certify that:
(a) The contribution was created in whole or in part by me and I
have the right to submit it under the open source license
indicated in the file; or
(b) The contribution is based upon previous work that, to the best
of my knowledge, is covered under an appropriate open source
license and I have the right under that license to submit that
work with modifications, whether created in whole or in part
by me, under the same open source license (unless I am
permitted to submit under a different license), as indicated
in the file; or
(c) The contribution was provided directly to me by some other
person who certified (a), (b) or (c) and I have not modified
it.
(d) I understand and agree that this project and the contribution
are public and that a record of the contribution (including all
personal information I submit with it, including my sign-off) is
maintained indefinitely and may be redistributed consistent with
this project or the open source license(s) involved.
-22
View File
@@ -1,22 +0,0 @@
The MIT License (MIT)
Copyright (c) 2015 Klaus Post
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
-512
View File
@@ -1,512 +0,0 @@
# cpuid
Package cpuid provides information about the CPU running the current program.
CPU features are detected on startup, and kept for fast access through the life of the application.
Currently x86 / x64 (AMD64/i386) and ARM (ARM64) is supported, and no external C (cgo) code is used, which should make the library very easy to use.
You can access the CPU information by accessing the shared CPU variable of the cpuid library.
Package home: https://github.com/klauspost/cpuid
[![PkgGoDev](https://pkg.go.dev/badge/github.com/klauspost/cpuid)](https://pkg.go.dev/github.com/klauspost/cpuid/v2)
[![Go](https://github.com/klauspost/cpuid/actions/workflows/go.yml/badge.svg)](https://github.com/klauspost/cpuid/actions/workflows/go.yml)
## installing
`go get -u github.com/klauspost/cpuid/v2` using modules.
Drop `v2` for others.
Installing binary:
`go install github.com/klauspost/cpuid/v2/cmd/cpuid@latest`
Or download binaries from release page: https://github.com/klauspost/cpuid/releases
### Homebrew
For macOS/Linux users, you can install via [brew](https://brew.sh/)
```sh
$ brew install cpuid
```
## example
```Go
package main
import (
"fmt"
"strings"
. "github.com/klauspost/cpuid/v2"
)
func main() {
// Print basic CPU information:
fmt.Println("Name:", CPU.BrandName)
fmt.Println("PhysicalCores:", CPU.PhysicalCores)
fmt.Println("ThreadsPerCore:", CPU.ThreadsPerCore)
fmt.Println("LogicalCores:", CPU.LogicalCores)
fmt.Println("Family", CPU.Family, "Model:", CPU.Model, "Vendor ID:", CPU.VendorID)
fmt.Println("Features:", strings.Join(CPU.FeatureSet(), ","))
fmt.Println("Cacheline bytes:", CPU.CacheLine)
fmt.Println("L1 Data Cache:", CPU.Cache.L1D, "bytes")
fmt.Println("L1 Instruction Cache:", CPU.Cache.L1I, "bytes")
fmt.Println("L2 Cache:", CPU.Cache.L2, "bytes")
fmt.Println("L3 Cache:", CPU.Cache.L3, "bytes")
fmt.Println("Frequency", CPU.Hz, "hz")
// Test if we have these specific features:
if CPU.Supports(SSE, SSE2) {
fmt.Println("We have Streaming SIMD 2 Extensions")
}
}
```
Sample output:
```
>go run main.go
Name: AMD Ryzen 9 3950X 16-Core Processor
PhysicalCores: 16
ThreadsPerCore: 2
LogicalCores: 32
Family 23 Model: 113 Vendor ID: AMD
Features: ADX,AESNI,AVX,AVX2,BMI1,BMI2,CLMUL,CMOV,CX16,F16C,FMA3,HTT,HYPERVISOR,LZCNT,MMX,MMXEXT,NX,POPCNT,RDRAND,RDSEED,RDTSCP,SHA,SSE,SSE2,SSE3,SSE4,SSE42,SSE4A,SSSE3
Cacheline bytes: 64
L1 Data Cache: 32768 bytes
L1 Instruction Cache: 32768 bytes
L2 Cache: 524288 bytes
L3 Cache: 16777216 bytes
Frequency 0 hz
We have Streaming SIMD 2 Extensions
```
# usage
The `cpuid.CPU` provides access to CPU features. Use `cpuid.CPU.Supports()` to check for CPU features.
A faster `cpuid.CPU.Has()` is provided which will usually be inlined by the gc compiler.
To test a larger number of features, they can be combined using `f := CombineFeatures(CMOV, CMPXCHG8, X87, FXSR, MMX, SYSCALL, SSE, SSE2)`, etc.
This can be using with `cpuid.CPU.HasAll(f)` to quickly test if all features are supported.
Note that for some cpu/os combinations some features will not be detected.
`amd64` has rather good support and should work reliably on all platforms.
Note that hypervisors may not pass through all CPU features through to the guest OS,
so even if your host supports a feature it may not be visible on guests.
## arm64 feature detection
Not all operating systems provide ARM features directly
and there is no safe way to do so for the rest.
Currently `arm64/linux` and `arm64/freebsd` should be quite reliable.
`arm64/darwin` adds features expected from the M1 processor, but a lot remains undetected.
A `DetectARM()` can be used if you are able to control your deployment,
it will detect CPU features, but may crash if the OS doesn't intercept the calls.
A `-cpu.arm` flag for detecting unsafe ARM features can be added. See below.
Note that currently only features are detected on ARM,
no additional information is currently available.
## flags
It is possible to add flags that affects cpu detection.
For this the `Flags()` command is provided.
This must be called *before* `flag.Parse()` AND after the flags have been parsed `Detect()` must be called.
This means that any detection used in `init()` functions will not contain these flags.
Example:
```Go
package main
import (
"flag"
"fmt"
"strings"
"github.com/klauspost/cpuid/v2"
)
func main() {
cpuid.Flags()
flag.Parse()
cpuid.Detect()
// Test if we have these specific features:
if cpuid.CPU.Supports(cpuid.SSE, cpuid.SSE2) {
fmt.Println("We have Streaming SIMD 2 Extensions")
}
}
```
## commandline
Download as binary from: https://github.com/klauspost/cpuid/releases
Install from source:
`go install github.com/klauspost/cpuid/v2/cmd/cpuid@latest`
### Example
```
λ cpuid
Name: AMD Ryzen 9 3950X 16-Core Processor
Vendor String: AuthenticAMD
Vendor ID: AMD
PhysicalCores: 16
Threads Per Core: 2
Logical Cores: 32
CPU Family 23 Model: 113
Features: ADX,AESNI,AVX,AVX2,BMI1,BMI2,CLMUL,CLZERO,CMOV,CMPXCHG8,CPBOOST,CX16,F16C,FMA3,FXSR,FXSROPT,HTT,HYPERVISOR,LAHF,LZCNT,MCAOVERFLOW,MMX,MMXEXT,MOVBE,NX,OSXSAVE,POPCNT,RDRAND,RDSEED,RDTSCP,SCE,SHA,SSE,SSE2,SSE3,SSE4,SSE42,SSE4A,SSSE3,SUCCOR,X87,XSAVE
Microarchitecture level: 3
Cacheline bytes: 64
L1 Instruction Cache: 32768 bytes
L1 Data Cache: 32768 bytes
L2 Cache: 524288 bytes
L3 Cache: 16777216 bytes
```
### JSON Output:
```
λ cpuid --json
{
"BrandName": "AMD Ryzen 9 3950X 16-Core Processor",
"VendorID": 2,
"VendorString": "AuthenticAMD",
"PhysicalCores": 16,
"ThreadsPerCore": 2,
"LogicalCores": 32,
"Family": 23,
"Model": 113,
"CacheLine": 64,
"Hz": 0,
"BoostFreq": 0,
"Cache": {
"L1I": 32768,
"L1D": 32768,
"L2": 524288,
"L3": 16777216
},
"SGX": {
"Available": false,
"LaunchControl": false,
"SGX1Supported": false,
"SGX2Supported": false,
"MaxEnclaveSizeNot64": 0,
"MaxEnclaveSize64": 0,
"EPCSections": null
},
"Features": [
"ADX",
"AESNI",
"AVX",
"AVX2",
"BMI1",
"BMI2",
"CLMUL",
"CLZERO",
"CMOV",
"CMPXCHG8",
"CPBOOST",
"CX16",
"F16C",
"FMA3",
"FXSR",
"FXSROPT",
"HTT",
"HYPERVISOR",
"LAHF",
"LZCNT",
"MCAOVERFLOW",
"MMX",
"MMXEXT",
"MOVBE",
"NX",
"OSXSAVE",
"POPCNT",
"RDRAND",
"RDSEED",
"RDTSCP",
"SCE",
"SHA",
"SSE",
"SSE2",
"SSE3",
"SSE4",
"SSE42",
"SSE4A",
"SSSE3",
"SUCCOR",
"X87",
"XSAVE"
],
"X64Level": 3
}
```
### Check CPU microarch level
```
λ cpuid --check-level=3
2022/03/18 17:04:40 AMD Ryzen 9 3950X 16-Core Processor
2022/03/18 17:04:40 Microarchitecture level 3 is supported. Max level is 3.
Exit Code 0
λ cpuid --check-level=4
2022/03/18 17:06:18 AMD Ryzen 9 3950X 16-Core Processor
2022/03/18 17:06:18 Microarchitecture level 4 not supported. Max level is 3.
Exit Code 1
```
## Available flags
### x86 & amd64
| Feature Flag | Description |
|--------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| ADX | Intel ADX (Multi-Precision Add-Carry Instruction Extensions) |
| AESNI | Advanced Encryption Standard New Instructions |
| AMD3DNOW | AMD 3DNOW |
| AMD3DNOWEXT | AMD 3DNowExt |
| AMXBF16 | Tile computational operations on BFLOAT16 numbers |
| AMXINT8 | Tile computational operations on 8-bit integers |
| AMXFP16 | Tile computational operations on FP16 numbers |
| AMXFP8 | Tile computational operations on FP8 numbers |
| AMXCOMPLEX | Tile computational operations on complex numbers |
| AMXTILE | Tile architecture |
| AMXTF32 | Matrix Multiplication of TF32 Tiles into Packed Single Precision Tile |
| AMXTRANSPOSE | Tile multiply where the first operand is transposed |
| APX_F | Intel APX |
| AVX | AVX functions |
| AVX10 | If set the Intel AVX10 Converged Vector ISA is supported |
| AVX10_128 | If set indicates that AVX10 128-bit vector support is present |
| AVX10_256 | If set indicates that AVX10 256-bit vector support is present |
| AVX10_512 | If set indicates that AVX10 512-bit vector support is present |
| AVX2 | AVX2 functions |
| AVX512BF16 | AVX-512 BFLOAT16 Instructions |
| AVX512BITALG | AVX-512 Bit Algorithms |
| AVX512BW | AVX-512 Byte and Word Instructions |
| AVX512CD | AVX-512 Conflict Detection Instructions |
| AVX512DQ | AVX-512 Doubleword and Quadword Instructions |
| AVX512ER | AVX-512 Exponential and Reciprocal Instructions |
| AVX512F | AVX-512 Foundation |
| AVX512FP16 | AVX-512 FP16 Instructions |
| AVX512IFMA | AVX-512 Integer Fused Multiply-Add Instructions |
| AVX512PF | AVX-512 Prefetch Instructions |
| AVX512VBMI | AVX-512 Vector Bit Manipulation Instructions |
| AVX512VBMI2 | AVX-512 Vector Bit Manipulation Instructions, Version 2 |
| AVX512VL | AVX-512 Vector Length Extensions |
| AVX512VNNI | AVX-512 Vector Neural Network Instructions |
| AVX512VP2INTERSECT | AVX-512 Intersect for D/Q |
| AVX512VPOPCNTDQ | AVX-512 Vector Population Count Doubleword and Quadword |
| AVXIFMA | AVX-IFMA instructions |
| AVXNECONVERT | AVX-NE-CONVERT instructions |
| AVXSLOW | Indicates the CPU performs 2 128 bit operations instead of one |
| AVXVNNI | AVX (VEX encoded) VNNI neural network instructions |
| AVXVNNIINT8 | AVX-VNNI-INT8 instructions |
| AVXVNNIINT16 | AVX-VNNI-INT16 instructions |
| BHI_CTRL | Branch History Injection and Intra-mode Branch Target Injection / CVE-2022-0001, CVE-2022-0002 / INTEL-SA-00598 |
| BMI1 | Bit Manipulation Instruction Set 1 |
| BMI2 | Bit Manipulation Instruction Set 2 |
| CETIBT | Intel CET Indirect Branch Tracking |
| CETSS | Intel CET Shadow Stack |
| CLDEMOTE | Cache Line Demote |
| CLMUL | Carry-less Multiplication |
| CLZERO | CLZERO instruction supported |
| CMOV | i686 CMOV |
| CMPCCXADD | CMPCCXADD instructions |
| CMPSB_SCADBS_SHORT | Fast short CMPSB and SCASB |
| CMPXCHG8 | CMPXCHG8 instruction |
| CPBOOST | Core Performance Boost |
| CPPC | AMD: Collaborative Processor Performance Control |
| CX16 | CMPXCHG16B Instruction |
| EFER_LMSLE_UNS | AMD: =Core::X86::Msr::EFER[LMSLE] is not supported, and MBZ |
| ENQCMD | Enqueue Command |
| ERMS | Enhanced REP MOVSB/STOSB |
| F16C | Half-precision floating-point conversion |
| FLUSH_L1D | Flush L1D cache |
| FMA3 | Intel FMA 3. Does not imply AVX. |
| FMA4 | Bulldozer FMA4 functions |
| FP128 | AMD: When set, the internal FP/SIMD execution datapath is 128-bits wide |
| FP256 | AMD: When set, the internal FP/SIMD execution datapath is 256-bits wide |
| FSRM | Fast Short Rep Mov |
| FXSR | FXSAVE, FXRESTOR instructions, CR4 bit 9 |
| FXSROPT | FXSAVE/FXRSTOR optimizations |
| GFNI | Galois Field New Instructions. May require other features (AVX, AVX512VL,AVX512F) based on usage. |
| HLE | Hardware Lock Elision |
| HRESET | If set CPU supports history reset and the IA32_HRESET_ENABLE MSR |
| HTT | Hyperthreading (enabled) |
| HWA | Hardware assert supported. Indicates support for MSRC001_10 |
| HYBRID_CPU | This part has CPUs of more than one type. |
| HYPERVISOR | This bit has been reserved by Intel & AMD for use by hypervisors |
| IA32_ARCH_CAP | IA32_ARCH_CAPABILITIES MSR (Intel) |
| IA32_CORE_CAP | IA32_CORE_CAPABILITIES MSR |
| IBPB | Indirect Branch Restricted Speculation (IBRS) and Indirect Branch Predictor Barrier (IBPB) |
| IBRS | AMD: Indirect Branch Restricted Speculation |
| IBRS_PREFERRED | AMD: IBRS is preferred over software solution |
| IBRS_PROVIDES_SMP | AMD: IBRS provides Same Mode Protection |
| IBS | Instruction Based Sampling (AMD) |
| IBSBRNTRGT | Instruction Based Sampling Feature (AMD) |
| IBSFETCHSAM | Instruction Based Sampling Feature (AMD) |
| IBSFFV | Instruction Based Sampling Feature (AMD) |
| IBSOPCNT | Instruction Based Sampling Feature (AMD) |
| IBSOPCNTEXT | Instruction Based Sampling Feature (AMD) |
| IBSOPSAM | Instruction Based Sampling Feature (AMD) |
| IBSRDWROPCNT | Instruction Based Sampling Feature (AMD) |
| IBSRIPINVALIDCHK | Instruction Based Sampling Feature (AMD) |
| IBS_FETCH_CTLX | AMD: IBS fetch control extended MSR supported |
| IBS_OPDATA4 | AMD: IBS op data 4 MSR supported |
| IBS_OPFUSE | AMD: Indicates support for IbsOpFuse |
| IBS_PREVENTHOST | Disallowing IBS use by the host supported |
| IBS_ZEN4 | Fetch and Op IBS support IBS extensions added with Zen4 |
| IDPRED_CTRL | IPRED_DIS |
| INT_WBINVD | WBINVD/WBNOINVD are interruptible. |
| INVLPGB | NVLPGB and TLBSYNC instruction supported |
| KEYLOCKER | Key locker |
| KEYLOCKERW | Key locker wide |
| LAHF | LAHF/SAHF in long mode |
| LAM | If set, CPU supports Linear Address Masking |
| LBRVIRT | LBR virtualization |
| LZCNT | LZCNT instruction |
| MCAOVERFLOW | MCA overflow recovery support. |
| MCDT_NO | Processor do not exhibit MXCSR Configuration Dependent Timing behavior and do not need to mitigate it. |
| MCOMMIT | MCOMMIT instruction supported |
| MD_CLEAR | VERW clears CPU buffers |
| MMX | standard MMX |
| MMXEXT | SSE integer functions or AMD MMX ext |
| MOVBE | MOVBE instruction (big-endian) |
| MOVDIR64B | Move 64 Bytes as Direct Store |
| MOVDIRI | Move Doubleword as Direct Store |
| MOVSB_ZL | Fast Zero-Length MOVSB |
| MPX | Intel MPX (Memory Protection Extensions) |
| MOVU | MOVU SSE instructions are more efficient and should be preferred to SSE MOVL/MOVH. MOVUPS is more efficient than MOVLPS/MOVHPS. MOVUPD is more efficient than MOVLPD/MOVHPD |
| MSRIRC | Instruction Retired Counter MSR available |
| MSRLIST | Read/Write List of Model Specific Registers |
| MSR_PAGEFLUSH | Page Flush MSR available |
| NRIPS | Indicates support for NRIP save on VMEXIT |
| NX | NX (No-Execute) bit |
| OSXSAVE | XSAVE enabled by OS |
| PCONFIG | PCONFIG for Intel Multi-Key Total Memory Encryption |
| POPCNT | POPCNT instruction |
| PPIN | AMD: Protected Processor Inventory Number support. Indicates that Protected Processor Inventory Number (PPIN) capability can be enabled |
| PREFETCHI | PREFETCHIT0/1 instructions |
| PSFD | Predictive Store Forward Disable |
| RDPRU | RDPRU instruction supported |
| RDRAND | RDRAND instruction is available |
| RDSEED | RDSEED instruction is available |
| RDTSCP | RDTSCP Instruction |
| RRSBA_CTRL | Restricted RSB Alternate |
| RTM | Restricted Transactional Memory |
| RTM_ALWAYS_ABORT | Indicates that the loaded microcode is forcing RTM abort. |
| SERIALIZE | Serialize Instruction Execution |
| SEV | AMD Secure Encrypted Virtualization supported |
| SEV_64BIT | AMD SEV guest execution only allowed from a 64-bit host |
| SEV_ALTERNATIVE | AMD SEV Alternate Injection supported |
| SEV_DEBUGSWAP | Full debug state swap supported for SEV-ES guests |
| SEV_ES | AMD SEV Encrypted State supported |
| SEV_RESTRICTED | AMD SEV Restricted Injection supported |
| SEV_SNP | AMD SEV Secure Nested Paging supported |
| SGX | Software Guard Extensions |
| SGXLC | Software Guard Extensions Launch Control |
| SGXPQC | Software Guard Extensions 256-bit Encryption |
| SHA | Intel SHA Extensions |
| SME | AMD Secure Memory Encryption supported |
| SME_COHERENT | AMD Hardware cache coherency across encryption domains enforced |
| SM3_X86 | SM3 instructions |
| SM4_X86 | SM4 instructions |
| SPEC_CTRL_SSBD | Speculative Store Bypass Disable |
| SRBDS_CTRL | SRBDS mitigation MSR available |
| SSE | SSE functions |
| SSE2 | P4 SSE functions |
| SSE3 | Prescott SSE3 functions |
| SSE4 | Penryn SSE4.1 functions |
| SSE42 | Nehalem SSE4.2 functions |
| SSE4A | AMD Barcelona microarchitecture SSE4a instructions |
| SSSE3 | Conroe SSSE3 functions |
| STIBP | Single Thread Indirect Branch Predictors |
| STIBP_ALWAYSON | AMD: Single Thread Indirect Branch Prediction Mode has Enhanced Performance and may be left Always On |
| STOSB_SHORT | Fast short STOSB |
| SUCCOR | Software uncorrectable error containment and recovery capability. |
| SVM | AMD Secure Virtual Machine |
| SVMDA | Indicates support for the SVM decode assists. |
| SVMFBASID | SVM, Indicates that TLB flush events, including CR3 writes and CR4.PGE toggles, flush only the current ASID's TLB entries. Also indicates support for the extended VMCBTLB_Control |
| SVML | AMD SVM lock. Indicates support for SVM-Lock. |
| SVMNP | AMD SVM nested paging |
| SVMPF | SVM pause intercept filter. Indicates support for the pause intercept filter |
| SVMPFT | SVM PAUSE filter threshold. Indicates support for the PAUSE filter cycle count threshold |
| SYSCALL | System-Call Extension (SCE): SYSCALL and SYSRET instructions. |
| SYSEE | SYSENTER and SYSEXIT instructions |
| TBM | AMD Trailing Bit Manipulation |
| TDX_GUEST | Intel Trust Domain Extensions Guest |
| TLB_FLUSH_NESTED | AMD: Flushing includes all the nested translations for guest translations |
| TME | Intel Total Memory Encryption. The following MSRs are supported: IA32_TME_CAPABILITY, IA32_TME_ACTIVATE, IA32_TME_EXCLUDE_MASK, and IA32_TME_EXCLUDE_BASE. |
| TOPEXT | TopologyExtensions: topology extensions support. Indicates support for CPUID Fn8000_001D_EAX_x[N:0]-CPUID Fn8000_001E_EDX. |
| TSA_L1_NO | AMD only: Not vulnerable to TSA-L1 |
| TSA_SQ_NO | AMD only: Not vulnerable to TSA-SQ |
| TSA_VERW_CLEAR | AMD: If set, the memory form of the VERW instruction may be used to help mitigate TSA |
| TSCRATEMSR | MSR based TSC rate control. Indicates support for MSR TSC ratio MSRC000_0104 |
| TSXLDTRK | Intel TSX Suspend Load Address Tracking |
| VAES | Vector AES. AVX(512) versions requires additional checks. |
| VMCBCLEAN | VMCB clean bits. Indicates support for VMCB clean bits. |
| VMPL | AMD VM Permission Levels supported |
| VMSA_REGPROT | AMD VMSA Register Protection supported |
| VMX | Virtual Machine Extensions |
| VPCLMULQDQ | Carry-Less Multiplication Quadword. Requires AVX for 3 register versions. |
| VTE | AMD Virtual Transparent Encryption supported |
| WAITPKG | TPAUSE, UMONITOR, UMWAIT |
| WBNOINVD | Write Back and Do Not Invalidate Cache |
| WRMSRNS | Non-Serializing Write to Model Specific Register |
| X87 | FPU |
| XGETBV1 | Supports XGETBV with ECX = 1 |
| XOP | Bulldozer XOP functions |
| XSAVE | XSAVE, XRESTOR, XSETBV, XGETBV |
| XSAVEC | Supports XSAVEC and the compacted form of XRSTOR. |
| XSAVEOPT | XSAVEOPT available |
| XSAVES | Supports XSAVES/XRSTORS and IA32_XSS |
# ARM features:
| Feature Flag | Description |
|--------------|------------------------------------------------------------------|
| AESARM | AES instructions |
| ARMCPUID | Some CPU ID registers readable at user-level |
| ASIMD | Advanced SIMD |
| ASIMDDP | SIMD Dot Product |
| ASIMDHP | Advanced SIMD half-precision floating point |
| ASIMDRDM | Rounding Double Multiply Accumulate/Subtract (SQRDMLAH/SQRDMLSH) |
| ATOMICS | Large System Extensions (LSE) |
| CRC32 | CRC32/CRC32C instructions |
| DCPOP | Data cache clean to Point of Persistence (DC CVAP) |
| EVTSTRM | Generic timer |
| FCMA | Floatin point complex number addition and multiplication |
| FHM | FMLAL and FMLSL instructions |
| FP | Single-precision and double-precision floating point |
| FPHP | Half-precision floating point |
| GPA | Generic Pointer Authentication |
| JSCVT | Javascript-style double->int convert (FJCVTZS) |
| LRCPC | Weaker release consistency (LDAPR, etc) |
| PMULL | Polynomial Multiply instructions (PMULL/PMULL2) |
| RNDR | Random Number instructions |
| TLB | Outer Shareable and TLB range maintenance instructions |
| TS | Flag manipulation instructions |
| SHA1 | SHA-1 instructions (SHA1C, etc) |
| SHA2 | SHA-2 instructions (SHA256H, etc) |
| SHA3 | SHA-3 instructions (EOR3, RAXI, XAR, BCAX) |
| SHA512 | SHA512 instructions |
| SM3 | SM3 instructions |
| SM4 | SM4 instructions |
| SVE | Scalable Vector Extension |
# license
This code is published under an MIT license. See LICENSE file for more information.

Some files were not shown because too many files have changed in this diff Show More