Compare commits

..
26 Commits
Author SHA1 Message Date
devydave bb5158c803 Merge branch 'main' into fix/492 2026-11-01 14:35:14 +01:00
devydave 82a5dc35cc chore: adds missing translation deps 2026-08-14 18:14:16 +02:00
devydave 16a789fd64 chore: update translations 2026-08-14 18:13:52 +02:00
devydave 2733b20d42 docs: explains reading stackname 2026-08-14 18:09:01 +02:00
devydave d01c5e7828 refactor: simplify code 2026-08-14 16:53:44 +02:00
devydave 8eba47a3e1 refactor: adjust replace to replaceall and log file name that will be written 2026-08-14 16:28:44 +02:00
devydave 527813c924 fix: updating labels and tags 2026-08-14 12:46:45 +02:00
devydave 38f7720bd6 fix: prefixes volumes with stack name instead of default fallback for deployments 2026-08-14 11:09:12 +02:00
devydave 8c72890a63 fix: deploy labels nil check 2026-08-14 01:02:28 +02:00
devydave 0e5b42da17 chore: mod vendor 2026-08-13 23:09:45 +02:00
devydave 4c72dcb49c merge 2026-08-13 23:08:36 +02:00
devydave a28cc7afce docs: more formatting and typo 2026-08-13 20:41:04 +02:00
devydave 21dccc895d docs: formatting 2026-08-13 20:38:20 +02:00
devydave bb3c7576f8 docs: add setup documentation 2026-08-13 20:34:14 +02:00
devydave 02cbf7c014 refactor: make path editable without nix and impure 2026-08-13 19:57:51 +02:00
devydave a8c8df8604 fix: remove personal terminal choice 2026-08-12 23:01:12 +02:00
devydave 5766a25144 feat: adds host module 2026-08-12 22:43:16 +02:00
devydave 1ccbb31b70 refactor: simplify config and remove hom-manager dependency 2026-08-07 15:15:04 +02:00
devydave 5e0dac6508 refactor: adjusting hosts generation 2026-08-07 14:11:00 +02:00
devydave 38d13eae73 refactor: better variable name 2026-08-07 13:42:22 +02:00
devydave b5e22790be refactor: better name for inspect function 2026-08-07 13:36:25 +02:00
devydave beee7bd3f2 refactor: only export ssh auth socket when not existing 2026-08-07 11:09:30 +02:00
devydave 0c0e16914a refactor: remove traefik deployment because it is not needed 2026-08-07 10:36:23 +02:00
devydave 9bd8f4962a feat: adds running integrations test suite 2026-08-06 22:00:29 +02:00
devydave 4e42d4fb9e feat: adds local vm setup that can reach each other 2026-07-30 16:53:34 +02:00
decentral1se 94624bb16d WIP: feat: use compose-go
See toolshed/abra#492
2026-04-04 19:39:24 +02:00
364 changed files with 46182 additions and 32111 deletions
-2
View File
@@ -8,5 +8,3 @@
dist/
tests/integration/.bats
tests/resources/path_to_repo
advisories
.devbox
-37
View File
@@ -1,37 +0,0 @@
# AGENTS.md
This is the C4DT fork of [`toolshed/abra`](https://git.coopcloud.tech/toolshed/abra), the Co-op Cloud command-line tool. What differs from upstream is described in [README_C4DT.md](README_C4DT.md); keep that file up to date when adding a change.
## Branches
| Branch | Content |
| --- | --- |
| `main` | Mirror of upstream `toolshed/abra` `main`. No C4DT commits. |
| `c4dt` | The fork's release line: upstream plus all C4DT changes. Release tags are made here. |
| feature branches, e.g. `audit_v1` | Work in progress, merged into `c4dt` when ready. |
- Start new work from `c4dt`, in a feature branch.
- To follow upstream, update `main` from `toolshed/abra`, then merge or rebase `c4dt` onto it. Update the base commit and date in `README.md` and `README_C4DT.md`.
- Changes taken from other forks are cherry-picked with `git cherry-pick -x`, so the original commit stays recorded.
## Tags
Releases are annotated tags on `c4dt`, named `<upstream version>-c4dt.<n>`, e.g. `0.13.0-beta-c4dt.1`:
- `<upstream version>` is the latest upstream tag the fork is based on.
- `<n>` counts the C4DT releases on that base, starting at 1, and restarts at 1 when the upstream version changes.
- Tags are never moved or deleted: users pin them, so a tag must always build the same code.
Build a release with the tag as version, so `abra --version` prints `<tag>-<commit>`:
```bash
go build -ldflags "-X main.Version=<tag> -X main.Commit=$(git rev-parse HEAD)" -o abra ./cmd/abra
```
## Development
- Toolchain: `devbox.json`. Use `devbox run build` and `devbox run test`; the test script sets a temporary `ABRA_DIR`, which the tests require.
- Dependencies are vendored: after changing `go.mod`, run `go mod vendor`.
- `abra audit` is designed in [`doc/DesignAudit.md`](doc/DesignAudit.md), which also lists what later versions will add. Keep the design and the code in sync.
- Advisories for `abra audit` live in a separate repository, [`c4dt/advisories`](https://git.coopcloud.tech/c4dt/advisories), under `advisories/`.
- User-facing strings are wrapped in `i18n.G`.
-4
View File
@@ -5,10 +5,6 @@
[![Go Reference](https://pkg.go.dev/badge/coopcloud.tech/abra.svg)](https://pkg.go.dev/coopcloud.tech/abra)
[![Translation status](https://translate.coopcloud.tech/widget/co-op-cloud/svg-badge.svg)](https://translate.coopcloud.tech/engage/co-op-cloud/)
> [!NOTE]
> This is an LLM-modified version of `abra`, based on upstream [`toolshed/abra`](https://git.coopcloud.tech/toolshed/abra) `main` of 2026-11-18 (`0.13.0-beta` + 163 commits, `9023bb87`).
> See [README_C4DT.md](README_C4DT.md) for the differences.
The Co-op Cloud utility belt 🎩🐇
<a href="https://github.com/egonelbre/gophers"><img align="right" width="150" src="https://github.com/egonelbre/gophers/raw/master/.thumb/sketch/adventure/poking-fire.png"/></a>
-39
View File
@@ -1,39 +0,0 @@
# C4DT changes to `abra`
This fork of [`toolshed/abra`](https://git.coopcloud.tech/toolshed/abra) is maintained by C4DT. The changes were made with the help of an LLM coding assistant.
It is based on upstream `main` at commit `9023bb87` of 2026-11-18, which is 163 commits after the `0.13.0-beta` tag.
Releases are tags on the `c4dt` branch, named `<upstream version>-c4dt.<n>`, e.g. `0.13.0-beta-c4dt.1`. Pin a tag rather than following a branch. See [AGENTS.md](AGENTS.md) for the branch and tag conventions.
## Overview
| Change | Summary |
| --- | --- |
| [Digest pinning](#digest-pinning) | Recipes can pin images to a digest; `abra recipe upgrade` keeps the pin up to date |
| [`abra audit`](#abra-audit) | Checks all deployed apps against reviewed advisories and suggests fixes that are verified to exist |
| [Type remote](#type-remote) | Recipes can come from any git URL, and be pinned to a branch, in `RECIPE=` |
| [Tables without a terminal](#tables-without-a-terminal) | Table output works in pipes and cron jobs instead of failing |
| [Devbox](#devbox) | `devbox.json` with the toolchain, and `build` and `test` scripts |
## Details
### Digest pinning
Some upstream images, such as `nextcloud`, are rebuilt under the same tag, so the tag alone cannot tell a vulnerable image from a fixed one. Recipes can now pin an image to a digest, for example `nextcloud:34.0.5-fpm@sha256:...`. `abra recipe upgrade` keeps the digest in sync: when it moves a pinned image to a new tag, it writes the new tag's digest, and when the current tag was re-published upstream, it warns and offers to update the pin. A pinned digest also makes a failed registry lookup at deploy time visible, instead of Docker silently keeping an old local image. To support this, `pkg/client/registry.go` gained `GetRegistryDigest`, which returns the index digest that Docker resolves `image:tag` to, and now requests OCI image indexes, which many multi-arch images on Docker Hub require. Commit `e5804d22`.
### `abra audit`
`abra audit` checks every deployed app on every server against the reviewed advisories in [`c4dt/advisories`](https://git.coopcloud.tech/c4dt/advisories), which it pulls on every run. It reads the recipe version and the image digests from each service spec, and the image configs (build time, environment such as `PHP_VERSION`, version labels) from the server's Docker daemon, or from the registry for the server's platform. Advisories match by recipe version range, environment version range, image version, build time, and known-good or known-bad digests. Before suggesting a fix, the audit checks that it works: a redeploy is only suggested if the tag's current image is fixed, and a recipe upgrade names the oldest release whose images are fixed. If no release fixes the problem yet, it says so and warns; recipe repositories are only read from their release tags, so local checkouts are never changed. The output is a short numbered table of failing apps followed by a numbered list of fixes, and the exit code is 1 when anything fails. Unlike the rest of abra, the audit needs an explicitly chosen abra directory (`$ABRA_DIR` or `abraDir` in `abra.yaml`) and never falls back to `~/.abra`. The design, including the planned later versions, is in [`doc/DesignAudit.md`](doc/DesignAudit.md). Apps deployed with `--chaos` are recognised: the table shows the deployed commit, a suggested redeploy keeps `--chaos` (without it, the deploy would replace the chaos deployment with the release in the `.env` file), what a `--chaos` redeploy would use (the app's `COMPOSE_FILE` with its overrides, loaded as deploy does) is checked for fixed images before the releases, and each finding has an action that says who has to act: redeploy, upgrade, recipe must update (a fixed image exists in the registry, found by listing the repository's tags and ordering rebuilds such as `v6.1.0-1` or `v6.1.0-fix1` by build time), upstream must update (no fixed image exists), check by hand, or follow advisory. A redeploy is only suggested if it deploys a fixed image, taking digests pinned in the recipe into account. The table shows what is affected (e.g. `nginx 1.31.8`) instead of the advisory ID, which moves to the end of the fix line together with the advisory's CVEs. Commits `e07454ec`, `d699e51c`, `165a81de`, `491ea47f`, `d6912414`, `6fc26afa`, `cef4dc2e`, `527d2a6a`, `82564ded`, `371b5aad`, `d19f174e`.
### Type remote
Recipes no longer have to come from the catalogue. `abra app new` accepts a git URL in any common form (`https://`, `ssh://`, `git@host:path`), which is normalized to a canonical `host/path` name, and the app's `.env` gets a `RECIPE=<host/path>` line so that a later deploy, possibly on another machine, fetches the recipe from the same source. The clone records its source in a `.abra-source` file, and `abra recipe ls` shows these external recipes in a source column. The `:<version>` suffix in `RECIPE=` may now also be a branch name: it is fetched on demand into `refs/remotes/origin/*` and checked out at its tip as a chaos commit, after tags and commit hashes, and fails clearly in offline mode if the branch was never fetched. The deployed version is now stored in a single `.env` field, and upgrade and rollback set the version label themselves, because git-URL recipes often carry a stale label in `compose.yml`. Cherry-picked from `ineiti/abra` branch `type_remote` (commits `73227c9e`, `0a157896`, `0c91585e`, `47d9f732`, `5f80d5e8`); `9170b4d2` fixes one of its tests, which did not test the unfetched case it was named for.
### Tables without a terminal
`formatter.PrintTable` read the width of stdin and failed with `operation not supported by device` when it was not a terminal, so `abra audit` and every other table command failed in a pipe or cron job. It now reads the width of stdout and prints the table without a width limit when there is no terminal. Commit `5c4ffaf2`.
### Devbox
A `devbox.json` pins the toolchain (Go 1.27, make, gettext): `devbox run build` builds abra, and `devbox run test` runs the tests with a temporary `ABRA_DIR`, which the existing tests require. Along with it, the `vendor/` directory was regenerated with `go mod vendor`, because it was out of sync with `go.mod` after an upstream dependency update and the default build failed. A binary built with a plain `go build`, without the `-ldflags` that set the commit, no longer panics: commit hashes are now truncated safely, and the default commit is `unknown-commit`. Commits `4e88ff9d`, `d699e51c`, `907a7eaa`, `2f0f3880`, `836971d6`.
+3 -1
View File
@@ -158,7 +158,9 @@ checkout as-is. Recipe commit hashes are also supported as values for
Detach: false,
SendRegistryAuth: true,
}
compose, err := appPkg.GetAppComposeConfig(app.Name, deployOpts, app.Env)
app.Env["STACK_NAME"] = stackName
compose, err := appPkg.GetAppComposeConfig(composeFiles, app.Env)
if err != nil {
log.Fatal(err)
}
+1 -7
View File
@@ -222,13 +222,7 @@ synchronize your local app environment values with what is deployed live.`),
mergedEnv[k] = v
}
// The recipe version is always carried by TYPE (RECIPE, when present,
// is a bare source pointer). Stamp the deployed version onto TYPE and
// leave RECIPE untouched so the two never drift apart.
if t, ok := mergedEnv["TYPE"]; ok {
name, _, _ := strings.Cut(t, ":")
mergedEnv["TYPE"] = fmt.Sprintf("%s:%s", name, version)
} else {
if !strings.Contains(recipeEnvVar, ":") {
mergedEnv[recipeKey] = fmt.Sprintf("%s:%s", mergedEnv[recipeKey], version)
}
+3 -3
View File
@@ -13,7 +13,7 @@ import (
"coopcloud.tech/abra/pkg/i18n"
"coopcloud.tech/abra/pkg/log"
"coopcloud.tech/abra/pkg/upstream/convert"
composetypes "github.com/docker/cli/cli/compose/types"
composeGoTypes "github.com/compose-spec/compose-go/v2/types"
"github.com/docker/docker/api/types"
"github.com/docker/docker/api/types/filters"
dockerClient "github.com/docker/docker/client"
@@ -80,13 +80,13 @@ var AppLabelsCommand = &cobra.Command{
rows = append(rows, []string{i18n.G("RECIPE LABELS"), "---"})
config, err := app.Recipe.GetComposeConfig(app.Env)
config, err := app.Recipe.GetComposeConfig()
if err != nil {
log.Fatal(err)
}
var localLabelKeys []string
var appServiceConfig composetypes.ServiceConfig
var appServiceConfig composeGoTypes.ServiceConfig
for _, service := range config.Services {
if service.Name == "app" {
appServiceConfig = service
+2 -2
View File
@@ -262,8 +262,8 @@ func getAppResources(cl *dockerclient.Client, app app.App) (*AppResources, error
return nil, err
}
opts := stack.Deploy{Composefiles: composeFiles, Namespace: app.StackName()}
compose, err := appPkg.GetAppComposeConfig(app.Name, opts, app.Env)
app.Env["STACK_NAME"] = app.StackName()
compose, err := appPkg.GetAppComposeConfig(composeFiles, app.Env)
if err != nil {
return nil, err
}
+1 -17
View File
@@ -32,18 +32,6 @@ deploy <domain>" to do so.
You can see what recipes are available (i.e. values for the [recipe] argument)
by running "abra recipe ls".
In addition to short catalogue names, [recipe] also accepts arbitrary git
URLs to use a recipe from outside the catalogue (e.g. a fork or work in
progress). Any of these forms is accepted:
abra app new git.example.com/user/recipe
abra app new https://git.example.com/user/recipe
abra app new git@git.example.com:user/recipe
In that case a RECIPE=<canonical name> line is written to the app's .env
file so a subsequent "abra app deploy" (on this or another machine) will
re-fetch the recipe from the same git source.
Recipe commit hashes are supported values for "[version]".
Passing the "--secrets/-S" flag will automatically generate secrets for your
@@ -307,7 +295,7 @@ func ensureDomainFlag(recipe recipePkg.Recipe, server string) error {
if appDomain == "" && !internal.NoInput {
prompt := &survey.Input{
Message: i18n.G("Specify app domain"),
Default: fmt.Sprintf("%s.%s", recipe.ShortName(), server),
Default: fmt.Sprintf("%s.%s", recipe.Name, server),
}
if err := survey.AskOne(prompt, &appDomain); err != nil {
return err
@@ -318,10 +306,6 @@ func ensureDomainFlag(recipe recipePkg.Recipe, server string) error {
return errors.New(i18n.G("no domain provided"))
}
if strings.ContainsAny(appDomain, "/\\") {
return errors.New(i18n.G("invalid domain '%s': must not contain '/' or '\\'", appDomain))
}
return nil
}
+7 -12
View File
@@ -4,7 +4,8 @@ import (
"context"
"encoding/json"
"fmt"
"sort"
"maps"
"slices"
"strings"
"coopcloud.tech/abra/cli/internal"
@@ -87,26 +88,20 @@ func showPSOutput(app appPkg.App, cl *dockerClient.Client, deployedVersion, chao
return
}
deployOpts := stack.Deploy{
Composefiles: composeFiles,
Namespace: app.StackName(),
Prune: false,
ResolveImage: stack.ResolveImageAlways,
}
compose, err := appPkg.GetAppComposeConfig(app.Name, deployOpts, app.Env)
app.Env["STACK_NAME"] = app.StackName()
compose, err := appPkg.GetAppComposeConfig(composeFiles, app.Env)
if err != nil {
log.Fatal(err)
return
}
services := compose.Services
sort.Slice(services, func(i, j int) bool {
return services[i].Name < services[j].Name
})
var rows [][]string
allContainerStats := make(map[string]map[string]string)
for _, service := range services {
for _, serviceName := range slices.Sorted(maps.Keys(compose.Services)) {
service := services[serviceName]
filters := filters.NewArgs()
filters.Add("name", fmt.Sprintf("^%s_%s", app.StackName(), service.Name))
+2 -7
View File
@@ -174,7 +174,8 @@ beforehand. See "abra app backup" for more.`),
SendRegistryAuth: true,
}
compose, err := appPkg.GetAppComposeConfig(app.Name, deployOpts, app.Env)
app.Env["STACK_NAME"] = stackName
compose, err := appPkg.GetAppComposeConfig(composeFiles, app.Env)
if err != nil {
log.Fatal(err)
}
@@ -188,12 +189,6 @@ beforehand. See "abra app backup" for more.`),
appPkg.SetChaosVersionLabel(compose, stackName, chosenDowngrade)
}
// NOTE: stamp the deployed version label ourselves rather than relying
// on the value baked into the recipe's compose.yml. Git-URL/WIP recipes
// often carry a stale or missing label, which would otherwise leave the
// live deployment reporting the wrong version after a rollback.
appPkg.SetVersionLabel(compose, stackName, chosenDowngrade)
// Gather secrets
secretInfo, err := deploy.GatherSecretsForDeploy(cl, app, internal.ShowUnchanged)
if err != nil {
+2 -2
View File
@@ -89,8 +89,8 @@ Passing "--prune/-p" does not remove those volumes.`),
log.Fatal(err)
}
opts := stack.Deploy{Composefiles: composeFiles, Namespace: stackName}
compose, err := appPkg.GetAppComposeConfig(app.Name, opts, app.Env)
app.Env["STACK_NAME"] = stackName
compose, err := appPkg.GetAppComposeConfig(composeFiles, app.Env)
if err != nil {
log.Fatal(err)
}
+6 -25
View File
@@ -186,7 +186,8 @@ beforehand. See "abra app backup" for more.`),
SendRegistryAuth: true,
}
compose, err := appPkg.GetAppComposeConfig(app.Name, deployOpts, app.Env)
app.Env["STACK_NAME"] = stackName
compose, err := appPkg.GetAppComposeConfig(composeFiles, app.Env)
if err != nil {
log.Fatal(err)
}
@@ -200,12 +201,6 @@ beforehand. See "abra app backup" for more.`),
appPkg.SetChaosVersionLabel(compose, stackName, chosenUpgrade)
}
// NOTE: stamp the deployed version label ourselves rather than relying
// on the value baked into the recipe's compose.yml. Git-URL/WIP recipes
// often carry a stale or missing label, which would otherwise leave the
// live deployment reporting the wrong version after an upgrade.
appPkg.SetVersionLabel(compose, stackName, chosenUpgrade)
envVars, err := appPkg.CheckEnv(app)
if err != nil {
log.Fatal(err)
@@ -352,17 +347,9 @@ func getReleaseNotes(
return errors.New(i18n.G("parsing chosen upgrade version failed: %s", err))
}
// The deployed version can be "unknown" (e.g. deployed without a version
// label, as is common for git-URL/work-in-progress recipes). In that case
// there is no lower bound to filter by, so we gather notes for every
// version below the chosen one instead of failing.
var parsedDeployedVersion tagcmp.Tag
haveDeployedVersion := deployMeta.Version != config.UNKNOWN_DEFAULT
if haveDeployedVersion {
parsedDeployedVersion, err = tagcmp.Parse(deployMeta.Version)
if err != nil {
return errors.New(i18n.G("parsing deployment version failed: %s", err))
}
parsedDeployedVersion, err := tagcmp.Parse(deployMeta.Version)
if err != nil {
return errors.New(i18n.G("parsing deployment version failed: %s", err))
}
for _, version := range internal.SortVersionsDesc(versions) {
@@ -371,7 +358,7 @@ func getReleaseNotes(
return errors.New(i18n.G("parsing recipe version failed: %s", err))
}
if (!haveDeployedVersion || parsedVersion.IsGreaterThan(parsedDeployedVersion)) &&
if parsedVersion.IsGreaterThan(parsedDeployedVersion) &&
parsedVersion.IsLessThan(parsedChosenUpgrade) {
note, err := app.Recipe.GetReleaseNotes(version, app.Domain)
if err != nil {
@@ -434,12 +421,6 @@ func validateUpgradeVersionArg(
return errors.New(i18n.G("'%s' is not a known version for %s", specificVersion, app.Recipe.Name))
}
// Without a known deployed version we cannot tell whether the requested
// version is an upgrade; trust the explicit choice rather than failing.
if deployMeta.Version == config.UNKNOWN_DEFAULT {
return nil
}
parsedDeployedVersion, err := tagcmp.Parse(deployMeta.Version)
if err != nil {
return errors.New(i18n.G("'%s' is not a known version", deployMeta.Version))
-186
View File
@@ -1,186 +0,0 @@
// Package audit implements the `abra audit` command.
package audit
import (
"context"
"fmt"
"os"
"path/filepath"
"sort"
"strings"
"coopcloud.tech/abra/cli/internal"
appPkg "coopcloud.tech/abra/pkg/app"
auditPkg "coopcloud.tech/abra/pkg/audit"
"coopcloud.tech/abra/pkg/config"
"coopcloud.tech/abra/pkg/i18n"
"coopcloud.tech/abra/pkg/log"
"coopcloud.tech/abra/pkg/recipe"
"github.com/spf13/cobra"
)
// AuditCommand audits all deployed apps on all servers.
var AuditCommand = &cobra.Command{
// translators: `abra audit` command
Use: i18n.G("audit [flags]"),
// translators: Short description for `abra audit` command
Short: i18n.G("Audit deployed apps against known advisories"),
Long: i18n.G(`Check every deployed app on every server against a curated list of
advisories, and suggest how to fix what fails.
The advisories are pulled from %s into $ABRA_DIR/advisories
on every run. If the repository cannot be reached, the local copy is used.
Only published advisories are applied; use --include-drafts to also apply
drafts, e.g. to try a new advisory before publishing it.
The audit needs an explicitly chosen abra directory: set $ABRA_DIR, or set
abraDir in an abra.yaml file in the current directory or one of its parents.
Only failing apps are listed. The exit code is 0 when nothing fails, and 1
otherwise.`, auditPkg.DefaultAdvisoriesURL),
Example: i18n.G(` # audit all apps on all servers
ABRA_DIR=~/.abra abra audit
# also apply advisories that are still drafts
abra audit --include-drafts`),
Args: cobra.NoArgs,
Run: func(cmd *cobra.Command, args []string) {
os.Exit(run())
},
}
func run() int {
abraDir := config.ABRA_DIR_EXPLICIT
if abraDir == "" {
log.Fatal(i18n.G("abra audit needs an explicitly chosen abra directory: set $ABRA_DIR, or abraDir in an abra.yaml file"))
}
advDir := filepath.Join(abraDir, "advisories")
if err := auditPkg.SyncAdvisories(advDir, auditPkg.DefaultAdvisoriesURL, internal.Offline); err != nil {
log.Fatal(err)
}
advisories, warnings, err := auditPkg.LoadAdvisories(filepath.Join(advDir, "advisories"))
if err != nil {
log.Fatal(err)
}
for _, w := range warnings {
log.Warn(w)
}
evalOpts := auditPkg.EvaluateOptions{IncludeDrafts: includeDrafts}
var used, drafts int
for _, a := range advisories {
if evalOpts.Applies(a) {
used++
if a.Status == auditPkg.StatusDraft {
drafts++
}
}
}
log.Debug(i18n.G("loaded %d advisories, using %d", len(advisories), used))
if includeDrafts {
log.Info(i18n.G("including %d draft advisories, which have not been reviewed yet", drafts))
}
if entries, err := os.ReadDir(config.SERVERS_DIR); err == nil && len(entries) == 0 {
log.Info(i18n.G("no apps found in %s", config.SERVERS_DIR))
return 0
}
appFiles, err := appPkg.LoadAppFiles("") // "" means all servers
if err != nil {
log.Fatal(err)
}
apps, err := appPkg.GetApps(appFiles, "")
if err != nil {
log.Fatal(err)
}
if len(apps) == 0 {
log.Info(i18n.G("no apps found in %s", config.SERVERS_DIR))
return 0
}
states, serverErrs := auditPkg.CollectState(apps, auditPkg.CollectOptions{Offline: internal.Offline})
findings := auditPkg.Evaluate(states, advisories, evalOpts)
if !internal.Offline {
var recipes []recipe.Recipe
for _, app := range apps {
recipes = append(recipes, app.Recipe)
}
byName := make(map[string]appPkg.App)
for _, app := range apps {
byName[app.Name] = app
}
checker := auditPkg.FixChecker{
Fetch: auditPkg.RegistryFetch,
Tags: auditPkg.RegistryTags,
Recipes: auditPkg.NewGitRecipes(recipes, internal.Offline),
Checkout: func(name string) ([]string, error) {
return auditPkg.CheckoutImages(byName[name])
},
}
checker.CheckFixes(context.Background(), findings)
}
for _, f := range findings {
for _, r := range f.Reasons {
log.Debug(fmt.Sprintf("%s %s: %s", f.App, f.Advisory.ID, r))
}
}
sort.SliceStable(findings, func(i, j int) bool {
if findings[i].Server != findings[j].Server {
return findings[i].Server < findings[j].Server
}
return findings[i].App < findings[j].App
})
if len(findings) > 0 || len(serverErrs) > 0 {
if err := printReport(findings, serverErrs); err != nil {
log.Fatal(err)
}
}
servers := make(map[string]bool)
for _, st := range states {
servers[st.Server] = true
}
log.Info(i18n.G(
"audit done: deployed apps: %d, servers: %d, advisories: %d, findings: %d, unreachable servers: %d",
len(states), len(servers), used, len(findings), len(serverErrs),
))
// findings that the operator cannot fix alone, by who has to act
waiting := make(map[string][]string)
for _, f := range findings {
switch f.Action {
case auditPkg.ActionRecipe, auditPkg.ActionUpstream, auditPkg.ActionCheck:
waiting[f.Action] = append(waiting[f.Action], fmt.Sprintf("%s (%s)", f.App, auditPkg.ShortRecipeName(f.Recipe)))
}
}
for _, action := range []string{auditPkg.ActionUpstream, auditPkg.ActionRecipe, auditPkg.ActionCheck} {
if apps := waiting[action]; len(apps) > 0 {
log.Warn(fmt.Sprintf("%s: %s", actionLabel(action), strings.Join(apps, ", ")))
}
}
if len(findings) > 0 || len(serverErrs) > 0 {
return 1
}
return 0
}
var includeDrafts bool
func init() {
AuditCommand.Flags().BoolVar(
&includeDrafts,
i18n.G("include-drafts"),
false,
i18n.G("also apply draft advisories, to try them before publishing"),
)
}
-194
View File
@@ -1,194 +0,0 @@
package audit
import (
"fmt"
"os"
"strconv"
"strings"
auditPkg "coopcloud.tech/abra/pkg/audit"
"coopcloud.tech/abra/pkg/formatter"
"coopcloud.tech/abra/pkg/i18n"
"golang.org/x/term"
)
// printReport prints a narrow overview table, followed by a numbered list
// with the fix for each row. Long texts do not fit in a table, so they go in
// the list, wrapped to the terminal width.
func printReport(findings []auditPkg.Finding, serverErrs []auditPkg.ServerError) error {
servers := make(map[string]bool)
for _, f := range findings {
servers[f.Server] = true
}
showServer := len(servers) > 1
if len(findings) > 0 {
table, err := formatter.CreateTable()
if err != nil {
return err
}
headers := []string{"#"}
if showServer {
headers = append(headers, i18n.G("SERVER"))
}
headers = append(headers, i18n.G("APP"), i18n.G("RECIPE"), i18n.G("VERSION"), i18n.G("AFFECTED"), i18n.G("SEVERITY"), i18n.G("ACTION"))
table.Headers(headers...)
for i, f := range findings {
row := []string{strconv.Itoa(i + 1)}
if showServer {
row = append(row, f.Server)
}
row = append(row, f.App, auditPkg.ShortRecipeName(f.Recipe), displayVersion(f), f.Affected, f.Advisory.Severity, displayAction(f))
table.Row(row...)
}
if err := formatter.PrintTable(table); err != nil {
return err
}
fmt.Println()
width := termWidth()
for i, f := range findings {
prefix := fmt.Sprintf("%d. ", i+1)
fmt.Println(wrap(prefix+f.App+": "+fixLine(f), width, strings.Repeat(" ", len(prefix))))
}
}
if len(serverErrs) > 0 {
if len(findings) > 0 {
fmt.Println()
}
width := termWidth()
for _, e := range serverErrs {
fmt.Println(wrap(i18n.G("unreachable: %s: %s", e.Server, shortError(e.Err)), width, " "))
}
}
return nil
}
// actionLabel translates a finding action.
func actionLabel(action string) string {
switch action {
case auditPkg.ActionRedeploy:
return i18n.G("redeploy")
case auditPkg.ActionUpgrade:
return i18n.G("upgrade")
case auditPkg.ActionRecipe:
return i18n.G("recipe must update")
case auditPkg.ActionUpstream:
return i18n.G("upstream must update")
case auditPkg.ActionCheck:
return i18n.G("check by hand")
case auditPkg.ActionAdvisory:
return i18n.G("follow advisory")
}
return action
}
// displayAction is the action shown in the table. A finding whose images
// could not all be checked (status unknown) needs a look first.
func displayAction(f auditPkg.Finding) string {
if f.Status == auditPkg.FindingUnknown {
return actionLabel(auditPkg.ActionCheck)
}
return actionLabel(f.Action)
}
// fixLine starts with the action, so that the first words say who acts,
// and ends with the references to look the advisory up.
func fixLine(f auditPkg.Finding) string {
line := actionLabel(f.Action) + ". " + f.Fix
if f.Status == auditPkg.FindingUnknown {
line = i18n.G("%s. Not every image could be checked (see abra audit -d). If it is affected: %s. %s",
actionLabel(auditPkg.ActionCheck), actionLabel(f.Action), f.Fix)
}
return line + " (" + advisoryRefs(f.Advisory) + ")"
}
// advisoryRefs names the advisory and its CVEs: "CCA-2026-0002,
// CVE-2026-123007 and 4 more". Drafts are marked.
func advisoryRefs(adv *auditPkg.Advisory) string {
refs := adv.ID
if adv.Status == auditPkg.StatusDraft {
refs += " " + i18n.G("(draft)")
}
var cves []string
for _, r := range adv.Refs {
if strings.HasPrefix(r, "CVE-") {
cves = append(cves, strings.Fields(r)[0])
}
}
switch len(cves) {
case 0:
case 1:
refs += ", " + cves[0]
default:
refs += ", " + i18n.G("%s and %d more", cves[0], len(cves)-1)
}
return refs
}
// displayVersion shows what is deployed: the recipe version, or for a
// --chaos deployment the commit it was deployed from, as abra app ls does.
func displayVersion(f auditPkg.Finding) string {
if !f.Chaos {
return f.Version
}
if f.ChaosVersion == "" {
return f.Version + " " + i18n.G("(chaos)")
}
return f.ChaosVersion + " " + i18n.G("(chaos)")
}
// termWidth returns the terminal width, or 0 (no wrapping) when stdout is not
// a terminal or ABRA_CI is set.
func termWidth() int {
if os.Getenv("ABRA_CI") == "1" {
return 0
}
w, _, err := term.GetSize(int(os.Stdout.Fd()))
if err != nil || w < 40 {
return 0
}
return w - 1
}
// wrap breaks text into lines of at most width characters at spaces,
// indenting continuation lines. A width of 0 disables wrapping.
func wrap(text string, width int, indent string) string {
if width <= 0 || len(text) <= width {
return text
}
var lines []string
line := ""
for _, word := range strings.Fields(text) {
switch {
case line == "":
line = word
case len(line)+1+len(word) > width:
lines = append(lines, line)
line = indent + word
default:
line += " " + word
}
}
if line != "" {
lines = append(lines, line)
}
return strings.Join(lines, "\n")
}
// shortError keeps the useful part of docker/SSH connection errors, which
// otherwise contain several lines of boilerplate.
func shortError(err error) string {
msg := err.Error()
if i := strings.LastIndex(msg, "stderr="); i >= 0 {
msg = msg[i+len("stderr="):]
}
return strings.TrimSpace(msg)
}
-53
View File
@@ -1,53 +0,0 @@
package audit
import (
"errors"
"testing"
auditPkg "coopcloud.tech/abra/pkg/audit"
"github.com/stretchr/testify/assert"
)
func TestWrap(t *testing.T) {
assert.Equal(t, "short", wrap("short", 40, " "))
assert.Equal(t, "1. one two\n three\n four five", wrap("1. one two three four five", 12, " "))
assert.Equal(t, "no wrapping at all here", wrap("no wrapping at all here", 0, " "))
}
func TestShortError(t *testing.T) {
err := errors.New(`error during connect: Get "http://docker.example.com/v1.51/info": command [ssh -- host docker system dial-stdio] has exited with exit status 255, please make sure the URL is valid, and Docker 18.09 or later is installed on the remote host: stderr=ssh: connect to host srv port 22: No route to host
`)
assert.Equal(t, "ssh: connect to host srv port 22: No route to host", shortError(err))
assert.Equal(t, "plain", shortError(errors.New("plain")))
}
func TestDisplayVersion(t *testing.T) {
assert.Equal(t, "1.0.0+1.2.3", displayVersion(auditPkg.Finding{Version: "1.0.0+1.2.3"}))
assert.Equal(t, "1234abcd+U (chaos)", displayVersion(auditPkg.Finding{Version: "1.0.0+1.2.3", Chaos: true, ChaosVersion: "1234abcd+U"}))
assert.Equal(t, "1.0.0+1.2.3 (chaos)", displayVersion(auditPkg.Finding{Version: "1.0.0+1.2.3", Chaos: true}))
}
func TestFixLine(t *testing.T) {
adv := &auditPkg.Advisory{ID: "CCA-1", Status: auditPkg.StatusPublished}
f := auditPkg.Finding{Advisory: adv, Status: auditPkg.FindingAffected, Action: auditPkg.ActionUpstream, Fix: "No fixed image exists yet."}
assert.Equal(t, "upstream must update", displayAction(f))
assert.Equal(t, "upstream must update. No fixed image exists yet. (CCA-1)", fixLine(f))
// an image that could not be checked needs a look first
f = auditPkg.Finding{Advisory: adv, Status: auditPkg.FindingUnknown, Action: auditPkg.ActionRedeploy, Fix: "abra app deploy app --force"}
assert.Equal(t, "check by hand", displayAction(f))
assert.Equal(t, "check by hand. Not every image could be checked (see abra audit -d). If it is affected: redeploy. abra app deploy app --force (CCA-1)", fixLine(f))
}
func TestAdvisoryRefs(t *testing.T) {
adv := &auditPkg.Advisory{ID: "CCA-2026-0002", Status: auditPkg.StatusPublished}
assert.Equal(t, "CCA-2026-0002", advisoryRefs(adv))
adv.Refs = []string{"https://nginx.org/en/security_advisories.html", "CVE-2026-123007"}
assert.Equal(t, "CCA-2026-0002, CVE-2026-123007", advisoryRefs(adv))
adv.Refs = append(adv.Refs, "CVE-2026-113425", "CVE-2026-123633")
adv.Status = auditPkg.StatusDraft
assert.Equal(t, "CCA-2026-0002 (draft), CVE-2026-123007 and 2 more", advisoryRefs(adv))
}
+2 -1
View File
@@ -92,10 +92,11 @@ func SetBumpType(bumpType string) {
func GetMainAppImage(recipe recipe.Recipe) (string, error) {
var path string
config, err := recipe.GetComposeConfig(nil)
config, err := recipe.GetComposeConfig()
if err != nil {
return "", err
}
for _, service := range config.Services {
if service.Name == "app" {
img, err := reference.ParseNormalizedNamed(service.Image)
+3 -24
View File
@@ -59,15 +59,9 @@ func ValidateRecipe(args []string, cmdName string) recipe.Recipe {
log.Fatal(i18n.G("no recipe name provided"))
}
recipeName = recipe.NormalizeRecipeName(recipeName)
lookupName := recipeName
if i := strings.LastIndex(lookupName, ":"); i >= 0 {
lookupName = lookupName[:i]
}
if _, ok := knownRecipes[lookupName]; !ok {
if !strings.Contains(lookupName, "/") {
log.Fatal(i18n.G("no recipe '%s' exists? pass a git URL (e.g. https://git.example.com/user/recipe) to use a recipe outside the catalogue", lookupName))
if _, ok := knownRecipes[recipeName]; !ok {
if !strings.Contains(recipeName, "/") {
log.Fatal(i18n.G("no recipe '%s' exists?", recipeName))
}
}
@@ -76,21 +70,6 @@ func ValidateRecipe(args []string, cmdName string) recipe.Recipe {
log.Fatal(err)
}
_, err = chosenRecipe.GetComposeConfig(nil)
if err != nil {
if cmdName == i18n.G("generate") {
if strings.Contains(err.Error(), "missing a compose") {
log.Fatal(err)
}
log.Warn(err)
} else {
if strings.Contains(err.Error(), "template_driver is not allowed") {
log.Warn(i18n.G("ensure %s recipe compose.* files include \"version: '3.8'\"", recipeName))
}
log.Fatal(i18n.G("unable to validate recipe: %s", err))
}
}
log.Debug(i18n.G("validated %s as recipe argument", recipeName))
return chosenRecipe
-45
View File
@@ -2,13 +2,11 @@ package recipe
import (
"fmt"
"path"
"sort"
"strconv"
"strings"
"coopcloud.tech/abra/cli/internal"
"coopcloud.tech/abra/pkg/config"
"coopcloud.tech/abra/pkg/formatter"
"coopcloud.tech/abra/pkg/i18n"
"coopcloud.tech/abra/pkg/log"
@@ -43,7 +41,6 @@ var RecipeListCommand = &cobra.Command{
headers := []string{
i18n.G("name"),
i18n.G("source"),
i18n.G("category"),
i18n.G("status"),
i18n.G("healthcheck"),
@@ -59,7 +56,6 @@ var RecipeListCommand = &cobra.Command{
for _, recipe := range recipes {
row := []string{
recipe.Name,
i18n.G("catalogue"),
recipe.Category,
strconv.Itoa(recipe.Features.Status),
recipe.Features.Healthcheck,
@@ -80,23 +76,6 @@ var RecipeListCommand = &cobra.Command{
}
}
externals := externalRecipes(catl)
sort.Strings(externals)
for _, name := range externals {
row := []string{
name,
i18n.G("external"),
"-", "-", "-", "-", "-", "-", "-",
}
if pattern != "" {
if !strings.Contains(name, pattern) {
continue
}
}
table.Row(row...)
rows = append(rows, row)
}
if len(rows) > 0 {
if internal.MachineReadable {
out, err := formatter.ToJSON(headers, rows)
@@ -114,30 +93,6 @@ var RecipeListCommand = &cobra.Command{
},
}
// externalRecipes returns canonical names of locally-cloned recipes that
// were sourced from an arbitrary git URL (i.e. they carry a .abra-source
// sidecar) and are not already listed in the catalogue.
func externalRecipes(catl recipe.RecipeCatalogue) []string {
dirs, err := recipe.GetRecipesLocal()
if err != nil {
log.Debug(i18n.G("can't read local recipes: %s", err))
return nil
}
var names []string
for _, dir := range dirs {
canonical := recipe.ReadRecipeSource(path.Join(config.RECIPES_DIR, dir))
if canonical == "" {
continue
}
if _, inCatalogue := catl[canonical]; inCatalogue {
continue
}
names = append(names, canonical)
}
return names
}
var (
pattern string
)
+2 -1
View File
@@ -312,10 +312,11 @@ likely to change.
func GetImageVersions(recipe recipePkg.Recipe) (map[string]string, error) {
services := make(map[string]string)
config, err := recipe.GetComposeConfig(nil)
config, err := recipe.GetComposeConfig()
if err != nil {
return nil, err
}
missingTag := false
for _, service := range config.Services {
if service.Image == "" {
+1 -52
View File
@@ -124,14 +124,11 @@ interface.`),
log.Debug(i18n.G("did not find versions file for %s", recipe.Name))
}
config, err := recipe.GetComposeConfig(nil)
config, err := recipe.GetComposeConfig()
if err != nil {
log.Fatal(err)
}
// digest pinned images shared between services are only checked once
digestChecked := make(map[string]bool)
for _, service := range config.Services {
img, err := reference.ParseNormalizedNamed(service.Image)
if err != nil {
@@ -181,13 +178,6 @@ interface.`),
sort.Sort(tagcmp.ByTagDesc(compatible))
if canonical, ok := img.(reference.Canonical); ok && !digestChecked[service.Image] {
digestChecked[service.Image] = true
if err := refreshDigest(recipe, img, image, canonical.Digest().String()); err != nil {
log.Fatal(err)
}
}
if len(compatible) == 0 && !allTags {
log.Info(i18n.G("no new versions available for %s, assuming %s is the latest (use -a/--all-tags to see all anyway)", image, tag))
continue // skip on to the next tag and don't update any compose files
@@ -374,47 +364,6 @@ interface.`),
},
}
// refreshDigest checks whether the current tag of a digest pinned image was
// re-published upstream with a new digest and offers to update the pin.
func refreshDigest(recipe recipePkg.Recipe, img reference.Named, image, digest string) error {
tag := img.(reference.NamedTagged).Tag()
registryDigest, err := client.GetRegistryDigest(img, tag)
if err != nil {
return err
}
if registryDigest == digest {
log.Debug(i18n.G("digest of %s:%s is up to date", image, tag))
return nil
}
log.Warn(i18n.G("%s:%s was re-published upstream (digest %s → %s)", image, tag, digest, registryDigest))
update := false
if !internal.NoInput {
prompt := &survey.Confirm{
Message: i18n.G("update digest of %s:%s?", image, tag),
Default: true,
}
if err := survey.AskOne(prompt, &update); err != nil {
return err
}
}
if !update {
return nil
}
if _, err := recipe.UpdateTag(image, tag); err != nil {
return err
}
log.Info(i18n.G("digest updated for %s:%s", image, tag))
return nil
}
var (
allTags bool
createCommit bool
+1 -5
View File
@@ -7,7 +7,6 @@ import (
"strings"
"coopcloud.tech/abra/cli/app"
"coopcloud.tech/abra/cli/audit"
"coopcloud.tech/abra/cli/catalogue"
"coopcloud.tech/abra/cli/internal"
"coopcloud.tech/abra/cli/recipe"
@@ -86,12 +85,10 @@ func Run(version, commit string) {
Config:
$ABRA_DIR: %s`, config.ABRA_DIR),
Version: fmt.Sprintf("%s-%s", version, formatter.Truncate(commit, 7)),
Version: fmt.Sprintf("%s-%s", version, commit[:7]),
ValidArgs: []string{
// translators: `abra app` command for autocompletion
i18n.G("app"),
// translators: `abra audit` command for autocompletion
i18n.G("audit"),
// translators: `abra autocomplete` command for autocompletion
i18n.G("autocomplete"),
// translators: `abra catalogue` command for autocompletion
@@ -257,7 +254,6 @@ Config:
AutocompleteCommand,
manCommand,
app.AppCommand,
audit.AuditCommand,
catalogue.CatalogueCommand,
server.ServerCommand,
recipe.RecipeCommand,
+1 -1
View File
@@ -16,7 +16,7 @@ func main() {
Version = "dev"
}
if Commit == "" {
Commit = "unknown-commit"
Commit = " "
}
cli.Run(Version, Commit)
-17
View File
@@ -1,17 +0,0 @@
{
"$schema": "https://raw.githubusercontent.com/jetify-com/devbox/main/.schema/devbox.schema.json",
"packages": [
"go@1.27",
"gnumake@latest",
"gettext@latest",
"gnused@latest"
],
"shell": {
"scripts": {
"build": "make build",
"test": "export ABRA_DIR=\"$(mktemp -d)/abra\"; rc=0; make test || rc=$?; rm -rf \"$(dirname \"$ABRA_DIR\")\"; exit $rc",
"update-pot": "make update-pot",
"i18n": "make i18n"
}
}
}
-305
View File
@@ -1,305 +0,0 @@
{
"lockfile_version": "1",
"packages": {
"gettext@latest": {
"last_modified": "2026-10-20T14:23:57Z",
"resolved": "github:NixOS/nixpkgs/5adc50831c259f87faeb09ddf930d43597197473#gettext",
"source": "devbox-search",
"version": "1.0",
"systems": {
"aarch64-darwin": {
"outputs": [
{
"name": "out",
"path": "/nix/store/36c168mc93w4mlmqknam03ffyi6qg0w6-gettext-1.0",
"default": true
},
{
"name": "man",
"path": "/nix/store/38mvyy6a3ga3xqbzzbzfhpmmd4dll8hl-gettext-1.0-man",
"default": true
},
{
"name": "doc",
"path": "/nix/store/4i2r8pmdirlpig74ymxmi03sgl0inyd4-gettext-1.0-doc"
},
{
"name": "info",
"path": "/nix/store/ml58bdjablvqamxwc28ws5y9222hdvpx-gettext-1.0-info"
}
],
"store_path": "/nix/store/36c168mc93w4mlmqknam03ffyi6qg0w6-gettext-1.0"
},
"aarch64-linux": {
"outputs": [
{
"name": "out",
"path": "/nix/store/qh35hki623iqrj2mqfxnzjqawxn93gk7-gettext-1.0",
"default": true
},
{
"name": "man",
"path": "/nix/store/v0arqmc3cx4ls1vwg4c62kz7apcn76ml-gettext-1.0-man",
"default": true
},
{
"name": "doc",
"path": "/nix/store/4ficj1yfkd8bn17mlv81jsib04hfyg8c-gettext-1.0-doc"
},
{
"name": "info",
"path": "/nix/store/w0slr4m7ywbkmlm8dbksjp22adcab935-gettext-1.0-info"
}
],
"store_path": "/nix/store/qh35hki623iqrj2mqfxnzjqawxn93gk7-gettext-1.0"
},
"x86_64-darwin": {
"outputs": [
{
"name": "out",
"path": "/nix/store/9mxzw5vwn99vx2b74ipjzk8x4b784fjy-gettext-1.0",
"default": true
},
{
"name": "man",
"path": "/nix/store/r87hlf9pn8q3a8rnplzhy9lbgaadqcr8-gettext-1.0-man",
"default": true
},
{
"name": "doc",
"path": "/nix/store/xmssbi5i2w3fv67hrv6c29wdiv8gxywi-gettext-1.0-doc"
},
{
"name": "info",
"path": "/nix/store/j4lz4zqicd1mw2l5c272ialxhl0nkc0b-gettext-1.0-info"
}
],
"store_path": "/nix/store/9mxzw5vwn99vx2b74ipjzk8x4b784fjy-gettext-1.0"
},
"x86_64-linux": {
"outputs": [
{
"name": "out",
"path": "/nix/store/8qdw9shgr7dd3315jim8sfva2dnhq86c-gettext-1.0",
"default": true
},
{
"name": "man",
"path": "/nix/store/4acc3iagl3xlljb1aqn8j25vx5n400bc-gettext-1.0-man",
"default": true
},
{
"name": "doc",
"path": "/nix/store/dnz19fkr7spnpc7affh8ivcmk32g9xda-gettext-1.0-doc"
},
{
"name": "info",
"path": "/nix/store/v3z399ilsdjn461y0l2gi8h0z7jzsy6a-gettext-1.0-info"
}
],
"store_path": "/nix/store/8qdw9shgr7dd3315jim8sfva2dnhq86c-gettext-1.0"
}
}
},
"github:NixOS/nixpkgs/nixpkgs-unstable": {
"last_modified": "2026-10-20T14:23:57Z",
"resolved": "github:NixOS/nixpkgs/5adc50831c259f87faeb09ddf930d43597197473?lastModified=1792506237&narHash=sha256-Q%2F8KLB6jNfpaUN1HAedE%2Fu%2FEt9feE%2FzYSsutBLKrCFk%3D"
},
"gnumake@latest": {
"last_modified": "2026-10-20T14:23:57Z",
"resolved": "github:NixOS/nixpkgs/5adc50831c259f87faeb09ddf930d43597197473#gnumake",
"source": "devbox-search",
"version": "4.4.1",
"systems": {
"aarch64-darwin": {
"outputs": [
{
"name": "out",
"path": "/nix/store/306s97mvw5slpgg7hnb26xy37dhy4xk0-gnumake-4.4.1",
"default": true
},
{
"name": "man",
"path": "/nix/store/g0hyk9rpf10di0l5x6gxjahh5rip30kn-gnumake-4.4.1-man",
"default": true
},
{
"name": "doc",
"path": "/nix/store/xl1nz3znzjb27wrig0vrf5fj3z7jqh5s-gnumake-4.4.1-doc"
},
{
"name": "info",
"path": "/nix/store/99bqn610vynhvhwg8srd9g2l990cfa7a-gnumake-4.4.1-info"
}
],
"store_path": "/nix/store/306s97mvw5slpgg7hnb26xy37dhy4xk0-gnumake-4.4.1"
},
"aarch64-linux": {
"outputs": [
{
"name": "out",
"path": "/nix/store/nk0i2hc4x8qpshj326jzn36hkbgi7j7z-gnumake-4.4.1",
"default": true
},
{
"name": "man",
"path": "/nix/store/nacivbyfzp1w0xydskbxrkapp2gn552s-gnumake-4.4.1-man",
"default": true
},
{
"name": "debug",
"path": "/nix/store/f7yag335k3y4ymn9hb5q1yzqc1njf1hi-gnumake-4.4.1-debug"
},
{
"name": "doc",
"path": "/nix/store/782lakmsm9nxf4hpnn8whmq541cphv6r-gnumake-4.4.1-doc"
},
{
"name": "info",
"path": "/nix/store/51cb4z2jrvqdnab9cc8hjf478c8f55hw-gnumake-4.4.1-info"
}
],
"store_path": "/nix/store/nk0i2hc4x8qpshj326jzn36hkbgi7j7z-gnumake-4.4.1"
},
"x86_64-darwin": {
"outputs": [
{
"name": "out",
"path": "/nix/store/0bg5b97b9hbhz8907bfcrch1i116xppa-gnumake-4.4.1",
"default": true
},
{
"name": "man",
"path": "/nix/store/wb4yn3d7vmi1hr4f8pz38rjvcl7f70ma-gnumake-4.4.1-man",
"default": true
},
{
"name": "doc",
"path": "/nix/store/3b9cjf16jx4ncpp3b4k6m8q01nxsrw1n-gnumake-4.4.1-doc"
},
{
"name": "info",
"path": "/nix/store/rriz5p4q8a6sca7sw3p3js1m5dl09ia8-gnumake-4.4.1-info"
}
],
"store_path": "/nix/store/0bg5b97b9hbhz8907bfcrch1i116xppa-gnumake-4.4.1"
},
"x86_64-linux": {
"outputs": [
{
"name": "out",
"path": "/nix/store/vs2sa1msvkjxbb70h47819zq8a9srj21-gnumake-4.4.1",
"default": true
},
{
"name": "man",
"path": "/nix/store/ncw4l9vr02fh2prknq5dhmdycb00rdmr-gnumake-4.4.1-man",
"default": true
},
{
"name": "debug",
"path": "/nix/store/k7lz9zpj52al794kkw0d80hwj2v7pn9h-gnumake-4.4.1-debug"
},
{
"name": "doc",
"path": "/nix/store/7c3i6ivjqgc7hz21bchima12lxk8c1s4-gnumake-4.4.1-doc"
},
{
"name": "info",
"path": "/nix/store/9lbfzcvwx1c859fy6k0l9cbkp6fcxd9l-gnumake-4.4.1-info"
}
],
"store_path": "/nix/store/vs2sa1msvkjxbb70h47819zq8a9srj21-gnumake-4.4.1"
}
}
},
"gnused@latest": {
"last_modified": "2026-10-20T14:23:57Z",
"resolved": "github:NixOS/nixpkgs/5adc50831c259f87faeb09ddf930d43597197473#gnused",
"source": "devbox-search",
"version": "4.10",
"systems": {
"aarch64-darwin": {
"outputs": [
{
"name": "out",
"path": "/nix/store/8fyr2rna4vqibkqsbzyfd4z6fna5c09p-gnused-4.10",
"default": true
},
{
"name": "info",
"path": "/nix/store/6c8gqzdibqfzgnp8876lp4zh1z9f13g1-gnused-4.10-info"
}
],
"store_path": "/nix/store/8fyr2rna4vqibkqsbzyfd4z6fna5c09p-gnused-4.10"
},
"aarch64-linux": {
"outputs": [
{
"name": "out",
"path": "/nix/store/03rgyjxclvhxnif2klf13nkr92rpxqqg-gnused-4.10",
"default": true
},
{
"name": "info",
"path": "/nix/store/sgxd46ij300v6jwrxj5cfjz7a8029jxy-gnused-4.10-info"
}
],
"store_path": "/nix/store/03rgyjxclvhxnif2klf13nkr92rpxqqg-gnused-4.10"
},
"x86_64-linux": {
"outputs": [
{
"name": "out",
"path": "/nix/store/9wc73n5izv0m7ppjbibi4hbr68rba1lj-gnused-4.10",
"default": true
},
{
"name": "info",
"path": "/nix/store/4gfax7m36hw70wn2cqy0dh3dr1r3rlr3-gnused-4.10-info"
}
],
"store_path": "/nix/store/9wc73n5izv0m7ppjbibi4hbr68rba1lj-gnused-4.10"
}
}
},
"go@1.27": {
"last_modified": "2026-10-20T14:23:57Z",
"resolved": "github:NixOS/nixpkgs/5adc50831c259f87faeb09ddf930d43597197473#go_1_27",
"source": "devbox-search",
"version": "1.27.3",
"systems": {
"aarch64-darwin": {
"outputs": [
{
"name": "out",
"path": "/nix/store/s5m10yz53f8778k4mi1irdqzy8iaqri1-go-1.27.3",
"default": true
}
],
"store_path": "/nix/store/s5m10yz53f8778k4mi1irdqzy8iaqri1-go-1.27.3"
},
"aarch64-linux": {
"outputs": [
{
"name": "out",
"path": "/nix/store/0940szmkaskjshd3z0gp2m2kfxcnc424-go-1.27.3",
"default": true
}
],
"store_path": "/nix/store/0940szmkaskjshd3z0gp2m2kfxcnc424-go-1.27.3"
},
"x86_64-linux": {
"outputs": [
{
"name": "out",
"path": "/nix/store/1fng2iqlx5y8rqvhjp9203mymzyny6cz-go-1.27.3",
"default": true
}
],
"store_path": "/nix/store/1fng2iqlx5y8rqvhjp9203mymzyny6cz-go-1.27.3"
}
}
}
}
}
-450
View File
@@ -1,450 +0,0 @@
# abra audit — design
2026-11-20 · Linus Gasser
## Summary
Add a top-level `abra audit` command. It checks every deployed app on every server against a curated list of advisories, and offers to fix what fails. A maintainer-side `abra audit update` turns upstream vulnerability sources into proposed advisories.
## Problem
Operators of Co-op Cloud servers cannot easily tell whether their apps are safe:
- **Mutable tags.** Images such as `php` or `nextcloud` are often rebuilt under the same tag. A server can run a vulnerable image while its compose file looks current.
- **Stale base images.** An image like `nextcloud` is built on `php`. When `php` is fixed, `nextcloud` is not fixed until upstream rebuilds it.
- **Old recipe versions.** Some recipe versions ship known-vulnerable upstream versions, and nothing tells the operator.
- **Unreliable redeploys.** A redeploy should pull the new image. If the registry is overloaded at deploy time, Docker falls back to the bare tag and each node may keep its old local image, with only a warning.
## Goals and non-goals
Goals:
- One command reports, for every deployed app, whether it matches a known advisory.
- Detect vulnerable images even when the tag has not changed, including vulnerable base images.
- Propose a fix for each finding and, on request, apply it and verify the result.
- Help advisory maintainers find new problems from upstream sources, without missing images that were fixed before anyone looked.
- Work on mixed amd64 and arm64 swarms.
Non-goals:
- A full CVE scanner. Deep scanning is delegated to Trivy or Grype as an option.
- Rebuilding upstream images. abra can detect a stale base image but not patch it.
- Publishing advisories automatically. A human reviews every advisory before operators see it.
## Commands
`abra audit` is a top-level command, because an audit spans servers, apps and recipes. `abra app check` already exists (it checks `.env` variables), so that name is not available.
| Command | Who | What it does |
| --- | --- | --- |
| `abra audit` | operator | Audit all deployed apps on all servers |
| `abra audit app <domain>` | operator | Audit one deployed app |
| `abra audit server <server>` | operator | Audit all apps on one server |
| `abra audit recipe <recipe>` | maintainer, CI | Static check of recipe versions and pinned images; nothing deployed needed |
| `abra audit list` | anyone | List advisories, filtered by `--recipe` or `--severity` |
| `abra audit show <id>` | anyone | Details and fix notes for one advisory |
| `abra audit update` | advisory maintainer | Propose new or changed advisories from a CVE, from minimum versions, or from upstream sources |
Two aliases are kept, because operators look under `app` and `server` first:
- `abra app audit <domain>` is the same as `abra audit app <domain>`.
- `abra server audit <server>` is the same as `abra audit server <server>`.
**Storage.** The audit needs an explicitly chosen abra directory. It is taken from, in order:
1. `$ABRA_DIR`;
2. the `abraDir` setting of an `abra.yaml` file in the current directory or one of its parents, as the rest of abra already supports.
If neither exists, all `abra audit` commands fail with a clear error. They do not fall back to `~/.abra`, unlike the rest of abra. Below, `$ABRA_DIR` means this directory. Everything the audit stores lives below it:
| Path | Contents |
| --- | --- |
| `$ABRA_DIR/servers` | Server and app `.env` files, as for the rest of abra |
| `$ABRA_DIR/advisories` | Checkout of the advisories repo: shared, reviewed data |
| `$ABRA_DIR/.cache/audit/images` | Local cache of image configs, by digest; from v2 on, safe to delete |
**Fetching.** There is no cache lifetime: every invocation checks whether the advisory list has changed, and pulls it if so. The list is a git repo, so this is a `git fetch` of a few KB when nothing changed. When the repo cannot be reached, the audit warns, says how old the local copy is, and uses it.
### Options for `audit`, `app` and `server`
```
-s, --server <name> only apps on this server
-r, --recipe <name> only apps of this recipe
-a, --advisories <url|path> advisory source (default: c4dt/advisories)
--severity <level> minimum severity: low, medium, high, critical
--include-drafts also apply draft advisories (marked as such)
-f, --fix propose fixes, apply after confirmation
-n, --no-input apply fixes without asking
--refresh-only only re-pull images, never upgrade the recipe
--retries <n> re-checks after a fix, with backoff (default 3)
-o, --offline no registry or advisory fetch; recipe checks only
--deep also scan running digests with Trivy or Grype
--report send unknown digests to the advisories repo (opt-in)
-m, --machine JSON output for cron and monitoring
```
`abra audit recipe` takes only `--machine`, so CI can fail on findings.
### Exit codes
The exit code describes the state **after** the run, so a finding that was fixed and verified no longer counts.
| Code | Meaning |
| --- | --- |
| 0 | No findings remain: none were found, or all were fixed and verified |
| 1 | Findings remain that were not fixed: no `--fix`, fix declined, or strategy `manual` |
| 2 | A fix was tried but could not be verified, for example because the registry was unreachable |
With several apps, the highest code wins. The report lists every finding with its final status (`fixed`, `affected`, `unknown`, `unverified`), so fixed findings stay visible in the output and in `--machine` JSON.
## Advisory format
Advisories are YAML files, one per advisory, in a separate git repo under `advisories/`. They are kept apart from the recipe catalogue because they change on a different schedule and need their own review. The repo starts as `c4dt/advisories` and can move to `coop-cloud/advisories` once the format has settled.
```yaml
id: CCA-2026-0012
title: PHP 8.3 CGI argument injection
severity: critical # low | medium | high | critical
status: published # draft | published | withdrawn
refs: [CVE-2026-12345, https://example.org/advisory]
published: 2026-10-20
# Recipe versions: semver range on the recipe part of the tag
recipes:
nextcloud:
affected: "<7.0.3+30.0.1"
fixed: "7.0.3+30.0.1"
# Images: an image is fixed if ANY condition holds
images:
- match: "nextcloud:30*-apache"
fixed_from: sha256:2222... # this digest and all first seen after it
affected_digests: [sha256:1111...]
- match: "*" # any image...
base: "php:8.3*" # ...built on php 8.3
env: { PHP_VERSION: "<8.3.14" } # read from the image config
fix:
strategy: refresh # upgrade | refresh | manual
notes: "Upstream images rebuilt on 2026-10-18; a redeploy is enough."
```
### Image conditions
Digests are hashes and have no order, so "newer than digest X" cannot be computed from the digest itself. Instead, the format uses what the image says about itself, in order of preference:
| Condition | Field | Reliability | Notes |
| --- | --- | --- | --- |
| Version in the image | `env`, `version` | Best | Tests the fix itself, per platform, without pulling layers |
| Build time | `created_after` | Good, if the image records its real build time | Covers rebuilds where no version changes; see below |
| Base ancestry | `base`, `base_digest_from` | Good | Docker Official Images record their base in OCI annotations |
| Allowlist | `fixed_digests` | Exact but closed | Later rebuilds need the list updated |
| Denylist | `affected_digests` | Exact but incomplete | For one known-bad digest |
| Digest history | `fixed_from` | Good | Fallback for images whose build time is unusable; later version |
An image that matches `match` but satisfies no condition is reported as **unknown**, never as OK.
How the conditions combine, for an image that matches `match`:
1. A digest in `affected_digests` is affected; a digest in `fixed_digests` is fixed.
2. `env` both selects and tests. If the image lacks one of the variables, the entry does not apply, because the image is not built on that runtime. Otherwise the ranges are **affected** ranges: if every variable is in its range, `env` says affected, else fixed. Write `">=8.3.0 <8.3.14"` to limit the entry to one release series.
3. `version` is an affected range on `org.opencontainers.image.version`.
4. `created_after` says fixed for images built at or after that time, and affected for older ones, unless the build time is unusable.
5. If any condition says fixed, the image is fixed. Otherwise, if one says affected, it is affected. Otherwise it is unknown.
`match` is a glob where `*` matches any characters. A pattern with a `:` is matched against `name:tag`, otherwise against the name only, using Docker's short names (`nextcloud`, not `docker.io/library/nextcloud`). Version ranges are space- or comma-separated constraints (`<`, `<=`, `>`, `>=`, `=`, `!=`) that must all hold, with `||` between alternatives. A recipe range without an upstream part (`<7.0.3`) only compares the recipe part of the tag.
### Ordering images by version and build time
Most fixes change a version that the image shows, so no digest bookkeeping is needed:
- **Application version.** `NEXTCLOUD_VERSION` in the config, or the annotation `org.opencontainers.image.version` (for example `35.0.3-apache`).
- **Inherited runtime version.** Images inherit environment variables from their base, so `nextcloud:apache` also carries `PHP_VERSION=8.5.13`. A nextcloud rebuilt on a fixed php shows the new `PHP_VERSION`, even though `NEXTCLOUD_VERSION` is unchanged.
That leaves rebuilds where every visible version stays the same, typically a Debian package such as openssl patched in the base image. For these, the build time decides: `created_after` is the `Created` time of the first fixed image, and any image of that tag built later counts as fixed.
**Is the build time monotonic?** Per tag, in practice yes: every rebuild is a new build run with a later timestamp. On Docker Hub on 2026-11-20, `php:8.3-apache` recorded its real build time (2026-11-19 14:40 UTC), and `nextcloud:apache` was built 33 minutes later on top of it. Both also carry the annotations `org.opencontainers.image.base.name` and `base.digest`. The exceptions are known:
| Case | Effect | Handling |
| --- | --- | --- |
| Reproducible builds (`SOURCE_DATE_EPOCH`, ko, Nix, Bazel) set `Created` to a commit date or to 1970 | A base-only rebuild keeps the same `Created`, so old and new cannot be told apart | Detect it, then report unknown |
| Upstream moves a tag back to an older image | The older `Created` is reported as affected | Correct as is |
| Clock skew between build machines | Minutes | Irrelevant next to fix dates days apart |
The audit treats `Created` as unusable, and reports unknown, when:
- it is before 2000, which catches the 1970 epoch;
- it is earlier than the `Created` of the base image named in `base.digest`, because an image cannot be built before its base;
- two different digests of the same tag have the same `Created`.
The advisory can then fall back to `fixed_digests`, or later to the digest history.
### Fix strategies
| Strategy | Meaning | What `--fix` does |
| --- | --- | --- |
| `upgrade` | A newer recipe version fixes it | `abra app upgrade <domain> <fixed version>` |
| `refresh` | The same tag now points to a fixed image | Redeploy the same version, then verify the running digest |
| `manual` | No fixed image exists yet | Report only, with the notes and options such as pinning a digest |
## Digest history (fallback, later version)
The digest history is only needed for images whose build time is unusable, as listed above. It is deferred to v4, and built only if such images turn out to matter.
The advisories repo would keep an append-only history per image tag, recording when each digest was first seen:
```yaml
# digests/nextcloud/30-apache.yml (append-only)
- digest: sha256:1111...
first_seen: 2026-10-02T08:00:00Z
config: { created: 2026-10-01, NEXTCLOUD_VERSION: 30.0.3, PHP_VERSION: 8.3.12 }
- digest: sha256:2222...
first_seen: 2026-10-18T14:00:00Z
config: { created: 2026-10-18, NEXTCLOUD_VERSION: 30.0.3, PHP_VERSION: 8.3.14 }
```
It would be filled from three sources:
1. **`abra audit update`** runs regularly from CI and appends every new digest it sees, with its image config. The config is stored because registries may delete old digests once they are untagged.
2. **Backfill from `docker-library/repo-info`**, which keeps years of per-tag digests for official images in git.
3. **Running servers.** With `--report`, `abra audit` sends digests it finds that are not in the history. This is opt-in.
## Images fixed before anyone looked
Suppose a CVE is published and upstream pushes a fixed image under the same tag before `update` runs. `update` never sees the vulnerable image, but old installations still run it. This is handled because advisories are driven by the CVE, not by the current image:
- The CVE creates the advisory, whether or not today's image is clean.
- A clean current image only sets the strategy to `refresh`. Its versions become the `env` condition, or its `Created` time becomes `created_after`.
- On a server, an image with older versions or an earlier build time is reported as affected. Both lead to a refresh and a verification.
`update` only needs to see the fixed image, and that is the one available when it runs.
## Audit and fix
`abra audit` checks each app in the same way, and a fix counts only after the running digest has been verified.
1. **Read state.** From the service spec, read the recipe version label and each image reference `image:tag@sha256:...`. For per-platform checks, read the node's platform and fetch that image config from the registry.
2. **Match advisories.** First recipe ranges, then image conditions.
3. **Classify** each finding as affected or unknown. No finding means clean.
4. **Fix** (with `--fix`) according to the advisory's strategy, then verify.
```mermaid
flowchart TD
read[Read service spec] --> match[Match advisories]
match --> finding{Finding?}
finding -- no --> clean[Clean, exit 0]
finding -- affected or unknown --> strategy[Pick fix strategy]
strategy -- manual --> report[Report, exit 1]
strategy -- upgrade or refresh --> apply[Upgrade / refresh]
apply --> verified{Digest fixed?}
verified -- yes --> fixed[Fixed, exit 0]
verified -- no --> retry[Retry, backoff]
retry -- within --retries --> apply
retry -- retries used up --> unverified[Unverified, exit 2]
```
Verification waits for the service to converge, then reads the spec again. It fails if the spec holds a bare tag (the registry lookup failed) or a digest that still matches the advisory. Retries use backoff, for example 30 s, 2 min and 10 min, before giving up with exit code 2.
## `abra audit update`
`abra audit update` is a maintainer tool. It compares the recipes in the catalogue with a vulnerability, and proposes new or changed advisories for a human to review. It has three modes; the first two come first (see [Versions](#versions)).
```
abra audit update --cve <CVE-ID> # one known vulnerability
abra audit update --require <lib>=<v>[,<v>...] # minimum patched versions
abra audit update --sources [recipe...] # scan upstream sources (later)
--repo <path> local checkout of the advisories repo (default $ABRA_DIR/advisories)
--source <list> with --sources: endoflife,osv,ghsa,kev,repo-info,trivy (default: all but trivy)
--since <date> with --sources: only upstream changes after this date (default: last run)
-n, --no-input write all proposals as drafts without asking
--pr commit accepted proposals on a branch and open a PR
```
### Sources
| Source | Gives | Becomes |
| --- | --- | --- |
| [endoflife.date](https://endoflife.date) | End-of-life dates for runtimes such as php and postgres | `medium` advisory, strategy `upgrade` |
| [OSV.dev](https://osv.dev) and [GitHub Advisory Database](https://github.com/advisories) | CVEs with affected and fixed upstream versions | Recipe ranges, mapped through the `+UPSTREAM` part of recipe tags |
| [CISA KEV](https://www.cisa.gov/known-exploited-vulnerabilities-catalog) | CVEs exploited in the wild | Severity raised to `critical` |
| [docker-library/repo-info](https://github.com/docker-library/repo-info) | Digest and package history of official images | Digest history; `fixed_from` for rebuilt images |
| Trivy (opt-in, slow) | CVEs found in current catalogue images | Proposals the other sources missed |
### Steps
1. **Inventory.** List every recipe version in the catalogue and its images. Fetch each image config for runtime versions (`PHP_VERSION`, `NODE_VERSION`) and find its base image.
2. **Record digests.** Append new digests and their configs to the digest history.
3. **Query sources** for the upstream apps and runtimes found in step 1.
4. **Classify.** For each CVE, write a condition that separates affected from fixed images. Look at the current image only to choose the strategy: `refresh` if the current image is clean, `manual` if it is not.
5. **Deduplicate.** Match on CVE ID and affected recipe or image. If an advisory already covers the finding, propose a change (wider range, new fixed digest, higher severity) instead of a new entry.
6. **Review.** Show each proposal as a YAML diff: accept, edit, skip, or ignore permanently. Ignored findings go to `ignore.yml` with a reason, so they do not return.
7. **Write** accepted proposals with `status: draft`, optionally on a branch with a PR. Only a human sets `status: published`.
A human is needed because the sources say what is vulnerable upstream. Only a maintainer can judge whether it matters for the recipe's configuration and which fix strategy applies.
## Multiple platforms
abra already works on mixed amd64 and arm64 swarms, because it leaves the platform choice to Docker. The audit must compare digests at the right level.
**Current behaviour.** abra has no platform-specific code. `deploy`, `upgrade` and `rollback` use `ResolveImageAlways` (`cli/app/deploy.go:157`), so Docker resolves `image:tag` to the index digest that covers all platforms. Each node then pulls its own variant. `GetRegistryDigest` in `pkg/client/registry.go` also returns the index digest, so pinned `image:tag@sha256:...` references work on every platform.
| Question | Compare | Where to read it |
| --- | --- | --- |
| Has the tag moved? | Index digest | Service spec, and a registry HEAD request |
| Which PHP version is inside? | Per-platform image config | Pick the node's platform from the index (`docker node inspect` gives `Description.Platform`) |
| Is it built on the fixed base? | Per-platform layers | Same per-platform manifest |
Pitfalls:
- **A new index digest does not prove the fix for your platform.** Any platform's rebuild changes the index digest. Some upstreams rebuild arm64 later or not at all, so important fixes are checked per platform.
- **Copied per-platform digests.** The Docker Hub web page shows per-platform digests most prominently. A maintainer who pins one breaks other platforms with `no matching manifest`. `abra recipe lint` should warn when a pinned digest is not an index but the image is multi-arch.
- **Containers show the image ID.** `docker inspect` on a container gives the config digest, not the manifest digest. Use the service spec for comparisons.
## What Docker allows
abra can detect almost any stale or vulnerable layer, but it can only fix an image that upstream has already rebuilt.
| Possible | How |
| --- | --- |
| Know exactly what is running | Swarm stores `image:tag@sha256:...` in the service spec |
| Tell whether a tag has moved | Registry HEAD request for the tag's manifest digest |
| Read versions without pulling | Fetch the image config (a few KB): `Created` and `Env`, which often inherits `PHP_VERSION` or `NGINX_VERSION` from the base |
| Find the base image | OCI annotations `org.opencontainers.image.base.name` and `base.digest` where set; otherwise layer-prefix matching against the base's layers |
| Go deeper, to libc or openssl | SBOM and provenance attestations, which Docker Official Images publish |
| Not possible | Consequence |
| --- | --- |
| Patch a base layer of someone else's image | If upstream has not rebuilt, the options are waiting, pinning a known-good older digest, or having the recipe build its own image |
| Trust a redeploy without checking | On registry failure Docker deploys the bare tag; every refresh must be verified |
**Undeploy is not needed.** `undeploy` then `deploy` has the same registry problem as a plain redeploy, and adds downtime. A redeploy of the same version with `ResolveImageAlways` already asks the registry again.
**Pinned digests make failures visible.** With `image: nextcloud:30-apache@sha256:...` in the recipe, a failed registry lookup fails the deploy instead of running a stale image. The `allow_digest` branch adds this support to `abra recipe upgrade`.
## Versions
The feature is built in four versions. Each one is usable on its own, and is tried on real servers before the next one starts.
| Version | Adds | Changes servers? |
| --- | --- | --- |
| v1 | `abra audit`: report failing apps and suggest commands | No |
| v2 | `abra audit update --cve` and `--require`: draft advisories for one vulnerability | No |
| v3 | Subcommands, options, `--fix` with verification, deploy warnings | Yes, with `--fix` |
| v4 | Automatic advisory proposals from upstream sources, digest history | No |
### v1: basic audit
Answers "which of my apps need attention, and what do I run?" using only a hand-written advisory list.
- **Command.** `abra audit`, without arguments. It goes through every server in `$ABRA_DIR/servers` and every deployed app on it, using the same data as `abra app ls -S`.
- **Advisories.** Hand-written YAML in `c4dt/advisories`, pulled on every run. The first advisory is the nextcloud same-tag rebuild. Only `published` advisories apply; `--include-drafts` also applies `draft` ones, marked "(draft)" in the output, so a maintainer can try an advisory on real servers before publishing it. `withdrawn` advisories never apply.
- **Conditions.** Only recipe ranges (`affected`, `fixed`), `env`, `version`, `created_after` (with the checks for unusable build times), `fixed_digests` and `affected_digests`. Fields v1 does not know are skipped with a warning, so newer advisories do not break older abra versions.
- **Reads only.** Service specs (version label and image digests) and image configs. On a single-node server, the common case, the image config is read from the node itself with `docker image inspect`, so no registry is needed.
- **Output.** Only failing apps. A narrow numbered table lists app, recipe, deployed version, what is affected, severity and action, with a server column only when several servers are audited. For an app deployed with `--chaos`, the version is the commit it was deployed from, marked `(chaos)`, as in `abra app ls`. The affected column names what is vulnerable, from the condition that matched: the runtime and its version for an `env` condition (`NGINX_VERSION=1.31.8` gives `nginx 1.31.8`), the image's version for a `version` condition, the build date for `created_after`, the recipe and its version for a recipe range, and the image otherwise. Advisory IDs such as `CCA-2026-0002` are local to the advisories repository and mean nothing at a glance, so they are not in the table. Below it, a numbered list gives the fix for each row, starting with the action and ending with the references to look the advisory up: its ID, marked if it is a draft, and its CVEs (`CCA-2026-0002, CVE-2026-123007 and 4 more`). The list is wrapped to the terminal width, because fixes are too long for a table column. Unreachable servers follow, with only the useful part of the SSH error. The findings that the operator cannot fix alone are summarised at the end, grouped by action. The action says who has to act:
| Action | Meaning | Fix shown |
| --- | --- | --- |
| redeploy | What a redeploy deploys is fixed: the tag's current image, or the recipe checkout of a `--chaos` deployment | `abra app deploy <domain> --force && abra audit`; for a `--chaos` deployment `abra app deploy <domain> --chaos`, because without `--chaos` the deploy would replace the chaos deployment with the release in the `.env` file |
| upgrade | A released recipe version uses fixed images | `abra app upgrade <domain> <version>` |
| recipe must update | A fixed image exists in the registry, but no recipe release uses it | The fixed image; for a `--chaos` deployment, use it in the checkout or the app's override and redeploy with `--chaos` |
| upstream must update | No fixed image exists yet | Why: the tag was not rebuilt, or rebuilt but still affected, and no later tag is fixed; plus a pointer to the advisory's mitigations |
| check by hand | The audit could not decide: an image could not be checked (status `unknown`), the registry could not be searched, or build times are unusable | The reason, and the fix that would apply otherwise |
| follow advisory | Strategy `manual` | The advisory's notes |
- **Fix check.** Every suggestion is checked before it is shown:
- A redeploy only helps if what it deploys is fixed. That is the tag's current image in the registry, unless the recipe pins a digest (`image:tag@sha256:...`): then a redeploy keeps the pinned image, which is checked instead. Upstreams stop rebuilding patch-pinned tags such as `nextcloud:35.0.2-fpm` once the next patch is out. If a redeploy does not help, an upgrade is looked for.
- A `--chaos` deployment runs the local checkout, so for it the audit first loads what `abra app deploy --chaos` would deploy: the files of the app's `COMPOSE_FILE` (`GetComposeFiles`), overrides outside the recipe directory included, as they are on disk with uncommitted changes, merged and filled in with the app's `.env` values by the same loader as deploy. If those images are fixed, the action is a `--chaos` redeploy. Images already deployed from an override are checked like any other deployed image.
- For an upgrade, the newer releases of the recipe are searched for the oldest one whose images are fixed. Images are read from the `compose*.yml` files of each release tag, without changing the local checkout.
- If no release is fixed, the registry is searched for a fixed image, to tell "recipe must update" from "upstream must update". The tags of the image's repository are listed, and candidates are checked in this order:
1. The deployed tag itself, if its current image is fixed but a redeploy does not pick it up because the recipe pins an older digest.
2. Later builds of the same version under a suffixed tag. Upstreams mark such rebuilds in many ways (`v6.1.0-1`, `v6.1.0-fix1`, `v6.1.0_r2`), and in semver `v6.1.0-1` even sorts before `v6.1.0`, so suffixes are not parsed. A candidate is the deployed tag, a separator, and a suffix containing a number; per-architecture copies (`-amd64`), pre-releases (`-preprod`, `-rc1`) and variants (`-fpm`, `-alpine3`) are left out by name. The build time then decides: a candidate must be built at least an hour after the registry's current image of the deployed tag. A pre-release is built before the release, and a variant in the same CI run as it, so neither passes; comparing with the deployed image instead would let a pre-release pushed after an old deployment through. Of the fixed candidates, the latest built is chosen. If the build times are unusable (see [Ordering images by version and build time](#ordering-images-by-version-and-build-time)), the action is "check by hand".
3. Newer versions of the same form: `v6.1.1` or `v6.2.0` for `v6.1.0`, but not `v6`, `latest` or `main`. As for releases, the newest is checked first; if it is fixed, the oldest fixed one is chosen.
Every candidate is checked against the advisory: whether a tag looks newer never decides whether it is fixed. The search is bounded to 20 rebuild and 10 newer-version candidates per image.
- For a recipe version range, the advisory's `fixed` version must be a released tag; otherwise the recipe must update.
- If the registry or the recipe cannot be read, the suggestion is kept and marked as unchecked.
- **Recipe names.** Recipes are shown relative to `git.coopcloud.tech/coop-cloud/`: `nextcloud` for the canonical recipe, `c4dt/nextcloud` for a fork on the same server, and the full path elsewhere. Advisories name recipes the same way, so `nextcloud` in an advisory does not match the fork `c4dt/nextcloud`, which has its own versions.
- **Exit code.** 0 when nothing fails, 1 otherwise.
- **Errors.** An unreachable server is listed as such and the run continues.
**Test.** On a test server, deploy one old nextcloud recipe version and one app with a stale image under a current tag. Write an advisory for each. Check that both are reported, that following the suggestions fixes them, and that a second `abra audit` is clean.
### v2: `abra audit update` for one vulnerability
Turns a single vulnerability into draft advisories, so the maintainer does not have to check every recipe by hand. There are two ways to describe the vulnerability.
**By CVE: `abra audit update --cve CVE-2026-12345`.** The CVE is looked up in [OSV.dev](https://osv.dev), which also covers GitHub advisories and package ecosystems such as Packagist, npm, PyPI, Debian and Alpine. This gives the affected packages and their fixed versions. Recipes are then matched in two ways:
- applications, such as nextcloud, through the `+UPSTREAM` part of the recipe tags;
- runtimes and libraries, such as php, through the image config (next paragraph).
**By minimum versions: `abra audit update --require php=8.5.12,8.4.27,8.3.36,8.2.35`.** One minimum version per release series. For every image in the catalogue that contains the library:
| Image contains | Result |
| --- | --- |
| A listed series, below its minimum (php 8.4.20) | Affected |
| A listed series, at or above its minimum (php 8.4.27) | Fixed |
| A series that is not listed (php 8.1.x) | Unsupported, proposed as a separate end-of-life advisory |
Libraries are found through environment variables in the image config, which images inherit from their base:
| Library | Variable |
| --- | --- |
| php | `PHP_VERSION` |
| node | `NODE_VERSION` |
| python | `PYTHON_VERSION` |
| nginx | `NGINX_VERSION` |
| postgres | `PG_VERSION` |
| redis | `REDIS_VERSION` |
An unknown library is an error that lists the known ones. Finding libraries through SBOMs comes later.
**Which recipe versions.** Every published version of each recipe that uses the library is checked, so the proposal gets a correct `affected` range. Image configs are a few KB and are cached by digest in `$ABRA_DIR/.cache/audit/images/`, so this is cheap after the first run. A digest never changes its content, so this cache never expires.
**Strategy.** For each affected recipe, the current image under the same tag is checked too:
- a newer recipe version uses a fixed image → `upgrade` to that version;
- the same tag now points to a fixed image → `refresh`, with the current digest in `fixed_digests`;
- nothing is fixed yet → `manual`.
**Output.** Draft advisory files in the local checkout, each shown as a diff to accept, edit or skip. v2 does not commit or open pull requests.
**Test.** Run `--require` with the versions of a recent php security release, and compare the proposals with a list made by hand. Run `--cve` with a recent nextcloud CVE and check the recipe range.
### v3: arguments and fixes
Makes the audit usable for single apps and for automation, and lets it apply fixes.
- **Subcommands.** `abra audit app`, `server`, `recipe`, `list` and `show`, plus the aliases `abra app audit` and `abra server audit`.
- **Options.** `--server`, `--recipe`, `--severity`, `--machine`, `--offline` and `--advisories`.
- **Fixes.** `--fix`, `--refresh-only`, `--no-input` and `--retries`, with verification after each fix and exit code 2 when verification fails.
- **Deploy warnings.** `abra app deploy` and `abra app upgrade` warn, without refusing, when the target version matches an advisory.
- **Platforms.** `env` conditions are checked per platform on mixed amd64 and arm64 swarms.
**Test.** Run `abra audit --fix` against the v1 test setup, with the registry blocked for one attempt (for example with a firewall rule), and check that the retry succeeds and that the exit code is right in each case.
### v4: more automatic updates
Moves advisory maintenance from "react to a CVE" to "be told about new problems".
- `abra audit update --sources`, run on a schedule in CI, with endoflife.date, OSV, GHSA, CISA KEV and docker-library/repo-info.
- Deduplication against existing advisories, `ignore.yml`, and `--pr`.
- Digest history and `fixed_from`, only if images with unusable build times (reproducible builds) turn out to matter.
- Optional extras: `--deep` scanning with Trivy or Grype, the lint rule for per-platform digests, a status column in `abra app list`, and `--report`.
- Moving the advisories repo to `coop-cloud/advisories`.
## Decisions
| Question | Decision |
| --- | --- |
| Who maintains the advisories, and how fast? | Linus Gasser, manually, in `c4dt/advisories`, for now |
| Should `--report` exist? | Not in the first versions |
| How long to cache the advisory list? | No cache: every invocation checks for the latest list |
| What does `abra app deploy` do when the target version matches an advisory? | It warns, and does not refuse |
## Open questions
- [ ] Do the registries we depend on keep untagged digests long enough to read their config later? This matters only for the digest history (v4), because the config of a running image can be read from the node. To find out, take old `php` digests from docker-library/repo-info and try to fetch their manifests from Docker Hub.
- [ ] Which images used by recipes are built reproducibly, so that their `Created` time is unusable? `abra audit update` could list them while scanning the catalogue.
+1
View File
@@ -99,6 +99,7 @@
go_1_26
gnumake
gopls
gettext
];
};
+11 -9
View File
@@ -9,6 +9,7 @@ require (
github.com/charmbracelet/bubbletea v1.3.10
github.com/charmbracelet/lipgloss v1.1.0
github.com/charmbracelet/log v1.0.0
github.com/compose-spec/compose-go/v2 v2.10.1
github.com/containers/image v3.0.2+incompatible
github.com/decentral1se/passgen v1.0.1
github.com/distribution/reference v0.6.0
@@ -21,14 +22,13 @@ require (
github.com/leonelquinteros/gotext v1.8.0
github.com/moby/sys/signal v0.7.1
github.com/moby/term v0.5.2
github.com/opencontainers/go-digest v1.0.0
github.com/opencontainers/image-spec v1.1.1
github.com/pkg/errors v0.9.1
github.com/schollz/progressbar/v3 v3.19.1
github.com/sirupsen/logrus v1.9.4
github.com/spf13/cobra v1.10.1
github.com/stretchr/testify v1.12.1
golang.org/x/sys v0.50.0
golang.org/x/term v0.48.0
golang.org/x/sys v0.49.0
golang.org/x/term v0.47.0
gopkg.in/yaml.v3 v3.0.1
gotest.tools/v3 v3.5.2
)
@@ -76,7 +76,6 @@ require (
github.com/go-logr/stdr v1.2.2 // indirect
github.com/go-viper/mapstructure/v2 v2.5.0 // indirect
github.com/golang/groupcache v0.0.0-20241129210726-2c02b8208cf8 // indirect
github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 // indirect
github.com/google/uuid v1.6.0 // indirect
github.com/gorilla/mux v1.8.1 // indirect
github.com/grpc-ecosystem/grpc-gateway/v2 v2.28.0 // indirect
@@ -91,6 +90,7 @@ require (
github.com/mattn/go-isatty v0.0.22 // indirect
github.com/mattn/go-localereader v0.0.1 // indirect
github.com/mattn/go-runewidth v0.0.21 // indirect
github.com/mattn/go-shellwords v1.0.12 // indirect
github.com/mgutz/ansi v0.0.0-20200706080929-d51e80ef957d // indirect
github.com/miekg/pkcs11 v1.1.1 // indirect
github.com/mitchellh/colorstring v0.0.0-20190213212951-d06e56a500db // indirect
@@ -106,6 +106,8 @@ require (
github.com/muesli/cancelreader v0.2.2 // indirect
github.com/muesli/termenv v0.16.0 // indirect
github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 // indirect
github.com/opencontainers/go-digest v1.0.0 // indirect
github.com/opencontainers/image-spec v1.1.1 // indirect
github.com/opencontainers/runc v1.1.13 // indirect
github.com/opencontainers/runtime-spec v1.1.0 // indirect
github.com/pjbgf/sha1cd v0.7.0 // indirect
@@ -115,15 +117,13 @@ require (
github.com/prometheus/procfs v0.20.1 // indirect
github.com/rivo/uniseg v0.4.7 // indirect
github.com/russross/blackfriday/v2 v2.1.0 // indirect
github.com/santhosh-tekuri/jsonschema/v6 v6.0.1 // indirect
github.com/sergi/go-diff v1.4.0 // indirect
github.com/sirupsen/logrus v1.9.4 // indirect
github.com/skeema/knownhosts v1.3.2 // indirect
github.com/spf13/pflag v1.0.10 // indirect
github.com/theupdateframework/notary v0.7.0 // indirect
github.com/xanzy/ssh-agent v0.3.3 // indirect
github.com/xeipuuv/gojsonpointer v0.0.0-20190905194746-02993c407bfb // indirect
github.com/xeipuuv/gojsonreference v0.0.0-20180127040603-bd5ef7bd5415 // indirect
github.com/xeipuuv/gojsonschema v1.2.0 // indirect
github.com/xhit/go-str2duration/v2 v2.1.0 // indirect
github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e // indirect
go.opentelemetry.io/auto/sdk v1.2.1 // indirect
go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.67.0 // indirect
@@ -139,9 +139,11 @@ require (
go.opentelemetry.io/proto/otlp v1.10.0 // indirect
go.yaml.in/yaml/v2 v2.4.4 // indirect
go.yaml.in/yaml/v3 v3.0.5 // indirect
go.yaml.in/yaml/v4 v4.0.0-rc.3 // indirect
golang.org/x/crypto v0.56.0 // indirect
golang.org/x/exp v0.0.0-20260410095643-746e56fc9e2f // indirect
golang.org/x/net v0.57.0 // indirect
golang.org/x/sync v0.22.0 // indirect
golang.org/x/text v0.41.0 // indirect
golang.org/x/time v0.15.0 // indirect
google.golang.org/genproto/googleapis/api v0.0.0-20260401024825-9d38bb4040a9 // indirect
+13 -11
View File
@@ -176,6 +176,8 @@ github.com/cncf/udpa/go v0.0.0-20191209042840-269d4d468f6f/go.mod h1:M8M6+tZqaGX
github.com/cncf/udpa/go v0.0.0-20201120205902-5459f2c99403/go.mod h1:WmhPx2Nbnhtbo57+VJT5O0JRkEi1Wbu0z5j0R8u5Hbk=
github.com/cncf/xds/go v0.0.0-20210312221358-fbca930ec8ed/go.mod h1:eXthEFrGJvWHgFFCl3hGmgk+/aYT6PnTQLykKQRLhEs=
github.com/cockroachdb/datadriven v0.0.0-20190809214429-80d97fb3cbaa/go.mod h1:zn76sxSg3SzpJ0PPJaLDCu+Bu0Lg3sKTORVIj19EIF8=
github.com/compose-spec/compose-go/v2 v2.10.1 h1:mFbXobojGRFIVi1UknrvaDAZ+PkJfyjqkA1yseh+vAU=
github.com/compose-spec/compose-go/v2 v2.10.1/go.mod h1:Ohac1SzhO/4fXXrzWIztIVB6ckmKBv1Nt5Z5mGVESUg=
github.com/containerd/aufs v0.0.0-20200908144142-dab0cbea06f4/go.mod h1:nukgQABAEopAHvB6j7cnP5zJ+/3aVcE7hCYqvIwAHyE=
github.com/containerd/aufs v0.0.0-20201003224125-76a6863f2989/go.mod h1:AkGGQs9NM2vtYHaUen+NljV0/baGCAPELGm2q9ZXpWU=
github.com/containerd/aufs v0.0.0-20210316121734-20793ff83c97/go.mod h1:kL5kd6KM5TzQjR79jljyi4olc1Vrx6XBlcyj3gNv2PU=
@@ -317,6 +319,8 @@ github.com/dgrijalva/jwt-go v0.0.0-20170104182250-a601269ab70c/go.mod h1:E3ru+11
github.com/dgrijalva/jwt-go v3.2.0+incompatible/go.mod h1:E3ru+11k8xSBh+hMPgOLZmtrrCbhqsmaPHjLKYnJCaQ=
github.com/distribution/reference v0.6.0 h1:0IXCQ5g4/QMHHkarYzh5l+u8T3t73zM5QvfrDyIgxBk=
github.com/distribution/reference v0.6.0/go.mod h1:BbU0aIcezP1/5jX/8MP0YiH4SdvB5Y4f/wlDRiLyi3E=
github.com/dlclark/regexp2 v1.11.0 h1:G/nrcoOa7ZXlpoa/91N3X7mM3r8eIlMBBJZvsz/mxKI=
github.com/dlclark/regexp2 v1.11.0/go.mod h1:DHkYz0B9wPfa6wondMfaivmHpzrQ3v9q8cnmRbL6yW8=
github.com/dnaeon/go-vcr v1.0.1/go.mod h1:aBB1+wY4s93YsC3HHjMBMrwTj2R9FHDzUr9KyGc8n1E=
github.com/docker/cli v0.0.0-20191017083524-a8ff7f821017/go.mod h1:JLrzqnKDaYBop7H2jaqPtU4hHvMKP+vjCwu2uszcLI8=
github.com/docker/distribution v0.0.0-20190905152932-14b96e55d84c/go.mod h1:0+TTO4EOBfRPhZXAeF1Vu+W3hHZ8eLp8PgKVZlcvtFY=
@@ -500,8 +504,6 @@ github.com/google/pprof v0.0.0-20200212024743-f11f1df84d12/go.mod h1:ZgVRPoUq/hf
github.com/google/pprof v0.0.0-20200229191704-1ebb73c60ed3/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM=
github.com/google/pprof v0.0.0-20200430221834-fc25d7d30c6d/go.mod h1:ZgVRPoUq/hfqzAqh7sHMqb3I9Rq5C59dIz2SbBwJ4eM=
github.com/google/renameio v0.1.0/go.mod h1:KWCgfxg9yswjAJkECMjeO8J8rahYeXnNhOm40UhjYkI=
github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510 h1:El6M4kTTCOh6aBiKaUGG7oYTSPP8MxqL4YI3kZKwcP4=
github.com/google/shlex v0.0.0-20191202100458-e7afc7fbc510/go.mod h1:pupxD2MaaD3pAXIBCelhxNneeOaAeabZDe5s4K6zSpQ=
github.com/google/uuid v1.0.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/google/uuid v1.1.1/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
github.com/google/uuid v1.1.2/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
@@ -629,6 +631,7 @@ github.com/mattn/go-runewidth v0.0.21 h1:jJKAZiQH+2mIinzCJIaIG9Be1+0NR+5sz/lYEEj
github.com/mattn/go-runewidth v0.0.21/go.mod h1:XBkDxAl56ILZc9knddidhrOlY5R/pDhgLpndooCuJAs=
github.com/mattn/go-shellwords v1.0.3/go.mod h1:3xCvwCdWdlDJUrvuMn7Wuy9eWs4pE8vqg+NOMyg4B2o=
github.com/mattn/go-shellwords v1.0.6/go.mod h1:3xCvwCdWdlDJUrvuMn7Wuy9eWs4pE8vqg+NOMyg4B2o=
github.com/mattn/go-shellwords v1.0.12 h1:M2zGm7EW6UQJvDeQxo4T51eKPurbeFbe8WtebGE2xrk=
github.com/mattn/go-shellwords v1.0.12/go.mod h1:EZzvwXDESEeg03EKmM+RmDnNOPKG4lLtQsUlTZDWQ8Y=
github.com/mattn/go-sqlite3 v1.6.0/go.mod h1:FPy6KqzDD04eiIsT53CuJW3U88zkxoIYsOqkbpncsNc=
github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0=
@@ -803,6 +806,8 @@ github.com/russross/blackfriday/v2 v2.0.1/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQD
github.com/russross/blackfriday/v2 v2.1.0 h1:JIOH55/0cWyOuilr9/qlrm0BSXldqnqwMsf35Ld67mk=
github.com/russross/blackfriday/v2 v2.1.0/go.mod h1:+Rmxgy9KzJVeS9/2gXHxylqXiyQDYRxCVz55jmeOWTM=
github.com/safchain/ethtool v0.0.0-20190326074333-42ed695e3de8/go.mod h1:Z0q5wiBQGYcxhMZ6gUqHn6pYNLypFAvaL3UvgZLR0U4=
github.com/santhosh-tekuri/jsonschema/v6 v6.0.1 h1:PKK9DyHxif4LZo+uQSgXNqs0jj5+xZwwfKHgph2lxBw=
github.com/santhosh-tekuri/jsonschema/v6 v6.0.1/go.mod h1:JXeL+ps8p7/KNMjDQk3TCwPpBy0wYklyWTfbkIzdIFU=
github.com/satori/go.uuid v1.2.0/go.mod h1:dA0hQrYB0VpLJoorglMZABFdXlWrHn1NEOzdhQKdks0=
github.com/schollz/progressbar/v3 v3.19.1 h1:iv8BgwOvdML/S3p84uBpy/IMigv4U9594vPZYa2EdrU=
github.com/schollz/progressbar/v3 v3.19.1/go.mod h1:LFL7jqimKxfhero4K1eCkUr/6R39AgQeiPCJtlTWIW8=
@@ -883,13 +888,10 @@ github.com/willf/bitset v1.1.11/go.mod h1:83CECat5yLh5zVOf4P1ErAgKA5UDvKtgyUABdr
github.com/xanzy/ssh-agent v0.3.3 h1:+/15pJfg/RsTxqYcX6fHqOXZwwMP+2VyYWJeWM2qQFM=
github.com/xanzy/ssh-agent v0.3.3/go.mod h1:6dzNDKs0J9rVPHPhaGCukekBHKqfl+L3KghI1Bc68Uw=
github.com/xeipuuv/gojsonpointer v0.0.0-20180127040702-4e3ac2762d5f/go.mod h1:N2zxlSyiKSe5eX1tZViRH5QA0qijqEDrYZiPEAiq3wU=
github.com/xeipuuv/gojsonpointer v0.0.0-20190905194746-02993c407bfb h1:zGWFAtiMcyryUHoUjUJX0/lt1H2+i2Ka2n+D3DImSNo=
github.com/xeipuuv/gojsonpointer v0.0.0-20190905194746-02993c407bfb/go.mod h1:N2zxlSyiKSe5eX1tZViRH5QA0qijqEDrYZiPEAiq3wU=
github.com/xeipuuv/gojsonreference v0.0.0-20180127040603-bd5ef7bd5415 h1:EzJWgHovont7NscjpAxXsDA8S8BMYve8Y5+7cuRE7R0=
github.com/xeipuuv/gojsonreference v0.0.0-20180127040603-bd5ef7bd5415/go.mod h1:GwrjFmJcFw6At/Gs6z4yjiIwzuJ1/+UwLxMQDVQXShQ=
github.com/xeipuuv/gojsonschema v0.0.0-20180618132009-1d523034197f/go.mod h1:5yf86TLmAcydyeJq5YvxkGPE2fm/u4myDekKRoLuqhs=
github.com/xeipuuv/gojsonschema v1.2.0 h1:LhYJRs+L4fBtjZUfuSZIKGeVu0QRy8e5Xi7D17UxZ74=
github.com/xeipuuv/gojsonschema v1.2.0/go.mod h1:anYRn/JVcOK2ZgGU+IjEV4nwlhoK5sQluxsYJ78Id3Y=
github.com/xhit/go-str2duration/v2 v2.1.0 h1:lxklc02Drh6ynqX+DdPyp5pCKLUQpRT8bp8Ydu2Bstc=
github.com/xhit/go-str2duration/v2 v2.1.0/go.mod h1:ohY8p+0f07DiV6Em5LKB0s2YpLtXVyJfNt1+BlmyAsU=
github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2/go.mod h1:UETIi67q53MR2AWcXfiuqkDkRtnGDLqkBTpCHuJHxtU=
github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e h1:JVG44RsyaB9T2KIHavMF/ppJZNG9ZpyihvCd0w101no=
github.com/xo/terminfo v0.0.0-20220910002029-abceb7e1c41e/go.mod h1:RbqR21r5mrJuqunuUZ/Dhy/avygyECGrLceyNeo4LiM=
@@ -945,6 +947,8 @@ go.yaml.in/yaml/v2 v2.4.4/go.mod h1:gMZqIpDtDqOfM0uNfy0SkpRhvUryYH0Z6wdMYcacYXQ=
go.yaml.in/yaml/v3 v3.0.4/go.mod h1:DhzuOOF2ATzADvBadXxruRBLzYTpT36CKvDb3+aBEFg=
go.yaml.in/yaml/v3 v3.0.5 h1:N6y/pJk8buWs9NY5ERU2HSMfm+IuD/OtfdAnq6kESPw=
go.yaml.in/yaml/v3 v3.0.5/go.mod h1:HVTZu1O7/Vkt2N+BFy8Zza+lnLsABggaTM2ZpNIGuKg=
go.yaml.in/yaml/v4 v4.0.0-rc.3 h1:3h1fjsh1CTAPjW7q/EMe+C8shx5d8ctzZTrLcs/j8Go=
go.yaml.in/yaml/v4 v4.0.0-rc.3/go.mod h1:aZqd9kCMsGL7AuUv/m/PvWLdg5sjJsZ4oHDEnfPPfY0=
golang.org/x/crypto v0.0.0-20171113213409-9f005a07e0d3/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
golang.org/x/crypto v0.0.0-20181009213950-7c1a557ab941/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
@@ -1059,6 +1063,8 @@ golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJ
golang.org/x/sync v0.0.0-20201207232520-09787c993a3a/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20210220032951-036812b2e83c/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.0.0-20220722155255-886fb9371eb4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
golang.org/x/sync v0.22.0 h1:SZjpbeLmrCk4xhRSZFNZW5gFUeCeFgjekvI/+gfScek=
golang.org/x/sync v0.22.0/go.mod h1:9xrNwdLfx4jkKbNva9FpL6vEN7evnE43NNNJQ2LF3+0=
golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
@@ -1138,15 +1144,11 @@ golang.org/x/sys v0.0.0-20220715151400-c0bba94af5f8/go.mod h1:oPkhp1MJrh7nUepCBc
golang.org/x/sys v0.0.0-20220722155257-8c9f86f7a55f/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
golang.org/x/sys v0.49.0 h1:XbzkgYJHdqh/8m2Uu0W/dQv8nktxx4BFHp1M0gROTXA=
golang.org/x/sys v0.49.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
golang.org/x/sys v0.50.0 h1:dkKZEFTEKaLr1sjhs0WiKqoi8Jj5nHbS3Njmx1UoMJs=
golang.org/x/sys v0.50.0/go.mod h1:hNLxWAXmnKAxqDtdwIYC4bM9oQPEecfsnNMuSxOs3og=
golang.org/x/term v0.0.0-20201117132131-f5c789dd3221/go.mod h1:Nr5EML6q2oocZ2LXRh80K7BxOlk5/8JxuGnuhpl+muw=
golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
golang.org/x/term v0.0.0-20210927222741-03fcf44c2211/go.mod h1:jbD1KX2456YbFQfuXm/mYQcufACuNUgVhRMnK/tPxf8=
golang.org/x/term v0.47.0 h1:cpyOwQkr2Nzxe2p6KnrTG7Ei2wtzjsuSjhpXXepZ/BM=
golang.org/x/term v0.47.0/go.mod h1:efAfAuG6RKABvg1ArGebRkvX6LXul14SM67o2e4mUCY=
golang.org/x/term v0.48.0 h1:G6UU1eiSB8nl3DzeUGt7Mg+2SctI8tPS6Z5/mBt6cOk=
golang.org/x/term v0.48.0/go.mod h1:BRDLHCZr2SUjIeBHHpRyQeC9xzSAVbzY2rvKgafnHf4=
golang.org/x/text v0.0.0-20170915032832-14c0d48ead0c/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
+65 -120
View File
@@ -18,10 +18,10 @@ import (
"coopcloud.tech/abra/pkg/recipe"
"coopcloud.tech/abra/pkg/upstream/convert"
"coopcloud.tech/abra/pkg/upstream/stack"
composeGoTypes "github.com/compose-spec/compose-go/v2/types"
"coopcloud.tech/abra/pkg/log"
loader "coopcloud.tech/abra/pkg/upstream/stack"
composetypes "github.com/docker/cli/cli/compose/types"
"github.com/docker/docker/api/types/filters"
"github.com/schollz/progressbar/v3"
)
@@ -179,8 +179,8 @@ func (a App) Filters(appendServiceNames, exactMatch bool, services ...string) (f
return filters, err
}
opts := stack.Deploy{Composefiles: composeFiles}
compose, err := GetAppComposeConfig(a.Recipe.Name, opts, a.Env)
a.Env["STACK_NAME"] = a.StackName()
compose, err := GetAppComposeConfig(composeFiles, a.Env)
if err != nil {
return filters, err
}
@@ -260,30 +260,12 @@ func ReadAppEnvFile(appFile AppFile, name AppName) (App, error) {
func NewApp(env envfile.AppEnv, name string, appFile AppFile) (App, error) {
domain := env["DOMAIN"]
typeVar, hasType := env["TYPE"]
recipeVar, hasRecipe := env["RECIPE"]
if !hasType && !hasRecipe {
return App{}, errors.New(i18n.G("%s is missing the TYPE env var?", name))
}
// The recipe identity is taken from RECIPE when present (it records the
// canonical git source for externally-sourced recipes), otherwise from
// TYPE. The deployed version is always carried by TYPE when a TYPE line
// exists; only legacy apps that set RECIPE alone carry it there. RECIPE
// itself must stay a bare source pointer, never a versioned value.
identity := typeVar
if hasRecipe {
identity = recipeVar
}
versionSource := identity
if hasType {
versionSource = typeVar
}
recipeName, _, _ := strings.Cut(identity, ":")
if _, version, found := strings.Cut(versionSource, ":"); found && version != "" {
recipeName = fmt.Sprintf("%s:%s", recipeName, version)
recipeName, exists := env["RECIPE"]
if !exists {
recipeName, exists = env["TYPE"]
if !exists {
return App{}, errors.New(i18n.G("%s is missing the TYPE env var?", name))
}
}
return App{
@@ -351,8 +333,8 @@ func GetAppServiceNames(appName string) ([]string, error) {
return serviceNames, err
}
opts := stack.Deploy{Composefiles: composeFiles}
compose, err := GetAppComposeConfig(app.Recipe.Name, opts, app.Env)
app.Env["STACK_NAME"] = app.StackName()
compose, err := GetAppComposeConfig(composeFiles, app.Env)
if err != nil {
return serviceNames, err
}
@@ -410,15 +392,11 @@ func TemplateAppEnvSample(r recipe.Recipe, appName, server, domain string) error
newContents := strings.Replace(
string(read),
fmt.Sprintf("%s.example.com", r.ShortName()),
fmt.Sprintf("%s.example.com", r.Name),
domain,
-1,
)
if strings.Contains(r.Name, "/") {
newContents = injectRecipeLine(newContents, r.Name)
}
err = os.WriteFile(appEnvPath, []byte(newContents), 0)
if err != nil {
return err
@@ -429,37 +407,6 @@ func TemplateAppEnvSample(r recipe.Recipe, appName, server, domain string) error
return nil
}
// injectRecipeLine ensures the env file carries a RECIPE=<canonical name>
// line so a downstream `abra app deploy` (potentially on another machine)
// can re-fetch the recipe from its original git source. If a RECIPE= line
// already exists in the copied .env.sample it is replaced; otherwise a new
// line is inserted immediately after the TYPE= line, or appended at the
// end if no TYPE= line is present.
func injectRecipeLine(contents, canonicalName string) string {
lines := strings.Split(contents, "\n")
for i, line := range lines {
trimmed := strings.TrimLeft(line, " \t")
if strings.HasPrefix(trimmed, "RECIPE=") {
lines[i] = "RECIPE=" + canonicalName
return strings.Join(lines, "\n")
}
}
for i, line := range lines {
trimmed := strings.TrimLeft(line, " \t")
if strings.HasPrefix(trimmed, "TYPE=") {
out := make([]string, 0, len(lines)+1)
out = append(out, lines[:i+1]...)
out = append(out, "RECIPE="+canonicalName)
out = append(out, lines[i+1:]...)
return strings.Join(out, "\n")
}
}
if contents != "" && !strings.HasSuffix(contents, "\n") {
contents += "\n"
}
return contents + "RECIPE=" + canonicalName + "\n"
}
// SanitiseAppName makes a app name usable with Docker by replacing illegal
// characters.
func SanitiseAppName(name string) string {
@@ -543,13 +490,18 @@ func GetAppStatuses(apps []App, MachineReadable bool) (map[string]map[string]str
// GetAppComposeConfig retrieves a compose specification for a recipe. This
// specification is the result of a merge of all the compose.**.yml files in
// the recipe repository.
func GetAppComposeConfig(recipe string, opts stack.Deploy, appEnv envfile.AppEnv) (*composetypes.Config, error) {
compose, err := loader.LoadComposefile(opts, appEnv)
func GetAppComposeConfig(composeFiles []string, appEnv envfile.AppEnv) (*composeGoTypes.Project, error) {
compose, err := loader.LoadCompose(loader.LoadConf{ComposeFiles: composeFiles, AppEnv: appEnv})
if err != nil {
return &composetypes.Config{}, err
return &composeGoTypes.Project{}, err
}
log.Debug(i18n.G("retrieved %s for %s", compose.Filename, recipe))
recipeName, exists := appEnv["RECIPE"]
if !exists {
recipeName, _ = appEnv["TYPE"]
}
log.Debug(i18n.G("retrieved %s for %s", compose.Name, recipeName))
return compose, nil
}
@@ -557,7 +509,7 @@ func GetAppComposeConfig(recipe string, opts stack.Deploy, appEnv envfile.AppEnv
// ExposeAllEnv exposes all env variables to the app container
func ExposeAllEnv(
stackName string,
compose *composetypes.Config,
compose *composeGoTypes.Project,
appEnv envfile.AppEnv,
toDeployVersion string) {
for _, service := range compose.Services {
@@ -567,22 +519,10 @@ func ExposeAllEnv(
_, exists := service.Environment[k]
if !exists {
value := v
switch k {
case "TYPE":
if k == "TYPE" || k == "RECIPE" {
// NOTE(d1): don't use the wrong version from the app env
// since we are deploying a new version. Keep the
// recipe name already recorded in TYPE and only
// swap in the version being deployed.
name, _, _ := strings.Cut(v, ":")
if _, version, found := strings.Cut(toDeployVersion, ":"); found {
value = fmt.Sprintf("%s:%s", name, version)
} else {
value = name
}
case "RECIPE":
// RECIPE records the canonical git source only; the
// version is carried by TYPE, so keep it bare here.
value, _, _ = strings.Cut(toDeployVersion, ":")
// since we are deploying a new version
value = toDeployVersion
}
service.Environment[k] = &value
log.Debug(i18n.G("%s: %s: %s", stackName, k, value))
@@ -707,55 +647,60 @@ func (a App) WriteRecipeVersion(version string, dryRun bool) error {
if err != nil {
return err
}
defer file.Close()
var (
lines []string
hasType bool
scanner = bufio.NewScanner(file)
dirtyVersion string
skipped bool
lines []string
scanner = bufio.NewScanner(file)
)
for scanner.Scan() {
line := scanner.Text()
lines = append(lines, line)
if strings.HasPrefix(line, "TYPE=") {
hasType = true
}
}
if err := scanner.Err(); err != nil {
file.Close()
log.Fatal(err)
}
file.Close()
// The recipe version is carried by a single field: TYPE when present (the
// standard recipe key), otherwise the legacy RECIPE key. When both exist
// (git-URL recipes) RECIPE stays a bare source pointer and only TYPE is
// versioned, so the two can never drift apart.
targetPrefix := "RECIPE="
if hasType {
targetPrefix = "TYPE="
}
for i, line := range lines {
if !strings.HasPrefix(line, targetPrefix) || strings.HasPrefix(line, "#") {
if !strings.HasPrefix(line, "RECIPE=") && !strings.HasPrefix(line, "TYPE=") {
lines = append(lines, line)
continue
}
name, _, _ := strings.Cut(line, ":")
lines[i] = fmt.Sprintf("%s:%s", name, version)
if strings.HasPrefix(line, "#") {
lines = append(lines, line)
continue
}
if strings.Contains(line, version) && !a.Recipe.Dirty && !strings.HasSuffix(line, config.DIRTY_DEFAULT) {
skipped = true
lines = append(lines, line)
continue
}
splitted := strings.Split(line, ":")
line = fmt.Sprintf("%s:%s", splitted[0], version)
lines = append(lines, line)
}
if dryRun {
log.Debug(i18n.G("skipping writing version %s because dry run", version))
return nil
}
if err := os.WriteFile(a.Path, []byte(strings.Join(lines, "\n")), os.ModePerm); err != nil {
if err := scanner.Err(); err != nil {
log.Fatal(err)
}
log.Debug(i18n.G("version %s saved to %s.env", version, a.Domain))
if a.Recipe.Dirty && dirtyVersion != "" {
version = dirtyVersion
}
if !dryRun {
if err := os.WriteFile(a.Path, []byte(strings.Join(lines, "\n")), os.ModePerm); err != nil {
log.Fatal(err)
}
} else {
log.Debug(i18n.G("skipping writing version %s because dry run", version))
}
if !skipped {
log.Debug(i18n.G("version %s saved to %s.env", version, a.Domain))
} else {
log.Debug(i18n.G("skipping version %s write as already exists in %s.env", version, a.Domain))
}
return nil
}
-56
View File
@@ -3,8 +3,6 @@ package app_test
import (
"encoding/json"
"fmt"
"os"
"path/filepath"
"reflect"
"testing"
@@ -273,60 +271,6 @@ func TestWriteRecipeVersionOverwrite(t *testing.T) {
assert.Equal(t, "foo", app.Recipe.EnvVersion)
}
// TestWriteRecipeVersionSingleField ensures the version is written to exactly
// one field: TYPE when present (even alongside a RECIPE source pointer, as with
// git-URL recipes), otherwise the legacy RECIPE field. RECIPE must stay bare
// when a TYPE line exists so the two can never drift apart.
func TestWriteRecipeVersionSingleField(t *testing.T) {
cases := []struct {
name string
input string
want string
}{
{
name: "git recipe: TYPE and RECIPE both present",
input: "TYPE=myrecipe\nRECIPE=git.example.com/user/myrecipe\nDOMAIN=x\n",
want: "TYPE=myrecipe:1.2.3+4.5.6\nRECIPE=git.example.com/user/myrecipe\nDOMAIN=x",
},
{
name: "git recipe: re-versioning replaces only TYPE",
input: "TYPE=myrecipe:0.1.0+1.0.0\nRECIPE=git.example.com/user/myrecipe\nDOMAIN=x\n",
want: "TYPE=myrecipe:1.2.3+4.5.6\nRECIPE=git.example.com/user/myrecipe\nDOMAIN=x",
},
{
name: "legacy recipe: only RECIPE present",
input: "RECIPE=test_recipe\nDOMAIN=x\n",
want: "RECIPE=test_recipe:1.2.3+4.5.6\nDOMAIN=x",
},
{
name: "catalogue recipe: only TYPE present",
input: "TYPE=nextcloud\nDOMAIN=x\n",
want: "TYPE=nextcloud:1.2.3+4.5.6\nDOMAIN=x",
},
}
for _, tc := range cases {
t.Run(tc.name, func(t *testing.T) {
p := filepath.Join(t.TempDir(), "app.env")
if err := os.WriteFile(p, []byte(tc.input), 0o644); err != nil {
t.Fatal(err)
}
app := appPkg.App{Path: p, Domain: "x"}
if err := app.WriteRecipeVersion("1.2.3+4.5.6", false); err != nil {
t.Fatal(err)
}
got, err := os.ReadFile(p)
if err != nil {
t.Fatal(err)
}
assert.Equal(t, tc.want, string(got))
})
}
}
func TestWriteRecipeVersionUnknown(t *testing.T) {
test.Setup()
t.Cleanup(func() { test.Teardown() })
+7 -7
View File
@@ -7,12 +7,12 @@ import (
"coopcloud.tech/abra/pkg/i18n"
"coopcloud.tech/abra/pkg/log"
composetypes "github.com/docker/cli/cli/compose/types"
composeGoTypes "github.com/compose-spec/compose-go/v2/types"
)
// SetRecipeLabel adds the label 'coop-cloud.${STACK_NAME}.recipe=${RECIPE}' to the app container
// to signal which recipe is connected to the deployed app
func SetRecipeLabel(compose *composetypes.Config, stackName string, recipe string) {
func SetRecipeLabel(compose *composeGoTypes.Project, stackName string, recipe string) {
for _, service := range compose.Services {
if service.Name == "app" {
log.Debug(i18n.G("set recipe label 'coop-cloud.%s.recipe' to %s for %s", stackName, recipe, stackName))
@@ -24,7 +24,7 @@ func SetRecipeLabel(compose *composetypes.Config, stackName string, recipe strin
// SetChaosLabel adds the label 'coop-cloud.${STACK_NAME}.chaos=true/false' to the app container
// to signal if the app is deployed in chaos mode
func SetChaosLabel(compose *composetypes.Config, stackName string, chaos bool) {
func SetChaosLabel(compose *composeGoTypes.Project, stackName string, chaos bool) {
for _, service := range compose.Services {
if service.Name == "app" {
log.Debug(i18n.G("set label 'coop-cloud.%s.chaos' to %v for %s", stackName, chaos, stackName))
@@ -35,7 +35,7 @@ func SetChaosLabel(compose *composetypes.Config, stackName string, chaos bool) {
}
// SetChaosVersionLabel adds the label 'coop-cloud.${STACK_NAME}.chaos-version=$(GIT_COMMIT)' to the app container
func SetChaosVersionLabel(compose *composetypes.Config, stackName string, chaosVersion string) {
func SetChaosVersionLabel(compose *composeGoTypes.Project, stackName string, chaosVersion string) {
for _, service := range compose.Services {
if service.Name == "app" {
log.Debug(i18n.G("set label 'coop-cloud.%s.chaos-version' to %v for %s", stackName, chaosVersion, stackName))
@@ -45,7 +45,7 @@ func SetChaosVersionLabel(compose *composetypes.Config, stackName string, chaosV
}
}
func SetVersionLabel(compose *composetypes.Config, stackName string, version string) {
func SetVersionLabel(compose *composeGoTypes.Project, stackName string, version string) {
for _, service := range compose.Services {
if service.Name == "app" {
log.Debug(i18n.G("set label 'coop-cloud.%s.version' to %v for %s", stackName, version, stackName))
@@ -56,7 +56,7 @@ func SetVersionLabel(compose *composetypes.Config, stackName string, version str
}
// GetLabel reads docker labels in the format of "coop-cloud.${STACK_NAME}.${LABEL}" from the local compose files
func GetLabel(compose *composetypes.Config, stackName string, label string) string {
func GetLabel(compose *composeGoTypes.Project, stackName string, label string) string {
for _, service := range compose.Services {
if service.Name == "app" {
labelKey := fmt.Sprintf("coop-cloud.%s.%s", stackName, label)
@@ -73,7 +73,7 @@ func GetLabel(compose *composetypes.Config, stackName string, label string) stri
// GetTimeoutFromLabel reads the timeout value from docker label
// `coop-cloud.${STACK_NAME}.timeout=...` if present. A value is present if the
// operator uses a `TIMEOUT=...` in their app env.
func GetTimeoutFromLabel(compose *composetypes.Config, stackName string) (int, error) {
func GetTimeoutFromLabel(compose *composeGoTypes.Project, stackName string) (int, error) {
var timeout int
if timeoutLabel := GetLabel(compose, stackName, "timeout"); timeoutLabel != "" {
+2 -10
View File
@@ -6,7 +6,6 @@ import (
appPkg "coopcloud.tech/abra/pkg/app"
"coopcloud.tech/abra/pkg/test"
testPkg "coopcloud.tech/abra/pkg/test"
stack "coopcloud.tech/abra/pkg/upstream/stack"
"github.com/stretchr/testify/assert"
)
@@ -40,15 +39,8 @@ func TestGetTimeoutFromLabel(t *testing.T) {
t.Fatal(err)
}
deployOpts := stack.Deploy{
Composefiles: composeFiles,
Namespace: app.StackName(),
Prune: false,
ResolveImage: stack.ResolveImageAlways,
Detach: false,
}
compose, err := appPkg.GetAppComposeConfig(app.Name, deployOpts, app.Env)
app.Env["STACK_NAME"] = app.StackName()
compose, err := appPkg.GetAppComposeConfig(composeFiles, app.Env)
if err != nil {
t.Fatal(err)
}
-30
View File
@@ -1,30 +0,0 @@
package app
import "testing"
func TestInjectRecipeLineAfterType(t *testing.T) {
in := "DOMAIN=example.com\nTYPE=foo\nVERSION=1.0\n"
want := "DOMAIN=example.com\nTYPE=foo\nRECIPE=org/foo\nVERSION=1.0\n"
got := injectRecipeLine(in, "org/foo")
if got != want {
t.Errorf("injectRecipeLine inserted RECIPE in wrong position\nwant:\n%q\ngot:\n%q", want, got)
}
}
func TestInjectRecipeLineReplacesExisting(t *testing.T) {
in := "TYPE=foo\nRECIPE=old\nDOMAIN=example.com\n"
want := "TYPE=foo\nRECIPE=org/foo\nDOMAIN=example.com\n"
got := injectRecipeLine(in, "org/foo")
if got != want {
t.Errorf("injectRecipeLine should replace existing RECIPE line\nwant:\n%q\ngot:\n%q", want, got)
}
}
func TestInjectRecipeLineNoTypeAppends(t *testing.T) {
in := "DOMAIN=example.com\n"
want := "DOMAIN=example.com\nRECIPE=org/foo\n"
got := injectRecipeLine(in, "org/foo")
if got != want {
t.Errorf("injectRecipeLine should append when TYPE is missing\nwant:\n%q\ngot:\n%q", want, got)
}
}
-310
View File
@@ -1,310 +0,0 @@
// Package audit checks deployed apps against a curated list of advisories.
package audit
import (
"errors"
"fmt"
"io/fs"
"os"
"path/filepath"
"slices"
"sort"
"strings"
"time"
"coopcloud.tech/abra/pkg/i18n"
"gopkg.in/yaml.v3"
)
// Advisory statuses.
const (
StatusDraft = "draft"
StatusPublished = "published"
StatusWithdrawn = "withdrawn"
)
// Fix strategies.
const (
StrategyUpgrade = "upgrade"
StrategyRefresh = "refresh"
StrategyManual = "manual"
)
var severities = []string{"low", "medium", "high", "critical"}
// Advisory describes one known problem and which recipe versions and images
// it affects.
type Advisory struct {
ID string `yaml:"id"`
Title string `yaml:"title"`
Severity string `yaml:"severity"`
Status string `yaml:"status"`
Refs []string `yaml:"refs"`
Published string `yaml:"published"`
Recipes map[string]RecipeRange `yaml:"recipes"`
Images []ImageRule `yaml:"images"`
Fix Fix `yaml:"fix"`
// Path is the file the advisory was read from.
Path string `yaml:"-"`
}
// RecipeRange gives the affected recipe versions of one recipe.
type RecipeRange struct {
Affected string `yaml:"affected"`
Fixed string `yaml:"fixed"`
affected Range
}
// ImageRule describes which images are affected. An image that matches
// Match is fixed if ANY of the fixing conditions holds. See Evaluate.
type ImageRule struct {
Match string `yaml:"match"`
Env map[string]string `yaml:"env"`
Version string `yaml:"version"`
CreatedAfter string `yaml:"created_after"`
FixedDigests []string `yaml:"fixed_digests"`
AffectedDigests []string `yaml:"affected_digests"`
// Skipped lists the fields this version of abra does not understand.
Skipped []string `yaml:"-"`
env map[string]Range
version *Range
createdAfter *time.Time
}
// Fix describes how to fix the advisory.
type Fix struct {
Strategy string `yaml:"strategy"`
Notes string `yaml:"notes"`
}
var (
knownTopLevel = []string{"id", "title", "severity", "status", "refs", "published", "recipes", "images", "fix"}
knownRecipe = []string{"affected", "fixed"}
knownImage = []string{"match", "env", "version", "created_after", "fixed_digests", "affected_digests"}
knownFix = []string{"strategy", "notes"}
)
// LoadAdvisories reads every *.yml / *.yaml file below dir. Advisories that
// cannot be parsed are skipped and reported as warnings, so that one broken
// file does not stop the audit. Fields that this version of abra does not
// know are skipped with a warning, so newer advisories do not break older
// abra versions.
func LoadAdvisories(dir string) ([]Advisory, []string, error) {
var advisories []Advisory
var warnings []string
if _, err := os.Stat(dir); err != nil {
return nil, nil, errors.New(i18n.G("no advisories found: %s", err))
}
err := filepath.WalkDir(dir, func(path string, d fs.DirEntry, err error) error {
if err != nil {
return err
}
if d.IsDir() {
return nil
}
ext := filepath.Ext(path)
if ext != ".yml" && ext != ".yaml" {
return nil
}
data, err := os.ReadFile(path)
if err != nil {
return err
}
adv, warns, err := ParseAdvisory(data)
rel, _ := filepath.Rel(dir, path)
for _, w := range warns {
warnings = append(warnings, fmt.Sprintf("%s: %s", rel, w))
}
if err != nil {
warnings = append(warnings, i18n.G("%s: skipping advisory: %s", rel, err))
return nil
}
adv.Path = path
advisories = append(advisories, adv)
return nil
})
if err != nil {
return nil, warnings, err
}
sort.Slice(advisories, func(i, j int) bool { return advisories[i].ID < advisories[j].ID })
return advisories, warnings, nil
}
// ParseAdvisory parses and validates one advisory.
func ParseAdvisory(data []byte) (Advisory, []string, error) {
var adv Advisory
var warnings []string
var root yaml.Node
if err := yaml.Unmarshal(data, &root); err != nil {
return adv, nil, err
}
if len(root.Content) == 0 {
return adv, nil, errors.New(i18n.G("empty file"))
}
doc := root.Content[0]
if doc.Kind != yaml.MappingNode {
return adv, nil, errors.New(i18n.G("advisory is not a mapping"))
}
unknown := func(n *yaml.Node, known []string, where string) []string {
var skipped []string
if n == nil || n.Kind != yaml.MappingNode {
return nil
}
for i := 0; i+1 < len(n.Content); i += 2 {
key := n.Content[i].Value
if !slices.Contains(known, key) {
skipped = append(skipped, key)
warnings = append(warnings, i18n.G("unknown field %s%s skipped", where, key))
}
}
return skipped
}
unknown(doc, knownTopLevel, "")
if n := mapValue(doc, "recipes"); n != nil && n.Kind == yaml.MappingNode {
for i := 0; i+1 < len(n.Content); i += 2 {
unknown(n.Content[i+1], knownRecipe, fmt.Sprintf("recipes.%s.", n.Content[i].Value))
}
}
if n := mapValue(doc, "fix"); n != nil {
unknown(n, knownFix, "fix.")
}
if err := doc.Decode(&adv); err != nil {
return adv, warnings, err
}
if n := mapValue(doc, "images"); n != nil && n.Kind == yaml.SequenceNode {
for i, img := range n.Content {
if i < len(adv.Images) {
adv.Images[i].Skipped = unknown(img, knownImage, fmt.Sprintf("images[%d].", i))
}
}
}
if err := adv.validate(); err != nil {
return adv, warnings, err
}
return adv, warnings, nil
}
func mapValue(n *yaml.Node, key string) *yaml.Node {
if n == nil || n.Kind != yaml.MappingNode {
return nil
}
for i := 0; i+1 < len(n.Content); i += 2 {
if n.Content[i].Value == key {
return n.Content[i+1]
}
}
return nil
}
func (a *Advisory) validate() error {
if a.ID == "" {
return errors.New(i18n.G("missing id"))
}
a.Severity = strings.ToLower(a.Severity)
if a.Severity != "" && !slices.Contains(severities, a.Severity) {
return errors.New(i18n.G("%s: unknown severity %s", a.ID, a.Severity))
}
a.Status = strings.ToLower(a.Status)
if a.Status == "" {
a.Status = StatusDraft
}
if !slices.Contains([]string{StatusDraft, StatusPublished, StatusWithdrawn}, a.Status) {
return errors.New(i18n.G("%s: unknown status %s", a.ID, a.Status))
}
a.Fix.Strategy = strings.ToLower(a.Fix.Strategy)
if a.Fix.Strategy != "" && !slices.Contains([]string{StrategyUpgrade, StrategyRefresh, StrategyManual}, a.Fix.Strategy) {
return errors.New(i18n.G("%s: unknown fix strategy %s", a.ID, a.Fix.Strategy))
}
if len(a.Recipes) == 0 && len(a.Images) == 0 {
return errors.New(i18n.G("%s: neither recipes nor images given", a.ID))
}
for name, rr := range a.Recipes {
raw := rr.Affected
if raw == "" {
if rr.Fixed == "" {
return errors.New(i18n.G("%s: recipe %s needs affected or fixed", a.ID, name))
}
raw = "<" + rr.Fixed
}
r, err := ParseRange(raw)
if err != nil {
return errors.New(i18n.G("%s: recipe %s: %s", a.ID, name, err))
}
rr.affected = r
a.Recipes[name] = rr
}
for i := range a.Images {
img := &a.Images[i]
if img.Match == "" {
return errors.New(i18n.G("%s: images[%d] has no match", a.ID, i))
}
if len(img.Env) > 0 {
img.env = make(map[string]Range)
for k, v := range img.Env {
r, err := ParseRange(v)
if err != nil {
return errors.New(i18n.G("%s: images[%d].env.%s: %s", a.ID, i, k, err))
}
img.env[k] = r
}
}
if img.Version != "" {
r, err := ParseRange(img.Version)
if err != nil {
return errors.New(i18n.G("%s: images[%d].version: %s", a.ID, i, err))
}
img.version = &r
}
if img.CreatedAfter != "" {
t, err := parseTime(img.CreatedAfter)
if err != nil {
return errors.New(i18n.G("%s: images[%d].created_after: %s", a.ID, i, err))
}
img.createdAfter = &t
}
for _, d := range append(slices.Clone(img.FixedDigests), img.AffectedDigests...) {
if !strings.HasPrefix(d, "sha256:") {
return errors.New(i18n.G("%s: images[%d]: digest %s must start with sha256:", a.ID, i, d))
}
}
}
return nil
}
func parseTime(s string) (time.Time, error) {
layouts := []string{time.RFC3339Nano, "2006-01-02T15:04:05", "2006-01-02 15:04:05", "2006-01-02"}
for _, l := range layouts {
if t, err := time.Parse(l, s); err == nil {
return t, nil
}
}
return time.Time{}, errors.New(i18n.G("unable to parse time %s", s))
}
-282
View File
@@ -1,282 +0,0 @@
package audit
import (
"os"
"path/filepath"
"strings"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestRange(t *testing.T) {
tests := []struct {
rng string
version string
want bool
}{
{"<7.0.3+30.0.1", "7.0.2+30.0.1", true},
{"<7.0.3+30.0.1", "7.0.3+30.0.1", false},
{"<7.0.3+30.0.1", "7.0.3+30.0.0", true},
{"<7.0.3", "7.0.2+31.0.0", true},
{"<7.0.3", "7.0.3+29.0.0", false},
{"<8.3.14", "8.3.12", true},
{">=8.3.0 <8.3.14", "8.2.20", false},
{">=8.3.0, <8.3.14", "8.3.13", true},
{">=8.3.0 <8.3.14 || >=8.4.0 <8.4.2", "8.4.1", true},
{">=8.3.0 <8.3.14 || >=8.4.0 <8.4.2", "8.4.2", false},
{"< 35.0.4", "35.0.3-apache", true},
{"8.3", "8.3.0", true},
{"!=1.2.3", "1.2.3", false},
{"<=v1.2", "1.2.0", true},
}
for _, tc := range tests {
r, err := ParseRange(tc.rng)
require.NoError(t, err, tc.rng)
v, err := ParseVersion(tc.version)
require.NoError(t, err, tc.version)
assert.Equal(t, tc.want, r.Contains(v), "%s in %s", tc.version, tc.rng)
}
for _, bad := range []string{"", "<", "<foo", "1.2 || "} {
_, err := ParseRange(bad)
assert.Error(t, err, bad)
}
}
const sampleAdvisory = `
id: CCA-2026-0012
title: PHP 8.3 CGI argument injection
severity: critical
status: published
refs: [CVE-2026-12345]
published: 2026-10-20
future_field: whatever
recipes:
nextcloud:
affected: "<7.0.3+30.0.1"
fixed: "7.0.3+30.0.1"
images:
- match: "nextcloud:30*-apache"
created_after: 2026-10-18
affected_digests: [sha256:1111111111111111111111111111111111111111111111111111111111111111]
fixed_digests: [sha256:2222222222222222222222222222222222222222222222222222222222222222]
- match: "*"
base: "php:8.3*"
env: { PHP_VERSION: ">=8.3.0 <8.3.14" }
fix:
strategy: refresh
notes: "Upstream images rebuilt on 2026-10-18; a redeploy is enough."
`
func TestParseAdvisory(t *testing.T) {
adv, warns, err := ParseAdvisory([]byte(sampleAdvisory))
require.NoError(t, err)
assert.Equal(t, "CCA-2026-0012", adv.ID)
assert.Equal(t, StatusPublished, adv.Status)
assert.Equal(t, "2026-10-20", adv.Published)
assert.Equal(t, "7.0.3+30.0.1", adv.Recipes["nextcloud"].Fixed)
require.Len(t, adv.Images, 2)
require.NotNil(t, adv.Images[0].createdAfter)
assert.Equal(t, time.Date(2026, 10, 18, 0, 0, 0, 0, time.UTC), *adv.Images[0].createdAfter)
assert.Equal(t, []string{"base"}, adv.Images[1].Skipped)
joined := strings.Join(warns, "\n")
assert.Contains(t, joined, "future_field")
assert.Contains(t, joined, "images[1].base")
}
func TestParseAdvisoryInvalid(t *testing.T) {
for name, doc := range map[string]string{
"no id": "status: published\nrecipes: {a: {fixed: 1.0.0}}",
"bad severity": "id: X\nseverity: extreme\nrecipes: {a: {fixed: 1.0.0}}",
"no conditions": "id: X",
"bad range": "id: X\nrecipes: {a: {affected: '<foo'}}",
"bad digest": "id: X\nimages: [{match: a, fixed_digests: [abc]}]",
"bad strategy": "id: X\nrecipes: {a: {fixed: 1.0.0}}\nfix: {strategy: pray}",
"image no match": "id: X\nimages: [{env: {A: '<1'}}]",
} {
_, _, err := ParseAdvisory([]byte(doc))
assert.Error(t, err, name)
}
}
func TestLoadAdvisories(t *testing.T) {
dir := t.TempDir()
require.NoError(t, os.MkdirAll(filepath.Join(dir, "2026"), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(dir, "2026", "good.yml"), []byte(sampleAdvisory), 0o644))
require.NoError(t, os.WriteFile(filepath.Join(dir, "broken.yaml"), []byte("id: [\n"), 0o644))
require.NoError(t, os.WriteFile(filepath.Join(dir, "README.md"), []byte("# hi"), 0o644))
advs, warns, err := LoadAdvisories(dir)
require.NoError(t, err)
require.Len(t, advs, 1)
assert.Contains(t, strings.Join(warns, "\n"), "broken.yaml")
}
func mustImage(t *testing.T, service, ref string, info *ImageInfo) ImageRef {
t.Helper()
img, err := ParseImageRef(service, ref)
require.NoError(t, err)
if info != nil {
cp := *info
img.Info = &cp
}
return img
}
func at(s string) *time.Time {
tm, err := time.Parse(time.RFC3339, s)
if err != nil {
panic(err)
}
return &tm
}
func TestEvaluate(t *testing.T) {
adv, _, err := ParseAdvisory([]byte(sampleAdvisory))
require.NoError(t, err)
advs := []Advisory{adv}
newImg := &ImageInfo{Created: at("2026-10-19T10:00:00Z"), Env: map[string]string{"PHP_VERSION": "8.3.14"}}
oldImg := &ImageInfo{Created: at("2026-10-01T10:00:00Z"), Env: map[string]string{"PHP_VERSION": "8.3.12"}}
noPHP := &ImageInfo{Created: at("2026-10-01T10:00:00Z"), Env: map[string]string{}}
states := []AppState{
{
// clean: fixed recipe, rebuilt image
Server: "s1", Name: "clean.example.com", Recipe: "nextcloud", Version: "7.0.3+30.0.1",
Images: []ImageRef{
mustImage(t, "app", "nextcloud:30.0.1-apache@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", newImg),
mustImage(t, "db", "postgres:15@sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", noPHP),
},
},
{
// old recipe version
Server: "s1", Name: "old.example.com", Recipe: "nextcloud", Version: "7.0.2+30.0.0",
Images: []ImageRef{mustImage(t, "app", "nextcloud:30.0.0-apache@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", newImg)},
},
{
// current recipe, stale image under the same tag
Server: "s1", Name: "stale.example.com", Recipe: "nextcloud", Version: "7.0.3+30.0.1",
Images: []ImageRef{mustImage(t, "app", "nextcloud:30.0.1-apache@sha256:cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc", oldImg)},
},
{
// some other recipe built on old php
Server: "s2", Name: "wp.example.com", Recipe: "wordpress", Version: "1.0.0+6.0.0",
Images: []ImageRef{mustImage(t, "app", "wordpress:6@sha256:dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd", oldImg)},
},
{
// known bad digest
Server: "s2", Name: "bad.example.com", Recipe: "nextcloud", Version: "7.0.3+30.0.1",
Images: []ImageRef{mustImage(t, "app", "nextcloud:30.0.1-apache@sha256:1111111111111111111111111111111111111111111111111111111111111111",
&ImageInfo{Created: at("2026-10-20T10:00:00Z"), Env: map[string]string{"PHP_VERSION": "8.3.14"}})},
},
{
// no image config and no digest: unknown
Server: "s2", Name: "nocfg.example.com", Recipe: "nextcloud", Version: "7.0.3+30.0.1",
Images: []ImageRef{mustImage(t, "app", "nextcloud:30.0.1-apache", nil)},
},
{
// reproducible build: unusable build time, php fixed by env
Server: "s2", Name: "repro.example.com", Recipe: "nextcloud", Version: "7.0.3+30.0.1",
Images: []ImageRef{mustImage(t, "app", "nextcloud:30.0.1-apache@sha256:eeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeeee",
&ImageInfo{Created: at("1970-01-01T00:00:00Z"), Env: map[string]string{"PHP_VERSION": "8.3.13"}})},
},
}
markUnusableCreated(states)
findings := Evaluate(states, advs, EvaluateOptions{})
got := map[string]Finding{}
for _, f := range findings {
got[f.App] = f
}
assert.NotContains(t, got, "clean.example.com")
require.Contains(t, got, "old.example.com")
assert.Equal(t, FindingAffected, got["old.example.com"].Status)
assert.Equal(t, "abra app upgrade old.example.com 7.0.3+30.0.1", got["old.example.com"].Fix)
require.Contains(t, got, "stale.example.com")
assert.Equal(t, FindingAffected, got["stale.example.com"].Status)
assert.Equal(t, "abra app deploy stale.example.com --force && abra audit", got["stale.example.com"].Fix)
require.Contains(t, got, "wp.example.com")
assert.Equal(t, FindingAffected, got["wp.example.com"].Status)
require.Contains(t, got, "bad.example.com")
assert.Equal(t, FindingAffected, got["bad.example.com"].Status)
require.Contains(t, got, "nocfg.example.com")
assert.Equal(t, FindingUnknown, got["nocfg.example.com"].Status)
require.Contains(t, got, "repro.example.com")
assert.Equal(t, FindingAffected, got["repro.example.com"].Status, "env still says affected")
assert.Contains(t, strings.Join(got["repro.example.com"].Reasons, " "), "PHP_VERSION=8.3.13")
// drafts are ignored, unless asked for
advs[0].Status = StatusDraft
assert.Empty(t, Evaluate(states, advs, EvaluateOptions{}))
assert.Len(t, Evaluate(states, advs, EvaluateOptions{IncludeDrafts: true}), len(findings))
// withdrawn advisories are always ignored
advs[0].Status = StatusWithdrawn
assert.Empty(t, Evaluate(states, advs, EvaluateOptions{IncludeDrafts: true}))
}
func TestCreatedUnusable(t *testing.T) {
adv, _, err := ParseAdvisory([]byte(`
id: X
status: published
images:
- match: "app:1"
created_after: 2026-10-18
fix: {strategy: refresh}
`))
require.NoError(t, err)
same := at("2026-10-01T00:00:00Z")
states := []AppState{
{Server: "a", Name: "a1", Images: []ImageRef{mustImage(t, "app", "app:1@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa", &ImageInfo{Created: same})}},
{Server: "b", Name: "b1", Images: []ImageRef{mustImage(t, "app", "app:1@sha256:bbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbbb", &ImageInfo{Created: same})}},
{Server: "c", Name: "c1", Images: []ImageRef{mustImage(t, "app", "app:1@sha256:cccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccccc",
&ImageInfo{Created: at("2026-10-01T00:00:00Z"), BaseCreated: at("2026-10-05T00:00:00Z")})}},
{Server: "d", Name: "d1", Images: []ImageRef{mustImage(t, "app", "app:1@sha256:dddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddddd", &ImageInfo{Created: at("2026-10-02T00:00:00Z")})}},
{Server: "e", Name: "e1", Images: []ImageRef{mustImage(t, "app", "app:1@sha256:ffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffffff", &ImageInfo{Created: at("2026-10-20T00:00:00Z")})}},
}
markUnusableCreated(states)
got := map[string]string{}
for _, f := range Evaluate(states, []Advisory{adv}, EvaluateOptions{}) {
got[f.App] = f.Status
}
assert.Equal(t, map[string]string{
"a1": FindingUnknown, // shares build time with b1
"b1": FindingUnknown, // shares build time with a1
"c1": FindingUnknown, // built before its base
"d1": FindingAffected, // usable and old
}, got)
}
func TestGlobMatch(t *testing.T) {
img, err := ParseImageRef("app", "git.coopcloud.tech/coop-cloud/foo:1.2-apache@sha256:aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa")
require.NoError(t, err)
assert.True(t, globMatch("*", img))
assert.True(t, globMatch("git.coopcloud.tech/coop-cloud/foo", img))
assert.True(t, globMatch("*/foo:1.*-apache", img))
assert.False(t, globMatch("foo:*", img))
img, err = ParseImageRef("app", "docker.io/library/nextcloud:30.0.1-apache")
require.NoError(t, err)
assert.Equal(t, "nextcloud", img.Name)
assert.True(t, globMatch("nextcloud:30*-apache", img))
}
-415
View File
@@ -1,415 +0,0 @@
package audit
import (
"fmt"
"regexp"
"slices"
"strings"
"coopcloud.tech/abra/pkg/i18n"
)
// Finding statuses.
const (
FindingAffected = "affected"
FindingUnknown = "unknown"
)
// Finding actions say who has to act, and how.
const (
// ActionRedeploy: redeploying the app picks up a fixed image.
ActionRedeploy = "redeploy"
// ActionUpgrade: a released recipe version fixes it.
ActionUpgrade = "upgrade"
// ActionRecipe: a fixed image exists, but no recipe release uses it.
ActionRecipe = "recipe must update"
// ActionUpstream: no fixed image exists yet.
ActionUpstream = "upstream must update"
// ActionCheck: the audit could not decide.
ActionCheck = "check by hand"
// ActionAdvisory: the advisory's notes say what to do.
ActionAdvisory = "follow advisory"
)
// Finding is one advisory that applies to one deployed app.
type Finding struct {
Server string `json:"server"`
App string `json:"app"`
Recipe string `json:"recipe"`
Version string `json:"version"`
// Chaos and ChaosVersion are copied from AppState.
Chaos bool `json:"chaos"`
ChaosVersion string `json:"chaosVersion,omitempty"`
Advisory *Advisory `json:"-"`
Status string `json:"status"`
// Affected names what is vulnerable, e.g. "nginx 1.31.8" or
// "lasuite-docs 2.0.0+v6.0.4", from the conditions that matched.
Affected string `json:"affected"`
// Reasons explains, per recipe or image, why the finding was raised.
Reasons []string `json:"reasons"`
// Fix is a suggested command, or the advisory's notes for manual fixes.
Fix string `json:"fix"`
// Action says who has to act, one of the Action constants.
Action string `json:"action"`
// NoFix is set when no recipe release fixes the finding yet.
NoFix bool `json:"noFix"`
recipeAffected bool
// images that are affected or unknown, for VerifyRefresh
images []ImageRef
os, arch string
}
// verdict of one image rule for one image.
type verdict int
const (
notApplicable verdict = iota
fixed
affected
unknown
)
// EvaluateOptions configures Evaluate.
type EvaluateOptions struct {
// IncludeDrafts also applies advisories with status draft, so that a
// maintainer can try an advisory before publishing it.
IncludeDrafts bool
}
// Applies reports whether an advisory is used: published advisories always,
// drafts only on request, withdrawn advisories never.
func (o EvaluateOptions) Applies(adv Advisory) bool {
switch adv.Status {
case StatusPublished:
return true
case StatusDraft:
return o.IncludeDrafts
}
return false
}
// Evaluate matches every app against every advisory that applies.
func Evaluate(states []AppState, advisories []Advisory, opts EvaluateOptions) []Finding {
var findings []Finding
for _, st := range states {
for i := range advisories {
adv := &advisories[i]
if !opts.Applies(*adv) {
continue
}
if f, ok := evaluateApp(st, adv); ok {
findings = append(findings, f)
}
}
}
return findings
}
func evaluateApp(st AppState, adv *Advisory) (Finding, bool) {
f := Finding{
Server: st.Server,
App: st.Name,
Recipe: st.Recipe,
Version: st.Version,
Chaos: st.Chaos,
ChaosVersion: st.ChaosVersion,
Advisory: adv,
}
var gotAffected, gotUnknown bool
var affectedLabels []string
// 1. recipe ranges
if rr, ok := recipeRange(adv, st.Recipe); ok && st.Version != "" {
v, err := ParseVersion(st.Version)
if err == nil && rr.affected.Contains(v) {
gotAffected = true
f.recipeAffected = true
f.Reasons = append(f.Reasons, i18n.G("recipe version %s is in %s", st.Version, rr.affected.Raw))
affectedLabels = appendUnique(affectedLabels, ShortRecipeName(st.Recipe)+" "+st.Version)
}
}
// 2. image conditions
for _, img := range st.Images {
for _, rule := range adv.Images {
if !globMatch(rule.Match, img) {
continue
}
v, reason := evaluateImage(rule, img)
reason = fmt.Sprintf("%s (%s): %s", img.Service, img.NameTag(), reason)
switch v {
case affected:
gotAffected = true
case unknown:
gotUnknown = true
default:
continue
}
if !slices.Contains(f.Reasons, reason) {
f.Reasons = append(f.Reasons, reason)
}
affectedLabels = appendUnique(affectedLabels, affectedLabel(rule, img))
if !slices.ContainsFunc(f.images, func(i ImageRef) bool { return i.Ref == img.Ref }) {
f.images = append(f.images, img)
}
}
}
switch {
case gotAffected:
f.Status = FindingAffected
case gotUnknown:
f.Status = FindingUnknown
default:
return f, false
}
f.Affected = strings.Join(affectedLabels, ", ")
f.Action, f.Fix = suggestFix(f, st)
f.os, f.arch = st.OS, st.Arch
return f, true
}
// affectedLabel names what makes img affected under rule, in a few words:
// the runtime and its version for env conditions ("nginx 1.31.8"), the
// image's own version for version conditions, its build date for
// created_after, and the image itself otherwise.
func affectedLabel(rule ImageRule, img ImageRef) string {
info := img.Info
if info == nil || (img.Digest != "" && slices.Contains(rule.AffectedDigests, img.Digest)) {
return img.NameTag()
}
if len(rule.env) > 0 {
var keys []string
for k := range rule.env {
if _, ok := info.Env[k]; ok {
keys = append(keys, k)
}
}
if len(keys) == len(rule.env) {
slices.Sort(keys)
var parts []string
for _, k := range keys {
parts = append(parts, componentName(k)+" "+info.Env[k])
}
return strings.Join(parts, ", ")
}
}
if rule.version != nil && info.Version() != "" {
name := img.Name
if i := strings.LastIndex(name, "/"); i >= 0 {
name = name[i+1:]
}
return name + " " + info.Version()
}
if rule.createdAfter != nil && info.Created != nil && info.createdUnusable == "" {
return i18n.G("%s built %s", img.NameTag(), info.Created.UTC().Format("2006-01-02"))
}
return img.NameTag()
}
// componentName turns a version variable into the name of what it
// versions: NGINX_VERSION -> nginx, PHP_VERSION -> php.
func componentName(envVar string) string {
name := strings.TrimSuffix(envVar, "_VERSION")
return strings.ReplaceAll(strings.ToLower(name), "_", "-")
}
// evaluateImage applies one rule to one image that matches it:
//
// 1. a digest in affected_digests is affected;
// 2. a digest in fixed_digests is fixed;
// 3. the env block selects images: if a variable is missing, the rule does
// not apply (the image is not built on that runtime). If every variable
// is in its range, env votes affected, otherwise fixed;
// 4. version is an affected range on the image version, and votes the same way;
// 5. created_after votes fixed for images built at or after that time, and
// affected for older images, but only when the build time is usable.
//
// The image is fixed if any condition votes fixed, affected if one votes
// affected, and unknown otherwise: an image that matches but satisfies no
// condition is never reported as OK.
func evaluateImage(rule ImageRule, img ImageRef) (verdict, string) {
if img.Digest != "" && slices.Contains(rule.AffectedDigests, img.Digest) {
return affected, i18n.G("digest %s is known to be affected", shortDigest(img.Digest))
}
if img.Digest != "" && slices.Contains(rule.FixedDigests, img.Digest) {
return fixed, ""
}
needsConfig := len(rule.env) > 0 || rule.version != nil || rule.createdAfter != nil
info := img.Info
if needsConfig && info == nil {
if img.InfoErr != nil {
return unknown, i18n.G("image config unavailable: %s", img.InfoErr)
}
return unknown, i18n.G("image config unavailable")
}
var votedAffected []string
var notes []string
if len(rule.env) > 0 {
for k := range rule.env {
if _, ok := info.Env[k]; !ok {
return notApplicable, ""
}
}
allIn := true
var vals []string
for k, r := range rule.env {
val := info.Env[k]
vals = append(vals, fmt.Sprintf("%s=%s", k, val))
v, err := ParseVersion(val)
if err != nil {
allIn = false
notes = append(notes, i18n.G("unable to parse %s=%s", k, val))
continue
}
if !r.Contains(v) {
return fixed, ""
}
}
if allIn {
slices.Sort(vals)
votedAffected = append(votedAffected, strings.Join(vals, ", "))
}
}
if rule.version != nil {
raw := info.Version()
if raw == "" {
notes = append(notes, i18n.G("image has no version"))
} else if v, err := ParseVersion(raw); err != nil {
notes = append(notes, i18n.G("unable to parse image version %s", raw))
} else if rule.version.Contains(v) {
votedAffected = append(votedAffected, i18n.G("version %s is in %s", raw, rule.version.Raw))
} else {
return fixed, ""
}
}
if rule.createdAfter != nil {
switch {
case info.Created == nil:
notes = append(notes, i18n.G("image has no build time"))
case info.createdUnusable != "":
notes = append(notes, info.createdUnusable)
case !info.Created.Before(*rule.createdAfter):
return fixed, ""
default:
votedAffected = append(votedAffected, i18n.G("built %s, before %s", info.Created.UTC().Format("2006-01-02 15:04"), rule.createdAfter.UTC().Format("2006-01-02 15:04")))
}
}
if len(votedAffected) > 0 {
return affected, strings.Join(votedAffected, "; ")
}
if img.Digest == "" {
notes = append(notes, i18n.G("service spec has no digest (registry lookup failed at deploy?)"))
}
if len(rule.Skipped) > 0 {
notes = append(notes, i18n.G("conditions not supported by this abra: %s", strings.Join(rule.Skipped, ", ")))
}
if len(notes) == 0 {
notes = append(notes, i18n.G("no condition proves digest %s fixed", shortDigest(img.Digest)))
}
return unknown, strings.Join(notes, "; ")
}
// recipeRange looks up the advisory's range for a recipe. Advisories name
// recipes in the short form of ShortRecipeName, so "nextcloud" covers
// git.coopcloud.tech/coop-cloud/nextcloud but not a fork such as
// git.coopcloud.tech/c4dt/nextcloud, which has its own versions.
func recipeRange(adv *Advisory, recipe string) (RecipeRange, bool) {
if rr, ok := adv.Recipes[recipe]; ok {
return rr, true
}
rr, ok := adv.Recipes[ShortRecipeName(recipe)]
return rr, ok
}
func upgradeCommand(adv *Advisory, app, recipe string) string {
if rr, ok := recipeRange(adv, recipe); ok && rr.Fixed != "" {
return fmt.Sprintf("abra app upgrade %s %s", app, rr.Fixed)
}
return fmt.Sprintf("abra app upgrade %s", app)
}
// suggestFix returns the action and fix for a finding as the advisory
// suggests it. FixChecker then verifies the fix and may change both.
func suggestFix(f Finding, st AppState) (string, string) {
adv := f.Advisory
upgrade := func() string { return upgradeCommand(adv, st.Name, st.Recipe) }
// a recipe version in the affected range is only fixed by upgrading
if f.recipeAffected {
return ActionUpgrade, upgrade()
}
switch adv.Fix.Strategy {
case StrategyUpgrade:
return ActionUpgrade, upgrade()
case StrategyRefresh:
// without --chaos, deploy would replace the chaos deployment with
// the release in the .env file
if st.Chaos {
return ActionRedeploy, fmt.Sprintf("abra app deploy %s --chaos && abra audit", st.Name)
}
return ActionRedeploy, fmt.Sprintf("abra app deploy %s --force && abra audit", st.Name)
default:
if adv.Fix.Notes != "" {
return ActionAdvisory, strings.TrimSpace(adv.Fix.Notes)
}
return ActionAdvisory, i18n.G("No fix is known yet.")
}
}
// globMatch matches an advisory pattern against an image. "*" matches any
// sequence of characters, "?" a single one. Patterns with a ":" are matched
// against "name:tag", others against the name only.
func globMatch(pattern string, img ImageRef) bool {
target := img.Name
if strings.Contains(pattern, ":") {
target = img.NameTag()
}
return globToRegexp(pattern).MatchString(target)
}
func globToRegexp(pattern string) *regexp.Regexp {
var b strings.Builder
b.WriteString("^")
for _, r := range pattern {
switch r {
case '*':
b.WriteString(".*")
case '?':
b.WriteString(".")
default:
b.WriteString(regexp.QuoteMeta(string(r)))
}
}
b.WriteString("$")
return regexp.MustCompile(b.String())
}
func shortDigest(d string) string {
if d == "" {
return i18n.G("(none)")
}
d = strings.TrimPrefix(d, "sha256:")
if len(d) > 12 {
d = d[:12]
}
return "sha256:" + d
}
-442
View File
@@ -1,442 +0,0 @@
package audit
import (
"context"
"fmt"
"strings"
"time"
"coopcloud.tech/abra/pkg/i18n"
"coopcloud.tech/abra/pkg/log"
)
// FetchFunc reads the config of an image for a platform from a registry.
type FetchFunc func(ctx context.Context, img ImageRef, os, arch string) (*ImageInfo, error)
// RegistryFetch is the default FetchFunc.
func RegistryFetch(ctx context.Context, img ImageRef, os, arch string) (*ImageInfo, error) {
return FetchFromRegistry(ctx, img, os, arch)
}
// TagsFunc lists the tags of an image's repository.
type TagsFunc func(ctx context.Context, img ImageRef) ([]string, error)
// FixChecker checks that the suggested fixes actually exist, and decides
// who has to act on each finding (see the Action constants).
type FixChecker struct {
Fetch FetchFunc
// Tags lists repository tags, to find fixed images that no recipe
// release uses yet. If nil, no such search is made.
Tags TagsFunc
Recipes RecipeSource
// Checkout returns the images that abra app deploy --chaos would deploy
// for an app, see CheckoutImages. If nil, checkouts are not checked.
Checkout func(app string) ([]string, error)
cache map[string]fetchResult
tagsCache map[string]tagsResult
}
type fetchResult struct {
img *ImageRef
err error
}
type tagsResult struct {
tags []string
err error
}
// CheckFixes verifies the suggested fix of each finding, and sets its
// action:
//
// - a redeploy only helps if what it deploys is fixed: the tag's current
// image in the registry, or the digest the recipe pins. Upstreams stop
// rebuilding patch-pinned tags such as nextcloud:35.0.2-fpm once the
// next patch is out. If a redeploy does not help, an upgrade is looked
// for instead;
// - for an app deployed with --chaos, the recipe checkout and the app's
// overrides are checked first: if they use fixed images, a --chaos
// redeploy is the fix;
// - for an upgrade, the newer releases of the recipe are searched for the
// oldest one whose images are fixed;
// - if no release is fixed, the registry is searched for a fixed image
// (see findFixedTag): if one exists, the recipe must update, otherwise
// upstream must;
// - for a recipe version range, the advisory's fixed version must be a
// released tag of the recipe.
//
// When the registry or the recipe cannot be read, the suggestion is kept and
// marked as unchecked.
func (c *FixChecker) CheckFixes(ctx context.Context, findings []Finding) {
if c.cache == nil {
c.cache = make(map[string]fetchResult)
}
if c.tagsCache == nil {
c.tagsCache = make(map[string]tagsResult)
}
for i := range findings {
f := &findings[i]
switch {
case f.recipeAffected:
c.checkRecipeRange(f)
case len(f.images) == 0:
case f.Advisory.Fix.Strategy == StrategyRefresh:
c.checkRefresh(ctx, f)
case f.Advisory.Fix.Strategy == StrategyUpgrade:
c.checkUpgrade(ctx, f, f.images)
}
}
}
func (c *FixChecker) checkRecipeRange(f *Finding) {
rr, ok := recipeRange(f.Advisory, f.Recipe)
if !ok || rr.Fixed == "" || c.Recipes == nil {
return
}
versions, err := c.Recipes.Versions(f.Recipe)
if err != nil {
log.Debug(i18n.G("%s: unable to list versions of %s: %s", f.App, f.Recipe, err))
f.Fix += " " + i18n.G("(unchecked: %s)", err)
return
}
want, _ := ParseVersion(rr.Fixed)
for _, v := range versions {
if pv, err := ParseVersion(v); err == nil && pv.Compare(want) >= 0 {
return
}
}
short := ShortRecipeName(f.Recipe)
f.Action = ActionRecipe
f.Fix = i18n.G("The advisory names %s %s as fixed, but the latest release is %s. A release of %s with the fix is needed.",
short, rr.Fixed, lastOr(versions, i18n.G("none")), short)
f.NoFix = true
}
func (c *FixChecker) checkRefresh(ctx context.Context, f *Finding) {
if f.arch == "" {
f.Fix += " " + i18n.G("(unchecked: platform unknown)")
return
}
var stale []ImageRef
var fetchErr error
for _, img := range f.images {
// a redeploy deploys the digest the recipe pins, if any, otherwise
// whatever the tag points to now
target, byDigest := img, false
if d := c.pinnedDigest(f, img); d != "" {
target.Digest, byDigest = d, true
} else if img.Tag == "" {
stale = append(stale, img)
continue
}
cur, err := c.current(ctx, target, f.os, f.arch, byDigest)
if err != nil {
fetchErr = err
log.Debug(i18n.G("%s: unable to read current %s from registry: %s", f.App, img.NameTag(), err))
continue
}
if cur.Digest == img.Digest || !fixedByAdvisory(f.Advisory, *cur) {
if byDigest {
f.Reasons = append(f.Reasons, i18n.G("redeploy would not help: the recipe pins %s to %s", img.NameTag(), shortDigest(target.Digest)))
} else {
f.Reasons = append(f.Reasons, i18n.G("redeploy would not help: the current image of %s (%s) is not fixed either", img.NameTag(), shortDigest(cur.Digest)))
}
stale = append(stale, img)
}
}
switch {
case len(stale) > 0:
c.checkUpgrade(ctx, f, stale)
case fetchErr != nil:
f.Fix += " " + i18n.G("(unchecked: registry unreachable)")
default:
var names []string
for _, img := range f.images {
names = appendUnique(names, img.NameTag())
}
f.Fix = i18n.G("The registry's %s now has a fixed image: %s", strings.Join(names, ", "), f.Fix)
}
}
// pinnedDigest returns the digest that a redeploy of the app would use for
// the image, if its compose files pin one: those of the deployed release,
// or for a --chaos deployment those of the checkout.
func (c *FixChecker) pinnedDigest(f *Finding, img ImageRef) string {
var refs []string
var err error
switch {
case f.Chaos && c.Checkout != nil:
refs, err = c.Checkout(f.App)
case !f.Chaos && c.Recipes != nil && f.Version != "":
refs, err = c.Recipes.Images(f.Recipe, f.Version)
default:
return ""
}
if err != nil {
return ""
}
for _, ref := range refs {
if r, err := ParseImageRef("", ref); err == nil && r.Name == img.Name && r.Digest != "" {
return r.Digest
}
}
return ""
}
// checkUpgrade looks for the oldest recipe release newer than the deployed
// one whose images are all fixed. stale are the images a redeploy does not
// fix.
func (c *FixChecker) checkUpgrade(ctx context.Context, f *Finding, stale []ImageRef) {
f.Action = ActionUpgrade
generic := upgradeCommand(f.Advisory, f.App, f.Recipe)
if c.Recipes == nil || f.arch == "" {
f.Fix = generic + " " + i18n.G("(unchecked: no fixed release looked up)")
return
}
names := make(map[string]bool)
for _, img := range stale {
names[img.Name] = true
}
// A chaos deployment runs the recipe checkout: if the checkout already
// uses fixed images, redeploying it is the fix.
if f.Chaos && c.Checkout != nil {
if fixedImgs, ok := c.checkoutFixed(ctx, f, names); ok {
f.Action = ActionRedeploy
f.Fix = i18n.G("The recipe checkout already uses %s, which is fixed: %s",
strings.Join(fixedImgs, ", "), fmt.Sprintf("abra app deploy %s --chaos && abra audit", f.App))
return
}
}
versions, err := c.Recipes.Versions(f.Recipe)
if err != nil {
log.Debug(i18n.G("%s: unable to list versions of %s: %s", f.App, f.Recipe, err))
f.Fix = generic + " " + i18n.G("(unchecked: %s)", err)
return
}
current, curErr := ParseVersion(f.Version)
var newer []string
for _, v := range versions {
pv, _ := ParseVersion(v)
if curErr != nil || pv.Compare(current) > 0 {
newer = append(newer, v)
}
}
if len(newer) == 0 {
f.Reasons = append(f.Reasons, i18n.G("no newer release of %s exists", ShortRecipeName(f.Recipe)))
c.noRelease(ctx, f, stale)
return
}
// Check the latest release first: if it is not fixed, no release is.
fixedAt := func(v string) (bool, error) { return c.versionFixed(ctx, f, v, names) }
latest := newer[len(newer)-1]
ok, err := fixedAt(latest)
if err != nil {
f.Fix = generic + " " + i18n.G("(unchecked: %s)", err)
return
}
if !ok {
f.Reasons = append(f.Reasons, i18n.G("the latest release of %s, %s, is not fixed", ShortRecipeName(f.Recipe), latest))
c.noRelease(ctx, f, stale)
return
}
best := latest
for _, v := range newer[:len(newer)-1] {
if ok, err := fixedAt(v); err == nil && ok {
best = v
break
}
}
f.Fix = i18n.G("Release %s uses fixed images: %s", best, fmt.Sprintf("abra app upgrade %s %s", f.App, best))
}
// noRelease handles a finding that no recipe release fixes: it searches the
// registry for a fixed image of each stale image. If every one has a fixed
// image, the recipe must update, otherwise upstream must.
func (c *FixChecker) noRelease(ctx context.Context, f *Finding, stale []ImageRef) {
f.NoFix = true
short := ShortRecipeName(f.Recipe)
var found, missing []string
var searchErr error
for _, img := range stale {
res, err := c.findFixedTag(ctx, f, img)
switch {
case err != nil:
log.Debug(i18n.G("%s: unable to search for a fixed %s: %s", f.App, img.Name, err))
searchErr = err
case res.tag == "":
missing = append(missing, upstreamState(img, res.current))
case res.sameTag:
found = append(found, i18n.G("%s (its current image)", img.Name+":"+res.tag))
default:
found = append(found, img.Name+":"+res.tag)
}
}
switch {
case len(missing) > 0:
f.Action = ActionUpstream
f.Fix = i18n.G("No fixed image exists yet: %s.", strings.Join(missing, "; "))
if f.Advisory.Fix.Notes != "" {
f.Fix += " " + i18n.G("Until then, see the advisory's mitigations.")
}
case searchErr != nil:
f.Action = ActionCheck
f.Fix = i18n.G("No release of %s fixes it, and the registry could not be searched for a fixed image: %s", short, searchErr)
case f.Chaos:
f.Action = ActionRecipe
checkout := i18n.G("the recipe checkout")
if f.ChaosVersion != "" {
checkout += " (" + i18n.G("deployed with --chaos from %s", f.ChaosVersion) + ")"
}
f.Fix = i18n.G("Upstream published a fixed image, %s. Use it in %s or the app's override, then run abra app deploy %s --chaos.",
strings.Join(found, ", "), checkout, f.App)
default:
f.Action = ActionRecipe
f.Fix = i18n.G("Upstream published a fixed image, %s, but no release of %s uses it yet. Release %s with it, then run abra app upgrade %s.",
strings.Join(found, ", "), short, short, f.App)
}
}
// upstreamState says why there is no fixed image of img: its tag was not
// rebuilt, or rebuilt but still affected.
func upstreamState(img ImageRef, current *ImageRef) string {
if current != nil && current.Digest != img.Digest {
if current.Info != nil && current.Info.Created != nil && current.Info.createdUnusable == "" {
return i18n.G("%s was rebuilt on %s but is still affected, and no later tag is fixed",
img.NameTag(), current.Info.Created.UTC().Format(time.DateOnly))
}
return i18n.G("%s was rebuilt but is still affected, and no later tag is fixed", img.NameTag())
}
return i18n.G("%s has not been rebuilt with a fix, and no later tag is fixed", img.NameTag())
}
// versionFixed reports whether every image of the recipe version that has
// one of the given names is fixed. An image that is no longer used counts as
// fixed.
func (c *FixChecker) versionFixed(ctx context.Context, f *Finding, version string, names map[string]bool) (bool, error) {
refs, err := c.Recipes.Images(f.Recipe, version)
if err != nil {
return false, err
}
ok, _, err := c.imagesFixed(ctx, f, version, refs, names)
return ok, err
}
// checkoutFixed reports whether a --chaos redeploy of the app, from its
// recipe checkout and overrides, would use fixed images, and which. A
// checkout that drops the affected images is not reported: that is more
// likely an unrelated change than a fix.
func (c *FixChecker) checkoutFixed(ctx context.Context, f *Finding, names map[string]bool) ([]string, bool) {
refs, err := c.Checkout(f.App)
if err != nil {
log.Debug(i18n.G("%s: unable to read the recipe checkout of %s: %s", f.App, f.Recipe, err))
return nil, false
}
ok, used, err := c.imagesFixed(ctx, f, i18n.G("checkout"), refs, names)
if err != nil {
log.Debug(i18n.G("%s: unable to check the recipe checkout of %s: %s", f.App, f.Recipe, err))
return nil, false
}
return used, ok && len(used) > 0
}
// imagesFixed reports whether every image in refs that has one of the given
// names is fixed, and returns those images. where names the recipe version
// for debug messages.
func (c *FixChecker) imagesFixed(ctx context.Context, f *Finding, where string, refs []string, names map[string]bool) (bool, []string, error) {
var used []string
for _, ref := range refs {
img, err := ParseImageRef("", ref)
if err != nil || !names[img.Name] {
continue
}
used = append(used, img.NameTag())
cur, err := c.current(ctx, img, f.os, f.arch, true)
if err != nil {
return false, used, err
}
if !fixedByAdvisory(f.Advisory, *cur) {
log.Debug(i18n.G("%s %s: %s is not fixed", ShortRecipeName(f.Recipe), where, img.NameTag()))
return false, used, nil
}
}
return true, used, nil
}
// current reads the image a reference points to now: by digest if byDigest
// is set and the reference has one, by tag otherwise.
func (c *FixChecker) current(ctx context.Context, img ImageRef, os, arch string, byDigest bool) (*ImageRef, error) {
ref := img
if !byDigest {
ref.Digest = ""
}
key := strings.Join([]string{ref.NameTag(), ref.Digest, os, arch}, "|")
if r, ok := c.cache[key]; ok {
return r.img, r.err
}
ref.Info, ref.InfoErr = nil, nil
fctx, cancel := context.WithTimeout(ctx, 60*time.Second)
defer cancel()
info, err := c.Fetch(fctx, ref, os, arch)
if err != nil {
c.cache[key] = fetchResult{err: err}
return nil, err
}
info.createdUnusable = createdUnusable(info)
if info.Digest != "" {
ref.Digest = info.Digest
}
ref.Info = info
c.cache[key] = fetchResult{img: &ref}
return &ref, nil
}
// fixedByAdvisory reports whether no image rule of the advisory flags img.
func fixedByAdvisory(adv *Advisory, img ImageRef) bool {
for _, rule := range adv.Images {
if !globMatch(rule.Match, img) {
continue
}
if v, _ := evaluateImage(rule, img); v == affected || v == unknown {
return false
}
}
return true
}
func appendUnique(list []string, s string) []string {
if containsString(list, s) {
return list
}
return append(list, s)
}
func lastOr(list []string, def string) string {
if len(list) == 0 {
return def
}
return list[len(list)-1]
}
-332
View File
@@ -1,332 +0,0 @@
package audit
import (
"context"
"errors"
"os"
"path/filepath"
"strings"
"testing"
"time"
appPkg "coopcloud.tech/abra/pkg/app"
"coopcloud.tech/abra/pkg/recipe"
"github.com/go-git/go-git/v5"
"github.com/go-git/go-git/v5/plumbing/object"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
const phpAdvisory = `
id: PHP
status: published
recipes:
ranged: {fixed: 2.0.0+1.0.0}
images:
- match: "*"
env: { PHP_VERSION: ">=8.5.0 <8.5.13" }
fix: {strategy: refresh}
`
// fakeRecipes maps recipe -> version -> images.
type fakeRecipes map[string]map[string][]string
func (f fakeRecipes) Versions(name string) ([]string, error) {
vs, ok := f[name]
if !ok {
return nil, errors.New("no such recipe")
}
var out []string
for v := range vs {
out = append(out, v)
}
sortVersions(out)
return out, nil
}
func (f fakeRecipes) Images(name, version string) ([]string, error) {
return f[name][version], nil
}
func digestOf(c string) string { return "sha256:" + strings.Repeat(c, 64) }
func TestCheckFixes(t *testing.T) {
adv, _, err := ParseAdvisory([]byte(phpAdvisory))
require.NoError(t, err)
old := &ImageInfo{Env: map[string]string{"PHP_VERSION": "8.5.12"}}
app := func(name, recipe, version, image string, platform bool) AppState {
st := AppState{Server: "s", Name: name, Recipe: recipe, Version: version,
Images: []ImageRef{mustImage(t, "app", image, old)}}
if platform {
st.OS, st.Arch = "linux", "amd64"
}
return st
}
states := []AppState{
app("floating", "nc", "1.0.0+35.0.2", "nextcloud:35-fpm@"+digestOf("a"), true),
app("pinned", "nc", "1.0.0+35.0.2", "nextcloud:35.0.2-fpm@"+digestOf("b"), true),
app("fork", "git.coopcloud.tech/c4dt/nextcloud", "16.2.2+35.0.2-fpm", "nextcloud:35.0.2-fpm@"+digestOf("b"), true),
app("stale", "stale", "1.0.0+35.0.1", "nextcloud:35.0.1-fpm@"+digestOf("e"), true),
app("offline", "nc", "1.0.0+35.0.2", "other:1@"+digestOf("c"), true),
app("noplatform", "nc", "1.0.0+35.0.2", "nextcloud:35-fpm@"+digestOf("a"), false),
app("ranged", "ranged", "1.0.0+1.0.0", "postgres:15@"+digestOf("f"), true),
app("upstream", "up", "1.0.0+1.0.0", "frozen:1.0.0@"+digestOf("9"), true),
app("digestpin", "dp", "1.0.0+35", "nextcloud:35-fpm@"+digestOf("a"), true),
app("unsearchable", "us", "1.0.0+1.0.0", "private:1.0.0@"+digestOf("8"), true),
}
// the same apps, deployed with --chaos
for _, name := range []string{"floating", "fork", "stale", "pinned"} {
for _, st := range states {
if st.Name == name {
st.Name, st.Chaos, st.ChaosVersion = "chaos-"+name, true, "1234abcd+U"
states = append(states, st)
break
}
}
}
recipes := fakeRecipes{
"nc": {
"1.0.0+35.0.2": {"nextcloud:35.0.2-fpm", "postgres:15"},
"1.0.1+35.0.3": {"nextcloud:35.0.3-fpm", "postgres:15"},
"1.1.0+35.0.4": {"nextcloud:35.0.4-fpm", "postgres:15"},
},
"git.coopcloud.tech/c4dt/nextcloud": {
"16.2.1+35.0.2-fpm": {"nextcloud:35.0.2-fpm"},
"16.2.2+35.0.2-fpm": {"nextcloud:35.0.2-fpm"},
},
"stale": {
"1.0.0+35.0.1": {"nextcloud:35.0.1-fpm"},
"1.0.1+35.0.2": {"nextcloud:35.0.2-fpm"},
},
"ranged": {"1.0.0+1.0.0": {"postgres:15"}},
"up": {"1.0.0+1.0.0": {"frozen:1.0.0"}},
// the recipe pins the old digest of a tag that has been rebuilt
"dp": {"1.0.0+35": {"nextcloud:35-fpm@" + digestOf("a")}},
"us": {"1.0.0+1.0.0": {"private:1.0.0"}},
}
// PHP version per tag; a pinned digest a is the old 35-fpm image
php := map[string]string{
"nextcloud:35-fpm": "8.5.13",
"nextcloud:35.0.2-fpm": "8.5.12",
"nextcloud:35.0.1-fpm": "8.5.12",
"nextcloud:35.0.3-fpm": "8.5.13",
"nextcloud:35.0.4-fpm": "8.5.13",
"frozen:1.0.0": "8.5.12",
"frozen:1.0.1": "8.5.12",
"private:1.0.0": "8.5.12",
}
var fetched []string
fetch := func(_ context.Context, img ImageRef, os, arch string) (*ImageInfo, error) {
fetched = append(fetched, img.NameTag()+"@"+img.Digest)
if img.Digest == digestOf("a") {
return &ImageInfo{Digest: digestOf("a"), Env: map[string]string{"PHP_VERSION": "8.5.12"}}, nil
}
v, ok := php[img.NameTag()]
if !ok {
return nil, errors.New("registry down")
}
d := digestOf("d")
switch img.NameTag() {
case "nextcloud:35.0.2-fpm":
d = digestOf("b")
case "frozen:1.0.0":
d = digestOf("9")
case "private:1.0.0":
d = digestOf("8")
}
return &ImageInfo{Digest: d, Env: map[string]string{"PHP_VERSION": v}}, nil
}
tags := func(_ context.Context, img ImageRef) ([]string, error) {
switch img.Name {
case "nextcloud":
return []string{"35-fpm", "35.0.1-fpm", "35.0.2-fpm", "35.0.3-fpm", "35.0.4-fpm", "latest"}, nil
case "frozen":
return []string{"1.0.0", "1.0.1"}, nil
}
return nil, errors.New("unauthorized")
}
// what a --chaos redeploy would use: chaos-fork's checkout already moved
// to a fixed image, chaos-stale's still uses the deployed one, and
// chaos-pinned's cannot be read
checkout := func(app string) ([]string, error) {
switch app {
case "chaos-fork":
return []string{"nextcloud:35.0.3-fpm"}, nil
case "chaos-stale":
return []string{"nextcloud:35.0.1-fpm"}, nil
case "chaos-floating":
return []string{"nextcloud:35-fpm"}, nil
}
return nil, errors.New("no checkout")
}
findings := Evaluate(states, []Advisory{adv}, EvaluateOptions{})
require.Len(t, findings, 14)
checker := FixChecker{Fetch: fetch, Tags: tags, Recipes: recipes, Checkout: checkout}
checker.CheckFixes(context.Background(), findings)
got := map[string]Finding{}
for _, f := range findings {
got[f.App] = f
}
check := func(app, action, fix string) {
t.Helper()
assert.Equal(t, action, got[app].Action, app)
assert.Equal(t, fix, got[app].Fix, app)
assert.Equal(t, action == ActionRecipe || action == ActionUpstream || action == ActionCheck, got[app].NoFix, app)
}
check("floating", ActionRedeploy, "The registry's nextcloud:35-fpm now has a fixed image: abra app deploy floating --force && abra audit")
check("pinned", ActionUpgrade, "Release 1.0.1+35.0.3 uses fixed images: abra app upgrade pinned 1.0.1+35.0.3")
check("fork", ActionRecipe, "Upstream published a fixed image, nextcloud:35.0.3-fpm, but no release of c4dt/nextcloud uses it yet. Release c4dt/nextcloud with it, then run abra app upgrade fork.")
// 35.0.2-fpm is newer but not fixed: the oldest fixed tag is suggested
check("stale", ActionRecipe, "Upstream published a fixed image, nextcloud:35.0.3-fpm, but no release of stale uses it yet. Release stale with it, then run abra app upgrade stale.")
check("offline", ActionRedeploy, "abra app deploy offline --force && abra audit (unchecked: registry unreachable)")
check("noplatform", ActionRedeploy, "abra app deploy noplatform --force && abra audit (unchecked: platform unknown)")
check("ranged", ActionRecipe, "The advisory names ranged 2.0.0+1.0.0 as fixed, but the latest release is 1.0.0+1.0.0. A release of ranged with the fix is needed.")
check("upstream", ActionUpstream, "No fixed image exists yet: frozen:1.0.0 has not been rebuilt with a fix, and no later tag is fixed.")
check("digestpin", ActionRecipe, "Upstream published a fixed image, nextcloud:35-fpm (its current image), but no release of dp uses it yet. Release dp with it, then run abra app upgrade digestpin.")
check("unsearchable", ActionCheck, "No release of us fixes it, and the registry could not be searched for a fixed image: unauthorized")
// a chaos deployment is redeployed with --chaos, and fixed in the checkout
check("chaos-floating", ActionRedeploy, "The registry's nextcloud:35-fpm now has a fixed image: abra app deploy chaos-floating --chaos && abra audit")
check("chaos-fork", ActionRedeploy, "The recipe checkout already uses nextcloud:35.0.3-fpm, which is fixed: abra app deploy chaos-fork --chaos && abra audit")
// without a readable checkout, the releases are checked as usual
check("chaos-pinned", ActionUpgrade, "Release 1.0.1+35.0.3 uses fixed images: abra app upgrade chaos-pinned 1.0.1+35.0.3")
check("chaos-stale", ActionRecipe, "Upstream published a fixed image, nextcloud:35.0.3-fpm. Use it in the recipe checkout (deployed with --chaos from 1234abcd+U) or the app's override, then run abra app deploy chaos-stale --chaos.")
// each image is fetched once, by tag or by digest
seen := map[string]int{}
for _, f := range fetched {
seen[f]++
}
for img, n := range seen {
if !strings.HasPrefix(img, "other:1@") {
assert.Equal(t, 1, n, img)
}
}
}
func TestShortRecipeName(t *testing.T) {
for in, want := range map[string]string{
"nextcloud": "nextcloud",
"git.coopcloud.tech/coop-cloud/nextcloud": "nextcloud",
"git.coopcloud.tech/c4dt/nextcloud": "c4dt/nextcloud",
"https://git.coopcloud.tech/c4dt/nextcloud.git": "c4dt/nextcloud",
"github.com/foo/nextcloud": "github.com/foo/nextcloud",
} {
assert.Equal(t, want, ShortRecipeName(in), in)
}
}
func TestRecipeRangeForks(t *testing.T) {
adv, _, err := ParseAdvisory([]byte("id: X\nstatus: published\nrecipes: {nextcloud: {fixed: 7.0.0}}\n"))
require.NoError(t, err)
_, ok := recipeRange(&adv, "nextcloud")
assert.True(t, ok)
_, ok = recipeRange(&adv, "git.coopcloud.tech/coop-cloud/nextcloud")
assert.True(t, ok)
_, ok = recipeRange(&adv, "git.coopcloud.tech/c4dt/nextcloud")
assert.False(t, ok, "a fork has its own versions")
}
func TestComposeImages(t *testing.T) {
assert.Equal(t, []string{"nextcloud:35.0.2-fpm", "nginx:1.29", "postgres:15"}, composeImages(`
services:
app: {image: nextcloud:35.0.2-fpm}
web: {image: "${WEB_IMAGE:-nginx:1.29}"}
db: {image: postgres:15}
custom: {image: "${CUSTOM_IMAGE}"}
build: {build: .}
`))
}
func TestGitRecipes(t *testing.T) {
dir := t.TempDir()
repo, err := git.PlainInit(dir, false)
require.NoError(t, err)
sig := &object.Signature{Name: "t", Email: "t@example.com", When: time.Now()}
release := func(version, image string) {
commitFile(t, repo, dir, "compose.yml", "services:\n app:\n image: "+image+"\n")
head, err := repo.Head()
require.NoError(t, err)
_, err = repo.CreateTag(version, head.Hash(), &git.CreateTagOptions{Tagger: sig, Message: version})
require.NoError(t, err)
}
release("1.0.0+35.0.2-fpm", "nextcloud:35.0.2-fpm")
release("1.0.1+35.0.3-fpm", "nextcloud:35.0.3-fpm")
// a local edit must survive
require.NoError(t, os.WriteFile(filepath.Join(dir, "compose.yml"), []byte("local edit\n"), 0o644))
g := NewGitRecipes([]recipe.Recipe{{Name: "nc", Dir: dir}}, true)
versions, err := g.Versions("nc")
require.NoError(t, err)
assert.Equal(t, []string{"1.0.0+35.0.2-fpm", "1.0.1+35.0.3-fpm"}, versions)
images, err := g.Images("nc", "1.0.0+35.0.2-fpm")
require.NoError(t, err)
assert.Equal(t, []string{"nextcloud:35.0.2-fpm"}, images)
data, err := os.ReadFile(filepath.Join(dir, "compose.yml"))
require.NoError(t, err)
assert.Equal(t, "local edit\n", string(data))
}
func TestCheckoutImages(t *testing.T) {
root := t.TempDir()
recipeDir := filepath.Join(root, "recipes", "nc")
serverDir := filepath.Join(root, "servers", "srv")
require.NoError(t, os.MkdirAll(recipeDir, 0o755))
require.NoError(t, os.MkdirAll(serverDir, 0o755))
write := func(file, content string) {
require.NoError(t, os.WriteFile(file, []byte(content), 0o644))
}
write(filepath.Join(recipeDir, "compose.yml"), `version: "3.8"
services:
app:
image: nextcloud:35.0.4-fpm
web:
image: nginx:${NGINX_TAG:-1.31.8}
db:
image: postgres:15
`)
write(filepath.Join(recipeDir, "compose.smtp.yml"), `version: "3.8"
services:
smtp:
image: postfix:3
`)
// an override outside the recipe directory pins the app image
write(filepath.Join(serverDir, "nc.override.yaml"), `version: "3.8"
services:
app:
image: nextcloud:35.0.2-fpm
`)
app := appPkg.App{
Name: "nc.example.com",
Recipe: recipe.Recipe{Name: "nc", Dir: recipeDir,
ComposePath: filepath.Join(recipeDir, "compose.yml")},
Env: map[string]string{
"COMPOSE_FILE": "compose.yml:../../servers/srv/nc.override.yaml",
"NGINX_TAG": "1.31.9",
},
}
images, err := CheckoutImages(app)
require.NoError(t, err)
assert.Equal(t, []string{"nextcloud:35.0.2-fpm", "nginx:1.31.9", "postgres:15"}, images)
app.Env = map[string]string{"COMPOSE_FILE": "compose.yml:missing.yml"}
_, err = CheckoutImages(app)
assert.Error(t, err)
}
-326
View File
@@ -1,326 +0,0 @@
package audit
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"strings"
"time"
"coopcloud.tech/abra/pkg/i18n"
"github.com/containers/image/docker"
"github.com/containers/image/manifest"
"github.com/containers/image/pkg/blobinfocache/none"
"github.com/containers/image/types"
"github.com/distribution/reference"
dockerClient "github.com/docker/docker/client"
"github.com/opencontainers/go-digest"
imgspecv1 "github.com/opencontainers/image-spec/specs-go/v1"
)
// OCI annotations and labels read by the audit.
const (
AnnotationVersion = "org.opencontainers.image.version"
AnnotationBaseName = "org.opencontainers.image.base.name"
AnnotationBaseDigest = "org.opencontainers.image.base.digest"
)
// ImageRef is one image as referenced in a service spec.
type ImageRef struct {
Service string `json:"service"`
// Ref is the reference as stored in the service spec, typically
// image:tag@sha256:...
Ref string `json:"ref"`
// Name is the familiar image name, e.g. "nextcloud" or "git.coopcloud.tech/coop-cloud/foo".
Name string `json:"name"`
Tag string `json:"tag"`
Digest string `json:"digest"`
Info *ImageInfo `json:"-"`
InfoErr error `json:"-"`
named reference.Named
}
// ImageInfo holds what the image says about itself.
type ImageInfo struct {
// Digest is the manifest (index) digest, set when read from a registry.
Digest string
Created *time.Time
Env map[string]string
Labels map[string]string
Annotations map[string]string
// BaseCreated is the build time of the base image named in the
// base.digest annotation, if it could be read.
BaseCreated *time.Time
// Source is "node" or "registry".
Source string
// createdUnusable is set when the build time cannot be trusted.
createdUnusable string
}
// ParseImageRef parses an image reference from a service spec.
func ParseImageRef(service, ref string) (ImageRef, error) {
img := ImageRef{Service: service, Ref: ref}
named, err := reference.ParseNormalizedNamed(ref)
if err != nil {
return img, err
}
img.named = named
img.Name = reference.FamiliarName(named)
if t, ok := named.(reference.Tagged); ok {
img.Tag = t.Tag()
}
if d, ok := named.(reference.Digested); ok {
img.Digest = d.Digest().String()
}
return img, nil
}
// NameTag returns "name:tag", the string matched against advisory patterns.
func (i ImageRef) NameTag() string {
if i.Tag == "" {
return i.Name
}
return i.Name + ":" + i.Tag
}
// Version returns the image's version from its label or annotation.
func (i *ImageInfo) Version() string {
if v := i.Labels[AnnotationVersion]; v != "" {
return v
}
return i.Annotations[AnnotationVersion]
}
// InspectOnNode reads the image config from the docker daemon of the server.
// This needs no registry, and works on single-node servers where the running
// image is always present.
func InspectOnNode(ctx context.Context, cl *dockerClient.Client, img ImageRef) (*ImageInfo, error) {
ref := img.Ref
if img.Digest != "" && img.named != nil {
ref = fmt.Sprintf("%s@%s", reference.TrimNamed(img.named).String(), img.Digest)
}
resp, err := cl.ImageInspect(ctx, ref)
if err != nil {
return nil, err
}
info := &ImageInfo{Source: "node", Env: map[string]string{}, Labels: map[string]string{}}
if resp.Created != "" {
if t, err := time.Parse(time.RFC3339Nano, resp.Created); err == nil {
info.Created = &t
}
}
if resp.Config != nil {
info.Env = parseEnv(resp.Config.Env)
if resp.Config.Labels != nil {
info.Labels = resp.Config.Labels
}
}
return info, nil
}
// manifestDoc covers the fields needed from image indexes, Docker manifest
// lists and image manifests.
type manifestDoc struct {
MediaType string `json:"mediaType"`
Annotations map[string]string `json:"annotations"`
Config *struct {
Digest string `json:"digest"`
MediaType string `json:"mediaType"`
Size int64 `json:"size"`
} `json:"config"`
Manifests []struct {
Digest string `json:"digest"`
Annotations map[string]string `json:"annotations"`
Platform *struct {
Architecture string `json:"architecture"`
OS string `json:"os"`
Variant string `json:"variant"`
} `json:"platform"`
} `json:"manifests"`
}
// FetchFromRegistry reads the image config for the given platform from the
// registry. Only a few KB are transferred; no layers are pulled. When the
// image records its base in the base.digest annotation, the base image's
// build time is read as well.
func FetchFromRegistry(ctx context.Context, img ImageRef, os, arch string) (*ImageInfo, error) {
info, err := fetchConfig(ctx, img, os, arch)
if err != nil {
return nil, err
}
baseName := info.Annotations[AnnotationBaseName]
baseDigest := info.Annotations[AnnotationBaseDigest]
if baseName != "" && baseDigest != "" {
if base, err := ParseImageRef("", fmt.Sprintf("%s@%s", stripTagDigest(baseName), baseDigest)); err == nil {
if bi, err := fetchConfig(ctx, base, os, arch); err == nil {
info.BaseCreated = bi.Created
}
}
}
return info, nil
}
// RegistryTags lists the tags of the image's repository.
func RegistryTags(ctx context.Context, img ImageRef) ([]string, error) {
if img.named == nil {
return nil, errors.New(i18n.G("invalid image reference %s", img.Ref))
}
ref, err := docker.ParseReference("//" + reference.TrimNamed(img.named).String())
if err != nil {
return nil, err
}
return docker.GetRepositoryTags(ctx, &types.SystemContext{}, ref)
}
func stripTagDigest(name string) string {
named, err := reference.ParseNormalizedNamed(name)
if err != nil {
return name
}
return reference.TrimNamed(named).String()
}
func fetchConfig(ctx context.Context, img ImageRef, os, arch string) (*ImageInfo, error) {
if img.named == nil {
return nil, errors.New(i18n.G("invalid image reference %s", img.Ref))
}
// NOTE: containers/image refuses references with both tag and digest
refStr := reference.TrimNamed(img.named).String()
if img.Digest != "" {
refStr += "@" + img.Digest
} else if img.Tag != "" {
refStr += ":" + img.Tag
}
ref, err := docker.ParseReference("//" + refStr)
if err != nil {
return nil, err
}
src, err := ref.NewImageSource(ctx, &types.SystemContext{})
if err != nil {
return nil, err
}
defer src.Close()
raw, _, err := src.GetManifest(ctx, nil)
if err != nil {
return nil, err
}
topDigest, err := manifest.Digest(raw)
if err != nil {
return nil, err
}
var doc manifestDoc
if err := json.Unmarshal(raw, &doc); err != nil {
return nil, err
}
annotations := map[string]string{}
for k, v := range doc.Annotations {
annotations[k] = v
}
if len(doc.Manifests) > 0 {
var chosen string
for _, m := range doc.Manifests {
if m.Platform != nil && m.Platform.OS == os && m.Platform.Architecture == arch {
chosen = m.Digest
for k, v := range m.Annotations {
annotations[k] = v
}
break
}
}
if chosen == "" {
return nil, errors.New(i18n.G("%s has no image for %s/%s", img.Ref, os, arch))
}
d := digest.Digest(chosen)
raw, _, err = src.GetManifest(ctx, &d)
if err != nil {
return nil, err
}
doc = manifestDoc{}
if err := json.Unmarshal(raw, &doc); err != nil {
return nil, err
}
for k, v := range doc.Annotations {
annotations[k] = v
}
}
if doc.Config == nil || doc.Config.Digest == "" {
return nil, errors.New(i18n.G("%s: unsupported manifest type %s", img.Ref, doc.MediaType))
}
blob, _, err := src.GetBlob(ctx, types.BlobInfo{Digest: digest.Digest(doc.Config.Digest), Size: doc.Config.Size}, none.NoCache)
if err != nil {
return nil, err
}
defer blob.Close()
cfgBytes, err := io.ReadAll(io.LimitReader(blob, 4<<20))
if err != nil {
return nil, err
}
var cfg imgspecv1.Image
if err := json.Unmarshal(cfgBytes, &cfg); err != nil {
return nil, err
}
info := &ImageInfo{
Source: "registry",
Digest: topDigest.String(),
Created: cfg.Created,
Env: parseEnv(cfg.Config.Env),
Labels: cfg.Config.Labels,
Annotations: annotations,
}
if info.Labels == nil {
info.Labels = map[string]string{}
}
return info, nil
}
func parseEnv(env []string) map[string]string {
out := make(map[string]string, len(env))
for _, e := range env {
k, v, _ := strings.Cut(e, "=")
out[k] = v
}
return out
}
// NormaliseArch maps the architecture names reported by the docker daemon
// (uname -m style) to OCI platform names.
func NormaliseArch(arch string) string {
switch arch {
case "x86_64", "amd64":
return "amd64"
case "aarch64", "arm64":
return "arm64"
case "armv7l", "armv7", "armhf", "arm":
return "arm"
case "i386", "i686", "386":
return "386"
}
return arch
}
-256
View File
@@ -1,256 +0,0 @@
package audit
import (
"errors"
"path"
"regexp"
"sort"
"strings"
appPkg "coopcloud.tech/abra/pkg/app"
"coopcloud.tech/abra/pkg/i18n"
"coopcloud.tech/abra/pkg/log"
"coopcloud.tech/abra/pkg/recipe"
"coopcloud.tech/abra/pkg/upstream/stack"
"github.com/go-git/go-git/v5"
"github.com/go-git/go-git/v5/plumbing"
"github.com/go-git/go-git/v5/plumbing/object"
"gopkg.in/yaml.v3"
)
const canonicalRecipePrefix = "git.coopcloud.tech/coop-cloud/"
// ShortRecipeName returns the shortest unambiguous form of a recipe name,
// relative to the canonical git.coopcloud.tech/coop-cloud/ location:
//
// nextcloud -> nextcloud
// git.coopcloud.tech/coop-cloud/nextcloud -> nextcloud
// git.coopcloud.tech/c4dt/nextcloud -> c4dt/nextcloud
// github.com/foo/nextcloud -> github.com/foo/nextcloud
func ShortRecipeName(name string) string {
name = strings.TrimPrefix(name, "https://")
name = strings.TrimSuffix(name, ".git")
if strings.HasPrefix(name, canonicalRecipePrefix) {
return strings.TrimPrefix(name, canonicalRecipePrefix)
}
if strings.HasPrefix(name, "git.coopcloud.tech/") {
return strings.TrimPrefix(name, "git.coopcloud.tech/")
}
return name
}
// RecipeSource lists the released versions of a recipe and the images each
// version uses.
type RecipeSource interface {
Versions(recipeName string) ([]string, error)
Images(recipeName, version string) ([]string, error)
}
// GitRecipes reads recipe versions from the local recipe checkouts. It never
// changes the working tree: tags are fetched into the repository, and compose
// files are read from the tag's tree.
type GitRecipes struct {
Offline bool
recipes map[string]recipe.Recipe
repos map[string]*git.Repository
errs map[string]error
}
// NewGitRecipes prepares access to the recipes of the given recipes.
func NewGitRecipes(recipes []recipe.Recipe, offline bool) *GitRecipes {
g := &GitRecipes{
Offline: offline,
recipes: make(map[string]recipe.Recipe),
repos: make(map[string]*git.Repository),
errs: make(map[string]error),
}
for _, r := range recipes {
g.recipes[r.Name] = r
}
return g
}
func (g *GitRecipes) open(name string) (*git.Repository, error) {
if repo, ok := g.repos[name]; ok {
return repo, nil
}
if err, ok := g.errs[name]; ok {
return nil, err
}
repo, err := g.openUncached(name)
if err != nil {
g.errs[name] = err
return nil, err
}
g.repos[name] = repo
return repo, nil
}
func (g *GitRecipes) openUncached(name string) (*git.Repository, error) {
r, ok := g.recipes[name]
if !ok {
return nil, errors.New(i18n.G("unknown recipe %s", name))
}
repo, err := git.PlainOpen(r.Dir)
if err != nil {
if g.Offline {
return nil, err
}
if err := r.EnsureExists(); err != nil {
return nil, err
}
if repo, err = git.PlainOpen(r.Dir); err != nil {
return nil, err
}
}
if !g.Offline {
err := repo.Fetch(&git.FetchOptions{RemoteName: "origin", Tags: git.AllTags})
if err != nil && !errors.Is(err, git.NoErrAlreadyUpToDate) {
log.Debug(i18n.G("%s: unable to fetch tags, using local tags: %s", name, err))
}
}
return repo, nil
}
// Versions returns the parseable release tags, oldest first.
func (g *GitRecipes) Versions(name string) ([]string, error) {
repo, err := g.open(name)
if err != nil {
return nil, err
}
iter, err := repo.Tags()
if err != nil {
return nil, err
}
var tags []string
_ = iter.ForEach(func(ref *plumbing.Reference) error {
tag := ref.Name().Short()
if _, err := ParseVersion(tag); err == nil {
tags = append(tags, tag)
}
return nil
})
sortVersions(tags)
return tags, nil
}
// Images returns the images used by a recipe version, read from the
// compose.yml and compose.*.yml files of that tag.
func (g *GitRecipes) Images(name, version string) ([]string, error) {
repo, err := g.open(name)
if err != nil {
return nil, err
}
hash, err := repo.ResolveRevision(plumbing.Revision("refs/tags/" + version))
if err != nil {
return nil, err
}
commit, err := repo.CommitObject(*hash)
if err != nil {
return nil, err
}
tree, err := commit.Tree()
if err != nil {
return nil, err
}
var images []string
err = tree.Files().ForEach(func(f *object.File) error {
if strings.Contains(f.Name, "/") {
return nil
}
if ok, _ := path.Match("compose*yml", f.Name); !ok {
return nil
}
content, err := f.Contents()
if err != nil {
return err
}
for _, img := range composeImages(content) {
if !containsString(images, img) {
images = append(images, img)
}
}
return nil
})
return images, err
}
// CheckoutImages returns the images that abra app deploy --chaos would
// deploy for the app: the compose files of its COMPOSE_FILE, read from the
// recipe checkout as they are on disk, merged and with the app's .env
// values filled in, exactly as deploy does. Overrides outside the recipe
// directory are included.
func CheckoutImages(app appPkg.App) ([]string, error) {
files, err := app.Recipe.GetComposeFiles(app.Env)
if err != nil {
return nil, err
}
compose, err := appPkg.GetAppComposeConfig(app.Name, stack.Deploy{Composefiles: files}, app.Env)
if err != nil {
return nil, err
}
var images []string
for _, svc := range compose.Services {
if svc.Image != "" && !containsString(images, svc.Image) {
images = append(images, svc.Image)
}
}
sort.Strings(images)
return images, nil
}
var envDefault = regexp.MustCompile(`\$\{[A-Za-z0-9_]+:?-([^}]*)\}`)
// composeImages extracts services.*.image from a compose file. Variables with
// a default (${FOO:-bar}) are replaced by the default; images with other
// variables are skipped.
func composeImages(content string) []string {
var doc struct {
Services map[string]struct {
Image string `yaml:"image"`
} `yaml:"services"`
}
if err := yaml.Unmarshal([]byte(content), &doc); err != nil {
return nil
}
var images []string
for _, svc := range doc.Services {
img := envDefault.ReplaceAllString(svc.Image, "$1")
if img == "" || strings.Contains(img, "$") {
continue
}
images = append(images, img)
}
sort.Strings(images)
return images
}
func sortVersions(tags []string) {
sort.SliceStable(tags, func(i, j int) bool {
a, _ := ParseVersion(tags[i])
b, _ := ParseVersion(tags[j])
return a.Compare(b) < 0
})
}
func containsString(list []string, s string) bool {
for _, l := range list {
if l == s {
return true
}
}
return false
}
-33
View File
@@ -1,33 +0,0 @@
package audit
import (
"context"
"os"
"testing"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
// TestFetchFromRegistry talks to Docker Hub, so it only runs with
// ABRA_TEST_NETWORK=1.
func TestFetchFromRegistry(t *testing.T) {
if os.Getenv("ABRA_TEST_NETWORK") != "1" {
t.Skip("set ABRA_TEST_NETWORK=1 to run")
}
for _, arch := range []string{"amd64", "arm64"} {
img, err := ParseImageRef("app", "nextcloud:apache")
require.NoError(t, err)
info, err := FetchFromRegistry(context.Background(), img, "linux", arch)
require.NoError(t, err)
assert.NotEmpty(t, info.Env["PHP_VERSION"], arch)
assert.NotEmpty(t, info.Env["NEXTCLOUD_VERSION"], arch)
require.NotNil(t, info.Created, arch)
t.Logf("%s: created=%s php=%s nextcloud=%s base=%s baseCreated=%v",
arch, info.Created, info.Env["PHP_VERSION"], info.Env["NEXTCLOUD_VERSION"],
info.Annotations[AnnotationBaseName], info.BaseCreated)
}
}
-114
View File
@@ -1,114 +0,0 @@
package audit
import (
"errors"
"os"
"path/filepath"
"time"
gitPkg "coopcloud.tech/abra/pkg/git"
"coopcloud.tech/abra/pkg/i18n"
"coopcloud.tech/abra/pkg/log"
"github.com/go-git/go-git/v5"
)
// DefaultAdvisoriesURL is the git repository holding the reviewed advisories.
const DefaultAdvisoriesURL = "https://git.coopcloud.tech/c4dt/advisories.git"
// fetchMarker is touched after each successful fetch, so the age of the local
// copy can be reported when the remote cannot be reached.
const fetchMarker = "abra-audit-last-fetch"
// SyncAdvisories makes sure dir holds an up-to-date checkout of the
// advisories repository. There is no cache lifetime: every call checks the
// remote. When the remote cannot be reached, a warning with the age of the
// local copy is logged and the local copy is used.
//
// A directory that is not a git checkout is used as is, which is handy for
// testing advisories locally.
func SyncAdvisories(dir, url string, offline bool) error {
if _, err := os.Stat(dir); os.IsNotExist(err) {
if offline {
return errors.New(i18n.G("no local advisories in %s and offline mode is on", dir))
}
log.Info(i18n.G("fetching advisories from %s", url))
if err := gitPkg.Clone(dir, url); err != nil {
_ = os.RemoveAll(dir)
return errors.New(i18n.G("unable to fetch advisories from %s: %s", url, err))
}
if _, err := git.PlainOpen(dir); err != nil {
_ = os.RemoveAll(dir)
return errors.New(i18n.G("unable to fetch advisories from %s: %s", url, err))
}
touchMarker(dir)
return nil
}
repo, err := git.PlainOpen(dir)
if err != nil {
log.Warn(i18n.G("%s is not a git checkout, using it as is", dir))
return nil
}
if offline {
log.Info(i18n.G("offline mode: using local advisories (%s)", localAge(dir, repo)))
return nil
}
wt, err := repo.Worktree()
if err != nil {
return err
}
err = wt.Pull(&git.PullOptions{RemoteName: "origin"})
if err != nil && !errors.Is(err, git.NoErrAlreadyUpToDate) {
log.Warn(i18n.G("unable to update advisories: %s", err))
log.Warn(i18n.G("using local advisories (%s)", localAge(dir, repo)))
return nil
}
if err == nil {
log.Debug(i18n.G("advisories updated"))
} else {
log.Debug(i18n.G("advisories already up to date"))
}
touchMarker(dir)
return nil
}
func touchMarker(dir string) {
p := filepath.Join(dir, ".git", fetchMarker)
now := time.Now()
if err := os.WriteFile(p, []byte(now.UTC().Format(time.RFC3339)+"\n"), 0o644); err != nil {
log.Debug(i18n.G("unable to write %s: %s", p, err))
}
}
// localAge describes how old the local copy is: the time of the last
// successful fetch, or of the last commit if abra never fetched it.
func localAge(dir string, repo *git.Repository) string {
if st, err := os.Stat(filepath.Join(dir, ".git", fetchMarker)); err == nil {
return i18n.G("last fetched %s ago", humanAge(time.Since(st.ModTime())))
}
if head, err := repo.Head(); err == nil {
if c, err := repo.CommitObject(head.Hash()); err == nil {
return i18n.G("last commit %s ago", humanAge(time.Since(c.Committer.When)))
}
}
return i18n.G("age unknown")
}
func humanAge(d time.Duration) string {
switch {
case d < time.Hour:
return i18n.G("%d minutes", int(d.Minutes()))
case d < 48*time.Hour:
return i18n.G("%d hours", int(d.Hours()))
default:
return i18n.G("%d days", int(d.Hours()/24))
}
}
-59
View File
@@ -1,59 +0,0 @@
package audit
import (
"os"
"path/filepath"
"testing"
"time"
"github.com/go-git/go-git/v5"
"github.com/go-git/go-git/v5/plumbing/object"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func commitFile(t *testing.T, repo *git.Repository, dir, name, content string) {
t.Helper()
require.NoError(t, os.MkdirAll(filepath.Dir(filepath.Join(dir, name)), 0o755))
require.NoError(t, os.WriteFile(filepath.Join(dir, name), []byte(content), 0o644))
wt, err := repo.Worktree()
require.NoError(t, err)
_, err = wt.Add(name)
require.NoError(t, err)
_, err = wt.Commit("add "+name, &git.CommitOptions{
Author: &object.Signature{Name: "test", Email: "test@example.com", When: time.Now()},
})
require.NoError(t, err)
}
func TestSyncAdvisories(t *testing.T) {
upstream := t.TempDir()
repo, err := git.PlainInit(upstream, false)
require.NoError(t, err)
commitFile(t, repo, upstream, "advisories/a.yml", "id: A\nrecipes: {x: {fixed: 1.0.0}}\n")
local := filepath.Join(t.TempDir(), "advisories")
// offline without a local copy fails
require.Error(t, SyncAdvisories(local, upstream, true))
// first run clones
require.NoError(t, SyncAdvisories(local, upstream, false))
assert.FileExists(t, filepath.Join(local, "advisories", "a.yml"))
assert.FileExists(t, filepath.Join(local, ".git", fetchMarker))
// next run pulls new advisories
commitFile(t, repo, upstream, "advisories/b.yml", "id: B\nrecipes: {x: {fixed: 1.0.0}}\n")
require.NoError(t, SyncAdvisories(local, upstream, false))
assert.FileExists(t, filepath.Join(local, "advisories", "b.yml"))
// already up to date
require.NoError(t, SyncAdvisories(local, upstream, false))
// unreachable remote: warn and use the local copy
require.NoError(t, os.RemoveAll(upstream))
require.NoError(t, SyncAdvisories(local, upstream, false))
advs, _, err := LoadAdvisories(filepath.Join(local, "advisories"))
require.NoError(t, err)
assert.Len(t, advs, 2)
}
-273
View File
@@ -1,273 +0,0 @@
package audit
import (
"context"
"fmt"
"sort"
"sync"
"time"
appPkg "coopcloud.tech/abra/pkg/app"
"coopcloud.tech/abra/pkg/client"
"coopcloud.tech/abra/pkg/i18n"
"coopcloud.tech/abra/pkg/log"
"coopcloud.tech/abra/pkg/upstream/convert"
"coopcloud.tech/abra/pkg/upstream/stack"
dockerClient "github.com/docker/docker/client"
)
// AppState is what is deployed for one app.
type AppState struct {
Server string `json:"server"`
Name string `json:"name"`
Recipe string `json:"recipe"`
Version string `json:"version"`
// Chaos is set for apps deployed with --chaos. Version is then the
// release the checkout was based on, and ChaosVersion the commit.
Chaos bool `json:"chaos"`
ChaosVersion string `json:"chaosVersion,omitempty"`
Images []ImageRef `json:"images"`
// OS and Arch of the server, used to read per-platform image configs.
// Empty when unknown or offline.
OS string `json:"-"`
Arch string `json:"-"`
}
// ServerError records a server that could not be audited.
type ServerError struct {
Server string
Err error
}
// CollectOptions configures CollectState.
type CollectOptions struct {
// Offline disables registry lookups; image configs are then only read
// from the server's docker daemon.
Offline bool
}
// CollectState reads the deployed state of all given apps. Servers are
// queried in parallel. Apps that are not deployed are left out. A server that
// cannot be reached is returned as a ServerError, and the other servers are
// still audited.
func CollectState(apps []appPkg.App, opts CollectOptions) ([]AppState, []ServerError) {
byServer := make(map[string][]appPkg.App)
for _, app := range apps {
byServer[app.Server] = append(byServer[app.Server], app)
}
var (
mu sync.Mutex
wg sync.WaitGroup
states []AppState
errs []ServerError
)
for server, serverApps := range byServer {
wg.Add(1)
go func(server string, serverApps []appPkg.App) {
defer wg.Done()
s, err := collectServer(server, serverApps, opts)
mu.Lock()
defer mu.Unlock()
if err != nil {
errs = append(errs, ServerError{Server: server, Err: err})
return
}
states = append(states, s...)
}(server, serverApps)
}
wg.Wait()
sort.Slice(states, func(i, j int) bool {
if states[i].Server != states[j].Server {
return states[i].Server < states[j].Server
}
return states[i].Name < states[j].Name
})
sort.Slice(errs, func(i, j int) bool { return errs[i].Server < errs[j].Server })
markUnusableCreated(states)
return states, errs
}
func collectServer(server string, apps []appPkg.App, opts CollectOptions) ([]AppState, error) {
cl, err := client.New(server, client.WithTimeout(10))
if err != nil {
return nil, err
}
status := stack.GetAllDeployedServices(cl, server)
if status.Err != nil {
return nil, status.Err
}
ctx := context.Background()
var platformOS, platformArch string
if !opts.Offline {
if info, err := cl.Info(ctx); err == nil {
platformOS = info.OSType
platformArch = NormaliseArch(info.Architecture)
} else {
log.Debug(i18n.G("%s: unable to read platform: %s", server, err))
}
}
byStack := make(map[string]*AppState)
var order []string
for _, app := range apps {
byStack[app.StackName()] = &AppState{
Server: server,
Name: app.Name,
Recipe: app.Recipe.Name,
OS: platformOS,
Arch: platformArch,
}
}
deployed := make(map[string]bool)
cache := make(map[string]imageResult)
for _, svc := range status.Services {
stackName := svc.Spec.Labels[convert.LabelNamespace]
st, ok := byStack[stackName]
if !ok {
continue
}
if !deployed[stackName] {
deployed[stackName] = true
order = append(order, stackName)
}
if v, ok := svc.Spec.Labels[fmt.Sprintf("coop-cloud.%s.version", stackName)]; ok && v != "" {
st.Version = v
}
if c := svc.Spec.Labels[fmt.Sprintf("coop-cloud.%s.chaos", stackName)]; c == "true" {
st.Chaos = true
}
if v := svc.Spec.Labels[fmt.Sprintf("coop-cloud.%s.chaos-version", stackName)]; v != "" {
st.ChaosVersion = v
}
if svc.Spec.TaskTemplate.ContainerSpec == nil {
continue
}
ref := svc.Spec.TaskTemplate.ContainerSpec.Image
serviceName := svc.Spec.Name
if ns := stackName + "_"; len(serviceName) > len(ns) && serviceName[:len(ns)] == ns {
serviceName = serviceName[len(ns):]
}
img, err := ParseImageRef(serviceName, ref)
if err != nil {
img.InfoErr = err
st.Images = append(st.Images, img)
continue
}
res, ok := cache[ref]
if !ok {
res = inspectImage(ctx, cl, img, platformOS, platformArch, opts)
cache[ref] = res
}
img.Info, img.InfoErr = res.info, res.err
st.Images = append(st.Images, img)
}
var states []AppState
for _, stackName := range order {
st := byStack[stackName]
sort.Slice(st.Images, func(i, j int) bool { return st.Images[i].Service < st.Images[j].Service })
states = append(states, *st)
}
return states, nil
}
type imageResult struct {
info *ImageInfo
err error
}
func inspectImage(ctx context.Context, cl *dockerClient.Client, img ImageRef, platformOS, platformArch string, opts CollectOptions) imageResult {
nodeCtx, cancel := context.WithTimeout(ctx, 30*time.Second)
defer cancel()
info, nodeErr := InspectOnNode(nodeCtx, cl, img)
if nodeErr == nil {
return imageResult{info: info}
}
log.Debug(i18n.G("%s: not found on node: %s", img.Ref, nodeErr))
if opts.Offline || platformArch == "" {
return imageResult{err: nodeErr}
}
regCtx, cancel := context.WithTimeout(ctx, 60*time.Second)
defer cancel()
info, err := FetchFromRegistry(regCtx, img, platformOS, platformArch)
if err != nil {
return imageResult{err: fmt.Errorf("%s", i18n.G("node: %s; registry: %s", nodeErr, err))}
}
return imageResult{info: info}
}
// markUnusableCreated flags build times that cannot be trusted:
// - before 2000, which catches reproducible builds using the 1970 epoch;
// - earlier than the build time of the base image;
// - shared by two different digests of the same tag.
func markUnusableCreated(states []AppState) {
// tag -> created -> digests
seen := make(map[string]map[time.Time]map[string]bool)
for _, st := range states {
for _, img := range st.Images {
if img.Info == nil || img.Info.Created == nil || img.Digest == "" {
continue
}
c := img.Info.Created.UTC()
if seen[img.NameTag()] == nil {
seen[img.NameTag()] = make(map[time.Time]map[string]bool)
}
if seen[img.NameTag()][c] == nil {
seen[img.NameTag()][c] = make(map[string]bool)
}
seen[img.NameTag()][c][img.Digest] = true
}
}
for _, st := range states {
for _, img := range st.Images {
info := img.Info
if info == nil || info.Created == nil {
continue
}
c := info.Created.UTC()
switch {
case createdUnusable(info) != "":
info.createdUnusable = createdUnusable(info)
case len(seen[img.NameTag()][c]) > 1:
info.createdUnusable = i18n.G("several digests of %s share the build time %s", img.NameTag(), c.Format(time.RFC3339))
}
}
}
}
// createdUnusable runs the checks that need only the image itself: a build
// time before 2000, or earlier than the base image's.
func createdUnusable(info *ImageInfo) string {
if info.Created == nil {
return ""
}
c := info.Created.UTC()
switch {
case c.Before(time.Date(2000, 1, 1, 0, 0, 0, 0, time.UTC)):
return i18n.G("build time %s is before 2000 (reproducible build?)", c.Format(time.RFC3339))
case info.BaseCreated != nil && c.Before(*info.BaseCreated):
return i18n.G("build time %s is earlier than its base image", c.Format(time.RFC3339))
}
return ""
}
-290
View File
@@ -1,290 +0,0 @@
package audit
import (
"context"
"errors"
"regexp"
"sort"
"strconv"
"strings"
"time"
"coopcloud.tech/abra/pkg/i18n"
"coopcloud.tech/abra/pkg/log"
)
// Limits for the search of fixed tags, to keep registry requests bounded.
const (
maxRebuildCandidates = 20
maxNewerCandidates = 10
// rebuildMargin is how much later than the current image a rebuild must
// be built. Variants of one version (-fpm, -alpine) usually come out of
// the same CI run within minutes; rebuilds come hours or days later.
rebuildMargin = time.Hour
)
// tagSearch is the result of findFixedTag.
type tagSearch struct {
// tag is a fixed tag, or "" if none was found.
tag string
// sameTag is set when the deployed tag itself now points to a fixed
// image, which a redeploy does not pick up because the recipe pins an
// older digest.
sameTag bool
// current is the registry's current image of the deployed tag.
current *ImageRef
}
// findFixedTag searches the registry for a fixed image of img, in this
// order:
//
// 1. the deployed tag itself, if its current image is fixed;
// 2. later builds of the same version under a suffixed tag, such as
// v6.1.0-1 or v6.1.0-fix1 for v6.1.0. Suffixes are not parsed, since
// upstreams use many schemes. A candidate must be built at least
// rebuildMargin after the tag's current image: this keeps out
// pre-releases such as v6.1.0-preprod, built before the release, and
// variants built along with it. The latest-built fixed one is chosen;
// 3. newer versions of the same form, e.g. v6.1.1 or v6.2.0, but not
// v6 or latest: the oldest fixed one is chosen.
//
// Every candidate is checked against the advisory: whether a tag looks newer
// never decides whether it is fixed.
func (c *FixChecker) findFixedTag(ctx context.Context, f *Finding, img ImageRef) (tagSearch, error) {
var res tagSearch
if c.Tags == nil {
return res, errors.New(i18n.G("no tag search"))
}
if img.Tag == "" {
return res, errors.New(i18n.G("%s has no tag", img.Name))
}
cur, err := c.current(ctx, img, f.os, f.arch, false)
if err != nil {
return res, err
}
res.current = cur
if cur.Digest != img.Digest && fixedByAdvisory(f.Advisory, *cur) {
res.tag, res.sameTag = img.Tag, true
return res, nil
}
tags, err := c.tags(ctx, img)
if err != nil {
return res, err
}
tag, err := c.fixedRebuild(ctx, f, img, cur, tags)
if err != nil || tag != "" {
res.tag = tag
return res, err
}
res.tag = c.fixedNewer(ctx, f, img, tags)
return res, nil
}
// fixedRebuild returns the latest-built fixed rebuild of img's version.
func (c *FixChecker) fixedRebuild(ctx context.Context, f *Finding, img ImageRef, cur *ImageRef, tags []string) (string, error) {
var candidates []string
for _, t := range tags {
if isRebuildTag(img.Tag, t) {
candidates = append(candidates, t)
}
}
if len(candidates) == 0 {
return "", nil
}
if cur.Info == nil || cur.Info.Created == nil || cur.Info.createdUnusable != "" {
return "", errors.New(i18n.G("the build time of %s is unusable, so %s cannot be checked", img.NameTag(), strings.Join(candidates, ", ")))
}
after := cur.Info.Created.Add(rebuildMargin)
sort.Strings(candidates)
if len(candidates) > maxRebuildCandidates {
log.Debug(i18n.G("%s: checking only %d of %d rebuild tags", img.Name, maxRebuildCandidates, len(candidates)))
candidates = candidates[len(candidates)-maxRebuildCandidates:]
}
var best string
var bestTime time.Time
var unusable []string
for _, t := range candidates {
ref, err := ParseImageRef("", img.Name+":"+t)
if err != nil {
continue
}
cand, err := c.current(ctx, ref, f.os, f.arch, false)
if err != nil {
log.Debug(i18n.G("%s: skipping %s: %s", f.App, ref.NameTag(), err))
continue
}
info := cand.Info
switch {
case info.Created == nil || info.createdUnusable != "":
unusable = append(unusable, t)
continue
case info.Created.Before(after):
log.Debug(i18n.G("%s: skipping %s: built before %s", f.App, ref.NameTag(), img.NameTag()))
continue
case !fixedByAdvisory(f.Advisory, *cand):
log.Debug(i18n.G("%s: %s is not fixed", f.App, ref.NameTag()))
continue
}
if best == "" || info.Created.After(bestTime) {
best, bestTime = t, *info.Created
}
}
if best == "" && len(unusable) > 0 {
return "", errors.New(i18n.G("the build times of %s are unusable", strings.Join(unusable, ", ")))
}
return best, nil
}
// fixedNewer returns the oldest fixed newer version of img's tag, of the same
// form. As for recipe releases, the newest is checked first: if it is not
// fixed, none is.
func (c *FixChecker) fixedNewer(ctx context.Context, f *Finding, img ImageRef, tags []string) string {
base, ok := parseTagForm(img.Tag)
if !ok {
return ""
}
var newer []tagForm
for _, t := range tags {
if tf, ok := parseTagForm(t); ok && tf.sameForm(base) && tf.compare(base) > 0 {
newer = append(newer, tf)
}
}
if len(newer) == 0 {
return ""
}
sort.Slice(newer, func(i, j int) bool { return newer[i].compare(newer[j]) < 0 })
fixedAt := func(tf tagForm) bool {
ref, err := ParseImageRef("", img.Name+":"+tf.raw)
if err != nil {
return false
}
cand, err := c.current(ctx, ref, f.os, f.arch, false)
if err != nil {
log.Debug(i18n.G("%s: skipping %s: %s", f.App, ref.NameTag(), err))
return false
}
return fixedByAdvisory(f.Advisory, *cand)
}
latest := newer[len(newer)-1]
if !fixedAt(latest) {
return ""
}
for i, tf := range newer[:len(newer)-1] {
if i >= maxNewerCandidates {
break
}
if fixedAt(tf) {
return tf.raw
}
}
return latest.raw
}
func (c *FixChecker) tags(ctx context.Context, img ImageRef) ([]string, error) {
if r, ok := c.tagsCache[img.Name]; ok {
return r.tags, r.err
}
tctx, cancel := context.WithTimeout(ctx, 60*time.Second)
defer cancel()
tags, err := c.Tags(tctx, img)
c.tagsCache[img.Name] = tagsResult{tags: tags, err: err}
return tags, err
}
var (
// suffix words of pre-releases and of variants, which are not rebuilds
notRebuildWord = regexp.MustCompile(`^(?i)(pre|preprod|rc|beta|alpha|dev|devel|test|snapshot|nightly|edge|canary|pr|alpine|slim|fpm|apache|debian|ubuntu|bookworm|bullseye|trixie|windowsservercore|nanoserver)\d*$`)
archWord = regexp.MustCompile(`^(?i)(amd64|arm64|arm64v8|armv7|armv6|arm|386|i386|ppc64le|s390x|riscv64|x86_64|aarch64)$`)
digits = regexp.MustCompile(`\d`)
)
// isRebuildTag reports whether tag looks like a later build of base, such as
// v6.1.0-1 or v6.1.0-fix1 for v6.1.0: base, a separator, and a suffix with a
// number in it. Patch versions (1.2.3 for 1.2), pre-releases, variants and
// per-architecture tags are not rebuilds.
func isRebuildTag(base, tag string) bool {
if len(tag) < len(base)+2 || !strings.HasPrefix(tag, base) {
return false
}
sep, suffix := tag[len(base)], tag[len(base)+1:]
if !strings.ContainsRune("-_.+", rune(sep)) {
return false
}
if sep == '.' && suffix[0] >= '0' && suffix[0] <= '9' {
return false
}
if !digits.MatchString(suffix) {
return false
}
words := strings.FieldsFunc(suffix, func(r rune) bool { return strings.ContainsRune("-_.+", r) })
for _, w := range words {
if notRebuildWord.MatchString(w) {
return false
}
}
return !archWord.MatchString(words[len(words)-1])
}
// tagForm splits a tag such as v6.1.0 or 35.0.2-fpm into prefix ("v"),
// version numbers and the rest ("-fpm").
type tagForm struct {
raw string
prefix string
nums []int
rest string
}
var tagFormRe = regexp.MustCompile(`^([vV]?)(\d+(?:\.\d+)*)(.*)$`)
func parseTagForm(tag string) (tagForm, bool) {
m := tagFormRe.FindStringSubmatch(tag)
if m == nil {
return tagForm{}, false
}
tf := tagForm{raw: tag, prefix: m[1], rest: m[3]}
for _, p := range strings.Split(m[2], ".") {
n, err := strconv.Atoi(p)
if err != nil {
return tagForm{}, false
}
tf.nums = append(tf.nums, n)
}
return tf, true
}
// sameForm reports whether two tags differ only in their version numbers,
// with as many numbers: v6.1.1 has the form of v6.1.0, but v6.1 and
// v6.1.0-1 do not.
func (t tagForm) sameForm(o tagForm) bool {
return t.prefix == o.prefix && t.rest == o.rest && len(t.nums) == len(o.nums)
}
func (t tagForm) compare(o tagForm) int {
for i := range t.nums {
if i >= len(o.nums) {
return 1
}
if t.nums[i] != o.nums[i] {
if t.nums[i] < o.nums[i] {
return -1
}
return 1
}
}
if len(t.nums) < len(o.nums) {
return -1
}
return 0
}
-206
View File
@@ -1,206 +0,0 @@
package audit
import (
"context"
"errors"
"testing"
"time"
"github.com/stretchr/testify/assert"
"github.com/stretchr/testify/require"
)
func TestIsRebuildTag(t *testing.T) {
for tag, want := range map[string]bool{
"v6.1.0-1": true,
"v6.1.0-fix1": true,
"v6.1.0_r2": true,
"v6.1.0+build3": true,
"v6.1.0-1-amd64": false, // per-architecture copy
"v6.1.0-arm64": false,
"v6.1.0-preprod": false, // pre-release
"v6.1.0-rc1": false,
"v6.1.0-fpm": false, // variant, no number
"v6.1.0-alpine3": false, // variant
"v6.1.0.1": false, // a newer version, not a rebuild
"v6.1.0": false,
"v6.1.01": false,
"v6.1.1-1": false, // rebuild of another version
"v6.1.0-": false,
} {
assert.Equal(t, want, isRebuildTag("v6.1.0", tag), tag)
}
}
func TestTagForm(t *testing.T) {
base, ok := parseTagForm("35.0.2-fpm")
require.True(t, ok)
for tag, want := range map[string]bool{
"35.0.3-fpm": true,
"36.0.0-fpm": true,
"35.0.3-apache": false,
"35-fpm": false,
"v35.0.3-fpm": false,
} {
tf, ok := parseTagForm(tag)
require.True(t, ok, tag)
assert.Equal(t, want, tf.sameForm(base), tag)
}
_, ok = parseTagForm("latest")
assert.False(t, ok)
}
const nginxAdvisory = `
id: NGINX
status: published
images:
- match: "*"
env: { NGINX_VERSION: "<1.31.9" }
fix: {strategy: refresh}
`
// fakeImage is one tag in the fake registry.
type fakeImage struct {
created time.Time
nginx string
}
func TestFindFixedTag(t *testing.T) {
adv, _, err := ParseAdvisory([]byte(nginxAdvisory))
require.NoError(t, err)
release := time.Date(2026, 11, 13, 13, 0, 0, 0, time.UTC)
day := 24 * time.Hour
search := func(t *testing.T, deployed string, registry map[string]fakeImage) (tagSearch, error) {
t.Helper()
fetch := func(_ context.Context, img ImageRef, os, arch string) (*ImageInfo, error) {
fi, ok := registry[img.Tag]
if !ok {
return nil, errors.New("not found")
}
created := fi.created
return &ImageInfo{Digest: digestOf(string(rune('a' + len(img.Tag)%6))), Created: &created,
Env: map[string]string{"NGINX_VERSION": fi.nginx}}, nil
}
tags := func(_ context.Context, img ImageRef) ([]string, error) {
var out []string
for tag := range registry {
out = append(out, tag)
}
return out, nil
}
c := FixChecker{Fetch: fetch, Tags: tags, cache: map[string]fetchResult{}, tagsCache: map[string]tagsResult{}}
f := &Finding{App: "app", Advisory: &adv, os: "linux", arch: "amd64"}
img := mustImage(t, "app", "lasuite/frontend:"+deployed+"@"+digestOf("0"), nil)
return c.findFixedTag(context.Background(), f, img)
}
t.Run("latest-built fixed rebuild", func(t *testing.T) {
res, err := search(t, "v6.1.0", map[string]fakeImage{
"v6.1.0": {release, "1.31.8"},
"v6.1.0-preprod": {release.Add(-day), "1.31.9"}, // built before the release
"v6.1.0-1": {release.Add(6 * day), "1.31.9"}, // fixed rebuild
"v6.1.0-1-amd64": {release.Add(6 * day), "1.31.9"}, // per-arch copy
"v6.1.0-fix2": {release.Add(7 * day), "1.31.9"}, // later fixed rebuild
"v6.1.0-fix3": {release.Add(8 * day), "1.31.8"}, // later, but not fixed
"v6.1.0-r1": {release.Add(10 * time.Minute), "1.31.9"}, // same CI run as the release
})
require.NoError(t, err)
assert.Equal(t, "v6.1.0-fix2", res.tag)
assert.False(t, res.sameTag)
})
t.Run("rebuild preferred to newer version", func(t *testing.T) {
res, err := search(t, "v6.1.0", map[string]fakeImage{
"v6.1.0": {release, "1.31.8"},
"v6.1.0-1": {release.Add(6 * day), "1.31.9"},
"v6.2.0": {release.Add(7 * day), "1.31.9"},
})
require.NoError(t, err)
assert.Equal(t, "v6.1.0-1", res.tag)
})
t.Run("oldest fixed newer version", func(t *testing.T) {
res, err := search(t, "v6.1.0", map[string]fakeImage{
"v6.1.0": {release, "1.31.8"},
"v6.1.1": {release.Add(day), "1.31.8"},
"v6.2.0": {release.Add(2 * day), "1.31.9"},
"v7.0.0": {release.Add(3 * day), "1.31.9"},
"v6": {release.Add(3 * day), "1.31.9"},
"latest": {release.Add(3 * day), "1.31.9"},
"main": {release.Add(4 * day), "1.31.9"},
})
require.NoError(t, err)
assert.Equal(t, "v6.2.0", res.tag)
})
t.Run("no fixed image", func(t *testing.T) {
res, err := search(t, "v6.1.0", map[string]fakeImage{
"v6.1.0": {release, "1.31.8"},
"v6.1.0-preprod": {release.Add(-day), "1.31.9"},
"v6.1.1": {release.Add(day), "1.31.8"},
"main": {release.Add(day), "1.31.9"},
})
require.NoError(t, err)
assert.Equal(t, "", res.tag)
require.NotNil(t, res.current)
})
t.Run("unusable build time of the release", func(t *testing.T) {
_, err := search(t, "v6.1.0", map[string]fakeImage{
"v6.1.0": {time.Unix(0, 0).UTC(), "1.31.8"},
"v6.1.0-1": {release.Add(6 * day), "1.31.9"},
})
assert.ErrorContains(t, err, "build time of lasuite/frontend:v6.1.0 is unusable")
})
t.Run("unusable build time of the rebuild", func(t *testing.T) {
_, err := search(t, "v6.1.0", map[string]fakeImage{
"v6.1.0": {release, "1.31.8"},
"v6.1.0-1": {time.Unix(0, 0).UTC(), "1.31.9"},
})
assert.ErrorContains(t, err, "build times of v6.1.0-1 are unusable")
})
}
func TestAffectedLabel(t *testing.T) {
adv, _, err := ParseAdvisory([]byte(`
id: X
status: published
recipes:
nc: {affected: "<2.0.0"}
images:
- match: "lasuite/*"
env: { NGINX_VERSION: "<1.31.9" }
- match: "grafana/*"
version: "<12.0.0"
- match: "debian:*"
created_after: "2026-11-01T00:00:00Z"
`))
require.NoError(t, err)
built := time.Date(2026, 10, 1, 0, 0, 0, 0, time.UTC)
states := []AppState{
{Name: "nginx", Recipe: "x", Images: []ImageRef{mustImage(t, "app", "lasuite/frontend:v6.1.0@"+digestOf("a"),
&ImageInfo{Env: map[string]string{"NGINX_VERSION": "1.31.8"}})}},
{Name: "version", Recipe: "x", Images: []ImageRef{mustImage(t, "app", "grafana/grafana:11.0@"+digestOf("b"),
&ImageInfo{Labels: map[string]string{AnnotationVersion: "11.0.1"}})}},
{Name: "built", Recipe: "x", Images: []ImageRef{mustImage(t, "app", "debian:12@"+digestOf("c"),
&ImageInfo{Created: &built})}},
{Name: "noinfo", Recipe: "x", Images: []ImageRef{mustImage(t, "app", "lasuite/backend:v6.1.0@"+digestOf("d"), nil)}},
{Name: "recipe", Recipe: "git.coopcloud.tech/coop-cloud/nc", Version: "1.0.0+30.0.1"},
}
got := map[string]string{}
for _, f := range Evaluate(states, []Advisory{adv}, EvaluateOptions{}) {
got[f.App] = f.Affected
}
assert.Equal(t, map[string]string{
"nginx": "nginx 1.31.8",
"version": "grafana 11.0.1",
"built": "debian:12 built 2026-10-01",
"noinfo": "lasuite/backend:v6.1.0",
"recipe": "nc 1.0.0+30.0.1",
}, got)
}
-169
View File
@@ -1,169 +0,0 @@
package audit
import (
"errors"
"regexp"
"strconv"
"strings"
"coopcloud.tech/abra/pkg/i18n"
)
// Version is a loosely parsed version. Recipe tags have the form
// RECIPE+UPSTREAM (e.g. 7.0.3+30.0.1), so a version is a list of groups
// separated by "+", each group a list of numbers. Anything after the leading
// numeric part of a group (e.g. "-apache" in "35.0.3-apache") is ignored.
type Version struct {
Raw string
Groups [][]int
}
var numericPrefix = regexp.MustCompile(`^[vV]?(\d+(?:\.\d+)*)`)
// ParseVersion parses a version such as "8.3.14", "v1.2", "35.0.3-apache" or
// "7.0.3+30.0.1".
func ParseVersion(raw string) (Version, error) {
v := Version{Raw: raw}
raw = strings.TrimSpace(raw)
if raw == "" {
return v, errors.New(i18n.G("empty version"))
}
for _, group := range strings.Split(raw, "+") {
m := numericPrefix.FindStringSubmatch(group)
if m == nil {
return v, errors.New(i18n.G("unable to parse version %s", raw))
}
var nums []int
for _, part := range strings.Split(m[1], ".") {
n, err := strconv.Atoi(part)
if err != nil {
return v, errors.New(i18n.G("unable to parse version %s", raw))
}
nums = append(nums, n)
}
v.Groups = append(v.Groups, nums)
}
return v, nil
}
// Compare returns -1, 0 or 1. Only the groups present in both versions are
// compared, so "7.0.3" equals "7.0.3+30.0.1": a constraint without an
// upstream part only constrains the recipe part. Missing numbers inside a
// group count as 0.
func (v Version) Compare(o Version) int {
groups := min(len(v.Groups), len(o.Groups))
for g := 0; g < groups; g++ {
a, b := v.Groups[g], o.Groups[g]
for i := 0; i < max(len(a), len(b)); i++ {
var x, y int
if i < len(a) {
x = a[i]
}
if i < len(b) {
y = b[i]
}
if x < y {
return -1
}
if x > y {
return 1
}
}
}
return 0
}
type constraint struct {
op string
version Version
}
// Range is a set of version constraints. Constraints separated by spaces or
// commas must all hold; alternatives are separated by "||". For example
// ">=8.3.0 <8.3.14 || >=8.4.0 <8.4.2".
type Range struct {
Raw string
alts [][]constraint
}
var constraintRe = regexp.MustCompile(`^(<=|>=|==|!=|<|>|=)?\s*(.+)$`)
// ParseRange parses a version range.
func ParseRange(raw string) (Range, error) {
r := Range{Raw: raw}
if strings.TrimSpace(raw) == "" {
return r, errors.New(i18n.G("empty version range"))
}
for _, alt := range strings.Split(raw, "||") {
// NOTE: allow "< 1.2" by gluing a lone operator to the next token
fields := strings.FieldsFunc(alt, func(r rune) bool { return r == ' ' || r == ',' || r == '\t' })
var tokens []string
for i := 0; i < len(fields); i++ {
if constraintRe.MatchString(fields[i]) && strings.Trim(fields[i], "<>=!") == "" && i+1 < len(fields) {
tokens = append(tokens, fields[i]+fields[i+1])
i++
continue
}
tokens = append(tokens, fields[i])
}
if len(tokens) == 0 {
return r, errors.New(i18n.G("empty alternative in version range %s", raw))
}
var cs []constraint
for _, tok := range tokens {
m := constraintRe.FindStringSubmatch(tok)
if m == nil {
return r, errors.New(i18n.G("unable to parse version range %s", raw))
}
op := m[1]
if op == "" || op == "==" {
op = "="
}
v, err := ParseVersion(m[2])
if err != nil {
return r, errors.New(i18n.G("unable to parse version range %s: %s", raw, err))
}
cs = append(cs, constraint{op: op, version: v})
}
r.alts = append(r.alts, cs)
}
return r, nil
}
// Contains reports whether the version satisfies the range.
func (r Range) Contains(v Version) bool {
for _, alt := range r.alts {
ok := true
for _, c := range alt {
cmp := v.Compare(c.version)
switch c.op {
case "<":
ok = cmp < 0
case "<=":
ok = cmp <= 0
case ">":
ok = cmp > 0
case ">=":
ok = cmp >= 0
case "=":
ok = cmp == 0
case "!=":
ok = cmp != 0
}
if !ok {
break
}
}
if ok {
return true
}
}
return false
}
+1 -1
View File
@@ -120,7 +120,7 @@ func CommandNameComplete(appName string) ([]string, cobra.ShellCompDirective) {
func SecretComplete(recipeName string) ([]string, cobra.ShellCompDirective) {
r := recipe.Get(recipeName)
config, err := r.GetComposeConfig(nil)
config, err := r.GetComposeConfig()
if err != nil {
err := i18n.G("autocomplete failed: %s", err)
return []string{err}, cobra.ShellCompDirectiveError
+1 -52
View File
@@ -4,35 +4,18 @@ import (
"context"
"errors"
"fmt"
"slices"
"coopcloud.tech/abra/pkg/i18n"
"github.com/containers/image/docker"
"github.com/containers/image/manifest"
"github.com/containers/image/types"
"github.com/distribution/reference"
imgspecv1 "github.com/opencontainers/image-spec/specs-go/v1"
)
func init() {
// NOTE: containers/image does not request OCI image indexes by default.
// Many multi-arch images (e.g. on Docker Hub) are published as OCI indexes
// and the registry refuses to serve them without the matching Accept header.
if !slices.Contains(manifest.DefaultRequestedManifestMIMETypes, imgspecv1.MediaTypeImageIndex) {
manifest.DefaultRequestedManifestMIMETypes = append(
manifest.DefaultRequestedManifestMIMETypes,
imgspecv1.MediaTypeImageIndex,
)
}
}
// GetRegistryTags retrieves all tags of an image from a container registry.
func GetRegistryTags(img reference.Named) ([]string, error) {
var tags []string
// NOTE: strip tag and digest, containers/image refuses references
// which have both (e.g. nextcloud:34.0.5-fpm@sha256:...)
ref, err := docker.ParseReference(fmt.Sprintf("//%s", reference.TrimNamed(img)))
ref, err := docker.ParseReference(fmt.Sprintf("//%s", img))
if err != nil {
return tags, errors.New(i18n.G("failed to parse image %s, saw: %s", img, err.Error()))
}
@@ -45,37 +28,3 @@ func GetRegistryTags(img reference.Named) ([]string, error) {
return tags, nil
}
// GetRegistryDigest retrieves the manifest digest of an image tag from a
// container registry. For multi-arch images, this is the digest of the image
// index, which is what Docker resolves "image:tag" to.
func GetRegistryDigest(img reference.Named, tag string) (string, error) {
tagged, err := reference.WithTag(reference.TrimNamed(img), tag)
if err != nil {
return "", errors.New(i18n.G("failed to parse image %s, saw: %s", img, err.Error()))
}
ref, err := docker.ParseReference(fmt.Sprintf("//%s", tagged))
if err != nil {
return "", errors.New(i18n.G("failed to parse image %s, saw: %s", tagged, err.Error()))
}
ctx := context.Background()
src, err := ref.NewImageSource(ctx, &types.SystemContext{})
if err != nil {
return "", err
}
defer src.Close()
manifestBytes, _, err := src.GetManifest(ctx, nil)
if err != nil {
return "", err
}
digest, err := manifest.Digest(manifestBytes)
if err != nil {
return "", err
}
return digest.String(), nil
}
+2 -13
View File
@@ -73,17 +73,6 @@ type Abra struct {
// 2. check if abraDir was set in a config file
// 3. use $HOME/.abra when above two options failed
func (a Abra) GetAbraDir() string {
if dir := a.GetExplicitAbraDir(); dir != "" {
return dir
}
log.Debug(i18n.G("using default abra dir"))
return os.ExpandEnv("$HOME/.abra")
}
// GetExplicitAbraDir returns the abra dir only when it was explicitly chosen,
// either through $ABRA_DIR or through the abraDir setting of a config file.
// It returns an empty string otherwise, and never falls back to $HOME/.abra.
func (a Abra) GetExplicitAbraDir() string {
if dir, exists := os.LookupEnv("ABRA_DIR"); exists && dir != "" {
log.Debug(i18n.G("read abra dir from $ABRA_DIR"))
return dir
@@ -96,7 +85,8 @@ func (a Abra) GetExplicitAbraDir() string {
// Make the path absolute
return path.Join(a.configPath, a.AbraDir)
}
return ""
log.Debug(i18n.G("using default abra dir"))
return os.ExpandEnv("$HOME/.abra")
}
func (a Abra) GetServersDir() string { return path.Join(a.GetAbraDir(), "servers") }
@@ -108,7 +98,6 @@ var config = LoadAbraConfig()
var (
ABRA_DIR = config.GetAbraDir()
ABRA_DIR_EXPLICIT = config.GetExplicitAbraDir()
SERVERS_DIR = config.GetServersDir()
RECIPES_DIR = config.GetRecipesDir()
LOGS_DIR = config.GetLogsDir()
+3 -3
View File
@@ -14,8 +14,8 @@ import (
"coopcloud.tech/abra/pkg/recipe"
"coopcloud.tech/abra/pkg/secret"
composeGoTypes "github.com/compose-spec/compose-go/v2/types"
"github.com/distribution/reference"
composetypes "github.com/docker/cli/cli/compose/types"
"github.com/docker/docker/api/types/swarm"
dockerClient "github.com/docker/docker/client"
)
@@ -229,7 +229,7 @@ func GatherSecretsForDeploy(cl *dockerClient.Client, app appPkg.App, showUnchang
return secretInfo, nil
}
func GatherConfigsForDeploy(cl *dockerClient.Client, app appPkg.App, compose *composetypes.Config, abraShEnv map[string]string, showUnchanged bool) ([]string, error) {
func GatherConfigsForDeploy(cl *dockerClient.Client, app appPkg.App, compose *composeGoTypes.Project, abraShEnv map[string]string, showUnchanged bool) ([]string, error) {
// Get current configs from existing deployment
currentConfigs, err := GetConfigsForStack(cl, app)
if err != nil {
@@ -268,7 +268,7 @@ func GatherConfigsForDeploy(cl *dockerClient.Client, app appPkg.App, compose *co
return configInfo, nil
}
func GatherImagesForDeploy(cl *dockerClient.Client, app appPkg.App, compose *composetypes.Config, showUnchanged bool) ([]string, error) {
func GatherImagesForDeploy(cl *dockerClient.Client, app appPkg.App, compose *composeGoTypes.Project, showUnchanged bool) ([]string, error) {
// Get current images from existing deployment
currentImages, err := GetImagesForStack(cl, app)
if err != nil {
+8 -18
View File
@@ -26,24 +26,12 @@ var BoldUnderlineStyle = lipgloss.NewStyle().
Bold(true).
Underline(true)
// ShortenID returns the first 12 characters of an ID, or the whole ID if it
// is shorter.
func ShortenID(str string) string {
return Truncate(str, 12)
return str[:12]
}
// SmallSHA returns the first 8 characters of a hash, or the whole hash if it
// is shorter (e.g. the placeholder commit of a build without ldflags).
func SmallSHA(hash string) string {
return Truncate(hash, 8)
}
// Truncate returns the first n characters of str, or str if it is shorter.
func Truncate(str string, n int) string {
if len(str) <= n {
return str
}
return str[:n]
return hash[:8]
}
// RemoveSha remove image sha from a string that are added in some docker outputs
@@ -96,10 +84,12 @@ func PrintTable(t *table.Table) error {
tWidth, _ := lipgloss.Size(t.String())
// Without a terminal (a pipe, cron), there is no width to fit: print the
// table as it is instead of failing.
width, _, err := term.GetSize(int(os.Stdout.Fd()))
if err == nil && tWidth > width {
width, _, err := term.GetSize(0)
if err != nil {
return err
}
if tWidth > width {
t.Width(width - 10)
}
-50
View File
@@ -1,9 +1,6 @@
package formatter
import (
"io"
"os"
"strings"
"testing"
"github.com/stretchr/testify/assert"
@@ -12,50 +9,3 @@ import (
func TestBoldDirtyDefault(t *testing.T) {
assert.Equal(t, "foo", BoldDirtyDefault("foo"))
}
func TestSmallSHA(t *testing.T) {
assert.Equal(t, "d699e51c", SmallSHA("d699e51c4c2a8f0e"))
assert.Equal(t, " ", SmallSHA(" "))
assert.Equal(t, "", SmallSHA(""))
}
func TestShortenID(t *testing.T) {
assert.Equal(t, "0123456789ab", ShortenID("0123456789abcdef"))
assert.Equal(t, "abc", ShortenID("abc"))
}
func TestTruncate(t *testing.T) {
assert.Equal(t, "abc", Truncate("abcdef", 3))
assert.Equal(t, "ab", Truncate("ab", 7))
}
func TestPrintTableWithoutTerminal(t *testing.T) {
t.Setenv("ABRA_CI", "")
// stdout is a pipe, not a terminal
r, w, err := os.Pipe()
if err != nil {
t.Fatal(err)
}
stdout := os.Stdout
os.Stdout = w
defer func() { os.Stdout = stdout }()
table, err := CreateTable()
if err != nil {
t.Fatal(err)
}
table.Headers("A", "B")
table.Row("one", "two")
err = PrintTable(table)
w.Close()
out, _ := io.ReadAll(r)
if err != nil {
t.Fatalf("PrintTable without a terminal: %s", err)
}
if !strings.Contains(string(out), "one") {
t.Fatalf("table not printed: %q", out)
}
}
-5
View File
@@ -36,11 +36,6 @@ func IsClean(repoPath string) (bool, error) {
return false, err
}
// Ignore the abra-managed sidecar file that records the canonical
// source URL for externally-cloned recipes; it lives alongside the
// recipe but is not part of the upstream tree.
patterns = append(patterns, gitignore.ParsePattern(".abra-source", nil))
if len(patterns) > 0 {
worktree.Excludes = append(patterns, worktree.Excludes...)
}
-36
View File
@@ -2,8 +2,6 @@ package git
import (
"errors"
"os"
"path/filepath"
"testing"
"github.com/go-git/go-git/v5"
@@ -15,37 +13,3 @@ func TestIsClean(t *testing.T) {
assert.Equal(t, isClean, false)
assert.True(t, errors.Is(err, git.ErrRepositoryNotExists))
}
// TestIsCleanIgnoresAbraSource confirms that the .abra-source sidecar
// file written by abra next to externally-cloned recipes does not cause
// IsClean to report the worktree as dirty.
func TestIsCleanIgnoresAbraSource(t *testing.T) {
dir := t.TempDir()
if _, err := git.PlainInit(dir, false); err != nil {
t.Fatalf("git init failed: %s", err)
}
sidecar := filepath.Join(dir, ".abra-source")
if err := os.WriteFile(sidecar, []byte("git.example.com/u/recipe\n"), 0o644); err != nil {
t.Fatalf("writing sidecar failed: %s", err)
}
isClean, err := IsClean(dir)
if err != nil {
t.Fatalf("IsClean returned error: %s", err)
}
assert.True(t, isClean, "expected worktree with only .abra-source to be reported clean")
// Sanity check: an unrelated untracked file should still mark it dirty.
other := filepath.Join(dir, "random.txt")
if err := os.WriteFile(other, []byte("hello"), 0o644); err != nil {
t.Fatalf("writing extra file failed: %s", err)
}
isClean, err = IsClean(dir)
if err != nil {
t.Fatalf("IsClean returned error: %s", err)
}
assert.False(t, isClean, "expected worktree with unrelated untracked file to be reported dirty")
}
+243 -306
View File
File diff suppressed because it is too large Load Diff
Binary file not shown.
+1402 -583
View File
File diff suppressed because one or more lines are too long
+13 -29
View File
@@ -62,13 +62,6 @@ func (l LintRule) Skip(recipe recipe.Recipe) bool {
var LintRules = map[string][]LintRule{
"warn": {
{
Ref: "R001",
Level: i18n.G("warn"),
Description: i18n.G("compose config has expected version"),
HowToResolve: i18n.G("ensure 'version: \"3.8\"' in compose configs"),
Function: LintComposeVersion,
},
{
Ref: "R002",
Level: i18n.G("warn"),
@@ -217,18 +210,6 @@ func LintForErrors(recipe recipe.Recipe) error {
return nil
}
func LintComposeVersion(recipe recipe.Recipe) (bool, error) {
config, err := recipe.GetComposeConfig(nil)
if err != nil {
return false, err
}
if config.Version == "3.8" {
return true, nil
}
return true, nil
}
func LintEnvConfigPresent(r recipe.Recipe) (bool, error) {
if _, err := os.Stat(r.SampleEnvPath); !os.IsNotExist(err) {
return true, nil
@@ -238,7 +219,7 @@ func LintEnvConfigPresent(r recipe.Recipe) (bool, error) {
}
func LintAppService(recipe recipe.Recipe) (bool, error) {
config, err := recipe.GetComposeConfig(nil)
config, err := recipe.GetComposeConfig()
if err != nil {
return false, err
}
@@ -269,11 +250,14 @@ func LintTraefikEnabledSkipCondition(r recipe.Recipe) (bool, error) {
}
func LintTraefikEnabled(recipe recipe.Recipe) (bool, error) {
config, err := recipe.GetComposeConfig(nil)
config, err := recipe.GetComposeConfig()
if err != nil {
return false, err
}
for _, service := range config.Services {
if service.Deploy == nil {
continue
}
for label := range service.Deploy.Labels {
if label == "traefik.enable" {
if service.Deploy.Labels[label] == "true" {
@@ -287,7 +271,7 @@ func LintTraefikEnabled(recipe recipe.Recipe) (bool, error) {
}
func LintDeployLabelsPresent(recipe recipe.Recipe) (bool, error) {
config, err := recipe.GetComposeConfig(nil)
config, err := recipe.GetComposeConfig()
if err != nil {
return false, err
}
@@ -302,7 +286,7 @@ func LintDeployLabelsPresent(recipe recipe.Recipe) (bool, error) {
}
func LintHealthchecks(recipe recipe.Recipe) (bool, error) {
config, err := recipe.GetComposeConfig(nil)
config, err := recipe.GetComposeConfig()
if err != nil {
return false, err
}
@@ -316,7 +300,7 @@ func LintHealthchecks(recipe recipe.Recipe) (bool, error) {
}
func LintAllImagesTagged(recipe recipe.Recipe) (bool, error) {
config, err := recipe.GetComposeConfig(nil)
config, err := recipe.GetComposeConfig()
if err != nil {
return false, err
}
@@ -334,7 +318,7 @@ func LintAllImagesTagged(recipe recipe.Recipe) (bool, error) {
}
func LintNoUnstableTags(recipe recipe.Recipe) (bool, error) {
config, err := recipe.GetComposeConfig(nil)
config, err := recipe.GetComposeConfig()
if err != nil {
return false, err
}
@@ -361,7 +345,7 @@ func LintNoUnstableTags(recipe recipe.Recipe) (bool, error) {
}
func LintSemverLikeTags(recipe recipe.Recipe) (bool, error) {
config, err := recipe.GetComposeConfig(nil)
config, err := recipe.GetComposeConfig()
if err != nil {
return false, err
}
@@ -388,7 +372,7 @@ func LintSemverLikeTags(recipe recipe.Recipe) (bool, error) {
}
func LintImagePresent(recipe recipe.Recipe) (bool, error) {
config, err := recipe.GetComposeConfig(nil)
config, err := recipe.GetComposeConfig()
if err != nil {
return false, err
}
@@ -440,7 +424,7 @@ func LintMetadataFilledIn(r recipe.Recipe) (bool, error) {
}
func LintAbraShVendors(recipe recipe.Recipe) (bool, error) {
config, err := recipe.GetComposeConfig(nil)
config, err := recipe.GetComposeConfig()
if err != nil {
return false, err
}
@@ -472,7 +456,7 @@ func LintHasRecipeRepo(recipe recipe.Recipe) (bool, error) {
}
func LintSecretLengths(recipe recipe.Recipe) (bool, error) {
config, err := recipe.GetComposeConfig(nil)
config, err := recipe.GetComposeConfig()
if err != nil {
return false, err
}
+17 -84
View File
@@ -3,20 +3,16 @@ package recipe
import (
"errors"
"fmt"
"io/ioutil"
"os"
"path/filepath"
"regexp"
"strings"
"coopcloud.tech/abra/pkg/client"
"coopcloud.tech/abra/pkg/formatter"
"coopcloud.tech/abra/pkg/i18n"
"coopcloud.tech/abra/pkg/log"
"coopcloud.tech/abra/pkg/upstream/stack"
loader "coopcloud.tech/abra/pkg/upstream/stack"
composeGoTypes "github.com/compose-spec/compose-go/v2/types"
"github.com/distribution/reference"
composetypes "github.com/docker/cli/cli/compose/types"
)
// GetComposeFiles gets the list of compose files for an app (or recipe if you
@@ -63,7 +59,7 @@ func (r Recipe) GetComposeFiles(appEnv map[string]string) ([]string, error) {
return composeFiles, nil
}
func (r Recipe) GetComposeConfig(env map[string]string) (*composetypes.Config, error) {
func (r Recipe) GetComposeConfig() (*composeGoTypes.Project, error) {
pattern := fmt.Sprintf("%s/compose**yml", r.Dir)
composeFiles, err := filepath.Glob(pattern)
if err != nil {
@@ -74,25 +70,18 @@ func (r Recipe) GetComposeConfig(env map[string]string) (*composetypes.Config, e
return nil, errors.New(i18n.G("%s is missing a compose.yml or compose.*.yml file?", r.Name))
}
if env == nil {
env, err = r.SampleEnv()
if err != nil {
return nil, err
}
}
opts := stack.Deploy{Composefiles: composeFiles}
config, err := loader.LoadComposefile(opts, env)
config, err := loader.LoadCompose(loader.LoadConf{ComposeFiles: composeFiles})
if err != nil {
return nil, err
}
return config, nil
}
// GetVersionLabelLocal retrieves the version label on the local recipe config
func (r Recipe) GetVersionLabelLocal() (string, error) {
var label string
config, err := r.GetComposeConfig(nil)
config, err := r.GetComposeConfig()
if err != nil {
return "", err
}
@@ -112,14 +101,7 @@ func (r Recipe) GetVersionLabelLocal() (string, error) {
return label, nil
}
// getRegistryDigest is a variable so that it can be stubbed in tests.
var getRegistryDigest = client.GetRegistryDigest
// UpdateTag updates an image tag in-place on file system local compose files.
// Images which are pinned to a digest (e.g. "nextcloud:34.0.5-fpm@sha256:...")
// have their digest updated to the one of the new tag, as retrieved from the
// container registry. Passing the current tag of a digest pinned image
// refreshes the digest, e.g. when upstream re-published the same tag.
func (r Recipe) UpdateTag(image, tag string) (bool, error) {
fullPattern := fmt.Sprintf("%s/compose**yml", r.Dir)
image = formatter.StripTagMeta(image)
@@ -131,20 +113,8 @@ func (r Recipe) UpdateTag(image, tag string) (bool, error) {
log.Debug(i18n.G("considering %s config(s) for tag update", strings.Join(composeFiles, ", ")))
var (
updated bool
newDigest string
)
for _, composeFile := range composeFiles {
opts := stack.Deploy{Composefiles: []string{composeFile}}
sampleEnv, err := r.SampleEnv()
if err != nil {
return false, err
}
compose, err := loader.LoadComposefile(opts, sampleEnv)
compose, err := loader.LoadCompose(loader.LoadConf{ComposeFiles: []string{composeFile}, DisableConsistencyCheck: true})
if err != nil {
return false, err
}
@@ -154,7 +124,7 @@ func (r Recipe) UpdateTag(image, tag string) (bool, error) {
continue // may be a compose.$optional.yml file
}
img, err := reference.ParseNormalizedNamed(service.Image)
img, _ := reference.ParseNormalizedNamed(service.Image)
if err != nil {
return false, err
}
@@ -168,11 +138,6 @@ func (r Recipe) UpdateTag(image, tag string) (bool, error) {
continue
}
var composeDigest string
if canonical, ok := img.(reference.Canonical); ok {
composeDigest = canonical.Digest().String()
}
composeImage := formatter.StripTagMeta(reference.Path(img))
log.Debug(i18n.G("parsed %s from %s", composeTag, service.Image))
@@ -185,43 +150,18 @@ func (r Recipe) UpdateTag(image, tag string) (bool, error) {
old := fmt.Sprintf("%s:%s", composeImage, composeTag)
new := fmt.Sprintf("%s:%s", composeImage, tag)
replacedBytes := strings.ReplaceAll(string(bytes), old, new)
if composeDigest != "" {
if newDigest == "" {
newDigest, err = getRegistryDigest(img, tag)
if err != nil {
return false, errors.New(i18n.G("unable to retrieve digest for %s: %s", new, err))
}
}
log.Debug(i18n.G("updating %s to %s in %s", old, new, composeFile))
old = fmt.Sprintf("%s@%s", old, composeDigest)
new = fmt.Sprintf("%s@%s", new, newDigest)
}
if old == new {
continue
}
// NOTE: only match the full image reference, so that "foo:1.2" does
// not match "foo:1.2.3" or the "foo:1.2" part of "foo:1.2@sha256:..."
pattern := regexp.MustCompile(regexp.QuoteMeta(old) + `([^@\w.\-]|$)`)
replaced := pattern.ReplaceAllString(string(bytes), new+"${1}")
if replaced == string(bytes) {
continue
}
log.Debug(i18n.G("updating %s to %s in %s", old, new, compose.Filename))
if err := os.WriteFile(compose.Filename, []byte(replaced), 0o764); err != nil {
if err := os.WriteFile(composeFile, []byte(replacedBytes), 0o764); err != nil {
return false, err
}
updated = true
}
}
}
return updated, nil
return false, nil
}
// UpdateLabel updates a label in-place on file system local compose files.
@@ -235,20 +175,13 @@ func (r Recipe) UpdateLabel(pattern, serviceName, label string) error {
log.Debug(i18n.G("considering %s config(s) for label update", strings.Join(composeFiles, ", ")))
for _, composeFile := range composeFiles {
opts := stack.Deploy{Composefiles: []string{composeFile}}
sampleEnv, err := r.SampleEnv()
if err != nil {
return err
}
compose, err := loader.LoadComposefile(opts, sampleEnv)
compose, err := loader.LoadCompose(loader.LoadConf{ComposeFiles: []string{composeFile}, DisableConsistencyCheck: true})
if err != nil {
return err
}
serviceExists := false
var service composetypes.ServiceConfig
var service composeGoTypes.ServiceConfig
for _, s := range compose.Services {
if s.Name == serviceName {
service = s
@@ -265,22 +198,22 @@ func (r Recipe) UpdateLabel(pattern, serviceName, label string) error {
if strings.HasPrefix(oldLabel, "coop-cloud") && strings.Contains(oldLabel, "version") {
discovered = true
bytes, err := ioutil.ReadFile(composeFile)
bytes, err := os.ReadFile(composeFile)
if err != nil {
return err
}
old := i18n.G("coop-cloud.${STACK_NAME}.version=%s", value)
replacedBytes := strings.Replace(string(bytes), old, label, -1)
replacedBytes := strings.ReplaceAll(string(bytes), old, label)
if old == label {
log.Warnf(i18n.G("%s is already set, nothing to do?", label))
return nil
}
log.Debug(i18n.G("updating %s to %s in %s", old, label, compose.Filename))
log.Debug(i18n.G("updating %s to %s in %s", old, label, composeFile))
if err := ioutil.WriteFile(compose.Filename, []byte(replacedBytes), 0o764); err != nil {
if err := os.WriteFile(composeFile, []byte(replacedBytes), 0o764); err != nil {
return err
}
-127
View File
@@ -1,127 +0,0 @@
package recipe
import (
"os"
"path"
"testing"
"github.com/distribution/reference"
"github.com/stretchr/testify/assert"
)
const (
oldDigest = "sha256:1111111111111111111111111111111111111111111111111111111111111111"
newDigest = "sha256:2222222222222222222222222222222222222222222222222222222222222222"
)
func setupUpdateTagRecipe(t *testing.T, compose string) (Recipe, string) {
t.Helper()
dir := t.TempDir()
composePath := path.Join(dir, "compose.yml")
envPath := path.Join(dir, ".env.sample")
assert.NoError(t, os.WriteFile(composePath, []byte(compose), 0o644))
assert.NoError(t, os.WriteFile(envPath, []byte("TYPE=foo\n"), 0o644))
return Recipe{
Name: "foo",
Dir: dir,
ComposePath: composePath,
SampleEnvPath: envPath,
}, composePath
}
func stubRegistryDigest(t *testing.T, digest string) *[]string {
t.Helper()
var calls []string
orig := getRegistryDigest
getRegistryDigest = func(img reference.Named, tag string) (string, error) {
calls = append(calls, reference.Path(img)+":"+tag)
return digest, nil
}
t.Cleanup(func() { getRegistryDigest = orig })
return &calls
}
func TestUpdateTag(t *testing.T) {
testcases := []struct {
name string
compose string
image string
tag string
expected string
updated bool
lookups int
}{
{
name: "plain tag",
compose: "version: \"3.8\"\nservices:\n app:\n image: nginx:1.31.7\n",
image: "nginx",
tag: "1.31.8",
expected: "version: \"3.8\"\nservices:\n app:\n image: nginx:1.31.8\n",
updated: true,
},
{
name: "shorter tag does not mangle longer tag",
compose: "version: \"3.8\"\nservices:\n app:\n image: nginx:1.31\n web:\n image: nginx:1.31.7\n",
image: "nginx",
tag: "1.32",
expected: "version: \"3.8\"\nservices:\n app:\n image: nginx:1.32\n web:\n image: nginx:1.32\n",
updated: true,
},
{
name: "digest is updated with tag",
compose: "version: \"3.8\"\nservices:\n app:\n image: nextcloud:34.0.5-fpm@" + oldDigest + "\n",
image: "nextcloud",
tag: "34.0.6-fpm",
expected: "version: \"3.8\"\nservices:\n app:\n image: nextcloud:34.0.6-fpm@" + newDigest + "\n",
updated: true,
lookups: 1,
},
{
name: "digest is refreshed for same tag",
compose: "version: \"3.8\"\nservices:\n app:\n image: \"nextcloud:34.0.5-fpm@" + oldDigest + "\"\n",
image: "nextcloud",
tag: "34.0.5-fpm",
expected: "version: \"3.8\"\nservices:\n app:\n image: \"nextcloud:34.0.5-fpm@" + newDigest + "\"\n",
updated: true,
lookups: 1,
},
{
name: "same tag without digest is a no-op",
compose: "version: \"3.8\"\nservices:\n app:\n image: nextcloud:34.0.5-fpm\n",
image: "nextcloud",
tag: "34.0.5-fpm",
expected: "version: \"3.8\"\nservices:\n app:\n image: nextcloud:34.0.5-fpm\n",
updated: false,
},
{
name: "pinned and unpinned services sharing an image",
compose: "version: \"3.8\"\nservices:\n app:\n image: nextcloud:34.0.5-fpm@" + oldDigest + "\n cron:\n image: nextcloud:34.0.5-fpm\n",
image: "nextcloud",
tag: "34.0.6-fpm",
expected: "version: \"3.8\"\nservices:\n app:\n image: nextcloud:34.0.6-fpm@" + newDigest + "\n cron:\n image: nextcloud:34.0.6-fpm\n",
updated: true,
lookups: 1,
},
}
for _, tc := range testcases {
t.Run(tc.name, func(t *testing.T) {
r, composePath := setupUpdateTagRecipe(t, tc.compose)
calls := stubRegistryDigest(t, newDigest)
updated, err := r.UpdateTag(tc.image, tc.tag)
assert.NoError(t, err)
assert.Equal(t, tc.updated, updated)
assert.Len(t, *calls, tc.lookups)
content, err := os.ReadFile(composePath)
assert.NoError(t, err)
assert.Equal(t, tc.expected, string(content))
})
}
}
+11 -103
View File
@@ -4,7 +4,6 @@ import (
"errors"
"fmt"
"os"
"path"
"slices"
"sort"
"strings"
@@ -21,11 +20,6 @@ import (
"github.com/go-git/go-git/v5/plumbing"
)
// SourceFile is the sidecar file written next to an externally-cloned
// recipe so the canonical "host/path" name can be recovered later (the
// on-disk directory name escapes "/" and "." to "_", which is lossy).
const SourceFile = ".abra-source"
type EnsureContext struct {
Chaos bool
Offline bool
@@ -65,7 +59,7 @@ func (r Recipe) Ensure(ctx EnsureContext) error {
return errors.New(i18n.G(`cannot redeploy previous chaos version (%s), did you mean to use "--chaos"?`))
}
if _, err := r.EnsureVersionCtx(ctx, r.EnvVersion); err != nil {
if _, err := r.EnsureVersion(r.EnvVersion); err != nil {
return err
}
@@ -85,12 +79,6 @@ func (r Recipe) EnsureExists() error {
if err := gitPkg.Clone(r.Dir, r.GitURL); err != nil {
return err
}
if strings.Contains(r.Name, "/") {
sidecar := path.Join(r.Dir, SourceFile)
if err := os.WriteFile(sidecar, []byte(r.Name+"\n"), 0o644); err != nil {
log.Debug(i18n.G("failed to write recipe source sidecar %s: %s", sidecar, err))
}
}
}
if err := gitPkg.EnsureGitRepo(r.Dir); err != nil {
@@ -100,18 +88,6 @@ func (r Recipe) EnsureExists() error {
return nil
}
// ReadRecipeSource returns the canonical name recorded in the .abra-source
// sidecar inside the given recipe directory, or the empty string if no
// sidecar exists. This lets callers recover the unescaped "host/path"
// form for externally-cloned recipes.
func ReadRecipeSource(recipeDir string) string {
data, err := os.ReadFile(path.Join(recipeDir, SourceFile))
if err != nil {
return ""
}
return strings.TrimSpace(string(data))
}
// IsChaosCommit determines if a version sttring is a chaos commit or not.
func (r Recipe) IsChaosCommit(version string) (bool, error) {
isChaosCommit := false
@@ -149,13 +125,6 @@ func (r Recipe) IsChaosCommit(version string) (bool, error) {
// EnsureVersion checks whether a specific version exists for a recipe.
func (r Recipe) EnsureVersion(version string) (bool, error) {
return r.EnsureVersionCtx(EnsureContext{}, version)
}
// EnsureVersionCtx is like EnsureVersion but is aware of the surrounding
// EnsureContext, so that e.g. offline mode can skip network fetches when
// resolving a branch name that isn't already available locally.
func (r Recipe) EnsureVersionCtx(ctx EnsureContext, version string) (bool, error) {
isChaosCommit := false
if err := gitPkg.EnsureGitRepo(r.Dir); err != nil {
@@ -190,46 +159,30 @@ func (r Recipe) EnsureVersionCtx(ctx EnsureContext, version string) (bool, error
}
var opts *git.CheckoutOptions
var checkedOutRef string
if tagRef.String() == "" {
log.Debug(i18n.G("attempting to checkout '%s' as chaos commit", version))
hash, hashErr := repo.ResolveRevision(plumbing.Revision(version))
if hashErr != nil {
branchHash, found, branchErr := resolveBranch(repo, version, ctx.Offline)
if errors.Is(branchErr, errOfflineBranchUnresolved) {
return isChaosCommit, errors.New(i18n.G("'%s' is not available locally and cannot be fetched in offline mode; retry without --offline", version))
}
if branchErr != nil {
return isChaosCommit, branchErr
}
if !found {
return isChaosCommit, errors.New(i18n.G("unable to resolve '%s': %s", version, hashErr))
}
opts = &git.CheckoutOptions{Hash: branchHash, Create: false, Force: true}
isChaosCommit = true
checkedOutRef = version
} else {
opts = &git.CheckoutOptions{Hash: *hash, Create: false, Force: true}
isChaosCommit = true
checkedOutRef = hash.String()
hash, err := repo.ResolveRevision(plumbing.Revision(version))
if err != nil {
log.Fatal(i18n.G("unable to resolve '%s': %s", version, err))
}
opts = &git.CheckoutOptions{Hash: *hash, Create: false, Force: true}
isChaosCommit = true
} else {
opts = &git.CheckoutOptions{Branch: tagRef, Create: false, Force: true}
checkedOutRef = tagRef.Short()
}
worktree, err := repo.Worktree()
if err != nil {
return isChaosCommit, err
return isChaosCommit, nil
}
if err := worktree.Checkout(opts); err != nil {
return isChaosCommit, err
return isChaosCommit, nil
}
log.Debug(i18n.G("successfully checked %s out to %s in %s", r.Name, checkedOutRef, r.Dir))
log.Debug(i18n.G("successfully checked %s out to %s in %s", r.Name, tagRef.Short(), r.Dir))
return isChaosCommit, nil
}
@@ -469,7 +422,7 @@ func (r Recipe) GetRecipeVersions() (RecipeVersions, []string, error) {
log.Debug(i18n.G("git checkout: %s in %s", ref.Name(), r.Dir))
config, err := r.GetComposeConfig(nil)
config, err := r.GetComposeConfig()
if err != nil {
log.Debug(i18n.G("failed to get compose config for %s: %s", tag, err))
warnMsg = append(warnMsg, i18n.G("skipping tag %s: invalid compose config: %s", tag, err))
@@ -532,51 +485,6 @@ func (r Recipe) GetRecipeVersions() (RecipeVersions, []string, error) {
return versions, uniqueWarnings, nil
}
// resolveBranch resolves name as a branch on the "origin" remote, fetching
// it on demand into a remote-tracking ref (never into refs/heads/*, which
// would corrupt GetDefaultBranch's local-branch-name heuristic). It returns
// (hash, true, nil) if name is a branch, (_, false, nil) if name is
// confirmed not to be a branch, and a non-nil error for any other failure
// (e.g. network/auth issues) that should not be silently treated as "not a
// branch". When offline is true, no fetch is attempted and only an
// already-fetched remote-tracking ref is consulted.
func resolveBranch(repo *git.Repository, name string, offline bool) (plumbing.Hash, bool, error) {
remoteTrackingRef := plumbing.NewRemoteReferenceName("origin", name)
if !offline {
refSpec := gitCfg.RefSpec(fmt.Sprintf("+refs/heads/%s:%s", name, remoteTrackingRef))
err := repo.Fetch(&git.FetchOptions{
RemoteName: "origin",
RefSpecs: []gitCfg.RefSpec{refSpec},
Force: true,
})
if err != nil {
var noMatch git.NoMatchingRefSpecError
switch {
case errors.Is(err, git.NoErrAlreadyUpToDate):
case errors.As(err, &noMatch):
return plumbing.ZeroHash, false, nil
default:
return plumbing.ZeroHash, false, errors.New(i18n.G("unable to fetch branch '%s': %s", name, err))
}
}
}
ref, err := repo.Reference(remoteTrackingRef, true)
if err != nil {
if offline {
return plumbing.ZeroHash, false, errOfflineBranchUnresolved
}
return plumbing.ZeroHash, false, errors.New(i18n.G("unable to resolve fetched branch '%s': %s", name, err))
}
return ref.Hash(), true, nil
}
// errOfflineBranchUnresolved is returned by resolveBranch when offline mode
// prevents fetching a branch that isn't already available locally.
var errOfflineBranchUnresolved = errors.New(i18n.G("not fetched and cannot resolve offline"))
// Head retrieves latest HEAD metadata.
func (r Recipe) Head() (*plumbing.Reference, error) {
repo, err := git.PlainOpen(r.Dir)
-131
View File
@@ -6,8 +6,6 @@ import (
"testing"
"coopcloud.tech/abra/pkg/test"
"github.com/go-git/go-git/v5"
"github.com/go-git/go-git/v5/plumbing"
"github.com/stretchr/testify/assert"
)
@@ -38,132 +36,3 @@ func TestIsDirty(t *testing.T) {
assert.True(t, dirty)
}
func TestEnsureVersionBranch(t *testing.T) {
test.SetupWithUpstream()
t.Cleanup(func() { test.Teardown() })
upstream, err := git.PlainOpen(test.UpstreamRecipeDir)
if err != nil {
t.Fatal(err)
}
featureRef, err := upstream.Reference(plumbing.NewBranchReferenceName(test.FeatureBranch), true)
if err != nil {
t.Fatal(err)
}
r := Get(test.RecipeName)
isChaosCommit, err := r.EnsureVersion(test.FeatureBranch)
if err != nil {
t.Fatal(err)
}
assert.True(t, isChaosCommit)
head, err := r.Head()
if err != nil {
t.Fatal(err)
}
assert.Equal(t, featureRef.Hash(), head.Hash())
}
func TestEnsureVersionBranchRefetchesOnRedeploy(t *testing.T) {
test.SetupWithUpstream()
t.Cleanup(func() { test.Teardown() })
r := Get(test.RecipeName)
if _, err := r.EnsureVersion(test.FeatureBranch); err != nil {
t.Fatal(err)
}
if err := test.CreateBranchWithCommit(test.UpstreamRecipeDir, test.FeatureBranch, "feature2.txt", "more feature work"); err != nil {
t.Fatal(err)
}
upstream, err := git.PlainOpen(test.UpstreamRecipeDir)
if err != nil {
t.Fatal(err)
}
featureRef, err := upstream.Reference(plumbing.NewBranchReferenceName(test.FeatureBranch), true)
if err != nil {
t.Fatal(err)
}
if _, err := r.EnsureVersion(test.FeatureBranch); err != nil {
t.Fatal(err)
}
head, err := r.Head()
if err != nil {
t.Fatal(err)
}
assert.Equal(t, featureRef.Hash(), head.Hash())
}
func TestEnsureVersionOfflineUnfetchedBranch(t *testing.T) {
test.SetupWithUpstream()
t.Cleanup(func() { test.Teardown() })
r := Get(test.RecipeName)
// SetupWithUpstream creates the branch before cloning, so the clone
// already has it: remove the remote-tracking ref to make it unfetched.
repo, err := git.PlainOpen(r.Dir)
if err != nil {
t.Fatal(err)
}
if err := repo.Storer.RemoveReference(plumbing.NewRemoteReferenceName("origin", test.FeatureBranch)); err != nil {
t.Fatal(err)
}
headBefore, err := r.Head()
if err != nil {
t.Fatal(err)
}
_, err = r.EnsureVersionCtx(EnsureContext{Offline: true}, test.FeatureBranch)
assert.ErrorContains(t, err, "offline")
headAfter, err := r.Head()
if err != nil {
t.Fatal(err)
}
assert.Equal(t, headBefore.Hash(), headAfter.Hash())
}
func TestEnsureVersionUnknownNameErrors(t *testing.T) {
test.SetupWithUpstream()
t.Cleanup(func() { test.Teardown() })
r := Get(test.RecipeName)
_, err := r.EnsureVersion("does-not-exist-anywhere")
assert.Error(t, err)
}
func TestEnsureBranchPinnedRecipe(t *testing.T) {
test.SetupWithUpstream()
t.Cleanup(func() { test.Teardown() })
upstream, err := git.PlainOpen(test.UpstreamRecipeDir)
if err != nil {
t.Fatal(err)
}
featureRef, err := upstream.Reference(plumbing.NewBranchReferenceName(test.FeatureBranch), true)
if err != nil {
t.Fatal(err)
}
r := Get(test.RecipeName + ":" + test.FeatureBranch)
if err := r.Ensure(EnsureContext{}); err != nil {
t.Fatal(err)
}
head, err := r.Head()
if err != nil {
t.Fatal(err)
}
assert.Equal(t, featureRef.Hash(), head.Hash())
}
-88
View File
@@ -8,7 +8,6 @@ import (
"net/url"
"os"
"path"
"regexp"
"sort"
"strconv"
"strings"
@@ -122,84 +121,7 @@ type Features struct {
SSO string `json:"sso"`
}
// scpURLPattern matches SCP-style git URLs like git@host:path or git@host:port/path.
// Captures: 1=host(:port)?, 2=path.
var scpURLPattern = regexp.MustCompile(`^[\w.-]+@([\w.-]+(?::\d+)?):(.+)$`)
// NormalizeRecipeName canonicalizes a recipe identifier to a stable
// "host/path" form (or returns short catalog names unchanged). Accepts:
//
// - https://host/path[.git][:version]
// - http://host/path[.git][:version]
// - ssh://git@host[:port]/path[.git][:version]
// - git@host:path[.git][:version] (SCP-style)
// - host/path[.git][:version] (already canonical)
// - short-name[:version] (catalog recipe, pass through)
//
// The optional trailing :version suffix is preserved verbatim. The .git
// suffix and trailing slashes on the path are stripped so the four URL
// forms of the same repository collapse to one canonical value.
func NormalizeRecipeName(input string) string {
input = strings.TrimSpace(input)
// Split off the version suffix first, but only if it's not part of a
// scheme (https://) or SCP-style git@host: prefix. The simplest way is
// to detect those prefixes and treat the rest as the path-with-version.
var (
body = input
version string
)
switch {
case strings.HasPrefix(input, "https://") || strings.HasPrefix(input, "http://") || strings.HasPrefix(input, "ssh://"):
u, err := url.Parse(input)
if err != nil {
return input
}
host := u.Hostname() // strip port to keep canonical form colon-free
p := strings.TrimPrefix(u.Path, "/")
p, version = splitVersion(p)
body = host + "/" + p
case scpURLPattern.MatchString(input):
m := scpURLPattern.FindStringSubmatch(input)
host := m[1]
if i := strings.Index(host, ":"); i >= 0 {
host = host[:i] // strip port
}
p, v := splitVersion(m[2])
body = host + "/" + p
version = v
default:
body, version = splitVersion(input)
}
body = strings.TrimSuffix(body, "/")
body = strings.TrimSuffix(body, ".git")
if version != "" {
return body + ":" + version
}
return body
}
// splitVersion separates a trailing ":version" suffix from a recipe path.
// It only treats the final colon-separated segment as a version when it is
// non-empty and contains no slash (a slash means the colon belonged to the
// path, e.g. a host:port or scheme separator).
func splitVersion(s string) (string, string) {
idx := strings.LastIndex(s, ":")
if idx < 0 {
return s, ""
}
candidate := s[idx+1:]
if candidate == "" || strings.Contains(candidate, "/") {
return s, ""
}
return s[:idx], candidate
}
func Get(name string) Recipe {
name = NormalizeRecipeName(name)
version := ""
versionRaw := ""
if strings.Contains(name, ":") {
@@ -287,16 +209,6 @@ func (r Recipe) String() string {
return out
}
// ShortName returns the final path segment of the recipe name, i.e. the
// bare recipe name without any "host/org/" prefix carried by externally
// sourced (git URL) recipes. For catalogue recipes it returns Name unchanged.
func (r Recipe) ShortName() string {
if i := strings.LastIndex(r.Name, "/"); i >= 0 {
return r.Name[i+1:]
}
return r.Name
}
func escapeRecipeName(recipeName string) string {
recipeName = strings.ReplaceAll(recipeName, "/", "_")
recipeName = strings.ReplaceAll(recipeName, ".", "_")
-125
View File
@@ -88,88 +88,6 @@ func TestGet(t *testing.T) {
AbraShPath: path.Join(cfg.GetAbraDir(), "recipes/mygit_org_myorg_cool-recipe/abra.sh"),
},
},
{
name: "https://mygit.org/myorg/cool-recipe",
recipe: Recipe{
Name: "mygit.org/myorg/cool-recipe",
Dir: path.Join(cfg.GetAbraDir(), "/recipes/mygit_org_myorg_cool-recipe"),
GitURL: "https://mygit.org/myorg/cool-recipe.git",
SSHURL: "ssh://git@mygit.org/myorg/cool-recipe.git",
ComposePath: path.Join(cfg.GetAbraDir(), "recipes/mygit_org_myorg_cool-recipe/compose.yml"),
ReadmePath: path.Join(cfg.GetAbraDir(), "recipes/mygit_org_myorg_cool-recipe/README.md"),
SampleEnvPath: path.Join(cfg.GetAbraDir(), "recipes/mygit_org_myorg_cool-recipe/.env.sample"),
AbraShPath: path.Join(cfg.GetAbraDir(), "recipes/mygit_org_myorg_cool-recipe/abra.sh"),
},
},
{
name: "https://mygit.org/myorg/cool-recipe.git",
recipe: Recipe{
Name: "mygit.org/myorg/cool-recipe",
Dir: path.Join(cfg.GetAbraDir(), "/recipes/mygit_org_myorg_cool-recipe"),
GitURL: "https://mygit.org/myorg/cool-recipe.git",
SSHURL: "ssh://git@mygit.org/myorg/cool-recipe.git",
ComposePath: path.Join(cfg.GetAbraDir(), "recipes/mygit_org_myorg_cool-recipe/compose.yml"),
ReadmePath: path.Join(cfg.GetAbraDir(), "recipes/mygit_org_myorg_cool-recipe/README.md"),
SampleEnvPath: path.Join(cfg.GetAbraDir(), "recipes/mygit_org_myorg_cool-recipe/.env.sample"),
AbraShPath: path.Join(cfg.GetAbraDir(), "recipes/mygit_org_myorg_cool-recipe/abra.sh"),
},
},
{
name: "https://mygit.org/myorg/cool-recipe.git:1.2.4",
recipe: Recipe{
Name: "mygit.org/myorg/cool-recipe",
EnvVersion: "1.2.4",
EnvVersionRaw: "1.2.4",
Dir: path.Join(cfg.GetAbraDir(), "/recipes/mygit_org_myorg_cool-recipe"),
GitURL: "https://mygit.org/myorg/cool-recipe.git",
SSHURL: "ssh://git@mygit.org/myorg/cool-recipe.git",
ComposePath: path.Join(cfg.GetAbraDir(), "recipes/mygit_org_myorg_cool-recipe/compose.yml"),
ReadmePath: path.Join(cfg.GetAbraDir(), "recipes/mygit_org_myorg_cool-recipe/README.md"),
SampleEnvPath: path.Join(cfg.GetAbraDir(), "recipes/mygit_org_myorg_cool-recipe/.env.sample"),
AbraShPath: path.Join(cfg.GetAbraDir(), "recipes/mygit_org_myorg_cool-recipe/abra.sh"),
},
},
{
name: "ssh://git@mygit.org/myorg/cool-recipe.git",
recipe: Recipe{
Name: "mygit.org/myorg/cool-recipe",
Dir: path.Join(cfg.GetAbraDir(), "/recipes/mygit_org_myorg_cool-recipe"),
GitURL: "https://mygit.org/myorg/cool-recipe.git",
SSHURL: "ssh://git@mygit.org/myorg/cool-recipe.git",
ComposePath: path.Join(cfg.GetAbraDir(), "recipes/mygit_org_myorg_cool-recipe/compose.yml"),
ReadmePath: path.Join(cfg.GetAbraDir(), "recipes/mygit_org_myorg_cool-recipe/README.md"),
SampleEnvPath: path.Join(cfg.GetAbraDir(), "recipes/mygit_org_myorg_cool-recipe/.env.sample"),
AbraShPath: path.Join(cfg.GetAbraDir(), "recipes/mygit_org_myorg_cool-recipe/abra.sh"),
},
},
{
name: "git@mygit.org:myorg/cool-recipe.git",
recipe: Recipe{
Name: "mygit.org/myorg/cool-recipe",
Dir: path.Join(cfg.GetAbraDir(), "/recipes/mygit_org_myorg_cool-recipe"),
GitURL: "https://mygit.org/myorg/cool-recipe.git",
SSHURL: "ssh://git@mygit.org/myorg/cool-recipe.git",
ComposePath: path.Join(cfg.GetAbraDir(), "recipes/mygit_org_myorg_cool-recipe/compose.yml"),
ReadmePath: path.Join(cfg.GetAbraDir(), "recipes/mygit_org_myorg_cool-recipe/README.md"),
SampleEnvPath: path.Join(cfg.GetAbraDir(), "recipes/mygit_org_myorg_cool-recipe/.env.sample"),
AbraShPath: path.Join(cfg.GetAbraDir(), "recipes/mygit_org_myorg_cool-recipe/abra.sh"),
},
},
{
name: "git@mygit.org:myorg/cool-recipe:1.2.4",
recipe: Recipe{
Name: "mygit.org/myorg/cool-recipe",
EnvVersion: "1.2.4",
EnvVersionRaw: "1.2.4",
Dir: path.Join(cfg.GetAbraDir(), "/recipes/mygit_org_myorg_cool-recipe"),
GitURL: "https://mygit.org/myorg/cool-recipe.git",
SSHURL: "ssh://git@mygit.org/myorg/cool-recipe.git",
ComposePath: path.Join(cfg.GetAbraDir(), "recipes/mygit_org_myorg_cool-recipe/compose.yml"),
ReadmePath: path.Join(cfg.GetAbraDir(), "recipes/mygit_org_myorg_cool-recipe/README.md"),
SampleEnvPath: path.Join(cfg.GetAbraDir(), "recipes/mygit_org_myorg_cool-recipe/.env.sample"),
AbraShPath: path.Join(cfg.GetAbraDir(), "recipes/mygit_org_myorg_cool-recipe/abra.sh"),
},
},
}
for _, tc := range testcases {
@@ -183,49 +101,6 @@ func TestGet(t *testing.T) {
}
}
func TestNormalizeRecipeName(t *testing.T) {
cases := []struct {
in, want string
}{
// catalog short names pass through
{"foo", "foo"},
{"foo:1.2.3", "foo:1.2.3"},
// bare host/path form
{"mygit.org/myorg/cool-recipe", "mygit.org/myorg/cool-recipe"},
{"mygit.org/myorg/cool-recipe.git", "mygit.org/myorg/cool-recipe"},
{"mygit.org/myorg/cool-recipe.git:1.2.3", "mygit.org/myorg/cool-recipe:1.2.3"},
{"mygit.org/myorg/cool-recipe/", "mygit.org/myorg/cool-recipe"},
// https://
{"https://mygit.org/myorg/cool-recipe", "mygit.org/myorg/cool-recipe"},
{"https://mygit.org/myorg/cool-recipe.git", "mygit.org/myorg/cool-recipe"},
{"https://mygit.org/myorg/cool-recipe.git:1.2.3", "mygit.org/myorg/cool-recipe:1.2.3"},
{"http://mygit.org/myorg/cool-recipe", "mygit.org/myorg/cool-recipe"},
// ssh://, with and without port
{"ssh://git@mygit.org/myorg/cool-recipe.git", "mygit.org/myorg/cool-recipe"},
{"ssh://git@mygit.org:2222/myorg/cool-recipe.git", "mygit.org/myorg/cool-recipe"},
// SCP-style git@host:path
{"git@mygit.org:myorg/cool-recipe", "mygit.org/myorg/cool-recipe"},
{"git@mygit.org:myorg/cool-recipe.git", "mygit.org/myorg/cool-recipe"},
{"git@mygit.org:myorg/cool-recipe:1.2.3", "mygit.org/myorg/cool-recipe:1.2.3"},
// whitespace
{" https://mygit.org/myorg/cool-recipe ", "mygit.org/myorg/cool-recipe"},
}
for _, tc := range cases {
t.Run(tc.in, func(t *testing.T) {
got := NormalizeRecipeName(tc.in)
if got != tc.want {
t.Errorf("NormalizeRecipeName(%q) = %q, want %q", tc.in, got, tc.want)
}
})
}
}
func TestGetVersionLabelLocalDoesNotUseTimeoutLabel(t *testing.T) {
test.Setup()
t.Cleanup(func() { test.Teardown() })
+2 -4
View File
@@ -20,7 +20,6 @@ import (
"coopcloud.tech/abra/pkg/envfile"
"coopcloud.tech/abra/pkg/i18n"
"coopcloud.tech/abra/pkg/log"
"coopcloud.tech/abra/pkg/upstream/stack"
loader "coopcloud.tech/abra/pkg/upstream/stack"
"github.com/decentral1se/passgen"
"github.com/docker/docker/api/types"
@@ -122,14 +121,13 @@ func ReadSecretsConfig(appEnvPath string, composeFiles []string, stackName strin
// Set the STACK_NAME to be able to generate the remote name correctly.
appEnv["STACK_NAME"] = stackName
opts := stack.Deploy{Composefiles: composeFiles}
composeConfig, err := loader.LoadComposefile(opts, appEnv)
composeConfig, err := loader.LoadCompose(loader.LoadConf{ComposeFiles: composeFiles, AppEnv: appEnv})
if err != nil {
return nil, err
}
// Read the compose files without injecting environment variables.
configWithoutEnv, err := loader.LoadComposefile(opts, map[string]string{}, loader.SkipInterpolation)
configWithoutEnv, err := loader.LoadCompose(loader.LoadConf{ComposeFiles: composeFiles})
if err != nil {
return nil, err
}
+1 -1
View File
@@ -3,7 +3,7 @@ version: "3.8"
services:
app:
image: nginx:1.31.9
image: nginx:1.31.8
secrets:
- test_pass_one
- test_pass_two
+11 -125
View File
@@ -5,16 +5,16 @@ import (
"log"
"os"
"path"
"path/filepath"
"runtime"
gitPkg "coopcloud.tech/abra/pkg/git"
"git.coopcloud.tech/toolshed/godotenv"
"github.com/go-git/go-git/v5"
"github.com/go-git/go-git/v5/plumbing"
"github.com/go-git/go-git/v5/plumbing/object"
)
var (
AppName = "test_app.example.com"
StackName = "test_app_example_com"
ServerName = "test_server"
RecipeName = "test_recipe"
@@ -62,13 +62,19 @@ func Setup() {
}
}
serverSrcDir := os.ExpandEnv("$PWD/../../tests/resources/test_server")
_, f, _, ok := runtime.Caller(0)
if !ok {
log.Fatal("Setup: unable to discover current working directory of file")
}
pwd := filepath.Dir(f)
serverSrcDir := filepath.Join(pwd, "/../../tests/resources/test_server")
serverDestDir := os.ExpandEnv("$ABRA_DIR/servers/test_server")
if err := os.CopyFS(serverDestDir, os.DirFS(serverSrcDir)); err != nil {
log.Fatal(err)
}
recipeSrcDir := os.ExpandEnv("$PWD/../../tests/resources/test_recipe")
recipeSrcDir := filepath.Join(pwd, "/../../tests/resources/test_recipe")
recipeDestDir := os.ExpandEnv("$ABRA_DIR/recipes/test_recipe")
if err := os.CopyFS(recipeDestDir, os.DirFS(recipeSrcDir)); err != nil {
log.Fatal(err)
@@ -79,126 +85,6 @@ func Setup() {
}
}
// UpstreamRecipeDir is the "remote" repo used by SetupWithUpstream, kept
// outside recipes/ so Teardown's blanket removal of $ABRA_DIR still cleans
// it up.
var UpstreamRecipeDir = os.ExpandEnv("$ABRA_DIR/_upstream/test_recipe")
// FeatureBranch is the non-default branch created by SetupWithUpstream.
const FeatureBranch = "feature"
// SetupWithUpstream is like Setup, but the recipe under $ABRA_DIR/recipes
// is a real clone of a separate "upstream" repo (UpstreamRecipeDir) rather
// than a git-inited copy of the fixture. The upstream repo additionally has
// a FeatureBranch with one extra commit not present on main, so tests can
// exercise resolving/fetching a non-default branch as a recipe :version.
func SetupWithUpstream() {
Teardown()
if err := os.Mkdir(os.ExpandEnv("$ABRA_DIR"), 0764); err != nil {
if !os.IsExist(err) {
log.Fatal(err)
}
}
if err := os.Mkdir(os.ExpandEnv("$ABRA_DIR/servers"), 0700); err != nil {
if !os.IsExist(err) {
log.Fatal(err)
}
}
if err := os.Mkdir(os.ExpandEnv("$ABRA_DIR/recipes"), 0764); err != nil {
if !os.IsExist(err) {
log.Fatal(err)
}
}
serverSrcDir := os.ExpandEnv("$PWD/../../tests/resources/test_server")
serverDestDir := os.ExpandEnv("$ABRA_DIR/servers/test_server")
if err := os.CopyFS(serverDestDir, os.DirFS(serverSrcDir)); err != nil {
log.Fatal(err)
}
if err := os.MkdirAll(path.Dir(UpstreamRecipeDir), 0764); err != nil {
log.Fatal(err)
}
recipeSrcDir := os.ExpandEnv("$PWD/../../tests/resources/test_recipe")
if err := os.CopyFS(UpstreamRecipeDir, os.DirFS(recipeSrcDir)); err != nil {
log.Fatal(err)
}
if err := os.WriteFile(path.Join(UpstreamRecipeDir, ".env.sample"), []byte("TYPE=test_recipe\n"), 0o644); err != nil {
log.Fatal(err)
}
if err := gitPkg.Init(UpstreamRecipeDir, true, "tester", "helo@coopcloud.tech"); err != nil {
log.Fatal(err)
}
if err := CreateBranchWithCommit(UpstreamRecipeDir, FeatureBranch, "feature.txt", "feature work"); err != nil {
log.Fatal(err)
}
recipeDestDir := os.ExpandEnv("$ABRA_DIR/recipes/test_recipe")
if err := gitPkg.Clone(recipeDestDir, UpstreamRecipeDir); err != nil {
log.Fatal(err)
}
}
// CreateBranchWithCommit checks out branchName in repoPath (creating it off
// the current HEAD if it doesn't exist yet, or extending it if it does),
// writes fileName with contents, commits it, then returns to the repo's
// default branch (leaving the new commit reachable only via the branch's
// ref, matching how a contributor would push a feature branch upstream
// without switching the repo's HEAD away from main).
func CreateBranchWithCommit(repoPath, branchName, fileName, contents string) error {
repo, err := git.PlainOpen(repoPath)
if err != nil {
return err
}
head, err := repo.Head()
if err != nil {
return err
}
worktree, err := repo.Worktree()
if err != nil {
return err
}
branchRef := plumbing.NewBranchReferenceName(branchName)
create := true
if _, err := repo.Reference(branchRef, true); err == nil {
create = false
}
checkoutOpts := &git.CheckoutOptions{Branch: branchRef, Create: create}
if create {
checkoutOpts.Hash = head.Hash()
}
if err := worktree.Checkout(checkoutOpts); err != nil {
return err
}
if err := os.WriteFile(path.Join(repoPath, fileName), []byte(contents), 0o644); err != nil {
return err
}
if _, err := worktree.Add(fileName); err != nil {
return err
}
if _, err := worktree.Commit(fileName, &git.CommitOptions{
Author: &object.Signature{Name: "tester", Email: "helo@coopcloud.tech"},
}); err != nil {
return err
}
return worktree.Checkout(&git.CheckoutOptions{Branch: head.Name()})
}
func AddEnv(envKey, envValue string) error {
filePath := os.ExpandEnv(fmt.Sprintf("$ABRA_DIR/servers/%s/%s.env", ServerName, AppName))
+13 -15
View File
@@ -4,7 +4,7 @@ import (
"io/ioutil"
"strings"
composetypes "github.com/docker/cli/cli/compose/types"
composeGoTypes "github.com/compose-spec/compose-go/v2/types"
networktypes "github.com/docker/docker/api/types/network"
"github.com/docker/docker/api/types/swarm"
)
@@ -48,19 +48,17 @@ func AddStackLabel(namespace Namespace, labels map[string]string) map[string]str
return labels
}
type networkMap map[string]composetypes.NetworkConfig
// Networks from the compose-file type to the engine API type
func Networks(namespace Namespace, networks networkMap, servicesNetworks map[string]struct{}) (map[string]networktypes.CreateOptions, []string) {
func Networks(namespace Namespace, networks map[string]composeGoTypes.NetworkConfig, servicesNetworks map[string]struct{}) (map[string]networktypes.CreateOptions, []string) {
if networks == nil {
networks = make(map[string]composetypes.NetworkConfig)
networks = make(map[string]composeGoTypes.NetworkConfig)
}
externalNetworks := []string{}
result := make(map[string]networktypes.CreateOptions)
for internalName := range servicesNetworks {
network := networks[internalName]
if network.External.External {
if network.External {
externalNetworks = append(externalNetworks, network.Name)
continue
}
@@ -98,19 +96,19 @@ func Networks(namespace Namespace, networks networkMap, servicesNetworks map[str
}
// Secrets converts secrets from the Compose type to the engine API type
func Secrets(namespace Namespace, secrets map[string]composetypes.SecretConfig) ([]swarm.SecretSpec, error) {
func Secrets(namespace Namespace, secrets map[string]composeGoTypes.SecretConfig) ([]swarm.SecretSpec, error) {
result := []swarm.SecretSpec{}
for name, secret := range secrets {
if secret.External.External {
if secret.External {
continue
}
var obj swarmFileObject
var err error
if secret.Driver != "" {
obj = driverObjectConfig(namespace, name, composetypes.FileObjectConfig(secret))
obj = driverObjectConfig(namespace, name, composeGoTypes.FileObjectConfig(secret))
} else {
obj, err = fileObjectConfig(namespace, name, composetypes.FileObjectConfig(secret))
obj, err = fileObjectConfig(namespace, name, composeGoTypes.FileObjectConfig(secret))
}
if err != nil {
return nil, err
@@ -133,14 +131,14 @@ func Secrets(namespace Namespace, secrets map[string]composetypes.SecretConfig)
}
// Configs converts config objects from the Compose type to the engine API type
func Configs(namespace Namespace, configs map[string]composetypes.ConfigObjConfig) ([]swarm.ConfigSpec, error) {
func Configs(namespace Namespace, configs map[string]composeGoTypes.ConfigObjConfig) ([]swarm.ConfigSpec, error) {
result := []swarm.ConfigSpec{}
for name, config := range configs {
if config.External.External {
if config.External {
continue
}
obj, err := fileObjectConfig(namespace, name, composetypes.FileObjectConfig(config))
obj, err := fileObjectConfig(namespace, name, composeGoTypes.FileObjectConfig(config))
if err != nil {
return nil, err
}
@@ -160,7 +158,7 @@ type swarmFileObject struct {
Data []byte
}
func driverObjectConfig(namespace Namespace, name string, obj composetypes.FileObjectConfig) swarmFileObject {
func driverObjectConfig(namespace Namespace, name string, obj composeGoTypes.FileObjectConfig) swarmFileObject {
if obj.Name != "" {
name = obj.Name
} else {
@@ -176,7 +174,7 @@ func driverObjectConfig(namespace Namespace, name string, obj composetypes.FileO
}
}
func fileObjectConfig(namespace Namespace, name string, obj composetypes.FileObjectConfig) (swarmFileObject, error) {
func fileObjectConfig(namespace Namespace, name string, obj composeGoTypes.FileObjectConfig) (swarmFileObject, error) {
data, err := ioutil.ReadFile(obj.File)
if err != nil {
return swarmFileObject{}, err
-170
View File
@@ -1,170 +0,0 @@
package convert // https://github.com/docker/cli/blob/master/cli/compose/convert/compose_test.go
import (
"testing"
composetypes "github.com/docker/cli/cli/compose/types"
"github.com/docker/docker/api/types/network"
"gotest.tools/v3/assert"
is "gotest.tools/v3/assert/cmp"
"gotest.tools/v3/fs"
)
func TestNamespaceScope(t *testing.T) {
scoped := Namespace{name: "foo"}.Scope("bar")
assert.Check(t, is.Equal("foo_bar", scoped))
}
func TestAddStackLabel(t *testing.T) {
labels := map[string]string{
"something": "labeled",
}
actual := AddStackLabel(Namespace{name: "foo"}, labels)
expected := map[string]string{
"something": "labeled",
LabelNamespace: "foo",
}
assert.Check(t, is.DeepEqual(expected, actual))
}
func TestNetworks(t *testing.T) {
namespace := Namespace{name: "foo"}
serviceNetworks := map[string]struct{}{
"normal": {},
"outside": {},
"default": {},
"attachablenet": {},
"named": {},
}
source := networkMap{
"normal": composetypes.NetworkConfig{
Driver: "overlay",
DriverOpts: map[string]string{
"opt": "value",
},
Ipam: composetypes.IPAMConfig{
Driver: "driver",
Config: []*composetypes.IPAMPool{
{
Subnet: "10.0.0.0",
},
},
},
Labels: map[string]string{
"something": "labeled",
},
},
"outside": composetypes.NetworkConfig{
External: composetypes.External{External: true},
Name: "special",
},
"attachablenet": composetypes.NetworkConfig{
Driver: "overlay",
Attachable: true,
},
"named": composetypes.NetworkConfig{
Name: "othername",
},
}
expected := map[string]network.CreateOptions{
"foo_default": {
Labels: map[string]string{
LabelNamespace: "foo",
},
},
"foo_normal": {
Driver: "overlay",
IPAM: &network.IPAM{
Driver: "driver",
Config: []network.IPAMConfig{
{
Subnet: "10.0.0.0",
},
},
},
Options: map[string]string{
"opt": "value",
},
Labels: map[string]string{
LabelNamespace: "foo",
"something": "labeled",
},
},
"foo_attachablenet": {
Driver: "overlay",
Attachable: true,
Labels: map[string]string{
LabelNamespace: "foo",
},
},
"othername": {
Labels: map[string]string{LabelNamespace: "foo"},
},
}
networks, externals := Networks(namespace, source, serviceNetworks)
assert.DeepEqual(t, expected, networks)
assert.DeepEqual(t, []string{"special"}, externals)
}
func TestSecrets(t *testing.T) {
namespace := Namespace{name: "foo"}
secretText := "this is the first secret"
secretFile := fs.NewFile(t, "convert-secrets", fs.WithContent(secretText))
defer secretFile.Remove()
source := map[string]composetypes.SecretConfig{
"one": {
File: secretFile.Path(),
Labels: map[string]string{"monster": "mash"},
},
"ext": {
External: composetypes.External{
External: true,
},
},
}
specs, err := Secrets(namespace, source)
assert.NilError(t, err)
assert.Assert(t, is.Len(specs, 1))
secret := specs[0]
assert.Check(t, is.Equal("foo_one", secret.Name))
assert.Check(t, is.DeepEqual(map[string]string{
"monster": "mash",
LabelNamespace: "foo",
}, secret.Labels))
assert.Check(t, is.DeepEqual([]byte(secretText), secret.Data))
}
func TestConfigs(t *testing.T) {
namespace := Namespace{name: "foo"}
configText := "this is the first config"
configFile := fs.NewFile(t, "convert-configs", fs.WithContent(configText))
defer configFile.Remove()
source := map[string]composetypes.ConfigObjConfig{
"one": {
File: configFile.Path(),
Labels: map[string]string{"monster": "mash"},
},
"ext": {
External: composetypes.External{
External: true,
},
},
}
specs, err := Configs(namespace, source)
assert.NilError(t, err)
assert.Assert(t, is.Len(specs, 1))
config := specs[0]
assert.Check(t, is.Equal("foo_one", config.Name))
assert.Check(t, is.DeepEqual(map[string]string{
"monster": "mash",
LabelNamespace: "foo",
}, config.Labels))
assert.Check(t, is.DeepEqual([]byte(configText), config.Data))
}
+148 -83
View File
@@ -5,11 +5,13 @@ import (
"fmt"
"os"
"sort"
"strconv"
"strings"
"time"
"unsafe"
"coopcloud.tech/abra/pkg/i18n"
composetypes "github.com/docker/cli/cli/compose/types"
composeGoTypes "github.com/compose-spec/compose-go/v2/types"
"github.com/docker/cli/opts"
"github.com/docker/docker/api/types"
"github.com/docker/docker/api/types/container"
@@ -178,7 +180,7 @@ func ParseConfigs(client client.ConfigAPIClient, requestedConfigs []*swarmtypes.
// Services from compose-file types to engine API types
func Services(
namespace Namespace,
config *composetypes.Config,
config *composeGoTypes.Project,
client client.CommonAPIClient,
) (map[string]swarm.ServiceSpec, error) {
result := make(map[string]swarm.ServiceSpec)
@@ -211,18 +213,54 @@ func Services(
func Service(
apiVersion string,
namespace Namespace,
service composetypes.ServiceConfig,
networkConfigs map[string]composetypes.NetworkConfig,
volumes map[string]composetypes.VolumeConfig,
service composeGoTypes.ServiceConfig,
networkConfigs map[string]composeGoTypes.NetworkConfig,
volumes map[string]composeGoTypes.VolumeConfig,
secrets []*swarm.SecretReference,
configs []*swarm.ConfigReference,
) (swarm.ServiceSpec, error) {
name := namespace.Scope(service.Name)
endpoint := convertEndpointSpec(service.Deploy.EndpointMode, service.Ports)
mode, err := convertDeployMode(service.Deploy.Mode, service.Deploy.Replicas)
if err != nil {
return swarm.ServiceSpec{}, err
var endpoint *swarmtypes.EndpointSpec
var mode swarmtypes.ServiceMode
var resources *swarmtypes.ResourceRequirements
var restartPolicy *swarmtypes.RestartPolicy
var labels map[string]string
var updateConfig *swarmtypes.UpdateConfig
var rollbackConfig *swarmtypes.UpdateConfig
var placement *swarmtypes.Placement
if service.Deploy != nil {
convertedEndpoint, err := convertEndpointSpec(service.Deploy.EndpointMode, service.Ports)
if err != nil {
return swarm.ServiceSpec{}, err
}
endpoint = convertedEndpoint
convertedMode, err := convertDeployMode(service.Deploy.Mode, service.Deploy.Replicas)
if err != nil {
return swarm.ServiceSpec{}, err
}
mode = convertedMode
convertResources, err := convertResources(service.Deploy.Resources)
if err != nil {
return swarm.ServiceSpec{}, err
}
resources = convertResources
convertedRestartPolicy, err := convertRestartPolicy(
service.Restart, service.Deploy.RestartPolicy)
if err != nil {
return swarm.ServiceSpec{}, err
}
restartPolicy = convertedRestartPolicy
labels = AddStackLabel(namespace, service.Deploy.Labels)
updateConfig = convertUpdateConfig(service.Deploy.UpdateConfig)
rollbackConfig = convertUpdateConfig(service.Deploy.RollbackConfig)
placement = &swarm.Placement{
Constraints: service.Deploy.Placement.Constraints,
Preferences: getPlacementPreference(service.Deploy.Placement.Preferences),
MaxReplicas: service.Deploy.Placement.MaxReplicas,
}
} else {
labels = AddStackLabel(namespace, nil)
}
mounts, err := Volumes(service.Volumes, volumes, namespace)
@@ -230,17 +268,6 @@ func Service(
return swarm.ServiceSpec{}, err
}
resources, err := convertResources(service.Deploy.Resources)
if err != nil {
return swarm.ServiceSpec{}, err
}
restartPolicy, err := convertRestartPolicy(
service.Restart, service.Deploy.RestartPolicy)
if err != nil {
return swarm.ServiceSpec{}, err
}
healthcheck, err := convertHealthcheck(service.HealthCheck)
if err != nil {
return swarm.ServiceSpec{}, err
@@ -254,9 +281,16 @@ func Service(
dnsConfig := convertDNSConfig(service.DNS, service.DNSSearch)
var privileges swarm.Privileges
credSpec := service.CredentialSpec
if credSpec == nil {
credSpec = &composeGoTypes.CredentialSpecConfig{}
}
privileges.CredentialSpec, err = convertCredentialSpec(
namespace, service.CredentialSpec, configs,
namespace, *credSpec, configs,
)
if err != nil {
return swarm.ServiceSpec{}, err
}
@@ -271,10 +305,15 @@ func Service(
capAdd, capDrop := opts.EffectiveCapAddCapDrop(service.CapAdd, service.CapDrop)
var stopGracePtr time.Duration
if service.StopGracePeriod != nil {
stopGracePtr = time.Duration(*service.StopGracePeriod)
}
serviceSpec := swarm.ServiceSpec{
Annotations: swarm.Annotations{
Name: name,
Labels: AddStackLabel(namespace, service.Deploy.Labels),
Labels: labels,
},
TaskTemplate: swarm.TaskSpec{
ContainerSpec: &swarm.ContainerSpec{
@@ -290,7 +329,7 @@ func Service(
Dir: service.WorkingDir,
User: service.User,
Mounts: mounts,
StopGracePeriod: composetypes.ConvertDurationPtr(service.StopGracePeriod),
StopGracePeriod: &stopGracePtr,
StopSignal: service.StopSignal,
TTY: service.Tty,
OpenStdin: service.StdinOpen,
@@ -308,16 +347,12 @@ func Service(
LogDriver: logDriver,
Resources: resources,
RestartPolicy: restartPolicy,
Placement: &swarm.Placement{
Constraints: service.Deploy.Placement.Constraints,
Preferences: getPlacementPreference(service.Deploy.Placement.Preferences),
MaxReplicas: service.Deploy.Placement.MaxReplicas,
},
Placement: placement,
},
EndpointSpec: endpoint,
Mode: mode,
UpdateConfig: convertUpdateConfig(service.Deploy.UpdateConfig),
RollbackConfig: convertUpdateConfig(service.Deploy.RollbackConfig),
UpdateConfig: updateConfig,
RollbackConfig: rollbackConfig,
}
// add an image label to serviceSpec
@@ -338,7 +373,7 @@ func Service(
return serviceSpec, nil
}
func getPlacementPreference(preferences []composetypes.PlacementPreferences) []swarm.PlacementPreference {
func getPlacementPreference(preferences []composeGoTypes.PlacementPreferences) []swarm.PlacementPreference {
result := []swarm.PlacementPreference{}
for _, preference := range preferences {
spreadDescriptor := preference.Spread
@@ -357,13 +392,13 @@ func sortStrings(strs []string) []string {
}
func convertServiceNetworks(
networks map[string]*composetypes.ServiceNetworkConfig,
networkConfigs networkMap,
networks map[string]*composeGoTypes.ServiceNetworkConfig,
networkConfigs map[string]composeGoTypes.NetworkConfig,
namespace Namespace,
name string,
) ([]swarm.NetworkAttachmentConfig, error) {
if len(networks) == 0 {
networks = map[string]*composetypes.ServiceNetworkConfig{
networks = map[string]*composeGoTypes.ServiceNetworkConfig{
defaultNetwork: {},
}
}
@@ -403,20 +438,20 @@ func convertServiceNetworks(
func convertServiceSecrets(
client client.SecretAPIClient,
namespace Namespace,
secrets []composetypes.ServiceSecretConfig,
secretSpecs map[string]composetypes.SecretConfig,
secrets []composeGoTypes.ServiceSecretConfig,
secretSpecs map[string]composeGoTypes.SecretConfig,
) ([]*swarm.SecretReference, error) {
refs := []*swarm.SecretReference{}
lookup := func(key string) (composetypes.FileObjectConfig, error) {
lookup := func(key string) (composeGoTypes.FileObjectConfig, error) {
secretSpec, exists := secretSpecs[key]
if !exists {
return composetypes.FileObjectConfig{}, errors.New(i18n.G("undefined secret %q", key))
return composeGoTypes.FileObjectConfig{}, errors.New(i18n.G("undefined secret %q", key))
}
return composetypes.FileObjectConfig(secretSpec), nil
return composeGoTypes.FileObjectConfig(secretSpec), nil
}
for _, secret := range secrets {
obj, err := convertFileObject(namespace, composetypes.FileReferenceConfig(secret), lookup)
obj, err := convertFileObject(namespace, composeGoTypes.FileReferenceConfig(secret), lookup)
if err != nil {
return nil, err
}
@@ -451,20 +486,20 @@ func convertServiceSecrets(
func convertServiceConfigObjs(
client client.ConfigAPIClient,
namespace Namespace,
service composetypes.ServiceConfig,
configSpecs map[string]composetypes.ConfigObjConfig,
service composeGoTypes.ServiceConfig,
configSpecs map[string]composeGoTypes.ConfigObjConfig,
) ([]*swarm.ConfigReference, error) {
refs := []*swarm.ConfigReference{}
lookup := func(key string) (composetypes.FileObjectConfig, error) {
lookup := func(key string) (composeGoTypes.FileObjectConfig, error) {
configSpec, exists := configSpecs[key]
if !exists {
return composetypes.FileObjectConfig{}, errors.New(i18n.G("undefined config %q", key))
return composeGoTypes.FileObjectConfig{}, errors.New(i18n.G("undefined config %q", key))
}
return composetypes.FileObjectConfig(configSpec), nil
return composeGoTypes.FileObjectConfig(configSpec), nil
}
for _, config := range service.Configs {
obj, err := convertFileObject(namespace, composetypes.FileReferenceConfig(config), lookup)
obj, err := convertFileObject(namespace, composeGoTypes.FileReferenceConfig(config), lookup)
if err != nil {
return nil, err
}
@@ -487,7 +522,7 @@ func convertServiceConfigObjs(
// if the credSpec uses a config, then we should grab the config name, and
// create a config reference for it. A File or Registry-type CredentialSpec
// does not need this operation.
if credSpec.Config != "" {
if credSpec != nil && credSpec.Config != "" {
// look up the config in the configSpecs.
obj, err := lookup(credSpec.Config)
if err != nil {
@@ -532,8 +567,8 @@ type swarmReferenceObject struct {
func convertFileObject(
namespace Namespace,
config composetypes.FileReferenceConfig,
lookup func(key string) (composetypes.FileObjectConfig, error),
config composeGoTypes.FileReferenceConfig,
lookup func(key string) (composeGoTypes.FileObjectConfig, error),
) (swarmReferenceObject, error) {
obj, err := lookup(config.Source)
if err != nil {
@@ -558,40 +593,37 @@ func convertFileObject(
if gid == "" {
gid = "0"
}
mode := config.Mode
if mode == nil {
mode = uint32Ptr(0444)
}
return swarmReferenceObject{
ref := swarmReferenceObject{
File: swarmReferenceTarget{
Name: target,
UID: uid,
GID: gid,
Mode: os.FileMode(*mode),
},
Name: source,
}, nil
}
}
func uint32Ptr(value uint32) *uint32 {
return &value
if config.Mode == nil {
defaultMode := 0444
ref.File.Mode = os.FileMode(defaultMode)
} else {
ref.File.Mode = os.FileMode(*config.Mode)
}
return ref, nil
}
// convertExtraHosts converts <host>:<ip> mappings to SwarmKit notation:
// "IP-address hostname(s)". The original order of mappings is preserved.
func convertExtraHosts(extraHosts composetypes.HostsList) []string {
func convertExtraHosts(extraHosts composeGoTypes.HostsList) []string {
hosts := []string{}
for _, hostIP := range extraHosts {
if v := strings.SplitN(hostIP, ":", 2); len(v) == 2 {
// Convert to SwarmKit notation: IP-address hostname(s)
hosts = append(hosts, fmt.Sprintf("%s %s", v[1], v[0]))
}
for hostName, hostIP := range extraHosts {
hosts = append(hosts, fmt.Sprintf("%s %s", hostIP, hostName))
}
return hosts
}
func convertHealthcheck(healthcheck *composetypes.HealthCheckConfig) (*container.HealthConfig, error) {
func convertHealthcheck(healthcheck *composeGoTypes.HealthCheckConfig) (*container.HealthConfig, error) {
if healthcheck == nil {
return nil, nil
}
@@ -629,7 +661,7 @@ func convertHealthcheck(healthcheck *composetypes.HealthCheckConfig) (*container
}, nil
}
func convertRestartPolicy(restart string, source *composetypes.RestartPolicy) (*swarm.RestartPolicy, error) {
func convertRestartPolicy(restart string, source *composeGoTypes.RestartPolicy) (*swarm.RestartPolicy, error) {
if source == nil {
policy, err := opts.ParseRestartPolicy(restart)
if err != nil {
@@ -653,15 +685,25 @@ func convertRestartPolicy(restart string, source *composetypes.RestartPolicy) (*
}
}
var windowPtr time.Duration
if source.Window != nil {
windowPtr = time.Duration(*source.Window)
}
var delayPtr time.Duration
if source.Delay != nil {
delayPtr = time.Duration(*source.Delay)
}
return &swarm.RestartPolicy{
Condition: swarm.RestartPolicyCondition(source.Condition),
Delay: composetypes.ConvertDurationPtr(source.Delay),
Delay: &delayPtr,
MaxAttempts: source.MaxAttempts,
Window: composetypes.ConvertDurationPtr(source.Window),
Window: &windowPtr,
}, nil
}
func convertUpdateConfig(source *composetypes.UpdateConfig) *swarm.UpdateConfig {
func convertUpdateConfig(source *composeGoTypes.UpdateConfig) *swarm.UpdateConfig {
if source == nil {
return nil
}
@@ -679,13 +721,13 @@ func convertUpdateConfig(source *composetypes.UpdateConfig) *swarm.UpdateConfig
}
}
func convertResources(source composetypes.Resources) (*swarm.ResourceRequirements, error) {
func convertResources(source composeGoTypes.Resources) (*swarm.ResourceRequirements, error) {
resources := &swarm.ResourceRequirements{}
var err error
if source.Limits != nil {
var cpus int64
if source.Limits.NanoCPUs != "" {
cpus, err = opts.ParseCPUs(source.Limits.NanoCPUs)
if source.Limits.NanoCPUs > 0 {
cpus, err = opts.ParseCPUs(fmt.Sprintf("%f", source.Limits.NanoCPUs))
if err != nil {
return nil, err
}
@@ -698,8 +740,8 @@ func convertResources(source composetypes.Resources) (*swarm.ResourceRequirement
}
if source.Reservations != nil {
var cpus int64
if source.Reservations.NanoCPUs != "" {
cpus, err = opts.ParseCPUs(source.Reservations.NanoCPUs)
if source.Reservations.NanoCPUs > 0 {
cpus, err = opts.ParseCPUs(fmt.Sprintf("%f", source.Reservations.NanoCPUs))
if err != nil {
return nil, err
}
@@ -728,13 +770,29 @@ func convertResources(source composetypes.Resources) (*swarm.ResourceRequirement
return resources, nil
}
func convertEndpointSpec(endpointMode string, source []composetypes.ServicePortConfig) *swarm.EndpointSpec {
func str2uint32(s string) (uint32, error) {
var u32 uint32
u64, err := strconv.ParseUint(s, 10, 32)
if err != nil {
return u32, err
}
return uint32(u64), nil
}
func convertEndpointSpec(endpointMode string, source []composeGoTypes.ServicePortConfig) (*swarm.EndpointSpec, error) {
portConfigs := []swarm.PortConfig{}
for _, port := range source {
published, err := str2uint32(port.Published)
if err != nil {
return &swarm.EndpointSpec{}, err
}
portConfig := swarm.PortConfig{
Protocol: swarm.PortConfigProtocol(port.Protocol),
TargetPort: port.Target,
PublishedPort: port.Published,
PublishedPort: published,
PublishMode: swarm.PortConfigPublishMode(port.Mode),
}
portConfigs = append(portConfigs, portConfig)
@@ -747,7 +805,7 @@ func convertEndpointSpec(endpointMode string, source []composetypes.ServicePortC
return &swarm.EndpointSpec{
Mode: swarm.ResolutionMode(strings.ToLower(endpointMode)),
Ports: portConfigs,
}
}, nil
}
func convertEnvironment(source map[string]*string) []string {
@@ -765,7 +823,7 @@ func convertEnvironment(source map[string]*string) []string {
return output
}
func convertDeployMode(mode string, replicas *uint64) (swarm.ServiceMode, error) {
func convertDeployMode(mode string, replicas *int) (swarm.ServiceMode, error) {
serviceMode := swarm.ServiceMode{}
switch mode {
@@ -775,7 +833,8 @@ func convertDeployMode(mode string, replicas *uint64) (swarm.ServiceMode, error)
}
serviceMode.Global = &swarm.GlobalService{}
case "replicated", "":
serviceMode.Replicated = &swarm.ReplicatedService{Replicas: replicas}
convReplicas := (*uint64)(unsafe.Pointer(replicas))
serviceMode.Replicated = &swarm.ReplicatedService{Replicas: convReplicas}
default:
return serviceMode, errors.New(i18n.G("unknown mode: %s", mode))
}
@@ -792,7 +851,7 @@ func convertDNSConfig(DNS []string, DNSSearch []string) *swarm.DNSConfig {
return nil
}
func convertCredentialSpec(namespace Namespace, spec composetypes.CredentialSpecConfig, refs []*swarm.ConfigReference) (*swarm.CredentialSpec, error) {
func convertCredentialSpec(namespace Namespace, spec composeGoTypes.CredentialSpecConfig, refs []*swarm.ConfigReference) (*swarm.CredentialSpec, error) {
var o []string
// Config was added in API v1.40
@@ -814,7 +873,13 @@ func convertCredentialSpec(namespace Namespace, spec composetypes.CredentialSpec
case l > 2:
return nil, errors.New(i18n.G("invalid credential spec: cannot specify both %s, and %s", strings.Join(o[:l-1], ", "), o[l-1]))
}
swarmCredSpec := swarm.CredentialSpec(spec)
swarmCredSpec := swarm.CredentialSpec{
Config: spec.Config,
File: spec.File,
Registry: spec.Registry,
}
// if we're using a swarm Config for the credential spec, over-write it
// here with the config ID
if swarmCredSpec.Config != "" {
@@ -836,7 +901,7 @@ func convertCredentialSpec(namespace Namespace, spec composetypes.CredentialSpec
return &swarmCredSpec, nil
}
func convertUlimits(origUlimits map[string]*composetypes.UlimitsConfig) []*units.Ulimit {
func convertUlimits(origUlimits map[string]*composeGoTypes.UlimitsConfig) []*units.Ulimit {
newUlimits := make(map[string]*units.Ulimit)
for name, u := range origUlimits {
if u.Single != 0 {
-678
View File
@@ -1,678 +0,0 @@
package convert // https://github.com/docker/cli/blob/master/cli/compose/convert/service_test.go
import (
"context"
"os"
"sort"
"strings"
"testing"
"time"
composetypes "github.com/docker/cli/cli/compose/types"
"github.com/docker/docker/api/types"
"github.com/docker/docker/api/types/container"
"github.com/docker/docker/api/types/swarm"
"github.com/docker/docker/client"
"github.com/pkg/errors"
"gotest.tools/v3/assert"
is "gotest.tools/v3/assert/cmp"
)
func TestConvertRestartPolicyFromNone(t *testing.T) {
policy, err := convertRestartPolicy("no", nil)
assert.NilError(t, err)
assert.Check(t, is.DeepEqual((*swarm.RestartPolicy)(nil), policy))
}
func TestConvertRestartPolicyFromUnknown(t *testing.T) {
_, err := convertRestartPolicy("unknown", nil)
assert.Error(t, err, "unknown restart policy: unknown")
}
func TestConvertRestartPolicyFromAlways(t *testing.T) {
policy, err := convertRestartPolicy("always", nil)
expected := &swarm.RestartPolicy{
Condition: swarm.RestartPolicyConditionAny,
}
assert.NilError(t, err)
assert.Check(t, is.DeepEqual(expected, policy))
}
func TestConvertRestartPolicyFromFailure(t *testing.T) {
policy, err := convertRestartPolicy("on-failure:4", nil)
attempts := uint64(4)
expected := &swarm.RestartPolicy{
Condition: swarm.RestartPolicyConditionOnFailure,
MaxAttempts: &attempts,
}
assert.NilError(t, err)
assert.Check(t, is.DeepEqual(expected, policy))
}
func strPtr(val string) *string {
return &val
}
func TestConvertEnvironment(t *testing.T) {
source := map[string]*string{
"foo": strPtr("bar"),
"key": strPtr("value"),
}
env := convertEnvironment(source)
sort.Strings(env)
assert.Check(t, is.DeepEqual([]string{"foo=bar", "key=value"}, env))
}
func TestConvertExtraHosts(t *testing.T) {
source := composetypes.HostsList{
"zulu:127.0.0.2",
"alpha:127.0.0.1",
"zulu:ff02::1",
}
assert.Check(t, is.DeepEqual([]string{"127.0.0.2 zulu", "127.0.0.1 alpha", "ff02::1 zulu"}, convertExtraHosts(source)))
}
func TestConvertResourcesFull(t *testing.T) {
source := composetypes.Resources{
Limits: &composetypes.ResourceLimit{
NanoCPUs: "0.003",
MemoryBytes: composetypes.UnitBytes(300000000),
},
Reservations: &composetypes.Resource{
NanoCPUs: "0.002",
MemoryBytes: composetypes.UnitBytes(200000000),
},
}
resources, err := convertResources(source)
assert.NilError(t, err)
expected := &swarm.ResourceRequirements{
Limits: &swarm.Limit{
NanoCPUs: 3000000,
MemoryBytes: 300000000,
},
Reservations: &swarm.Resources{
NanoCPUs: 2000000,
MemoryBytes: 200000000,
},
}
assert.Check(t, is.DeepEqual(expected, resources))
}
func TestConvertResourcesOnlyMemory(t *testing.T) {
source := composetypes.Resources{
Limits: &composetypes.ResourceLimit{
MemoryBytes: composetypes.UnitBytes(300000000),
},
Reservations: &composetypes.Resource{
MemoryBytes: composetypes.UnitBytes(200000000),
},
}
resources, err := convertResources(source)
assert.NilError(t, err)
expected := &swarm.ResourceRequirements{
Limits: &swarm.Limit{
MemoryBytes: 300000000,
},
Reservations: &swarm.Resources{
MemoryBytes: 200000000,
},
}
assert.Check(t, is.DeepEqual(expected, resources))
}
func TestConvertHealthcheck(t *testing.T) {
retries := uint64(10)
timeout := composetypes.Duration(30 * time.Second)
interval := composetypes.Duration(2 * time.Millisecond)
source := &composetypes.HealthCheckConfig{
Test: []string{"EXEC", "touch", "/foo"},
Timeout: &timeout,
Interval: &interval,
Retries: &retries,
}
expected := &container.HealthConfig{
Test: source.Test,
Timeout: time.Duration(timeout),
Interval: time.Duration(interval),
Retries: 10,
}
healthcheck, err := convertHealthcheck(source)
assert.NilError(t, err)
assert.Check(t, is.DeepEqual(expected, healthcheck))
}
func TestConvertHealthcheckDisable(t *testing.T) {
source := &composetypes.HealthCheckConfig{Disable: true}
expected := &container.HealthConfig{
Test: []string{"NONE"},
}
healthcheck, err := convertHealthcheck(source)
assert.NilError(t, err)
assert.Check(t, is.DeepEqual(expected, healthcheck))
}
func TestConvertHealthcheckDisableWithTest(t *testing.T) {
source := &composetypes.HealthCheckConfig{
Disable: true,
Test: []string{"EXEC", "touch"},
}
_, err := convertHealthcheck(source)
assert.Error(t, err, "test and disable can't be set at the same time")
}
func TestConvertEndpointSpec(t *testing.T) {
source := []composetypes.ServicePortConfig{
{
Protocol: "udp",
Target: 53,
Published: 1053,
Mode: "host",
},
{
Target: 8080,
Published: 80,
},
}
endpoint := convertEndpointSpec("vip", source)
expected := swarm.EndpointSpec{
Mode: swarm.ResolutionMode(strings.ToLower("vip")),
Ports: []swarm.PortConfig{
{
TargetPort: 8080,
PublishedPort: 80,
},
{
Protocol: "udp",
TargetPort: 53,
PublishedPort: 1053,
PublishMode: "host",
},
},
}
assert.Check(t, is.DeepEqual(expected, *endpoint))
}
func TestConvertServiceNetworksOnlyDefault(t *testing.T) {
networkConfigs := networkMap{}
configs, err := convertServiceNetworks(
nil, networkConfigs, NewNamespace("foo"), "service")
expected := []swarm.NetworkAttachmentConfig{
{
Target: "foo_default",
Aliases: []string{"service"},
},
}
assert.NilError(t, err)
assert.Check(t, is.DeepEqual(expected, configs))
}
func TestConvertServiceNetworks(t *testing.T) {
networkConfigs := networkMap{
"front": composetypes.NetworkConfig{
External: composetypes.External{External: true},
Name: "fronttier",
},
"back": composetypes.NetworkConfig{},
}
networks := map[string]*composetypes.ServiceNetworkConfig{
"front": {
Aliases: []string{"something"},
},
"back": {
Aliases: []string{"other"},
},
}
configs, err := convertServiceNetworks(
networks, networkConfigs, NewNamespace("foo"), "service")
expected := []swarm.NetworkAttachmentConfig{
{
Target: "foo_back",
Aliases: []string{"other", "service"},
},
{
Target: "fronttier",
Aliases: []string{"something", "service"},
},
}
assert.NilError(t, err)
assert.Check(t, is.DeepEqual(expected, configs))
}
func TestConvertServiceNetworksCustomDefault(t *testing.T) {
networkConfigs := networkMap{
"default": composetypes.NetworkConfig{
External: composetypes.External{External: true},
Name: "custom",
},
}
networks := map[string]*composetypes.ServiceNetworkConfig{}
configs, err := convertServiceNetworks(
networks, networkConfigs, NewNamespace("foo"), "service")
expected := []swarm.NetworkAttachmentConfig{
{
Target: "custom",
Aliases: []string{"service"},
},
}
assert.NilError(t, err)
assert.Check(t, is.DeepEqual(expected, configs))
}
func TestConvertDNSConfigEmpty(t *testing.T) {
dnsConfig := convertDNSConfig(nil, nil)
assert.Check(t, is.DeepEqual((*swarm.DNSConfig)(nil), dnsConfig))
}
var (
nameservers = []string{"8.8.8.8", "9.9.9.9"}
search = []string{"dc1.example.com", "dc2.example.com"}
)
func TestConvertDNSConfigAll(t *testing.T) {
dnsConfig := convertDNSConfig(nameservers, search)
assert.Check(t, is.DeepEqual(&swarm.DNSConfig{
Nameservers: nameservers,
Search: search,
}, dnsConfig))
}
func TestConvertDNSConfigNameservers(t *testing.T) {
dnsConfig := convertDNSConfig(nameservers, nil)
assert.Check(t, is.DeepEqual(&swarm.DNSConfig{
Nameservers: nameservers,
Search: nil,
}, dnsConfig))
}
func TestConvertDNSConfigSearch(t *testing.T) {
dnsConfig := convertDNSConfig(nil, search)
assert.Check(t, is.DeepEqual(&swarm.DNSConfig{
Nameservers: nil,
Search: search,
}, dnsConfig))
}
func TestConvertCredentialSpec(t *testing.T) {
tests := []struct {
name string
in composetypes.CredentialSpecConfig
out *swarm.CredentialSpec
configs []*swarm.ConfigReference
expectedErr string
}{
{
name: "empty",
},
{
name: "config-and-file",
in: composetypes.CredentialSpecConfig{Config: "0bt9dmxjvjiqermk6xrop3ekq", File: "somefile.json"},
expectedErr: `invalid credential spec: cannot specify both "Config" and "File"`,
},
{
name: "config-and-registry",
in: composetypes.CredentialSpecConfig{Config: "0bt9dmxjvjiqermk6xrop3ekq", Registry: "testing"},
expectedErr: `invalid credential spec: cannot specify both "Config" and "Registry"`,
},
{
name: "file-and-registry",
in: composetypes.CredentialSpecConfig{File: "somefile.json", Registry: "testing"},
expectedErr: `invalid credential spec: cannot specify both "File" and "Registry"`,
},
{
name: "config-and-file-and-registry",
in: composetypes.CredentialSpecConfig{Config: "0bt9dmxjvjiqermk6xrop3ekq", File: "somefile.json", Registry: "testing"},
expectedErr: `invalid credential spec: cannot specify both "Config", "File", and "Registry"`,
},
{
name: "missing-config-reference",
in: composetypes.CredentialSpecConfig{Config: "missing"},
expectedErr: "invalid credential spec: spec specifies config missing, but no such config can be found",
configs: []*swarm.ConfigReference{
{
ConfigName: "someName",
ConfigID: "missing",
},
},
},
{
name: "namespaced-config",
in: composetypes.CredentialSpecConfig{Config: "name"},
configs: []*swarm.ConfigReference{
{
ConfigName: "namespaced-config_name",
ConfigID: "someID",
},
},
out: &swarm.CredentialSpec{Config: "someID"},
},
{
name: "config",
in: composetypes.CredentialSpecConfig{Config: "someName"},
configs: []*swarm.ConfigReference{
{
ConfigName: "someOtherName",
ConfigID: "someOtherID",
}, {
ConfigName: "someName",
ConfigID: "someID",
},
},
out: &swarm.CredentialSpec{Config: "someID"},
},
{
name: "file",
in: composetypes.CredentialSpecConfig{File: "somefile.json"},
out: &swarm.CredentialSpec{File: "somefile.json"},
},
{
name: "registry",
in: composetypes.CredentialSpecConfig{Registry: "testing"},
out: &swarm.CredentialSpec{Registry: "testing"},
},
}
for _, tc := range tests {
tc := tc
t.Run(tc.name, func(t *testing.T) {
namespace := NewNamespace(tc.name)
swarmSpec, err := convertCredentialSpec(namespace, tc.in, tc.configs)
if tc.expectedErr != "" {
assert.Error(t, err, tc.expectedErr)
} else {
assert.NilError(t, err)
}
assert.DeepEqual(t, swarmSpec, tc.out)
})
}
}
func TestConvertUpdateConfigOrder(t *testing.T) {
// test default behavior
updateConfig := convertUpdateConfig(&composetypes.UpdateConfig{})
assert.Check(t, is.Equal("", updateConfig.Order))
// test start-first
updateConfig = convertUpdateConfig(&composetypes.UpdateConfig{
Order: "start-first",
})
assert.Check(t, is.Equal(updateConfig.Order, "start-first"))
// test stop-first
updateConfig = convertUpdateConfig(&composetypes.UpdateConfig{
Order: "stop-first",
})
assert.Check(t, is.Equal(updateConfig.Order, "stop-first"))
}
func TestConvertFileObject(t *testing.T) {
namespace := NewNamespace("testing")
config := composetypes.FileReferenceConfig{
Source: "source",
Target: "target",
UID: "user",
GID: "group",
Mode: uint32Ptr(0644),
}
swarmRef, err := convertFileObject(namespace, config, lookupConfig)
assert.NilError(t, err)
expected := swarmReferenceObject{
Name: "testing_source",
File: swarmReferenceTarget{
Name: config.Target,
UID: config.UID,
GID: config.GID,
Mode: os.FileMode(0644),
},
}
assert.Check(t, is.DeepEqual(expected, swarmRef))
}
func lookupConfig(key string) (composetypes.FileObjectConfig, error) {
if key != "source" {
return composetypes.FileObjectConfig{}, errors.New("bad key")
}
return composetypes.FileObjectConfig{}, nil
}
func TestConvertFileObjectDefaults(t *testing.T) {
namespace := NewNamespace("testing")
config := composetypes.FileReferenceConfig{Source: "source"}
swarmRef, err := convertFileObject(namespace, config, lookupConfig)
assert.NilError(t, err)
expected := swarmReferenceObject{
Name: "testing_source",
File: swarmReferenceTarget{
Name: config.Source,
UID: "0",
GID: "0",
Mode: os.FileMode(0444),
},
}
assert.Check(t, is.DeepEqual(expected, swarmRef))
}
func TestServiceConvertsIsolation(t *testing.T) {
src := composetypes.ServiceConfig{
Isolation: "hyperv",
}
result, err := Service("1.35", Namespace{name: "foo"}, src, nil, nil, nil, nil)
assert.NilError(t, err)
assert.Check(t, is.Equal(container.IsolationHyperV, result.TaskTemplate.ContainerSpec.Isolation))
}
func TestConvertServiceSecrets(t *testing.T) {
namespace := Namespace{name: "foo"}
secrets := []composetypes.ServiceSecretConfig{
{Source: "foo_secret"},
{Source: "bar_secret"},
}
secretSpecs := map[string]composetypes.SecretConfig{
"foo_secret": {
Name: "foo_secret",
},
"bar_secret": {
Name: "bar_secret",
},
}
client := &fakeClient{
secretListFunc: func(opts types.SecretListOptions) ([]swarm.Secret, error) {
assert.Check(t, is.Contains(opts.Filters.Get("name"), "foo_secret"))
assert.Check(t, is.Contains(opts.Filters.Get("name"), "bar_secret"))
return []swarm.Secret{
{Spec: swarm.SecretSpec{Annotations: swarm.Annotations{Name: "foo_secret"}}},
{Spec: swarm.SecretSpec{Annotations: swarm.Annotations{Name: "bar_secret"}}},
}, nil
},
}
refs, err := convertServiceSecrets(client, namespace, secrets, secretSpecs)
assert.NilError(t, err)
expected := []*swarm.SecretReference{
{
SecretName: "bar_secret",
File: &swarm.SecretReferenceFileTarget{
Name: "bar_secret",
UID: "0",
GID: "0",
Mode: 0444,
},
},
{
SecretName: "foo_secret",
File: &swarm.SecretReferenceFileTarget{
Name: "foo_secret",
UID: "0",
GID: "0",
Mode: 0444,
},
},
}
assert.DeepEqual(t, expected, refs)
}
func TestConvertServiceConfigs(t *testing.T) {
namespace := Namespace{name: "foo"}
service := composetypes.ServiceConfig{
Configs: []composetypes.ServiceConfigObjConfig{
{Source: "foo_config"},
{Source: "bar_config"},
},
CredentialSpec: composetypes.CredentialSpecConfig{
Config: "baz_config",
},
}
configSpecs := map[string]composetypes.ConfigObjConfig{
"foo_config": {
Name: "foo_config",
},
"bar_config": {
Name: "bar_config",
},
"baz_config": {
Name: "baz_config",
},
}
client := &fakeClient{
configListFunc: func(opts types.ConfigListOptions) ([]swarm.Config, error) {
assert.Check(t, is.Contains(opts.Filters.Get("name"), "foo_config"))
assert.Check(t, is.Contains(opts.Filters.Get("name"), "bar_config"))
assert.Check(t, is.Contains(opts.Filters.Get("name"), "baz_config"))
return []swarm.Config{
{Spec: swarm.ConfigSpec{Annotations: swarm.Annotations{Name: "foo_config"}}},
{Spec: swarm.ConfigSpec{Annotations: swarm.Annotations{Name: "bar_config"}}},
{Spec: swarm.ConfigSpec{Annotations: swarm.Annotations{Name: "baz_config"}}},
}, nil
},
}
refs, err := convertServiceConfigObjs(client, namespace, service, configSpecs)
assert.NilError(t, err)
expected := []*swarm.ConfigReference{
{
ConfigName: "bar_config",
File: &swarm.ConfigReferenceFileTarget{
Name: "bar_config",
UID: "0",
GID: "0",
Mode: 0444,
},
},
{
ConfigName: "baz_config",
Runtime: &swarm.ConfigReferenceRuntimeTarget{},
},
{
ConfigName: "foo_config",
File: &swarm.ConfigReferenceFileTarget{
Name: "foo_config",
UID: "0",
GID: "0",
Mode: 0444,
},
},
}
assert.DeepEqual(t, expected, refs)
}
type fakeClient struct {
client.Client
secretListFunc func(types.SecretListOptions) ([]swarm.Secret, error)
configListFunc func(types.ConfigListOptions) ([]swarm.Config, error)
}
func (c *fakeClient) SecretList(ctx context.Context, options types.SecretListOptions) ([]swarm.Secret, error) {
if c.secretListFunc != nil {
return c.secretListFunc(options)
}
return []swarm.Secret{}, nil
}
func (c *fakeClient) ConfigList(ctx context.Context, options types.ConfigListOptions) ([]swarm.Config, error) {
if c.configListFunc != nil {
return c.configListFunc(options)
}
return []swarm.Config{}, nil
}
func TestConvertUpdateConfigParallelism(t *testing.T) {
parallel := uint64(4)
// test default behavior
updateConfig := convertUpdateConfig(&composetypes.UpdateConfig{})
assert.Check(t, is.Equal(uint64(1), updateConfig.Parallelism))
// Non default value
updateConfig = convertUpdateConfig(&composetypes.UpdateConfig{
Parallelism: &parallel,
})
assert.Check(t, is.Equal(parallel, updateConfig.Parallelism))
}
func TestConvertServiceCapAddAndCapDrop(t *testing.T) {
tests := []struct {
title string
in, out composetypes.ServiceConfig
}{
{
title: "default behavior",
},
{
title: "some values",
in: composetypes.ServiceConfig{
CapAdd: []string{"SYS_NICE", "CAP_NET_ADMIN"},
CapDrop: []string{"CHOWN", "CAP_NET_ADMIN", "DAC_OVERRIDE", "CAP_FSETID", "CAP_FOWNER"},
},
out: composetypes.ServiceConfig{
CapAdd: []string{"CAP_NET_ADMIN", "CAP_SYS_NICE"},
CapDrop: []string{"CAP_CHOWN", "CAP_DAC_OVERRIDE", "CAP_FOWNER", "CAP_FSETID"},
},
},
{
title: "adding ALL capabilities",
in: composetypes.ServiceConfig{
CapAdd: []string{"ALL", "CAP_NET_ADMIN"},
CapDrop: []string{"CHOWN", "CAP_NET_ADMIN", "DAC_OVERRIDE", "CAP_FSETID", "CAP_FOWNER"},
},
out: composetypes.ServiceConfig{
CapAdd: []string{"ALL"},
CapDrop: []string{"CAP_CHOWN", "CAP_DAC_OVERRIDE", "CAP_FOWNER", "CAP_FSETID", "CAP_NET_ADMIN"},
},
},
{
title: "dropping ALL capabilities",
in: composetypes.ServiceConfig{
CapAdd: []string{"CHOWN", "CAP_NET_ADMIN", "DAC_OVERRIDE", "CAP_FSETID", "CAP_FOWNER"},
CapDrop: []string{"ALL", "CAP_NET_ADMIN", "CAP_FOO"},
},
out: composetypes.ServiceConfig{
CapAdd: []string{"CAP_CHOWN", "CAP_DAC_OVERRIDE", "CAP_FOWNER", "CAP_FSETID", "CAP_NET_ADMIN"},
CapDrop: []string{"ALL"},
},
},
}
for _, tc := range tests {
tc := tc
t.Run(tc.title, func(t *testing.T) {
result, err := Service("1.41", Namespace{name: "foo"}, tc.in, nil, nil, nil, nil)
assert.NilError(t, err)
assert.Check(t, is.DeepEqual(result.TaskTemplate.ContainerSpec.CapabilityAdd, tc.out.CapAdd))
assert.Check(t, is.DeepEqual(result.TaskTemplate.ContainerSpec.CapabilityDrop, tc.out.CapDrop))
})
}
}
+11 -11
View File
@@ -2,15 +2,15 @@ package convert // https://github.com/docker/cli/blob/master/cli/compose/convert
import (
"coopcloud.tech/abra/pkg/i18n"
composetypes "github.com/docker/cli/cli/compose/types"
composeGoTypes "github.com/compose-spec/compose-go/v2/types"
"github.com/docker/docker/api/types/mount"
"github.com/pkg/errors"
)
type volumes map[string]composetypes.VolumeConfig
type volumes map[string]composeGoTypes.VolumeConfig
// Volumes from compose-file types to engine api types
func Volumes(serviceVolumes []composetypes.ServiceVolumeConfig, stackVolumes volumes, namespace Namespace) ([]mount.Mount, error) {
func Volumes(serviceVolumes []composeGoTypes.ServiceVolumeConfig, stackVolumes volumes, namespace Namespace) ([]mount.Mount, error) {
var mounts []mount.Mount
for _, volumeConfig := range serviceVolumes {
@@ -23,7 +23,7 @@ func Volumes(serviceVolumes []composetypes.ServiceVolumeConfig, stackVolumes vol
return mounts, nil
}
func createMountFromVolume(volume composetypes.ServiceVolumeConfig) mount.Mount {
func createMountFromVolume(volume composeGoTypes.ServiceVolumeConfig) mount.Mount {
return mount.Mount{
Type: mount.Type(volume.Type),
Target: volume.Target,
@@ -34,7 +34,7 @@ func createMountFromVolume(volume composetypes.ServiceVolumeConfig) mount.Mount
}
func handleVolumeToMount(
volume composetypes.ServiceVolumeConfig,
volume composeGoTypes.ServiceVolumeConfig,
stackVolumes volumes,
namespace Namespace,
) (mount.Mount, error) {
@@ -68,7 +68,7 @@ func handleVolumeToMount(
}
// External named volumes
if stackVolume.External.External {
if stackVolume.External {
return result, nil
}
@@ -83,7 +83,7 @@ func handleVolumeToMount(
return result, nil
}
func handleBindToMount(volume composetypes.ServiceVolumeConfig) (mount.Mount, error) {
func handleBindToMount(volume composeGoTypes.ServiceVolumeConfig) (mount.Mount, error) {
result := createMountFromVolume(volume)
if volume.Source == "" {
@@ -103,7 +103,7 @@ func handleBindToMount(volume composetypes.ServiceVolumeConfig) (mount.Mount, er
return result, nil
}
func handleTmpfsToMount(volume composetypes.ServiceVolumeConfig) (mount.Mount, error) {
func handleTmpfsToMount(volume composeGoTypes.ServiceVolumeConfig) (mount.Mount, error) {
result := createMountFromVolume(volume)
if volume.Source != "" {
@@ -117,13 +117,13 @@ func handleTmpfsToMount(volume composetypes.ServiceVolumeConfig) (mount.Mount, e
}
if volume.Tmpfs != nil {
result.TmpfsOptions = &mount.TmpfsOptions{
SizeBytes: volume.Tmpfs.Size,
SizeBytes: int64(volume.Tmpfs.Size),
}
}
return result, nil
}
func handleNpipeToMount(volume composetypes.ServiceVolumeConfig) (mount.Mount, error) {
func handleNpipeToMount(volume composeGoTypes.ServiceVolumeConfig) (mount.Mount, error) {
result := createMountFromVolume(volume)
if volume.Source == "" {
@@ -144,7 +144,7 @@ func handleNpipeToMount(volume composetypes.ServiceVolumeConfig) (mount.Mount, e
}
func convertVolumeToMount(
volume composetypes.ServiceVolumeConfig,
volume composeGoTypes.ServiceVolumeConfig,
stackVolumes volumes,
namespace Namespace,
) (mount.Mount, error) {
-361
View File
@@ -1,361 +0,0 @@
package convert // https://github.com/docker/cli/blob/master/cli/compose/convert/volume_test.go
import (
"testing"
composetypes "github.com/docker/cli/cli/compose/types"
"github.com/docker/docker/api/types/mount"
"gotest.tools/v3/assert"
is "gotest.tools/v3/assert/cmp"
)
func TestConvertVolumeToMountAnonymousVolume(t *testing.T) {
config := composetypes.ServiceVolumeConfig{
Type: "volume",
Target: "/foo/bar",
}
expected := mount.Mount{
Type: mount.TypeVolume,
Target: "/foo/bar",
}
mount, err := convertVolumeToMount(config, volumes{}, NewNamespace("foo"))
assert.NilError(t, err)
assert.Check(t, is.DeepEqual(expected, mount))
}
func TestConvertVolumeToMountAnonymousBind(t *testing.T) {
config := composetypes.ServiceVolumeConfig{
Type: "bind",
Target: "/foo/bar",
Bind: &composetypes.ServiceVolumeBind{
Propagation: "slave",
},
}
_, err := convertVolumeToMount(config, volumes{}, NewNamespace("foo"))
assert.Error(t, err, "invalid bind source, source cannot be empty")
}
func TestConvertVolumeToMountUnapprovedType(t *testing.T) {
config := composetypes.ServiceVolumeConfig{
Type: "foo",
Target: "/foo/bar",
}
_, err := convertVolumeToMount(config, volumes{}, NewNamespace("foo"))
assert.Error(t, err, "volume type must be volume, bind, tmpfs or npipe")
}
func TestConvertVolumeToMountConflictingOptionsBindInVolume(t *testing.T) {
namespace := NewNamespace("foo")
config := composetypes.ServiceVolumeConfig{
Type: "volume",
Source: "foo",
Target: "/target",
Bind: &composetypes.ServiceVolumeBind{
Propagation: "slave",
},
}
_, err := convertVolumeToMount(config, volumes{}, namespace)
assert.Error(t, err, "bind options are incompatible with type volume")
}
func TestConvertVolumeToMountConflictingOptionsTmpfsInVolume(t *testing.T) {
namespace := NewNamespace("foo")
config := composetypes.ServiceVolumeConfig{
Type: "volume",
Source: "foo",
Target: "/target",
Tmpfs: &composetypes.ServiceVolumeTmpfs{
Size: 1000,
},
}
_, err := convertVolumeToMount(config, volumes{}, namespace)
assert.Error(t, err, "tmpfs options are incompatible with type volume")
}
func TestConvertVolumeToMountConflictingOptionsVolumeInBind(t *testing.T) {
namespace := NewNamespace("foo")
config := composetypes.ServiceVolumeConfig{
Type: "bind",
Source: "/foo",
Target: "/target",
Volume: &composetypes.ServiceVolumeVolume{
NoCopy: true,
},
}
_, err := convertVolumeToMount(config, volumes{}, namespace)
assert.Error(t, err, "volume options are incompatible with type bind")
}
func TestConvertVolumeToMountConflictingOptionsTmpfsInBind(t *testing.T) {
namespace := NewNamespace("foo")
config := composetypes.ServiceVolumeConfig{
Type: "bind",
Source: "/foo",
Target: "/target",
Tmpfs: &composetypes.ServiceVolumeTmpfs{
Size: 1000,
},
}
_, err := convertVolumeToMount(config, volumes{}, namespace)
assert.Error(t, err, "tmpfs options are incompatible with type bind")
}
func TestConvertVolumeToMountConflictingOptionsBindInTmpfs(t *testing.T) {
namespace := NewNamespace("foo")
config := composetypes.ServiceVolumeConfig{
Type: "tmpfs",
Target: "/target",
Bind: &composetypes.ServiceVolumeBind{
Propagation: "slave",
},
}
_, err := convertVolumeToMount(config, volumes{}, namespace)
assert.Error(t, err, "bind options are incompatible with type tmpfs")
}
func TestConvertVolumeToMountConflictingOptionsVolumeInTmpfs(t *testing.T) {
namespace := NewNamespace("foo")
config := composetypes.ServiceVolumeConfig{
Type: "tmpfs",
Target: "/target",
Volume: &composetypes.ServiceVolumeVolume{
NoCopy: true,
},
}
_, err := convertVolumeToMount(config, volumes{}, namespace)
assert.Error(t, err, "volume options are incompatible with type tmpfs")
}
func TestConvertVolumeToMountNamedVolume(t *testing.T) {
stackVolumes := volumes{
"normal": composetypes.VolumeConfig{
Driver: "glusterfs",
DriverOpts: map[string]string{
"opt": "value",
},
Labels: map[string]string{
"something": "labeled",
},
},
}
namespace := NewNamespace("foo")
expected := mount.Mount{
Type: mount.TypeVolume,
Source: "foo_normal",
Target: "/foo",
ReadOnly: true,
VolumeOptions: &mount.VolumeOptions{
Labels: map[string]string{
LabelNamespace: "foo",
"something": "labeled",
},
DriverConfig: &mount.Driver{
Name: "glusterfs",
Options: map[string]string{
"opt": "value",
},
},
NoCopy: true,
},
}
config := composetypes.ServiceVolumeConfig{
Type: "volume",
Source: "normal",
Target: "/foo",
ReadOnly: true,
Volume: &composetypes.ServiceVolumeVolume{
NoCopy: true,
},
}
mount, err := convertVolumeToMount(config, stackVolumes, namespace)
assert.NilError(t, err)
assert.Check(t, is.DeepEqual(expected, mount))
}
func TestConvertVolumeToMountNamedVolumeWithNameCustomizd(t *testing.T) {
stackVolumes := volumes{
"normal": composetypes.VolumeConfig{
Name: "user_specified_name",
Driver: "vsphere",
DriverOpts: map[string]string{
"opt": "value",
},
Labels: map[string]string{
"something": "labeled",
},
},
}
namespace := NewNamespace("foo")
expected := mount.Mount{
Type: mount.TypeVolume,
Source: "user_specified_name",
Target: "/foo",
ReadOnly: true,
VolumeOptions: &mount.VolumeOptions{
Labels: map[string]string{
LabelNamespace: "foo",
"something": "labeled",
},
DriverConfig: &mount.Driver{
Name: "vsphere",
Options: map[string]string{
"opt": "value",
},
},
NoCopy: true,
},
}
config := composetypes.ServiceVolumeConfig{
Type: "volume",
Source: "normal",
Target: "/foo",
ReadOnly: true,
Volume: &composetypes.ServiceVolumeVolume{
NoCopy: true,
},
}
mount, err := convertVolumeToMount(config, stackVolumes, namespace)
assert.NilError(t, err)
assert.Check(t, is.DeepEqual(expected, mount))
}
func TestConvertVolumeToMountNamedVolumeExternal(t *testing.T) {
stackVolumes := volumes{
"outside": composetypes.VolumeConfig{
Name: "special",
External: composetypes.External{External: true},
},
}
namespace := NewNamespace("foo")
expected := mount.Mount{
Type: mount.TypeVolume,
Source: "special",
Target: "/foo",
VolumeOptions: &mount.VolumeOptions{NoCopy: false},
}
config := composetypes.ServiceVolumeConfig{
Type: "volume",
Source: "outside",
Target: "/foo",
}
mount, err := convertVolumeToMount(config, stackVolumes, namespace)
assert.NilError(t, err)
assert.Check(t, is.DeepEqual(expected, mount))
}
func TestConvertVolumeToMountNamedVolumeExternalNoCopy(t *testing.T) {
stackVolumes := volumes{
"outside": composetypes.VolumeConfig{
Name: "special",
External: composetypes.External{External: true},
},
}
namespace := NewNamespace("foo")
expected := mount.Mount{
Type: mount.TypeVolume,
Source: "special",
Target: "/foo",
VolumeOptions: &mount.VolumeOptions{
NoCopy: true,
},
}
config := composetypes.ServiceVolumeConfig{
Type: "volume",
Source: "outside",
Target: "/foo",
Volume: &composetypes.ServiceVolumeVolume{
NoCopy: true,
},
}
mount, err := convertVolumeToMount(config, stackVolumes, namespace)
assert.NilError(t, err)
assert.Check(t, is.DeepEqual(expected, mount))
}
func TestConvertVolumeToMountBind(t *testing.T) {
stackVolumes := volumes{}
namespace := NewNamespace("foo")
expected := mount.Mount{
Type: mount.TypeBind,
Source: "/bar",
Target: "/foo",
ReadOnly: true,
BindOptions: &mount.BindOptions{Propagation: mount.PropagationShared},
}
config := composetypes.ServiceVolumeConfig{
Type: "bind",
Source: "/bar",
Target: "/foo",
ReadOnly: true,
Bind: &composetypes.ServiceVolumeBind{Propagation: "shared"},
}
mount, err := convertVolumeToMount(config, stackVolumes, namespace)
assert.NilError(t, err)
assert.Check(t, is.DeepEqual(expected, mount))
}
func TestConvertVolumeToMountVolumeDoesNotExist(t *testing.T) {
namespace := NewNamespace("foo")
config := composetypes.ServiceVolumeConfig{
Type: "volume",
Source: "unknown",
Target: "/foo",
ReadOnly: true,
}
_, err := convertVolumeToMount(config, volumes{}, namespace)
assert.Error(t, err, "undefined volume \"unknown\"")
}
func TestConvertTmpfsToMountVolume(t *testing.T) {
config := composetypes.ServiceVolumeConfig{
Type: "tmpfs",
Target: "/foo/bar",
Tmpfs: &composetypes.ServiceVolumeTmpfs{
Size: 1000,
},
}
expected := mount.Mount{
Type: mount.TypeTmpfs,
Target: "/foo/bar",
TmpfsOptions: &mount.TmpfsOptions{SizeBytes: 1000},
}
mount, err := convertVolumeToMount(config, volumes{}, NewNamespace("foo"))
assert.NilError(t, err)
assert.Check(t, is.DeepEqual(expected, mount))
}
func TestConvertTmpfsToMountVolumeWithSource(t *testing.T) {
config := composetypes.ServiceVolumeConfig{
Type: "tmpfs",
Source: "/bar",
Target: "/foo/bar",
Tmpfs: &composetypes.ServiceVolumeTmpfs{
Size: 1000,
},
}
_, err := convertVolumeToMount(config, volumes{}, NewNamespace("foo"))
assert.Error(t, err, "invalid tmpfs source, source must be empty")
}
func TestConvertVolumeToMountAnonymousNpipe(t *testing.T) {
config := composetypes.ServiceVolumeConfig{
Type: "npipe",
Source: `\\.\pipe\foo`,
Target: `\\.\pipe\foo`,
}
expected := mount.Mount{
Type: mount.TypeNamedPipe,
Source: `\\.\pipe\foo`,
Target: `\\.\pipe\foo`,
}
mount, err := convertVolumeToMount(config, volumes{}, NewNamespace("foo"))
assert.NilError(t, err)
assert.Check(t, is.DeepEqual(expected, mount))
}
+42 -132
View File
@@ -1,148 +1,58 @@
package stack // https://github.com/docker/cli/blob/master/cli/command/stack/loader/loader.go
import (
"context"
"fmt"
"io/ioutil"
"path/filepath"
"sort"
"strings"
"io"
"coopcloud.tech/abra/pkg/i18n"
"coopcloud.tech/abra/pkg/log"
"github.com/docker/cli/cli/compose/loader"
"github.com/docker/cli/cli/compose/schema"
composetypes "github.com/docker/cli/cli/compose/types"
composeGoCli "github.com/compose-spec/compose-go/v2/cli"
composeGoTypes "github.com/compose-spec/compose-go/v2/types"
"github.com/pkg/errors"
"github.com/sirupsen/logrus"
)
// DontSkipValidation ensures validation is done for compose file loading
func DontSkipValidation(opts *loader.Options) {
opts.SkipValidation = false
type LoadConf struct {
ComposeFiles []string
AppEnv map[string]string
DisableConsistencyCheck bool
}
// SkipInterpolation skip interpolating environment variables.
func SkipInterpolation(opts *loader.Options) {
opts.SkipInterpolation = true
}
func LoadCompose(conf LoadConf) (*composeGoTypes.Project, error) {
// NOTE(d1): silence compose-go internal logger
logrus.SetOutput(io.Discard)
if len(conf.ComposeFiles) == 0 {
return nil, errors.New(i18n.G("LoadCompose: provide compose files"))
}
// LoadComposefile parse the composefile specified in the cli and returns its Config and version.
func LoadComposefile(opts Deploy, appEnv map[string]string, options ...func(*loader.Options)) (*composetypes.Config, error) {
configDetails, err := getConfigDetails(opts.Composefiles, appEnv)
hasEnvVariables := len(conf.AppEnv) > 0
newProjectOptions := []composeGoCli.ProjectOptionsFn{
composeGoCli.WithInterpolation(hasEnvVariables),
}
if hasEnvVariables {
var env []string
for k, v := range conf.AppEnv {
env = append(env, fmt.Sprintf("%s=%s", k, v))
}
newProjectOptions = append(newProjectOptions, composeGoCli.WithEnv(env))
// prefixes secrets and volumes with stack name
stackName, ok := conf.AppEnv["STACK_NAME"]
if ok && stackName != "" {
newProjectOptions = append(newProjectOptions, composeGoCli.WithName(stackName))
}
}
// this is needed for updating tags and labels, because each file is loaded one by one
// and the consistency check requires every loaded file to have an image with tag
if conf.DisableConsistencyCheck {
newProjectOptions = append(newProjectOptions, composeGoCli.WithConsistency(false))
}
projectOptions, err := composeGoCli.NewProjectOptions(conf.ComposeFiles, newProjectOptions...)
if err != nil {
return nil, err
}
if options == nil {
options = []func(*loader.Options){DontSkipValidation}
}
dicts := getDictsFrom(configDetails.ConfigFiles)
config, err := loader.Load(configDetails, options...)
if err != nil {
if fpe, ok := err.(*loader.ForbiddenPropertiesError); ok {
return nil, errors.New(i18n.G("compose file contains unsupported options: %s", propertyWarnings(fpe.Properties)))
}
return nil, err
}
recipeName, exists := appEnv["RECIPE"]
if !exists {
recipeName, _ = appEnv["TYPE"]
}
unsupportedProperties := loader.GetUnsupportedProperties(dicts...)
if len(unsupportedProperties) > 0 {
log.Warn(i18n.G("%s: ignoring unsupported options: %s", recipeName, strings.Join(unsupportedProperties, ", ")))
}
deprecatedProperties := loader.GetDeprecatedProperties(dicts...)
if len(deprecatedProperties) > 0 {
log.Warn(i18n.G("%s: ignoring deprecated options: %s", recipeName, propertyWarnings(deprecatedProperties)))
}
return config, nil
}
func getDictsFrom(configFiles []composetypes.ConfigFile) []map[string]interface{} {
dicts := []map[string]interface{}{}
for _, configFile := range configFiles {
dicts = append(dicts, configFile.Config)
}
return dicts
}
func propertyWarnings(properties map[string]string) string {
var msgs []string
for name, description := range properties {
msgs = append(msgs, fmt.Sprintf("%s: %s", name, description))
}
sort.Strings(msgs)
return strings.Join(msgs, "\n\n")
}
func getConfigDetails(composefiles []string, appEnv map[string]string) (composetypes.ConfigDetails, error) {
var details composetypes.ConfigDetails
absPath, err := filepath.Abs(composefiles[0])
if err != nil {
return details, err
}
details.WorkingDir = filepath.Dir(absPath)
details.ConfigFiles, err = loadConfigFiles(composefiles)
if err != nil {
return details, err
}
// Take the first file version (2 files can't have different version)
details.Version = schema.Version(details.ConfigFiles[0].Config)
details.Environment = appEnv
return details, err
}
func buildEnvironment(env []string) (map[string]string, error) {
result := make(map[string]string, len(env))
for _, s := range env {
// if value is empty, s is like "K=", not "K".
if !strings.Contains(s, "=") {
return result, errors.New(i18n.G("unexpected environment %q", s))
}
kv := strings.SplitN(s, "=", 2)
result[kv[0]] = kv[1]
}
return result, nil
}
func loadConfigFiles(filenames []string) ([]composetypes.ConfigFile, error) {
var configFiles []composetypes.ConfigFile
for _, filename := range filenames {
configFile, err := loadConfigFile(filename)
if err != nil {
return configFiles, err
}
configFiles = append(configFiles, *configFile)
}
return configFiles, nil
}
func loadConfigFile(filename string) (*composetypes.ConfigFile, error) {
var bytes []byte
var err error
bytes, err = ioutil.ReadFile(filename)
if err != nil {
return nil, err
}
config, err := loader.ParseYAML(bytes)
if err != nil {
return nil, fmt.Errorf("%s: %s", filename, err)
}
return &composetypes.ConfigFile{
Filename: filename,
Config: config,
}, nil
return projectOptions.LoadProject(context.Background())
}
+26
View File
@@ -0,0 +1,26 @@
package stack_test // https://github.com/docker/cli/blob/master/cli/command/stack/loader/loader.go
import (
"testing"
"coopcloud.tech/abra/pkg/app"
"coopcloud.tech/abra/pkg/test"
)
func TestSkipInterpolation(t *testing.T) {
test.Setup()
t.Cleanup(func() { test.Teardown() })
a, err := app.Get(test.AppName)
if err != nil {
t.Fatal(err)
}
_, err = a.Recipe.GetComposeConfig()
if err != nil {
t.Fatal(err)
}
// TODO: ensure compose has a port with no interpolated value
// TODO: ensure compose has port with interpolated value
}
+4 -4
View File
@@ -12,6 +12,7 @@ import (
stdlibErr "errors"
tea "github.com/charmbracelet/bubbletea"
composeGoTypes "github.com/compose-spec/compose-go/v2/types"
"coopcloud.tech/abra/pkg/config"
"coopcloud.tech/abra/pkg/i18n"
@@ -20,7 +21,6 @@ import (
"coopcloud.tech/abra/pkg/upstream/convert"
"github.com/docker/cli/cli/command"
"github.com/docker/cli/cli/command/stack/formatter"
composetypes "github.com/docker/cli/cli/compose/types"
"github.com/docker/docker/api/types"
"github.com/docker/docker/api/types/container"
"github.com/docker/docker/api/types/filters"
@@ -197,7 +197,7 @@ func pruneServices(ctx context.Context, cl *dockerClient.Client, namespace conve
func RunDeploy(
cl *dockerClient.Client,
opts Deploy,
cfg *composetypes.Config,
cfg *composeGoTypes.Project,
appName string,
serverName string,
dontWait bool,
@@ -246,7 +246,7 @@ func deployCompose(
ctx context.Context,
cl *dockerClient.Client,
opts Deploy,
config *composetypes.Config,
config *composeGoTypes.Project,
appName string,
serverName string,
dontWait bool,
@@ -325,7 +325,7 @@ func deployCompose(
return nil
}
func getServicesDeclaredNetworks(serviceConfigs []composetypes.ServiceConfig) map[string]struct{} {
func getServicesDeclaredNetworks(serviceConfigs map[string]composeGoTypes.ServiceConfig) map[string]struct{} {
serviceNetworks := map[string]struct{}{}
for _, serviceConfig := range serviceConfigs {
if len(serviceConfig.Networks) == 0 {
+2 -1
View File
@@ -18,7 +18,8 @@ teardown(){
@test "recipe lint" {
run $ABRA recipe lint gitea
assert_success
assert_output --partial 'compose config has expected version'
# needs to be adjusted when upstream changes
assert_output --partial 'secret lfs_jwt_secret is longer than 12 characters'
}
@test "retrieve recipe if missing" {
@@ -0,0 +1,7 @@
---
services:
app:
ports:
- target: 22
published: ${PORT}
+1 -2
View File
@@ -1,9 +1,8 @@
---
version: "3.8"
services:
app:
image: nginx:1.31.9
image: nginx:1.31.8
networks:
- proxy
deploy:
@@ -1,7 +1,7 @@
Apache License
Version 2.0, January 2004
http://www.apache.org/licenses/
https://www.apache.org/licenses/
TERMS AND CONDITIONS FOR USE, REPRODUCTION, AND DISTRIBUTION
@@ -176,24 +176,13 @@
END OF TERMS AND CONDITIONS
APPENDIX: How to apply the Apache License to your work.
To apply the Apache License to your work, attach the following
boilerplate notice, with the fields enclosed by brackets "[]"
replaced with your own identifying information. (Don't include
the brackets!) The text should be enclosed in the appropriate
comment syntax for the file format. We also recommend that a
file or class name and description of purpose be included on the
same "printed page" as the copyright notice for easier
identification within third-party archives.
Copyright 2015 xeipuuv
Copyright 2020 The Compose Specification Authors
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
https://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
+2
View File
@@ -0,0 +1,2 @@
The Compose Specification
Copyright 2020 The Compose Specification Authors
+590
View File
@@ -0,0 +1,590 @@
/*
Copyright 2020 The Compose Specification Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package cli
import (
"context"
"io"
"os"
"path/filepath"
"strconv"
"strings"
"github.com/sirupsen/logrus"
"go.yaml.in/yaml/v4"
"github.com/compose-spec/compose-go/v2/consts"
"github.com/compose-spec/compose-go/v2/dotenv"
"github.com/compose-spec/compose-go/v2/loader"
"github.com/compose-spec/compose-go/v2/types"
"github.com/compose-spec/compose-go/v2/utils"
)
// ProjectOptions provides common configuration for loading a project.
type ProjectOptions struct {
// Name is a valid Compose project name to be used or empty.
//
// If empty, the project loader will automatically infer a reasonable
// project name if possible.
Name string
// WorkingDir is a file path to use as the project directory or empty.
//
// If empty, the project loader will automatically infer a reasonable
// working directory if possible.
WorkingDir string
// ConfigPaths are file paths to one or more Compose files.
//
// These are applied in order by the loader following the override logic
// as described in the spec.
//
// The first entry is required and is the primary Compose file.
// For convenience, WithConfigFileEnv and WithDefaultConfigPath
// are provided to populate this in a predictable manner.
ConfigPaths []string
// Environment are additional environment variables to make available
// for interpolation.
//
// NOTE: For security, the loader does not automatically expose any
// process environment variables. For convenience, WithOsEnv can be
// used if appropriate.
Environment types.Mapping
// EnvFiles are file paths to ".env" files with additional environment
// variable data.
//
// These are loaded in-order, so it is possible to override variables or
// in subsequent files.
//
// This field is optional, but any file paths that are included here must
// exist or an error will be returned during load.
EnvFiles []string
loadOptions []func(*loader.Options)
// Callbacks to retrieve metadata information during parse defined before
// creating the project
Listeners []loader.Listener
// ResourceLoaders manages support for remote resources
ResourceLoaders []loader.ResourceLoader
}
type ProjectOptionsFn func(*ProjectOptions) error
// NewProjectOptions creates ProjectOptions
func NewProjectOptions(configs []string, opts ...ProjectOptionsFn) (*ProjectOptions, error) {
options := &ProjectOptions{
ConfigPaths: configs,
Environment: map[string]string{},
Listeners: []loader.Listener{},
}
for _, o := range opts {
err := o(options)
if err != nil {
return nil, err
}
}
return options, nil
}
// WithName defines ProjectOptions' name
func WithName(name string) ProjectOptionsFn {
return func(o *ProjectOptions) error {
// a project (once loaded) cannot have an empty name
// however, on the options object, the name is optional: if unset,
// a name will be inferred by the loader, so it's legal to set the
// name to an empty string here
if name != loader.NormalizeProjectName(name) {
return loader.InvalidProjectNameErr(name)
}
o.Name = name
return nil
}
}
// WithWorkingDirectory defines ProjectOptions' working directory
func WithWorkingDirectory(wd string) ProjectOptionsFn {
return func(o *ProjectOptions) error {
if wd == "" {
return nil
}
abs, err := filepath.Abs(wd)
if err != nil {
return err
}
o.WorkingDir = abs
return nil
}
}
// WithConfigFileEnv allow to set compose config file paths by COMPOSE_FILE environment variable
func WithConfigFileEnv(o *ProjectOptions) error {
if len(o.ConfigPaths) > 0 {
return nil
}
sep := o.Environment[consts.ComposePathSeparator]
if sep == "" {
sep = string(os.PathListSeparator)
}
f, ok := o.Environment[consts.ComposeFilePath]
if ok {
paths, err := absolutePaths(strings.Split(f, sep))
o.ConfigPaths = paths
return err
}
return nil
}
// WithDefaultConfigPath searches for default config files from working directory
func WithDefaultConfigPath(o *ProjectOptions) error {
if len(o.ConfigPaths) > 0 {
return nil
}
pwd, err := o.GetWorkingDir()
if err != nil {
return err
}
for {
candidates := findFiles(DefaultFileNames, pwd)
if len(candidates) > 0 {
winner := candidates[0]
if len(candidates) > 1 {
logrus.Warnf("Found multiple config files with supported names: %s", strings.Join(candidates, ", "))
logrus.Warnf("Using %s", winner)
}
o.ConfigPaths = append(o.ConfigPaths, winner)
overrides := findFiles(DefaultOverrideFileNames, pwd)
if len(overrides) > 0 {
if len(overrides) > 1 {
logrus.Warnf("Found multiple override files with supported names: %s", strings.Join(overrides, ", "))
logrus.Warnf("Using %s", overrides[0])
}
o.ConfigPaths = append(o.ConfigPaths, overrides[0])
}
return nil
}
parent := filepath.Dir(pwd)
if parent == pwd {
// no config file found, but that's not a blocker if caller only needs project name
return nil
}
pwd = parent
}
}
// WithEnv defines a key=value set of variables used for compose file interpolation
func WithEnv(env []string) ProjectOptionsFn {
return func(o *ProjectOptions) error {
for k, v := range utils.GetAsEqualsMap(env) {
o.Environment[k] = v
}
return nil
}
}
// WithDiscardEnvFile sets discards the `env_file` section after resolving to
// the `environment` section
func WithDiscardEnvFile(o *ProjectOptions) error {
o.loadOptions = append(o.loadOptions, loader.WithDiscardEnvFiles)
return nil
}
// WithLoadOptions provides a hook to control how compose files are loaded
func WithLoadOptions(loadOptions ...func(*loader.Options)) ProjectOptionsFn {
return func(o *ProjectOptions) error {
o.loadOptions = append(o.loadOptions, loadOptions...)
return nil
}
}
// WithDefaultProfiles uses the provided profiles (if any), and falls back to
// profiles specified via the COMPOSE_PROFILES environment variable otherwise.
func WithDefaultProfiles(profiles ...string) ProjectOptionsFn {
return func(o *ProjectOptions) error {
if len(profiles) == 0 {
for _, s := range strings.Split(o.Environment[consts.ComposeProfiles], ",") {
profiles = append(profiles, strings.TrimSpace(s))
}
}
o.loadOptions = append(o.loadOptions, loader.WithProfiles(profiles))
return nil
}
}
// WithProfiles sets profiles to be activated
func WithProfiles(profiles []string) ProjectOptionsFn {
return func(o *ProjectOptions) error {
o.loadOptions = append(o.loadOptions, loader.WithProfiles(profiles))
return nil
}
}
// WithOsEnv imports environment variables from OS
func WithOsEnv(o *ProjectOptions) error {
for k, v := range utils.GetAsEqualsMap(os.Environ()) {
if _, set := o.Environment[k]; set {
continue
}
o.Environment[k] = v
}
return nil
}
// WithEnvFile sets an alternate env file.
//
// Deprecated: use WithEnvFiles instead.
func WithEnvFile(file string) ProjectOptionsFn {
var files []string
if file != "" {
files = []string{file}
}
return WithEnvFiles(files...)
}
// WithEnvFiles set env file(s) to be loaded to set project environment.
// defaults to local .env file if no explicit file is selected, until COMPOSE_DISABLE_ENV_FILE is set
func WithEnvFiles(file ...string) ProjectOptionsFn {
return func(o *ProjectOptions) error {
if len(file) > 0 {
o.EnvFiles = file
return nil
}
if v, ok := os.LookupEnv(consts.ComposeDisableDefaultEnvFile); ok {
b, err := strconv.ParseBool(v)
if err != nil {
return err
}
if b {
return nil
}
}
wd, err := o.GetWorkingDir()
if err != nil {
return err
}
defaultDotEnv := filepath.Join(wd, ".env")
s, err := os.Stat(defaultDotEnv)
if os.IsNotExist(err) {
return nil
}
if err != nil {
return err
}
if !s.IsDir() {
o.EnvFiles = []string{defaultDotEnv}
}
return nil
}
}
// WithDotEnv imports environment variables from .env file
func WithDotEnv(o *ProjectOptions) error {
envMap, err := dotenv.GetEnvFromFile(o.Environment, o.EnvFiles)
if err != nil {
return err
}
o.Environment.Merge(envMap)
return nil
}
// WithInterpolation set ProjectOptions to enable/skip interpolation
func WithInterpolation(interpolation bool) ProjectOptionsFn {
return func(o *ProjectOptions) error {
o.loadOptions = append(o.loadOptions, func(options *loader.Options) {
options.SkipInterpolation = !interpolation
})
return nil
}
}
// WithNormalization set ProjectOptions to enable/skip normalization
func WithNormalization(normalization bool) ProjectOptionsFn {
return func(o *ProjectOptions) error {
o.loadOptions = append(o.loadOptions, func(options *loader.Options) {
options.SkipNormalization = !normalization
})
return nil
}
}
// WithConsistency set ProjectOptions to enable/skip consistency
func WithConsistency(consistency bool) ProjectOptionsFn {
return func(o *ProjectOptions) error {
o.loadOptions = append(o.loadOptions, func(options *loader.Options) {
options.SkipConsistencyCheck = !consistency
})
return nil
}
}
// WithResolvedPaths set ProjectOptions to enable paths resolution
func WithResolvedPaths(resolve bool) ProjectOptionsFn {
return func(o *ProjectOptions) error {
o.loadOptions = append(o.loadOptions, func(options *loader.Options) {
options.ResolvePaths = resolve
})
return nil
}
}
// WithResourceLoader register support for ResourceLoader to manage remote resources
func WithResourceLoader(r loader.ResourceLoader) ProjectOptionsFn {
return func(o *ProjectOptions) error {
o.ResourceLoaders = append(o.ResourceLoaders, r)
o.loadOptions = append(o.loadOptions, func(options *loader.Options) {
options.ResourceLoaders = o.ResourceLoaders
})
return nil
}
}
// WithExtension register a know extension `x-*` with the go struct type to decode into
func WithExtension(name string, typ any) ProjectOptionsFn {
return func(o *ProjectOptions) error {
o.loadOptions = append(o.loadOptions, func(options *loader.Options) {
if options.KnownExtensions == nil {
options.KnownExtensions = map[string]any{}
}
options.KnownExtensions[name] = typ
})
return nil
}
}
// Append listener to event
func (o *ProjectOptions) WithListeners(listeners ...loader.Listener) {
o.Listeners = append(o.Listeners, listeners...)
}
// WithoutEnvironmentResolution disable environment resolution
func WithoutEnvironmentResolution(o *ProjectOptions) error {
o.loadOptions = append(o.loadOptions, func(options *loader.Options) {
options.SkipResolveEnvironment = true
})
return nil
}
// DefaultFileNames defines the Compose file names for auto-discovery (in order of preference)
var DefaultFileNames = []string{"compose.yaml", "compose.yml", "docker-compose.yml", "docker-compose.yaml"}
// DefaultOverrideFileNames defines the Compose override file names for auto-discovery (in order of preference)
var DefaultOverrideFileNames = []string{"compose.override.yml", "compose.override.yaml", "docker-compose.override.yml", "docker-compose.override.yaml"}
func (o *ProjectOptions) GetWorkingDir() (string, error) {
if o.WorkingDir != "" {
return filepath.Abs(o.WorkingDir)
}
PATH:
for _, path := range o.ConfigPaths {
if path != "-" {
for _, l := range o.ResourceLoaders {
if l.Accept(path) {
break PATH
}
}
absPath, err := filepath.Abs(path)
if err != nil {
return "", err
}
return filepath.Dir(absPath), nil
}
}
return os.Getwd()
}
// ReadConfigFiles reads ConfigFiles and populates the content field
func (o *ProjectOptions) ReadConfigFiles(ctx context.Context, workingDir string, options *ProjectOptions) (*types.ConfigDetails, error) {
config, err := loader.LoadConfigFiles(ctx, options.ConfigPaths, workingDir, options.loadOptions...)
if err != nil {
return nil, err
}
configs := make([][]byte, len(config.ConfigFiles))
for i, c := range config.ConfigFiles {
var err error
var b []byte
if c.IsStdin() {
b, err = io.ReadAll(os.Stdin)
if err != nil {
return nil, err
}
} else {
f, err := filepath.Abs(c.Filename)
if err != nil {
return nil, err
}
b, err = os.ReadFile(f)
if err != nil {
return nil, err
}
}
configs[i] = b
}
for i, c := range configs {
config.ConfigFiles[i].Content = c
}
return config, nil
}
// LoadProject loads compose file according to options and bind to types.Project go structs
func (o *ProjectOptions) LoadProject(ctx context.Context) (*types.Project, error) {
config, err := o.prepare(ctx)
if err != nil {
return nil, err
}
project, err := loader.LoadWithContext(ctx, types.ConfigDetails{
ConfigFiles: config.ConfigFiles,
WorkingDir: config.WorkingDir,
Environment: o.Environment,
}, o.loadOptions...)
if err != nil {
return nil, err
}
for _, config := range config.ConfigFiles {
project.ComposeFiles = append(project.ComposeFiles, config.Filename)
}
return project, nil
}
// LoadModel loads compose file according to options and returns a raw (yaml tree) model
func (o *ProjectOptions) LoadModel(ctx context.Context) (map[string]any, error) {
configDetails, err := o.prepare(ctx)
if err != nil {
return nil, err
}
return loader.LoadModelWithContext(ctx, *configDetails, o.loadOptions...)
}
// prepare converts ProjectOptions into loader's types.ConfigDetails and configures default load options
func (o *ProjectOptions) prepare(ctx context.Context) (*types.ConfigDetails, error) {
defaultDir, err := o.GetWorkingDir()
if err != nil {
return &types.ConfigDetails{}, err
}
configDetails, err := o.ReadConfigFiles(ctx, defaultDir, o)
if err != nil {
return configDetails, err
}
isNamed := false
if o.Name == "" {
type named struct {
Name string `yaml:"name,omitempty"`
}
// if any of the compose file is named, this is equivalent to user passing --project-name
for _, cfg := range configDetails.ConfigFiles {
var n named
err = yaml.Unmarshal(cfg.Content, &n)
if err != nil {
return nil, err
}
if n.Name != "" {
isNamed = true
break
}
}
}
o.loadOptions = append(o.loadOptions,
withNamePrecedenceLoad(defaultDir, isNamed, o),
withConvertWindowsPaths(o),
withListeners(o))
return configDetails, nil
}
// ProjectFromOptions load a compose project based on command line options
// Deprecated: use ProjectOptions.LoadProject or ProjectOptions.LoadModel
func ProjectFromOptions(ctx context.Context, options *ProjectOptions) (*types.Project, error) {
return options.LoadProject(ctx)
}
func withNamePrecedenceLoad(absWorkingDir string, namedInYaml bool, options *ProjectOptions) func(*loader.Options) {
return func(opts *loader.Options) {
if options.Name != "" {
opts.SetProjectName(options.Name, true)
} else if nameFromEnv, ok := options.Environment[consts.ComposeProjectName]; ok && nameFromEnv != "" {
opts.SetProjectName(nameFromEnv, true)
} else if !namedInYaml {
dirname := filepath.Base(absWorkingDir)
symlink, err := filepath.EvalSymlinks(absWorkingDir)
if err == nil && filepath.Base(symlink) != dirname {
logrus.Warnf("project has been loaded without an explicit name from a symlink. Using name %q", dirname)
}
opts.SetProjectName(
loader.NormalizeProjectName(dirname),
false,
)
}
}
}
func withConvertWindowsPaths(options *ProjectOptions) func(*loader.Options) {
return func(o *loader.Options) {
if o.ResolvePaths {
o.ConvertWindowsPaths = utils.StringToBool(options.Environment["COMPOSE_CONVERT_WINDOWS_PATHS"])
}
}
}
// save listeners from ProjectOptions (compose) to loader.Options
func withListeners(options *ProjectOptions) func(*loader.Options) {
return func(opts *loader.Options) {
opts.Listeners = append(opts.Listeners, options.Listeners...)
}
}
func findFiles(names []string, pwd string) []string {
candidates := []string{}
for _, n := range names {
f := filepath.Join(pwd, n)
if _, err := os.Stat(f); err == nil {
candidates = append(candidates, f)
}
}
return candidates
}
func absolutePaths(p []string) ([]string, error) {
var paths []string
for _, f := range p {
if f == "-" {
paths = append(paths, f)
continue
}
abs, err := filepath.Abs(f)
if err != nil {
return nil, err
}
f = abs
if _, err := os.Stat(f); err != nil {
return nil, err
}
paths = append(paths, f)
}
return paths, nil
}
+29
View File
@@ -0,0 +1,29 @@
/*
Copyright 2020 The Compose Specification Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package consts
const (
ComposeProjectName = "COMPOSE_PROJECT_NAME"
ComposePathSeparator = "COMPOSE_PATH_SEPARATOR"
ComposeFilePath = "COMPOSE_FILE"
ComposeDisableDefaultEnvFile = "COMPOSE_DISABLE_ENV_FILE"
ComposeProfiles = "COMPOSE_PROFILES"
)
const Extensions = "#extensions" // Using # prefix, we prevent risk to conflict with an actual yaml key
type ComposeFileKey struct{}
+22
View File
@@ -0,0 +1,22 @@
Copyright (c) 2013 John Barton
MIT License
Permission is hereby granted, free of charge, to any person obtaining
a copy of this software and associated documentation files (the
"Software"), to deal in the Software without restriction, including
without limitation the rights to use, copy, modify, merge, publish,
distribute, sublicense, and/or sell copies of the Software, and to
permit persons to whom the Software is furnished to do so, subject to
the following conditions:
The above copyright notice and this permission notice shall be
included in all copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+73
View File
@@ -0,0 +1,73 @@
/*
Copyright 2020 The Compose Specification Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package dotenv
import (
"bytes"
"fmt"
"os"
"path/filepath"
)
func GetEnvFromFile(currentEnv map[string]string, filenames []string) (map[string]string, error) {
envMap := make(map[string]string)
for _, dotEnvFile := range filenames {
abs, err := filepath.Abs(dotEnvFile)
if err != nil {
return envMap, err
}
dotEnvFile = abs
s, err := os.Stat(dotEnvFile)
if os.IsNotExist(err) {
return envMap, fmt.Errorf("couldn't find env file: %s", dotEnvFile)
}
if err != nil {
return envMap, err
}
if s.IsDir() {
if len(filenames) == 0 {
return envMap, nil
}
return envMap, fmt.Errorf("%s is a directory", dotEnvFile)
}
b, err := os.ReadFile(dotEnvFile)
if os.IsNotExist(err) {
return nil, fmt.Errorf("couldn't read env file: %s", dotEnvFile)
}
if err != nil {
return envMap, err
}
err = parseWithLookup(bytes.NewReader(b), envMap, func(k string) (string, bool) {
v, ok := currentEnv[k]
if ok {
return v, true
}
v, ok = envMap[k]
return v, ok
})
if err != nil {
return envMap, fmt.Errorf("failed to read %s: %w", dotEnvFile, err)
}
}
return envMap, nil
}
+51
View File
@@ -0,0 +1,51 @@
/*
Copyright 2020 The Compose Specification Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package dotenv
import (
"fmt"
"io"
)
const DotEnv = ".env"
var formats = map[string]Parser{
DotEnv: func(r io.Reader, filename string, vars map[string]string, lookup func(key string) (string, bool)) error {
err := parseWithLookup(r, vars, lookup)
if err != nil {
return fmt.Errorf("failed to read %s: %w", filename, err)
}
return nil
},
}
type Parser func(r io.Reader, filename string, vars map[string]string, lookup func(key string) (string, bool)) error
func RegisterFormat(format string, p Parser) {
formats[format] = p
}
func ParseWithFormat(r io.Reader, filename string, vars map[string]string, resolve LookupFn, format string) error {
if format == "" {
format = DotEnv
}
fn, ok := formats[format]
if !ok {
return fmt.Errorf("unsupported env_file format %q", format)
}
return fn(r, filename, vars, resolve)
}
+182
View File
@@ -0,0 +1,182 @@
// Package dotenv is a go port of the ruby dotenv library (https://github.com/bkeepers/dotenv)
//
// Examples/readme can be found on the github page at https://github.com/joho/godotenv
//
// The TL;DR is that you make a .env file that looks something like
//
// SOME_ENV_VAR=somevalue
//
// and then in your go code you can call
//
// godotenv.Load()
//
// and all the env vars declared in .env will be available through os.Getenv("SOME_ENV_VAR")
package dotenv
import (
"bytes"
"io"
"os"
"regexp"
"strings"
"github.com/compose-spec/compose-go/v2/template"
)
var utf8BOM = []byte("\uFEFF")
var startsWithDigitRegex = regexp.MustCompile(`^\s*\d.*`) // Keys starting with numbers are ignored
// LookupFn represents a lookup function to resolve variables from
type LookupFn func(string) (string, bool)
var noLookupFn = func(_ string) (string, bool) {
return "", false
}
// Parse reads an env file from io.Reader, returning a map of keys and values.
func Parse(r io.Reader) (map[string]string, error) {
return ParseWithLookup(r, nil)
}
// ParseWithLookup reads an env file from io.Reader, returning a map of keys and values.
func ParseWithLookup(r io.Reader, lookupFn LookupFn) (map[string]string, error) {
vars := map[string]string{}
err := parseWithLookup(r, vars, lookupFn)
return vars, err
}
// ParseWithLookup reads an env file from io.Reader, returning a map of keys and values.
func parseWithLookup(r io.Reader, vars map[string]string, lookupFn LookupFn) error {
data, err := io.ReadAll(r)
if err != nil {
return err
}
// seek past the UTF-8 BOM if it exists (particularly on Windows, some
// editors tend to add it, and it'll cause parsing to fail)
data = bytes.TrimPrefix(data, utf8BOM)
return newParser().parse(string(data), vars, lookupFn)
}
// Load will read your env file(s) and load them into ENV for this process.
//
// Call this function as close as possible to the start of your program (ideally in main).
//
// If you call Load without any args it will default to loading .env in the current path.
//
// You can otherwise tell it which files to load (there can be more than one) like:
//
// godotenv.Load("fileone", "filetwo")
//
// It's important to note that it WILL NOT OVERRIDE an env variable that already exists - consider the .env file to set dev vars or sensible defaults
func Load(filenames ...string) error {
return load(false, filenames...)
}
func load(overload bool, filenames ...string) error {
filenames = filenamesOrDefault(filenames)
for _, filename := range filenames {
err := loadFile(filename, overload)
if err != nil {
return err
}
}
return nil
}
// ReadWithLookup gets all env vars from the files and/or lookup function and return values as
// a map rather than automatically writing values into env
func ReadWithLookup(lookupFn LookupFn, filenames ...string) (map[string]string, error) {
filenames = filenamesOrDefault(filenames)
envMap := make(map[string]string)
for _, filename := range filenames {
individualEnvMap, individualErr := ReadFile(filename, lookupFn)
if individualErr != nil {
return envMap, individualErr
}
for key, value := range individualEnvMap {
if startsWithDigitRegex.MatchString(key) {
continue
}
envMap[key] = value
}
}
return envMap, nil
}
// Read all env (with same file loading semantics as Load) but return values as
// a map rather than automatically writing values into env
func Read(filenames ...string) (map[string]string, error) {
return ReadWithLookup(nil, filenames...)
}
// UnmarshalBytesWithLookup parses env file from byte slice of chars, returning a map of keys and values.
func UnmarshalBytesWithLookup(src []byte, lookupFn LookupFn) (map[string]string, error) {
return UnmarshalWithLookup(string(src), lookupFn)
}
// UnmarshalWithLookup parses env file from string, returning a map of keys and values.
func UnmarshalWithLookup(src string, lookupFn LookupFn) (map[string]string, error) {
out := make(map[string]string)
err := newParser().parse(src, out, lookupFn)
return out, err
}
func filenamesOrDefault(filenames []string) []string {
if len(filenames) == 0 {
return []string{".env"}
}
return filenames
}
func loadFile(filename string, overload bool) error {
envMap, err := ReadFile(filename, nil)
if err != nil {
return err
}
currentEnv := map[string]bool{}
rawEnv := os.Environ()
for _, rawEnvLine := range rawEnv {
key := strings.Split(rawEnvLine, "=")[0]
currentEnv[key] = true
}
for key, value := range envMap {
if !currentEnv[key] || overload {
_ = os.Setenv(key, value)
}
}
return nil
}
func ReadFile(filename string, lookupFn LookupFn) (map[string]string, error) {
file, err := os.Open(filename)
if err != nil {
return nil, err
}
defer file.Close()
return ParseWithLookup(file, lookupFn)
}
func expandVariables(value string, envMap map[string]string, lookupFn LookupFn) (string, error) {
retVal, err := template.Substitute(value, func(k string) (string, bool) {
if v, ok := lookupFn(k); ok {
return v, true
}
v, ok := envMap[k]
return v, ok
})
if err != nil {
return value, err
}
return retVal, nil
}
+286
View File
@@ -0,0 +1,286 @@
package dotenv
import (
"errors"
"fmt"
"regexp"
"strconv"
"strings"
"unicode"
)
const (
charComment = '#'
prefixSingleQuote = '\''
prefixDoubleQuote = '"'
)
var (
escapeSeqRegex = regexp.MustCompile(`(\\(?:[abcfnrtv$"\\]|0\d{0,3}))`)
exportRegex = regexp.MustCompile(`^export\s+`)
)
type parser struct {
line int
}
func newParser() *parser {
return &parser{
line: 1,
}
}
func (p *parser) parse(src string, out map[string]string, lookupFn LookupFn) error {
cutset := src
if lookupFn == nil {
lookupFn = noLookupFn
}
for {
cutset = p.getStatementStart(cutset)
if cutset == "" {
// reached end of file
break
}
key, left, inherited, err := p.locateKeyName(cutset)
if err != nil {
return err
}
if strings.Contains(key, " ") {
return fmt.Errorf("line %d: key cannot contain a space", p.line)
}
if inherited {
value, ok := lookupFn(key)
if ok {
out[key] = value
}
cutset = left
continue
}
value, left, err := p.extractVarValue(left, out, lookupFn)
if err != nil {
return err
}
out[key] = value
cutset = left
}
return nil
}
// getStatementPosition returns position of statement begin.
//
// It skips any comment line or non-whitespace character.
func (p *parser) getStatementStart(src string) string {
pos := p.indexOfNonSpaceChar(src)
if pos == -1 {
return ""
}
src = src[pos:]
if src[0] != charComment {
return src
}
// skip comment section
pos = strings.IndexFunc(src, isCharFunc('\n'))
if pos == -1 {
return ""
}
return p.getStatementStart(src[pos:])
}
// locateKeyName locates and parses key name and returns rest of slice
func (p *parser) locateKeyName(src string) (string, string, bool, error) {
var key string
var inherited bool
// trim "export" and space at beginning
if exportRegex.MatchString(src) {
// we use a `strings.trim` to preserve the pointer to the same underlying memory.
// a regexp replace would copy the string.
src = strings.TrimLeftFunc(strings.TrimPrefix(src, "export"), isSpace)
}
// locate key name end and validate it in single loop
offset := 0
loop:
for i, rune := range src {
if isSpace(rune) {
continue
}
switch rune {
case '=', ':', '\n':
// library also supports yaml-style value declaration
key = src[0:i]
offset = i + 1
inherited = rune == '\n'
break loop
case '_', '.', '-', '[', ']':
default:
// variable name should match [A-Za-z0-9_.-]
if unicode.IsLetter(rune) || unicode.IsNumber(rune) {
continue
}
return "", "", inherited, fmt.Errorf(
`line %d: unexpected character %q in variable name %q`,
p.line, string(rune), strings.Split(src, "\n")[0])
}
}
if src == "" {
return "", "", inherited, errors.New("zero length string")
}
if inherited && strings.IndexByte(key, ' ') == -1 {
p.line++
}
// trim whitespace
key = strings.TrimRightFunc(key, unicode.IsSpace)
cutset := strings.TrimLeftFunc(src[offset:], isSpace)
return key, cutset, inherited, nil
}
// extractVarValue extracts variable value and returns rest of slice
func (p *parser) extractVarValue(src string, envMap map[string]string, lookupFn LookupFn) (string, string, error) {
quote, isQuoted := hasQuotePrefix(src)
if !isQuoted {
// unquoted value - read until new line
value, rest, _ := strings.Cut(src, "\n")
p.line++
// Remove inline comments on unquoted lines
value, _, _ = strings.Cut(value, " #")
value = strings.TrimRightFunc(value, unicode.IsSpace)
retVal, err := expandVariables(value, envMap, lookupFn)
return retVal, rest, err
}
previousCharIsEscape := false
// lookup quoted string terminator
var chars []byte
for i := 1; i < len(src); i++ {
char := src[i]
if char == '\n' {
p.line++
}
if char != quote {
if !previousCharIsEscape && char == '\\' {
previousCharIsEscape = true
continue
}
if previousCharIsEscape {
previousCharIsEscape = false
chars = append(chars, '\\')
}
chars = append(chars, char)
continue
}
// skip escaped quote symbol (\" or \', depends on quote)
if previousCharIsEscape {
previousCharIsEscape = false
chars = append(chars, char)
continue
}
// trim quotes
value := string(chars)
if quote == prefixDoubleQuote {
// expand standard shell escape sequences & then interpolate
// variables on the result
retVal, err := expandVariables(expandEscapes(value), envMap, lookupFn)
if err != nil {
return "", "", err
}
value = retVal
}
return value, src[i+1:], nil
}
// return formatted error if quoted string is not terminated
valEndIndex := strings.IndexFunc(src, isCharFunc('\n'))
if valEndIndex == -1 {
valEndIndex = len(src)
}
return "", "", fmt.Errorf("line %d: unterminated quoted value %s", p.line, src[:valEndIndex])
}
func expandEscapes(str string) string {
out := escapeSeqRegex.ReplaceAllStringFunc(str, func(match string) string {
if match == `\$` {
// `\$` is not a Go escape sequence, the expansion parser uses
// the special `$$` syntax
// both `FOO=\$bar` and `FOO=$$bar` are valid in an env file and
// will result in FOO w/ literal value of "$bar" (no interpolation)
return "$$"
}
if strings.HasPrefix(match, `\0`) {
// octal escape sequences in Go are not prefixed with `\0`, so
// rewrite the prefix, e.g. `\0123` -> `\123` -> literal value "S"
match = strings.Replace(match, `\0`, `\`, 1)
}
// use Go to unquote (unescape) the literal
// see https://go.dev/ref/spec#Rune_literals
//
// NOTE: Go supports ADDITIONAL escapes like `\x` & `\u` & `\U`!
// These are NOT supported, which is why we use a regex to find
// only matches we support and then use `UnquoteChar` instead of a
// `Unquote` on the entire value
v, _, _, err := strconv.UnquoteChar(match, '"')
if err != nil {
return match
}
return string(v)
})
return out
}
func (p *parser) indexOfNonSpaceChar(src string) int {
return strings.IndexFunc(src, func(r rune) bool {
if r == '\n' {
p.line++
}
return !unicode.IsSpace(r)
})
}
// hasQuotePrefix reports whether charset starts with single or double quote and returns quote character
func hasQuotePrefix(src string) (byte, bool) {
if src == "" {
return 0, false
}
switch quote := src[0]; quote {
case prefixDoubleQuote, prefixSingleQuote:
return quote, true // isQuoted
default:
return 0, false
}
}
func isCharFunc(char rune) func(rune) bool {
return func(v rune) bool {
return v == char
}
}
// isSpace reports whether the rune is a space character but not line break character
//
// this differs from unicode.IsSpace, which also applies line break as space
func isSpace(r rune) bool {
switch r {
case '\t', '\v', '\f', '\r', ' ', 0x85, 0xA0:
return true
}
return false
}
+56
View File
@@ -0,0 +1,56 @@
/*
Copyright 2020 The Compose Specification Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package errdefs
import "errors"
var (
// ErrNotFound is returned when an object is not found
ErrNotFound = errors.New("not found")
// ErrInvalid is returned when a compose project is invalid
ErrInvalid = errors.New("invalid compose project")
// ErrUnsupported is returned when a compose project uses an unsupported attribute
ErrUnsupported = errors.New("unsupported attribute")
// ErrIncompatible is returned when a compose project uses an incompatible attribute
ErrIncompatible = errors.New("incompatible attribute")
// ErrDisabled is returned when a resource was found in model but is disabled
ErrDisabled = errors.New("disabled")
)
// IsNotFoundError returns true if the unwrapped error is ErrNotFound
func IsNotFoundError(err error) bool {
return errors.Is(err, ErrNotFound)
}
// IsInvalidError returns true if the unwrapped error is ErrInvalid
func IsInvalidError(err error) bool {
return errors.Is(err, ErrInvalid)
}
// IsUnsupportedError returns true if the unwrapped error is ErrUnsupported
func IsUnsupportedError(err error) bool {
return errors.Is(err, ErrUnsupported)
}
// IsUnsupportedError returns true if the unwrapped error is ErrIncompatible
func IsIncompatibleError(err error) bool {
return errors.Is(err, ErrIncompatible)
}
+199
View File
@@ -0,0 +1,199 @@
/*
Copyright 2020 The Compose Specification Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package format
import (
"errors"
"fmt"
"strings"
"unicode"
"unicode/utf8"
"github.com/compose-spec/compose-go/v2/types"
)
const endOfSpec = rune(0)
// ParseVolume parses a volume spec without any knowledge of the target platform
func ParseVolume(spec string) (types.ServiceVolumeConfig, error) {
volume := types.ServiceVolumeConfig{}
switch len(spec) {
case 0:
return volume, errors.New("invalid empty volume spec")
case 1, 2:
volume.Target = spec
volume.Type = types.VolumeTypeVolume
return volume, nil
}
var buffer []rune
var inVarSubstitution int // Track nesting depth of ${...}
for i, char := range spec + string(endOfSpec) {
// Check if we're entering a variable substitution
if char == '$' && i+1 < len(spec) && rune(spec[i+1]) == '{' {
inVarSubstitution++
buffer = append(buffer, char)
continue
}
// Check if we're exiting a variable substitution
if char == '}' && inVarSubstitution > 0 {
inVarSubstitution--
buffer = append(buffer, char)
continue
}
switch {
case isWindowsDrive(buffer, char):
buffer = append(buffer, char)
case (char == ':' || char == endOfSpec) && inVarSubstitution == 0:
if err := populateFieldFromBuffer(char, buffer, &volume); err != nil {
populateType(&volume)
return volume, fmt.Errorf("invalid spec: %s: %w", spec, err)
}
buffer = nil
default:
buffer = append(buffer, char)
}
}
populateType(&volume)
return volume, nil
}
func isWindowsDrive(buffer []rune, char rune) bool {
return char == ':' && len(buffer) == 1 && unicode.IsLetter(buffer[0])
}
func populateFieldFromBuffer(char rune, buffer []rune, volume *types.ServiceVolumeConfig) error {
strBuffer := string(buffer)
switch {
case len(buffer) == 0:
return errors.New("empty section between colons")
// Anonymous volume
case volume.Source == "" && char == endOfSpec:
volume.Target = strBuffer
return nil
case volume.Source == "":
volume.Source = strBuffer
return nil
case volume.Target == "":
volume.Target = strBuffer
return nil
case char == ':':
return errors.New("too many colons")
}
for _, option := range strings.Split(strBuffer, ",") {
switch option {
case "ro":
volume.ReadOnly = true
case "rw":
volume.ReadOnly = false
case "nocopy":
volume.Volume = &types.ServiceVolumeVolume{NoCopy: true}
default:
if isBindOption(option) {
setBindOption(volume, option)
}
// ignore unknown options FIXME why not report an error here?
}
}
return nil
}
var Propagations = []string{
types.PropagationRPrivate,
types.PropagationPrivate,
types.PropagationRShared,
types.PropagationShared,
types.PropagationRSlave,
types.PropagationSlave,
}
type setBindOptionFunc func(bind *types.ServiceVolumeBind, option string)
var bindOptions = map[string]setBindOptionFunc{
types.PropagationRPrivate: setBindPropagation,
types.PropagationPrivate: setBindPropagation,
types.PropagationRShared: setBindPropagation,
types.PropagationShared: setBindPropagation,
types.PropagationRSlave: setBindPropagation,
types.PropagationSlave: setBindPropagation,
types.SELinuxShared: setBindSELinux,
types.SELinuxPrivate: setBindSELinux,
}
func setBindPropagation(bind *types.ServiceVolumeBind, option string) {
bind.Propagation = option
}
func setBindSELinux(bind *types.ServiceVolumeBind, option string) {
bind.SELinux = option
}
func isBindOption(option string) bool {
_, ok := bindOptions[option]
return ok
}
func setBindOption(volume *types.ServiceVolumeConfig, option string) {
if volume.Bind == nil {
volume.Bind = &types.ServiceVolumeBind{}
}
bindOptions[option](volume.Bind, option)
}
func populateType(volume *types.ServiceVolumeConfig) {
if isFilePath(volume.Source) {
volume.Type = types.VolumeTypeBind
if volume.Bind == nil {
volume.Bind = &types.ServiceVolumeBind{}
}
// For backward compatibility with docker-compose legacy, using short notation involves
// bind will create missing host path
volume.Bind.CreateHostPath = true
} else {
volume.Type = types.VolumeTypeVolume
if volume.Volume == nil {
volume.Volume = &types.ServiceVolumeVolume{}
}
}
}
func isFilePath(source string) bool {
if source == "" {
return false
}
switch source[0] {
case '.', '/', '~':
return true
}
// windows named pipes
if strings.HasPrefix(source, `\\`) {
return true
}
first, nextIndex := utf8.DecodeRuneInString(source)
if len(source) <= nextIndex {
return false
}
return isWindowsDrive([]rune{first}, rune(source[nextIndex]))
}
+63
View File
@@ -0,0 +1,63 @@
/*
Copyright 2020 The Compose Specification Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package graph
import (
"fmt"
"slices"
"strings"
"github.com/compose-spec/compose-go/v2/types"
"github.com/compose-spec/compose-go/v2/utils"
)
// CheckCycle analyze project's depends_on relation and report an error on cycle detection
func CheckCycle(project *types.Project) error {
g, err := newGraph(project)
if err != nil {
return err
}
return g.checkCycle()
}
func (g *graph[T]) checkCycle() error {
// iterate on vertices in a name-order to render a predicable error message
// this is required by tests and enforce command reproducibility by user, which otherwise could be confusing
names := utils.MapKeys(g.vertices)
for _, name := range names {
err := searchCycle([]string{name}, g.vertices[name])
if err != nil {
return err
}
}
return nil
}
func searchCycle[T any](path []string, v *vertex[T]) error {
names := utils.MapKeys(v.children)
for _, name := range names {
if i := slices.Index(path, name); i >= 0 {
return fmt.Errorf("dependency cycle detected: %s -> %s", strings.Join(path[i:], " -> "), name)
}
ch := v.children[name]
err := searchCycle(append(path, name), ch)
if err != nil {
return err
}
}
return nil
}
+75
View File
@@ -0,0 +1,75 @@
/*
Copyright 2020 The Compose Specification Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package graph
// graph represents project as service dependencies
type graph[T any] struct {
vertices map[string]*vertex[T]
}
// vertex represents a service in the dependencies structure
type vertex[T any] struct {
key string
service *T
children map[string]*vertex[T]
parents map[string]*vertex[T]
}
func (g *graph[T]) addVertex(name string, service T) {
g.vertices[name] = &vertex[T]{
key: name,
service: &service,
parents: map[string]*vertex[T]{},
children: map[string]*vertex[T]{},
}
}
func (g *graph[T]) addEdge(src, dest string) {
g.vertices[src].children[dest] = g.vertices[dest]
g.vertices[dest].parents[src] = g.vertices[src]
}
func (g *graph[T]) roots() []*vertex[T] {
var res []*vertex[T]
for _, v := range g.vertices {
if len(v.parents) == 0 {
res = append(res, v)
}
}
return res
}
func (g *graph[T]) leaves() []*vertex[T] {
var res []*vertex[T]
for _, v := range g.vertices {
if len(v.children) == 0 {
res = append(res, v)
}
}
return res
}
// descendents return all descendents for a vertex, might contain duplicates
func (v *vertex[T]) descendents() []string {
var vx []string
for _, n := range v.children {
vx = append(vx, n.key)
vx = append(vx, n.descendents()...)
}
return vx
}
+80
View File
@@ -0,0 +1,80 @@
/*
Copyright 2020 The Compose Specification Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package graph
import (
"context"
"fmt"
"github.com/compose-spec/compose-go/v2/types"
)
// InDependencyOrder walk the service graph an invoke VisitorFn in respect to dependency order
func InDependencyOrder(ctx context.Context, project *types.Project, fn VisitorFn[types.ServiceConfig], options ...func(*Options)) error {
_, err := CollectInDependencyOrder[any](ctx, project, func(ctx context.Context, s string, config types.ServiceConfig) (any, error) {
return nil, fn(ctx, s, config)
}, options...)
return err
}
// CollectInDependencyOrder walk the service graph an invoke CollectorFn in respect to dependency order, then return result for each call
func CollectInDependencyOrder[T any](ctx context.Context, project *types.Project, fn CollectorFn[types.ServiceConfig, T], options ...func(*Options)) (map[string]T, error) {
graph, err := newGraph(project)
if err != nil {
return nil, err
}
t := newTraversal(fn)
for _, option := range options {
option(t.Options)
}
err = walk(ctx, graph, t)
return t.results, err
}
// newGraph creates a service graph from project
func newGraph(project *types.Project) (*graph[types.ServiceConfig], error) {
g := &graph[types.ServiceConfig]{
vertices: map[string]*vertex[types.ServiceConfig]{},
}
for name, s := range project.Services {
g.addVertex(name, s)
}
for name, s := range project.Services {
src := g.vertices[name]
for dep, condition := range s.DependsOn {
dest, ok := g.vertices[dep]
if !ok {
if condition.Required {
if ds, exists := project.DisabledServices[dep]; exists {
return nil, fmt.Errorf("service %q is required by %q but is disabled. Can be enabled by profiles %s", dep, name, ds.Profiles)
}
return nil, fmt.Errorf("service %q depends on unknown service %q", name, dep)
}
delete(s.DependsOn, name)
project.Services[name] = s
continue
}
src.children[dep] = dest
dest.parents[name] = src
}
}
err := g.checkCycle()
return g, err
}
+211
View File
@@ -0,0 +1,211 @@
/*
Copyright 2020 The Compose Specification Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package graph
import (
"context"
"slices"
"sync"
"golang.org/x/sync/errgroup"
)
// CollectorFn executes on each graph vertex based on visit order and return associated value
type CollectorFn[S any, T any] func(context.Context, string, S) (T, error)
// VisitorFn executes on each graph nodes based on visit order
type VisitorFn[S any] func(context.Context, string, S) error
type traversal[S any, T any] struct {
*Options
visitor CollectorFn[S, T]
mu sync.Mutex
status map[string]int
results map[string]T
}
type Options struct {
// inverse reverse the traversal direction
inverse bool
// maxConcurrency limit the concurrent execution of visitorFn while walking the graph
maxConcurrency int
// after marks a set of node as starting points walking the graph
after []string
}
const (
vertexEntered = iota
vertexVisited
)
func newTraversal[S, T any](fn CollectorFn[S, T]) *traversal[S, T] {
return &traversal[S, T]{
Options: &Options{},
status: map[string]int{},
results: map[string]T{},
visitor: fn,
}
}
// WithMaxConcurrency configure traversal to limit concurrency walking graph nodes
func WithMaxConcurrency(concurrency int) func(*Options) {
return func(o *Options) {
o.maxConcurrency = concurrency
}
}
// InReverseOrder configure traversal to walk the graph in reverse dependency order
func InReverseOrder(o *Options) {
o.inverse = true
}
// WithRootNodesAndDown creates a graphTraversal to start from selected nodes
func WithRootNodesAndDown(nodes []string) func(*Options) {
return func(o *Options) {
o.after = nodes
}
}
func walk[S, T any](ctx context.Context, g *graph[S], t *traversal[S, T]) error {
expect := len(g.vertices)
if expect == 0 {
return nil
}
// nodeCh need to allow n=expect writers while reader goroutine could have returned after ctx.Done
nodeCh := make(chan *vertex[S], expect)
defer close(nodeCh)
eg, ctx := errgroup.WithContext(ctx)
if t.maxConcurrency > 0 {
eg.SetLimit(t.maxConcurrency + 1)
}
eg.Go(func() error {
for {
select {
case <-ctx.Done():
return nil
case node := <-nodeCh:
expect--
if expect == 0 {
return nil
}
for _, adj := range t.adjacentNodes(node) {
t.visit(ctx, eg, adj, nodeCh)
}
}
}
})
// select nodes to start walking the graph based on traversal.direction
for _, node := range t.extremityNodes(g) {
t.visit(ctx, eg, node, nodeCh)
}
return eg.Wait()
}
func (t *traversal[S, T]) visit(ctx context.Context, eg *errgroup.Group, node *vertex[S], nodeCh chan *vertex[S]) {
if !t.ready(node) {
// don't visit this service yet as dependencies haven't been visited
return
}
if !t.enter(node) {
// another worker already acquired this node
return
}
eg.Go(func() error {
var (
err error
result T
)
if !t.skip(node) {
result, err = t.visitor(ctx, node.key, *node.service)
}
t.done(node, result)
nodeCh <- node
return err
})
}
func (t *traversal[S, T]) extremityNodes(g *graph[S]) []*vertex[S] {
if t.inverse {
return g.roots()
}
return g.leaves()
}
func (t *traversal[S, T]) adjacentNodes(v *vertex[S]) map[string]*vertex[S] {
if t.inverse {
return v.children
}
return v.parents
}
func (t *traversal[S, T]) ready(v *vertex[S]) bool {
t.mu.Lock()
defer t.mu.Unlock()
depends := v.children
if t.inverse {
depends = v.parents
}
for name := range depends {
if t.status[name] != vertexVisited {
return false
}
}
return true
}
func (t *traversal[S, T]) enter(v *vertex[S]) bool {
t.mu.Lock()
defer t.mu.Unlock()
if _, ok := t.status[v.key]; ok {
return false
}
t.status[v.key] = vertexEntered
return true
}
func (t *traversal[S, T]) done(v *vertex[S], result T) {
t.mu.Lock()
defer t.mu.Unlock()
t.status[v.key] = vertexVisited
t.results[v.key] = result
}
func (t *traversal[S, T]) skip(node *vertex[S]) bool {
if len(t.after) == 0 {
return false
}
if slices.Contains(t.after, node.key) {
return false
}
// is none of our starting node is a descendent, skip visit
ancestors := node.descendents()
for _, name := range t.after {
if slices.Contains(ancestors, name) {
return false
}
}
return true
}
@@ -0,0 +1,137 @@
/*
Copyright 2020 The Compose Specification Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package interpolation
import (
"errors"
"fmt"
"os"
"github.com/compose-spec/compose-go/v2/template"
"github.com/compose-spec/compose-go/v2/tree"
)
// Options supported by Interpolate
type Options struct {
// LookupValue from a key
LookupValue LookupValue
// TypeCastMapping maps key paths to functions to cast to a type
TypeCastMapping map[tree.Path]Cast
// Substitution function to use
Substitute func(string, template.Mapping) (string, error)
}
// LookupValue is a function which maps from variable names to values.
// Returns the value as a string and a bool indicating whether
// the value is present, to distinguish between an empty string
// and the absence of a value.
type LookupValue func(key string) (string, bool)
// Cast a value to a new type, or return an error if the value can't be cast
type Cast func(value string) (interface{}, error)
// Interpolate replaces variables in a string with the values from a mapping
func Interpolate(config map[string]interface{}, opts Options) (map[string]interface{}, error) {
if opts.LookupValue == nil {
opts.LookupValue = os.LookupEnv
}
if opts.TypeCastMapping == nil {
opts.TypeCastMapping = make(map[tree.Path]Cast)
}
if opts.Substitute == nil {
opts.Substitute = template.Substitute
}
out := map[string]interface{}{}
for key, value := range config {
interpolatedValue, err := recursiveInterpolate(value, tree.NewPath(key), opts)
if err != nil {
return out, err
}
out[key] = interpolatedValue
}
return out, nil
}
func recursiveInterpolate(value interface{}, path tree.Path, opts Options) (interface{}, error) {
switch value := value.(type) {
case string:
newValue, err := opts.Substitute(value, template.Mapping(opts.LookupValue))
if err != nil {
return value, newPathError(path, err)
}
caster, ok := opts.getCasterForPath(path)
if !ok {
return newValue, nil
}
casted, err := caster(newValue)
if err != nil {
return casted, newPathError(path, fmt.Errorf("failed to cast to expected type: %w", err))
}
return casted, nil
case map[string]interface{}:
out := map[string]interface{}{}
for key, elem := range value {
interpolatedElem, err := recursiveInterpolate(elem, path.Next(key), opts)
if err != nil {
return nil, err
}
out[key] = interpolatedElem
}
return out, nil
case []interface{}:
out := make([]interface{}, len(value))
for i, elem := range value {
interpolatedElem, err := recursiveInterpolate(elem, path.Next(tree.PathMatchList), opts)
if err != nil {
return nil, err
}
out[i] = interpolatedElem
}
return out, nil
default:
return value, nil
}
}
func newPathError(path tree.Path, err error) error {
var ite *template.InvalidTemplateError
switch {
case err == nil:
return nil
case errors.As(err, &ite):
return fmt.Errorf(
"invalid interpolation format for %s.\nYou may need to escape any $ with another $.\n%s",
path, ite.Template)
default:
return fmt.Errorf("error while interpolating %s: %w", path, err)
}
}
func (o Options) getCasterForPath(path tree.Path) (Cast, bool) {
for pattern, caster := range o.TypeCastMapping {
if path.Matches(pattern) {
return caster, true
}
}
return nil, false
}
+110
View File
@@ -0,0 +1,110 @@
/*
Copyright 2020 The Compose Specification Authors.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
*/
package loader
import (
"fmt"
"github.com/compose-spec/compose-go/v2/types"
)
// ResolveEnvironment update the environment variables for the format {- VAR} (without interpolation)
func ResolveEnvironment(dict map[string]any, environment types.Mapping) {
resolveServicesEnvironment(dict, environment)
resolveSecretsEnvironment(dict, environment)
resolveConfigsEnvironment(dict, environment)
}
func resolveServicesEnvironment(dict map[string]any, environment types.Mapping) {
services, ok := dict["services"].(map[string]any)
if !ok {
return
}
for service, cfg := range services {
serviceConfig, ok := cfg.(map[string]any)
if !ok {
continue
}
serviceEnv, ok := serviceConfig["environment"].([]any)
if !ok {
continue
}
envs := []any{}
for _, env := range serviceEnv {
varEnv, ok := env.(string)
if !ok {
continue
}
if found, ok := environment[varEnv]; ok {
envs = append(envs, fmt.Sprintf("%s=%s", varEnv, found))
} else {
// either does not exist or it was already resolved in interpolation
envs = append(envs, varEnv)
}
}
serviceConfig["environment"] = envs
services[service] = serviceConfig
}
dict["services"] = services
}
func resolveSecretsEnvironment(dict map[string]any, environment types.Mapping) {
secrets, ok := dict["secrets"].(map[string]any)
if !ok {
return
}
for name, cfg := range secrets {
secret, ok := cfg.(map[string]any)
if !ok {
continue
}
env, ok := secret["environment"].(string)
if !ok {
continue
}
if found, ok := environment[env]; ok {
secret[types.SecretConfigXValue] = found
}
secrets[name] = secret
}
dict["secrets"] = secrets
}
func resolveConfigsEnvironment(dict map[string]any, environment types.Mapping) {
configs, ok := dict["configs"].(map[string]any)
if !ok {
return
}
for name, cfg := range configs {
config, ok := cfg.(map[string]any)
if !ok {
continue
}
env, ok := config["environment"].(string)
if !ok {
continue
}
if found, ok := environment[env]; ok {
config["content"] = found
}
configs[name] = config
}
dict["configs"] = configs
}
@@ -1,5 +1,7 @@
# passed through
FOO=foo_from_env_file
ENV.WITH.DOT=ok
ENV_WITH_UNDERSCORE=ok
# overridden in example2.env
BAR=bar_from_env_file
+10
View File
@@ -0,0 +1,10 @@
# passed through
FOO=foo_from_label_file
LABEL.WITH.DOT=ok
LABEL_WITH_UNDERSCORE=ok
# overridden in example2.label
BAR=bar_from_label_file
# overridden in full-example.yml
BAZ=baz_from_label_file

Some files were not shown because too many files have changed in this diff Show More