Commit Graph

7266 Commits

Author SHA1 Message Date
61cb016851 Merge pull request #2961 from thaJeztah/20.10_backport_fix_context_dockerfile_from_stdin_with_buildkit
[20.10 backport] Fix reading context and dockerfile from stdin with BuildKit
2021-02-18 12:02:09 +01:00
53c4602909 Merge pull request #2962 from thaJeztah/20.10_backport_go_md2man_binary_name
[20.10 backport] Rename bin/md2man to bin/go-md2man
2021-02-18 11:58:57 +01:00
830d6595e7 Merge pull request #2963 from thaJeztah/20.10_backport_fix_update_rollback_order
[20.10 backport] fix --update-order and --rollback-order flags
2021-02-18 11:58:23 +01:00
13048444cc Merge pull request #2964 from thaJeztah/20.10_backport_fix_swarm_rollback_exitcode
[20.10 backport] Fix swarm rollback exitcode, and fix skipping verify step
2021-02-18 11:55:40 +01:00
3e293e6bd1 Merge pull request #2958 from thaJeztah/20.10_backport_fix_homedir_warning
[20.10 backport] cli/config: prevent warning if HOME is not set
2021-02-12 10:27:24 +01:00
41b3ea7e47 Merge pull request #2960 from thaJeztah/20.10_backport_ignore_sigurg 2021-02-04 07:36:47 -08:00
d6eeeb6259 service rollback: always verify state
Prior to this change, progressbars would sometimes be hidden, and the function
would return early. In addition, the direction of the progressbars would sometimes
be "incrementing" (similar to "docker service update"), and sometimes be "decrementing"
(to indicate a "rollback" is being performed).

This fix makes sure that we always proceed with the "verifying" step, and now
prints a message _after_ the verifying stage was completed;

    $ docker service rollback foo
    foo
    overall progress: rolling back update: 5 out of 5 tasks
    1/5: running   [>                                                  ]
    2/5: starting  [===========>                                       ]
    3/5: starting  [===========>                                       ]
    4/5: running   [>                                                  ]
    5/5: running   [>                                                  ]
    verify: Service converged
    rollback: rollback completed

    $ docker service rollback foo
    foo
    overall progress: rolling back update: 1 out of 1 tasks
    1/1: running   [>                                                  ]
    verify: Service converged
    rollback: rollback completed

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 104469be0b)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-02-04 14:55:51 +01:00
3e157d5293 docker service rollback: fix non-zero exit code in some cases
Before this change:
--------------------------------------------

    $ docker service create --replicas=1 --name foo -p 8080:80 nginx:alpine
    t33qvykv8y0zbz266rxynsbo3
    overall progress: 1 out of 1 tasks
    1/1: running   [==================================================>]
    verify: Service converged

    $ echo $?
    0

    $ docker service update --replicas=5 foo
    foo
    overall progress: 5 out of 5 tasks
    1/5: running   [==================================================>]
    2/5: running   [==================================================>]
    3/5: running   [==================================================>]
    4/5: running   [==================================================>]
    5/5: running   [==================================================>]
    verify: Service converged

    $ echo $?
    0

    $ docker service rollback foo
    foo
    rollback: manually requested rollback
    overall progress: rolling back update: 1 out of 1 tasks
    1/1: running   [>                                                  ]
    verify: Service converged

    $ echo $?
    0

    $ docker service rollback foo
    foo
    service rolled back: rollback completed

    $ echo $?
    1

After this change:
--------------------------------------------

    $ docker service create --replicas=1 --name foo -p 8080:80 nginx:alpine
    t33qvykv8y0zbz266rxynsbo3
    overall progress: 1 out of 1 tasks
    1/1: running   [==================================================>]
    verify: Service converged

    $ echo $?
    0

    $ docker service update --replicas=5 foo
    foo
    overall progress: 5 out of 5 tasks
    1/5: running   [==================================================>]
    2/5: running   [==================================================>]
    3/5: running   [==================================================>]
    4/5: running   [==================================================>]
    5/5: running   [==================================================>]
    verify: Waiting 1 seconds to verify that tasks are stable...

    $ echo $?
    0

    $ docker service rollback foo
    foo
    rollback: manually requested rollback
    overall progress: rolling back update: 1 out of 1 tasks
    1/1: running   [>                                                  ]
    verify: Service converged

    $ echo $?
    0

    $ docker service rollback foo
    foo
    service rolled back: rollback completed

    $ echo $?
    0

    $ docker service ps foo
    ID             NAME      IMAGE          NODE             DESIRED STATE   CURRENT STATE           ERROR     PORTS
    4dt4ms4c5qfb   foo.1     nginx:alpine   docker-desktop   Running         Running 2 minutes ago

Remaining issues with reconciliation
--------------------------------------------

Note that both before, and after this change, the command sometimes terminates
early, and does not wait for the service to reconcile; this is most apparent
when rolling back is scaling up (so more tasks are deployed);

    $ docker service rollback foo
    foo
    service rolled back: rollback completed

    $ docker service rollback foo
    foo
    rollback: manually requested rollback
    overall progress: rolling back update: 1 out of 5 tasks
    1/5: pending   [=================================>                 ]
    2/5: running   [>                                                  ]
    3/5: pending   [=================================>                 ]
    4/5: pending   [=================================>                 ]
    5/5: pending   [=================================>                 ]
    service rolled back: rollback completed

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit ce26a165b0)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-02-04 14:55:49 +01:00
1fdf84b8e9 fix --update-order and --rollback-order flags
Signed-off-by: Jim Lin <b04705003@ntu.edu.tw>
(cherry picked from commit 26a6a724aa)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-02-04 14:53:34 +01:00
376b99c6d6 Rename bin/md2man to bin/go-md2man
In the recent PR !2877, some code was added to check if md2man is
already installed in the build environment. This is to cater to the
needs of Linux distributions.

However it turns out that Linux distributions install md2man as
bin/go-md2man instead of bin/md2man, hence the PR !2877 doesn't help
much.

This commit fixes it by settling on using the binary name go-md2man.

For reference, here the file list of the package go-md2man in several
distributions:

- Debian: <https://packages.debian.org/sid/amd64/go-md2man/filelist>
- Ubuntu: <https://packages.ubuntu.com/hirsute/amd64/go-md2man/filelist>
- Fedora: <https://fedora.pkgs.org/31/fedora-x86_64/golang-github-cpuguy83-md2man-2.0.0-0.4.20190624gitf79a8a8.fc31.x86_64.rpm.html>
- ArchLinux: <https://www.archlinux.org/packages/community/x86_64/go-md2man/>

Signed-off-by: Arnaud Rebillout <elboulangero@gmail.com>
(cherry picked from commit 6e2607c6a6)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-02-04 14:50:45 +01:00
0de4e6e9a7 Fix reading context and dockerfile from stdin with BuildKit
Signed-off-by: Alexey Igrychev <alexey.igrychev@flant.com>
(cherry picked from commit fc9ca9a94a)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-02-04 14:48:19 +01:00
3c87f01b18 Ignore SIGURG on Linux.
In go1.14+, SIGURG is used by the runtime to handle preemtable system
calls.
In practice this signal caught *frequently*.

For reference:

https://go.googlesource.com/proposal/+/master/design/24543-non-cooperative-preemption.md
https://github.com/golang/go/issues/37942

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit fff164c22e)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-02-04 14:44:39 +01:00
d513e46bfc cli/config: prevent warning if HOME is not set
commit c2626a8270 replaced the use of
github.com/docker/docker/pkg/homedir with Golang's os.UserHomeDir().

This change was partially reverted in 7a279af43d
to account for situations where `$HOME` is not set.

In  situations where no configuration file is present in `~/.config/`, the CLI
falls back to looking for the (deprecated) `~/.dockercfg` configuration file,
which was still using `os.UserHomeDir()`, which produces an error/warning if
`$HOME` is not set.

This patch introduces a helper function and a global variable to get the user's
home-directory. The global variable is used to prevent repeatedly looking up
the user's information (which, depending on the setup can be a costly operation).

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit c85a37dbb4)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2021-02-04 14:38:27 +01:00
48d30b5b32 Use golang.org/x/sys/execabs
On Windows, the os/exec.{Command,CommandContext,LookPath} functions
resolve command names that have neither path separators nor file extension
(e.g., "git") by first looking in the current working directory before
looking in the PATH environment variable.
Go maintainers intended to match cmd.exe's historical behavior.

However, this is pretty much never the intended behavior and as an abundance of precaution
this patch prevents that when executing commands.
Example of commands that docker.exe may execute: `git`, `docker-buildx` (or other cli plugin), `docker-credential-wincred`, `docker`.

Note that this was prompted by the [Go 1.15.7 security fixes](https://blog.golang.org/path-security), but unlike in `go.exe`,
the windows path lookups in docker are not in a code path allowing remote code execution, thus there is no security impact on docker.

Signed-off-by: Tibor Vass <tibor@docker.com>
(cherry picked from commit 8d199d5bba)
Signed-off-by: Tibor Vass <tibor@docker.com>
v20.10.3
2021-01-28 22:27:59 +00:00
5941f4104a vendor docker, docker-credential-helpers and golang/sys for execabs package
Signed-off-by: Tibor Vass <tibor@docker.com>
(cherry picked from commit 7bef248765)
Signed-off-by: Tibor Vass <tibor@docker.com>
2021-01-28 22:27:59 +00:00
18f33b337d context: Add tarball e2e tests
Signed-off-by: Chris Crone <christopher.crone@docker.com>
2021-01-28 22:27:59 +00:00
9ecc69d17e context: Ensure context name is valid on import
Signed-off-by: Chris Crone <christopher.crone@docker.com>
2021-01-28 22:27:59 +00:00
6f49197cab context: Ensure import paths are valid
Signed-off-by: Chris Crone <christopher.crone@docker.com>
2021-01-28 22:27:59 +00:00
2291f610ae Merge pull request #2886 from thaJeztah/bump_buildkit
vendor: BuildKit v0.8.1, Engine v20.10.1
v20.10.2
2020-12-22 09:32:20 -08:00
6c97008f67 Merge pull request #2896 from ccouzens/patch-2
Fix link to docker build with squash experiment
2020-12-22 16:27:26 +01:00
1f0dff6752 Merge pull request #2883 from elboulangero/fix-container-creation-time-in-tests
Fix container creation time in test/builders
2020-12-22 14:18:49 +01:00
5e430753f2 Fix link to docker build with squash experiment
Signed-off-by: Chris Couzens <ccouzens@gmail.com>
2020-12-18 22:33:02 +00:00
4d6bc775b3 Fix container creation time in test/builders
The created time of the containerd is initialized with nanoseconds,
it seems to be a mistake.

In other places of the code, this field is initialized with seconds:

    $ grep -rh 'time\.Now()\.Unix()' | grep Created
    Created: time.Now().Unix(),
    Created: time.Now().Unix(),
    return []image.HistoryResponseItem{{ID: img, Created: time.Now().Unix()}}, nil

We can also see the the formatter assumes it to be seconds:

    cli/command/formatter/container.go
    ----
    func (c *ContainerContext) CreatedAt() string {
        return time.Unix(c.c.Created, 0).String()
    }

Interestingly, initializing the field with nanoseconds seems to work,
except on mips architecture, where it causes some kind of overflow.

~~~~
=== Failed
=== FAIL: cli/command/container TestContainerListWithoutFormat (0.00s)
    list_test.go:183: assertion failed:
        --- expected
        +++ actual
        @@ -1,7 +1,7 @@
         CONTAINER ID   IMAGE            COMMAND   CREATED                  STATUS        PORTS                NAMES
        -container_id   busybox:latest   "top"     Less than a second ago   Up 1 second                        c1
        -container_id   busybox:latest   "top"     Less than a second ago   Up 1 second                        c2
        -container_id   busybox:latest   "top"     Less than a second ago   Up 1 second   80-82/tcp            c3
        -container_id   busybox:latest   "top"     Less than a second ago   Up 1 second   81/udp               c4
        -container_id   busybox:latest   "top"     Less than a second ago   Up 1 second   8.8.8.8:82->82/tcp   c5
        +container_id   busybox:latest   "top"     -153722867 minutes ago   Up 1 second                        c1
        +container_id   busybox:latest   "top"     -153722867 minutes ago   Up 1 second                        c2
        +container_id   busybox:latest   "top"     -153722867 minutes ago   Up 1 second   80-82/tcp            c3
        +container_id   busybox:latest   "top"     -153722867 minutes ago   Up 1 second   81/udp               c4
        +container_id   busybox:latest   "top"     -153722867 minutes ago   Up 1 second   8.8.8.8:82->82/tcp   c5

=== FAIL: cli/command/container TestContainerListNoTrunc (0.00s)
    list_test.go:198: assertion failed:
        --- expected
        +++ actual
        @@ -1,4 +1,4 @@
         CONTAINER ID   IMAGE            COMMAND   CREATED                  STATUS        PORTS     NAMES
        -container_id   busybox:latest   "top"     Less than a second ago   Up 1 second             c1
        -container_id   busybox:latest   "top"     Less than a second ago   Up 1 second             c2,foo/bar
        +container_id   busybox:latest   "top"     -153722867 minutes ago   Up 1 second             c1
        +container_id   busybox:latest   "top"     -153722867 minutes ago   Up 1 second             c2,foo/bar
~~~~

Logs above taken from:
https://buildd.debian.org/status/fetch.php?pkg=docker.io&arch=mipsel&ver=20.10.0%7Erc1%2Bdfsg3-1&stamp=1606895899

~~~~
=== RUN   TestChtimesLinux
    chtimes_linux_test.go:87: Expected: 2262-04-11 23:47:16 +0000 UTC, got: 1990-01-27 10:50:44 +0000 UTC
--- FAIL: TestChtimesLinux (0.00s)
=== RUN   TestChtimes
    chtimes_test.go:92: Expected: 2262-04-11 23:47:16 +0000 UTC, got: 1990-01-27 10:50:44 +0000 UTC
--- FAIL: TestChtimes (0.00s)
~~~~

Logs above taken from:
https://buildd.debian.org/status/fetch.php?pkg=docker.io&arch=mips64el&ver=20.10.0%7Erc1%2Bdfsg3-1&stamp=1606895622

Signed-off-by: Arnaud Rebillout <elboulangero@gmail.com>
2020-12-16 08:55:29 +07:00
47ecb13fe4 vendor: github.com/docker/docker v20.10.1
full diff: cf31b9622a...v20.10.1

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-12-15 18:14:29 +01:00
a2b4d9c74f vendor: github.com/moby/buildkit v0.8.1
full diff: https://github.com/moby/buildkit/compare/v0.8.0...v0.8.1

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-12-15 18:09:01 +01:00
831ebeae96 Merge pull request #2884 from thaJeztah/format_help_link
Help link: remove color, add "bold" style, and white-space
v20.10.1
2020-12-14 18:27:14 -08:00
fa3e0bcdaf Help link: remove color, add "bold" style, and white-space
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-12-14 14:28:29 +01:00
42d4489992 Merge pull request #2877 from Foxboron/morten/fix-man
generate-man.sh: Only build md2man if it's not installed
2020-12-10 09:41:38 +01:00
29bd82ba14 generate-man.sh: Only build md2man if it's not installed
Distributions uses generate-man.sh to create man pages, however it only
assumes a container environment and attempts to build binaries towards
`/go` which might be an illegal path in `fakeroot`.

This patch ensures we only build `md2man` from the vendored sources if
the command is not present. This allows distributions to supply thier
packaged `md2man`.

Signed-off-by: Morten Linderud <morten@linderud.pw>
2020-12-09 23:06:18 +01:00
12164ff157 Merge pull request #2874 from thaJeztah/fix_deprecation_table
Deprecation: add pulling from non-compliant registries to table
2020-12-08 18:27:42 -08:00
06c60dea90 Deprecation: add pulling from non-compliant registries to table
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-12-09 02:36:30 +01:00
7287ab3890 Merge pull request #2872 from thaJeztah/deprecate_non_compliant_registries
Add deprecation note for non-compliant registries
v20.10.0
2020-12-07 15:36:43 -08:00
029ab0255a Merge pull request #2857 from gtardif/help_guides_link
Help guides link
2020-12-07 15:34:45 -08:00
37f45817b6 Merge pull request #2869 from thaJeztah/fix_yaml_formatting
docs: un-wrap line to prevent YAML docs from using "compact" formatting
2020-12-07 14:15:07 +01:00
cd409fc76a Merge pull request #2871 from thaJeztah/bump_engine_buildkit
vendor: docker cf31b9622ae9de4b5fb7a1c9456460980ab80281, buildkit v0.8.0
2020-12-07 14:12:54 +01:00
aa91af81c6 Add deprecation note for non-compliant registries
Docker Engine v20.10 and up includes optimizations to verify if images in the
local image cache need updating before pulling, preventing the Docker Engine
from making unnecessary API requests. These optimizations require the container
image registry to conform to the Open Container Initiative Distribution Specification
(https://github.com/opencontainers/distribution-spec).

While most registries conform to the specification, we encountered some registries
to be non-compliant, resulting in `docker pull` to fail.

As a temporary solution, Docker Engine v20.10 includes a fallback mechanism to
allow `docker pull` to be functional when using a non-compliant registry. A
warning message is printed in this situation:

    WARNING Failed to pull manifest by the resolved digest. This registry does not
            appear to conform to the distribution registry specification; falling back to
            pull by tag. This fallback is DEPRECATED, and will be removed in a future
            release.

The fallback is added to allow users to either migrate their images to a compliant
registry, or for these registries to become compliant.

Note that this fallback only addresses failures on `docker pull`. Other commands,
such as `docker stack deploy`, or pulling images with `containerd` will continue
to fail.

Given that other functionality is still broken with these registries, we consider
this fallback a _temporary_ solution, and will remove the fallback in an upcoming
major release.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-12-07 13:34:50 +01:00
c54aec5c85 vendor: BuildKit v0.8.0
full diff: https://github.com/moby/buildkit/compare/v0.8.0-rc2...v0.8.0

note that this is currently a few commits "behind" the version used in docker,
but changes since v0.8.0 do not affect the code that's vendored in the CLI,
so prefering to use a tagged version.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-12-07 12:50:36 +01:00
aae8d2dfde vendor: github.com/docker/docker cf31b9622ae9de4b5fb7a1c9456460980ab80281
full diff: 6c0a036dce...cf31b9622a

changes affecting vendored code:

- client: Fix error handling

Other changes:

- handleContainerExit: timeout on containerd DeleteTask
- vendor libnetwork to fix mix up between IPv4 and IPv6
- update containerd binary to v1.4.2
- update containerd binary to v1.4.3 (CVE-2020-15257)
- vendor: BuildKit v0.8.0-rc2, containerd, and dependencies
- Add fallback for pull by tag
- vendor: BuildKit 950603da215ae03b843f3f66fbe86c4876a6f5a1
- IPv6 iptables config option

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-12-07 12:43:03 +01:00
3455643194 docs: un-wrap line to prevent YAML docs from using "compact" formatting
Looks like the YAML conversion doesn't like lines starting with `[`, and
causing it to use the "compact" formatting in the generated YAML.

This patch un-wraps these lines to prevent this.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-12-07 12:24:44 +01:00
beab92999a Merge pull request #2862 from rumpl/chore-remove-kubernetes-dependency
Remove k8s.io/kubernetes dependency
2020-12-04 21:15:39 +01:00
9f9c4b7f3b Remove k8s.io/kubernetes dependency
We are only using the `IsPodReady` function that can be rewritten easily.

Signed-off-by: Djordje Lukic <djordje.lukic@docker.com>
2020-12-04 00:27:40 +01:00
98625314fa Merge pull request #2846 from bboehmke/dockerd_ip6tables
added ip6tables to daemon CLI and config file documentation
2020-12-03 20:56:21 +01:00
dca98e3c87 Merge pull request #2843 from thaJeztah/bump_docker_buildkit
vendor: buildkit v0.8.0-rc2, docker, containerd, and dependencies
v20.10.0-rc2
2020-12-02 14:02:57 -08:00
40ec81a79a build: display [auth] output
Signed-off-by: Tibor Vass <tibor@docker.com>
2020-12-02 21:22:36 +00:00
9a0a071d55 vendor: buildkit v0.8.0-rc2, docker
diffs:

- full diff: af34b94a78...6c0a036dce
- full diff: 4d1f260e84...v0.8.0-rc2

New dependencies:

- go.opencensus.io v0.22.3
- github.com/containerd/typeurl v1.0.1
- github.com/golang/groupcache 869f871628b6baa9cfbc11732cdf6546b17c1298

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-12-02 21:01:12 +00:00
4cab568abb vendor: github.com/moby/term bea5bbe245bf407372d477f1361d2ff042d2f556
full diff: 7f0af18e79...bea5bbe245

- Fix windows integer overflow on GOOS=windows, GOARCH=arm
- go.mod: github.com/creack/pty v1.1.11
  - v1.1.11: Add arm support for OpenBSD
  - v1.1.10: Fix CTTY to work with go1.15
- CI: fix Go version matrix, and drop go 1.12, add go 1.15
- CI: remove "sudo" to fix incorrect Go versions (incorrect PATH, GOROOT)

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-12-02 21:01:12 +00:00
f8a4060111 vendor: containerd 0edc412565dcc6e3d6125ff9e4b009ad4b89c638
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-12-02 21:01:12 +00:00
53471faac2 vendor: github.com/containerd/cgroups 0b889c03f102012f1d93a97ddd3ef71cd6f4f510
full diff: 318312a373...0b889c03f1

- v1/stats: add all fields of memory.oom_control
- memory: remove wrong memory.kmem.limit_in_bytes check
- CI: test against Go 1.15

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-12-02 21:01:12 +00:00
2c694b3233 vendor: github.com/opentracing/opentracing-go v1.2.0
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-12-02 21:01:12 +00:00
74d93d1fa2 vendor: github.com/tonistiigi/fsutil 0834f99b7b85462efb69b4f571a4fa3ca7da5ac9
full diff: ae3a8d7530...0834f99b7b

- walker: fix notadir error
- improving error returns
    - more typed errors
    - remove extra verbosity (eg. PathError already contains action and path)
    - ensure stack traces are added to errors
- various testing and linting fixes
- copy: use Clonefileat from golang.org/x/sys/unix on macOS
- go.mod: update opencontainers/go-digest v1.0.0
- github: test go1.15

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2020-12-02 21:01:12 +00:00