For now, just verifying that an error is returned, but not checking the
error message itself, because those are not under our control, and may
change with different Go versions.
```
=== Failed
=== FAIL: opts TestParseDockerDaemonHost (0.00s)
hosts_test.go:87: tcp tcp:a.b.c.d address expected error "Invalid bind address format: tcp:a.b.c.d" return, got "parse tcp://tcp:a.b.c.d: invalid port \":a.b.c.d\" after host" and addr
hosts_test.go:87: tcp tcp:a.b.c.d/path address expected error "Invalid bind address format: tcp:a.b.c.d/path" return, got "parse tcp://tcp:a.b.c.d/path: invalid port \":a.b.c.d\" after host" and addr
=== FAIL: opts TestParseTCP (0.00s)
hosts_test.go:129: tcp tcp:a.b.c.d address expected error Invalid bind address format: tcp:a.b.c.d return, got parse tcp://tcp:a.b.c.d: invalid port ":a.b.c.d" after host and addr
hosts_test.go:129: tcp tcp:a.b.c.d/path address expected error Invalid bind address format: tcp:a.b.c.d/path return, got parse tcp://tcp:a.b.c.d/path: invalid port ":a.b.c.d" after host and addr
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit de1523d221)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Upstream-commit: feb68f9055
Component: cli
go1.11.13 (released 2019/08/13) includes security fixes to the net/http and net/url packages.
See the Go 1.11.13 milestone on our issue tracker for details:
https://github.com/golang/go/issues?q=milestone%3AGo1.11.13
- net/http: Denial of Service vulnerabilities in the HTTP/2 implementation
net/http and golang.org/x/net/http2 servers that accept direct connections from untrusted
clients could be remotely made to allocate an unlimited amount of memory, until the program
crashes. Servers will now close connections if the send queue accumulates too many control
messages.
The issues are CVE-2019-9512 and CVE-2019-9514, and Go issue golang.org/issue/33606.
Thanks to Jonathan Looney from Netflix for discovering and reporting these issues.
This is also fixed in version v0.0.0-20190813141303-74dc4d7220e7 of golang.org/x/net/http2.
net/url: parsing validation issue
- url.Parse would accept URLs with malformed hosts, such that the Host field could have arbitrary
suffixes that would appear in neither Hostname() nor Port(), allowing authorization bypasses
in certain applications. Note that URLs with invalid, not numeric ports will now return an error
from url.Parse.
The issue is CVE-2019-14809 and Go issue golang.org/issue/29098.
Thanks to Julian Hector and Nikolai Krein from Cure53, and Adi Cohen (adico.me) for discovering
and reporting this issue.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Upstream-commit: aed09dc7eb
Component: cli
1. Use "in-place" variables for if statements to limit their scope to
the respectful `if` block.
2. Report the error returned from sd_journal_* by using CErr().
3. Use errors.New() instead of fmt.Errorf().
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 20a0e58a794cfb9b1a1f757d222248e22555f7f0)
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Upstream-commit: b32bc11a00a82584138ae78e083930918dcb0b36
Component: engine
From the first glance, `docker logs --tail 0` does not make sense,
as it is supposed to produce no output, but `tail -n 0` from GNU
coreutils is working like that, plus there is even a test case
(`TestLogsTail` in integration-cli/docker_cli_logs_test.go).
Now, something like `docker logs --follow --tail 0` makes total
sense, so let's make it work.
(NOTE if --tail is not used, config.Tail is set to -1)
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit dd4bfe30a8ac1b31630310090dc36ae3d9253444)
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Upstream-commit: 4427414624c56c459431d3dbbde81f3396cb02b0
Component: engine
If we take a long time to process log messages, and during that time
journal file rotation occurs, the journald client library will keep
those rotated files open until sd_journal_process() is called.
By periodically calling sd_journal_process() during the processing
loop we shrink the window of time a client instance has open file
descriptors for rotated (deleted) journal files.
This code is modelled after that of journalctl [1]; the above explanation
as well as the value of 1024 is taken from there.
[v2: fix CErr() argument]
[1] https://github.com/systemd/systemd/blob/dc16327c48d/src/journal/journalctl.c#L2676
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit b73fb8fd5d521081c92b5c2cce334c21b2e0ff5f)
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Upstream-commit: 48160719f2c5734b513aec7cedbbca0c72f4a085
Component: engine
TL;DR: simplify the code, fix --follow hanging indefinitely
Do the following to simplify the followJournal() code:
1. Use Go-native select instead of C-native polling.
2. Use Watch{Producer,Consumer}Gone(), eliminating the need
to have journald.closed variable, and an extra goroutine.
3. Use sd_journal_wait(). In the words of its own man page:
> A synchronous alternative for using sd_journal_get_fd(),
> sd_journal_get_events(), sd_journal_get_timeout() and
> sd_journal_process() is sd_journal_wait().
Unfortunately, the logic is still not as simple as it
could be; the reason being, once the container has exited,
journald might still be writing some logs from its internal
buffers onto journal file(s), and there is no way to
figure out whether it's done so we are guaranteed to
read all of it back. This bug can be reproduced with
something like
> $ ID=$(docker run -d busybox seq 1 150000); docker logs --follow $ID
> ...
> 128123
> $
(The last expected output line should be `150000`).
To avoid exiting from followJournal() early, add the
following logic: once the container is gone, keep trying
to drain the journal until there's no new data for at
least `waitTimeout` time period.
Should fix https://github.com/docker/for-linux/issues/575
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit f091febc942859ffbc881f3a3aa327366603ae65)
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Upstream-commit: be568f93432ed1399a0aacca5fbd2c1046ccbc85
Component: engine
1. The journald client library initializes inotify watch(es)
during the first call to sd_journal_get_fd(), and it make sense
to open it earlier in order to not lose any journal file rotation
events.
2. It only makes sense to call this if we're going to use it
later on -- so add a check for config.Follow.
3. Remove the redundant call to sd_journal_get_fd().
NOTE that any subsequent calls to sd_journal_get_fd() return
the same file descriptor, so there's no real need to save it
for later use in wait_for_data_cancelable().
Based on earlier patch by Nalin Dahyabhai <nalin@redhat.com>.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 981c01665bcb2c9fc5e555c5b976995f31c2a6b4)
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Upstream-commit: 349e199eab5337f03b442f38720293143e1b1fca
Component: engine
In case the LogConsumer is gone, the code that sends the message can
stuck forever. Wrap the code in select case, as all other loggers do.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 79039720c8b7352691350bd56be3cc226d67f205)
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Upstream-commit: 56a8a516127a2630c3a68ab31c416f223b91e9df
Component: engine
In case Tail=N parameter is requested, we need to show N lines.
It does not make sense to walk backwards one by one if we can
do it at once. Now, if Since=T is also provided, make sure we
haven't jumped too far (before T), and if we did, move forward.
The primary motivation for this was to make the code simpler.
This also fixes a tiny bug in the "since" implementation.
Before this commit:
> $ docker logs -t --tail=6000 --since="2019-03-10T03:54:25.00" $ID | head
> 2019-03-10T03:54:24.999821000Z 95981
After:
> $ docker logs -t --tail=6000 --since="2019-03-10T03:54:25.00" $ID | head
> 2019-03-10T03:54:25.000013000Z 95982
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit ff3cd167ea4d089b7695a263ba2fc4caa0a0750c)
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Upstream-commit: 2a124db7da440f1efd4c2957320d8b25d9d9ce36
Component: engine
Minor code simplification.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit e8f6166791c097deb15c39f8dddf6f97be65b224)
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Upstream-commit: 1d336dc53bd7bf5941596ffeb253d102de609a51
Component: engine
Clean up a deferred function call in the journal reading logic.
Signed-off-by: Nalin Dahyabhai <nalin@redhat.com>
(cherry picked from commit 1ada3e85bf89201910c28f2ff6892c00cee0f137)
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Upstream-commit: e700930ca521d0c004b6a3ed8bdd35a2d538aa15
Component: engine
As in other similar drivers (jsonlog, local), use a set
(i.e. `map[whatever]struct{}`), making the code simpler.
While at it, make sure we remove the reader from the set
after calling `ProducerGone()` on it.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit b2b169f13f681cd0d591ccb06d6cfff97933db77)
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Upstream-commit: fe85c72a2eac4cbf249d2c4c754684bb447eefdd
Component: engine
There are a few more places, apparently, that List operations against
Swarm exist, besides just in the List methods. This increases the max
received message size in those places.
Signed-off-by: Drew Erny <drew.erny@docker.com>
(cherry picked from commit a84a78e9767d82abd4744dad9ce4fb3f64141a8f)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Upstream-commit: 41da428d065ab15fd2d8aba1fbd0d0056136a43b
Component: engine
full diff: 19e791fd6d...142a73731c
included:
- docker/swarmkit#2872 [19.03 backport] Only update non-terminal tasks on node removal
- backport of docker/swarmkit#2867 Only update non-terminal tasks on node removal
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Upstream-commit: 7fcfdbaab6d8c6a2d55ad7b72a851c5e92b360ac
Component: engine
The Golang base images switch to buster, which causes some breakage
in networking and packages that are no longer available; (`btrfs-tools`
is now an empty package, and `libprotobuf-c0-dev` is gone).
Some of out tests also start faiilng on stretch, and will have to be
investigated further;
```
15:13:06 --- FAIL: TestRenameAnonymousContainer (3.37s)
15:13:06 rename_test.go:168: assertion failed: 0 (int) != 1 (inspect.State.ExitCode int): container a7fe866d588d65f353f42ffc5ea5288e52700384e1d90850e9c3d4dce8657666 exited with the wrong exitcode:
15:13:38 --- FAIL: TestHostnameDnsResolution (2.23s)
15:13:38 run_linux_test.go:128: assertion failed:
15:13:38 --- ←
15:13:38 +++ →
15:13:38 @@ -1 +1,2 @@
15:13:38 +ping: bad address 'foobar'
15:13:38
15:13:38
15:13:38 run_linux_test.go:129: assertion failed: 0 (int) != 1 (res.ExitCode int)
```
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit ed672bb523cb255d0b2b79837d9c45a7c3255000)
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Upstream-commit: 3c1bc29b2c372d0f1e37dc8aac33cdc01b623aa0
Component: engine
This check was used to make sure we don't bump Go versions independently
(Linux/Windows). The Dockerfile switched to using a build-arg to allow
overriding the Go version, which rendered this check non-functional.
It also fails if Linux versions use a specific variant of the image;
08:41:31 ERROR: Failed 'ERROR: Mismatched GO versions between Dockerfile and Dockerfile.windows. Update your PR to ensure that both files are updated and in sync. ${GO_VERSION}-stretch ${GO_VERSION}' at 07/20/2019 08:41:31
08:41:31 At C:\gopath\src\github.com\docker\docker\hack\ci\windows.ps1:448 char:9
08:41:31 + Throw "ERROR: Mismatched GO versions between Dockerfile and D ...
08:41:31 + ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
This patch fixes the check by looking for the value of `GO_VERSION` instead
of looking at the `FROM` line (which is harder to parse).
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 4fa57a8191b1d23c6466725b688519f83c0ac5dd)
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Upstream-commit: a2d32c5654a206954fd0d9a5fd08abd8c03d4523
Component: engine
This was an oversight when changing the Dockerfile to use a build-arg;
the Windows Dockerfile downloads the Go binaries, which never have a
trailing `.0`.
This patch makes sure that the trailing zero (if any) is removed.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit c5bd6e3dc7680d6c683496f63dafb1f30f87eaa7)
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Upstream-commit: 8f1a27c51f93698b5fe17f0fb043f47794e11cca
Component: engine
This allows overriding the version of Go without making modifications in the
source code, which can be useful to test against multiple versions.
For example:
make GO_VERSION=1.13beta1 shell
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit c6281bc4383b7f9eab617fd73601e8594c93365b)
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Conflicts:
* Dockerfile, Dockerfile.e2e, Dockerfile.simple, Dockerfile.windows:
(due to Go version difference, missing CROSS etc.)
Upstream-commit: d9ba337adb0c3880acd34ffe9b0a15717990876b
Component: engine
Protect access to q.quotas map, and lock around changing nextProjectID.
Techinically, the lock in findNextProjectID() is not needed as it is
only called during initialization, but one can never be too careful.
Fixes: 52897d1c092 ("projectquota: utility class for project quota controls")
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 1ac0a66a64a906911d0708cd0e5fa397a2f0b595)
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Upstream-commit: 7027bb9bedae63879c1e41894739ba0ea2deedc1
Component: engine
The docker-in-docker image now enables TLS by default (added in
docker-library/docker#166), which complicates testing in our
environment, and isn't needed for the tests we're running.
This patch sets the `DOCKER_TLS_CERTDIR` to an empty value to
disable TLS.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit b1a3c1aad1)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Upstream-commit: 9c8ac0a123
Component: cli
The edge channel is deprecated and no longer updated
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 08fd6dd63c)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Upstream-commit: a88330d9dd
Component: cli
