Commit Graph

6043 Commits

Author SHA1 Message Date
630cd7d11b Merge pull request #257 from thaJeztah/18.09_backport_increase_swarmkit_grpc
[18.09 backport] Increase max recv gRPC message size for nodes and secrets
Upstream-commit: ce2e2b672a7a3510419cb2706edab3fd7f12fd6c
Component: engine
2019-06-17 12:31:27 -07:00
eae44d6a4c Merge pull request #223 from thaJeztah/18.09_backport_devno
[18.09 backport] bugfix: fetch the right device number which great than 255
Upstream-commit: b236a1e78d35a62b164d00f4e19f59a556da8831
Component: engine
2019-06-17 12:27:17 -07:00
6a5e7a9ed6 Merge pull request #263 from thaJeztah/18.09_backport_39290alternate
[18.09 backport] Windows: Don't attempt detach VHD for R/O layers
Upstream-commit: ab9db72ae809597b5fb7a9e245f18fa3f99857ec
Component: engine
2019-06-17 12:23:41 -07:00
0041943819 aufs: retry auplink flush
Running a bundled aufs benchmark sometimes results in this warning:

> WARN[0001] Couldn't run auplink before unmount /tmp/aufs-tests/aufs/mnt/XXXXX  error="exit status 22" storage-driver=aufs

If we take a look at what aulink utility produces on stderr, we'll see:

> auplink:proc_mnt.c:96: /tmp/aufs-tests/aufs/mnt/XXXXX: Invalid argument

and auplink exits with exit code of 22 (EINVAL).

Looking into auplink source code, what happens is it tries to find a
record in /proc/self/mounts corresponding to the mount point (by using
setmntent()/getmntent_r() glibc functions), and it fails.

Some manual testing, as well as runtime testing with lots of printf
added on mount/unmount, as well as calls to check the superblock fs
magic on mount point (as in graphdriver.Mounted(graphdriver.FsMagicAufs, target)
confirmed that this record is in fact there, but sometimes auplink
can't find it. I was also able to reproduce the same error (inability
to find a mount in /proc/self/mounts that should definitely be there)
using a small C program, mocking what `auplink` does:

```c
 #include <stdio.h>
 #include <err.h>
 #include <mntent.h>
 #include <string.h>
 #include <stdlib.h>

int main(int argc, char **argv)
{
	FILE *fp;
	struct mntent m, *p;
	char a[4096];
	char buf[4096 + 1024];
	int found =0, lines = 0;

	if (argc != 2) {
		fprintf(stderr, "Usage: %s <mountpoint>\n", argv[0]);
		exit(1);
	}

	fp = setmntent("/proc/self/mounts", "r");
	if (!fp) {
		err(1, "setmntent");
	}
	setvbuf(fp, a, _IOLBF, sizeof(a));
	while ((p = getmntent_r(fp, &m, buf, sizeof(buf)))) {
		lines++;
		if (!strcmp(p->mnt_dir, argv[1])) {
			found++;
		}
	}
	printf("found %d entries for %s (%d lines seen)\n", found, argv[1], lines);
	return !found;
}
```

I have also wrote a few other C proggies -- one that reads
/proc/self/mounts directly, one that reads /proc/self/mountinfo instead.
They are also prone to the same occasional error.

It is not perfectly clear why this happens, but so far my best theory
is when a lot of mounts/unmounts happen in parallel with reading
contents of /proc/self/mounts, sometimes the kernel fails to provide
continuity (i.e. it skips some part of file or mixes it up in some
other way). In other words, this is a kernel bug (which is probably
hard to fix unless some other interface to get a mount entry is added).

Now, there is no real fix, and a workaround I was able to come up
with is to retry when we got EINVAL. It usually works on the second
attempt, although I've once seen it took two attempts to go through.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit ae431b10a9508e2bf3b1782e9d435855e3e13977)
Upstream-commit: c303e63ca6c3d25d29b8992451898734fa6e4e7c
Component: engine
2019-06-05 11:51:00 -07:00
c70089ebb3 aufs.Cleanup: optimize
Do not use filepath.Walk() as there's no requirement to recursively
go into every directory under mnt -- a (non-recursive) list of
directories in mnt is sufficient.

With filepath.Walk(), in case some container will fail to unmount,
it'll go through the whole container filesystem which is both
excessive and useless.

This is similar to commit f1a459229724f5e8e440b49f ("devmapper.shutdown:
optimize")

While at it, raise the priority of "unmount error" message from debug
to a warning. Note we don't have to explicitly add `m` as unmount error (from
pkg/mount) will have it.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 8fda12c6078ed5c86be0822a7a980c6fbc9220bf)
Upstream-commit: b85d4a4f09badf40201ef7fcbd5b930de216190d
Component: engine
2019-06-05 11:51:00 -07:00
3b8f3b67f2 aufs: optimize lots of layers case
In case there are a big number of layers, so that mount data won't fit
into a single memory page (4096 bytes on most platforms, which is good
enough for about 40 layers, depending on how long graphdriver root path
is), we supply additional layers with O_REMOUNT, as described in aufs
documentation.

Problem is, the current implementation does that one layer at a time
(i.e. there is one mount syscall per each additional layer).

Optimize the code to supply as many layers as we can fit in one page
(basically reusing the same code as for the original mount).

Note, per aufs docs, "[a]t remount-time, the options are interpreted
in the given order, e.g. left to right" so we should be good.

Tested on an image with ~100 layers.

Before (35 syscalls):
> [pid 22756] 1556919088.686955 mount("none", "/mnt/volume_sfo2_09/docker-aufs/aufs/mnt/a86f8c9dd0ec2486293119c20b0ec026e19bbc4d51332c554f7cf05d777c9866", "aufs", 0, "br:/mnt/volume_sfo2_09/docker-au"...) = 0 <0.000504>
> [pid 22756] 1556919088.687643 mount("none", "/mnt/volume_sfo2_09/docker-aufs/aufs/mnt/a86f8c9dd0ec2486293119c20b0ec026e19bbc4d51332c554f7cf05d777c9866", 0xc000c451b0, MS_REMOUNT, "append:/mnt/volume_sfo2_09/docke"...) = 0 <0.000105>
> [pid 22756] 1556919088.687851 mount("none", "/mnt/volume_sfo2_09/docker-aufs/aufs/mnt/a86f8c9dd0ec2486293119c20b0ec026e19bbc4d51332c554f7cf05d777c9866", 0xc000c451ba, MS_REMOUNT, "append:/mnt/volume_sfo2_09/docke"...) = 0 <0.000098>
> ..... (~30 lines skipped for clarity)
> [pid 22756] 1556919088.696182 mount("none", "/mnt/volume_sfo2_09/docker-aufs/aufs/mnt/a86f8c9dd0ec2486293119c20b0ec026e19bbc4d51332c554f7cf05d777c9866", 0xc000c45310, MS_REMOUNT, "append:/mnt/volume_sfo2_09/docke"...) = 0 <0.000266>

After (2 syscalls):
> [pid 24352] 1556919361.799889 mount("none", "/mnt/volume_sfo2_09/docker-aufs/aufs/mnt/8e7ba189e347a834e99eea4ed568f95b86cec809c227516afdc7c70286ff9a20", "aufs", 0, "br:/mnt/volume_sfo2_09/docker-au"...) = 0 <0.001717>
> [pid 24352] 1556919361.801761 mount("none", "/mnt/volume_sfo2_09/docker-aufs/aufs/mnt/8e7ba189e347a834e99eea4ed568f95b86cec809c227516afdc7c70286ff9a20", 0xc000dbecb0, MS_REMOUNT, "append:/mnt/volume_sfo2_09/docke"...) = 0 <0.001358>

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit d58c434bffef76e48bff75ede290937874488dd3)
Upstream-commit: 75488521735325e4564e66d9cf9c2b1bc1c2c64b
Component: engine
2019-06-05 11:51:00 -07:00
5ada897229 aufs: add lock around mount
Apparently there is some kind of race in aufs kernel module code,
which leads to the errors like:

[98221.158606] aufs au_xino_create2:186:dockerd[25801]: aufs.xino create err -17
[98221.162128] aufs au_xino_set:1229:dockerd[25801]: I/O Error, failed creating xino(-17).
[98362.239085] aufs au_xino_create2:186:dockerd[6348]: aufs.xino create err -17
[98362.243860] aufs au_xino_set:1229:dockerd[6348]: I/O Error, failed creating xino(-17).
[98373.775380] aufs au_xino_create:767:dockerd[27435]: open /dev/shm/aufs.xino(-17)
[98389.015640] aufs au_xino_create2:186:dockerd[26753]: aufs.xino create err -17
[98389.018776] aufs au_xino_set:1229:dockerd[26753]: I/O Error, failed creating xino(-17).
[98424.117584] aufs au_xino_create:767:dockerd[27105]: open /dev/shm/aufs.xino(-17)

So, we have to have a lock around mount syscall.

While at it, don't call the whole Unmount() on an error path, as
it leads to bogus error from auplink flush.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 5cd62852fa199f272b542d828c8c5d1db427ea53)
Upstream-commit: bb4b9fe29eba57ed71d8105ec0e5dacc8e72129e
Component: engine
2019-06-05 11:51:00 -07:00
5db6e4ad9b aufs: aufsMount: better errors for unix.Mount()
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 5873768dbe3be2733874b8cf68cb492817f81a94)
Upstream-commit: 4ab3020e8df18b5c12c668df6ef3e21b648ce04b
Component: engine
2019-06-05 11:51:00 -07:00
4fdc82012a aufs: use mount.Unmount
1. Use mount.Unmount() which ignores EINVAL ("not mounted") error,
and provides better error diagnostics (so we don't have to explicitly
add target to error messages).

2. Since we're ignoring "not mounted" error, we can call
multiple unmounts without any locking -- but since "auplink flush"
is still involved and can produce an error in logs, let's keep
the check for fs being mounted (it's just a statfs so should be fast).

2. While at it, improve the "can't unmount" error message in Put().

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 4beee98026feabe4f4f0468215b8fd9b56f90d5e)
Upstream-commit: 5b68d00abc4cad8b707a9a01dde9a420ca6eb995
Component: engine
2019-06-05 11:51:00 -07:00
a829a04518 aufs: remove extra locking
Both mount and unmount calls are already protected by fine-grained
(per id) locks in Get()/Put() introduced in commit fc1cf1911bb
("Add more locking to storage drivers"), so there's no point in
having a global lock in mount/unmount.

The only place from which unmount is called without any locking
is Cleanup() -- this is to be addressed in the next patch.

This reverts commit 824c24e6802ad3ed7e26b4f16e5ae81869b98185.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit f93750b2c4d5f6144f0790ffa89291da3c097b80)
Upstream-commit: 701939112efc71a0c867a626f0a56df9c56030db
Component: engine
2019-06-05 11:51:00 -07:00
e03614a156 aufs: get rid of mount()
The function is not needed as it's just a shallow wrapper around
unix.Mount().

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 2f98b5f51fb0a21e1bd930c0e660c4a7c4918aa5)
Upstream-commit: 023b63a0f276b79277c1c961d8c98d4d5b39525d
Component: engine
2019-06-05 11:51:00 -07:00
c81448bb56 pkg/mount: wrap mount/umount errors
The errors returned from Mount and Unmount functions are raw
syscall.Errno errors (like EPERM or EINVAL), which provides
no context about what has happened and why.

Similar to os.PathError type, introduce mount.Error type
with some context. The error messages will now look like this:

> mount /tmp/mount-tests/source:/tmp/mount-tests/target, flags: 0x1001: operation not permitted

or

> mount tmpfs:/tmp/mount-test-source-516297835: operation not permitted

Before this patch, it was just

> operation not permitted

[v2: add Cause()]
[v3: rename MountError to Error, document Cause()]
[v4: fixes; audited all users]
[v5: make Error type private; changes after @cpuguy83 reviews]

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 65331369617e89ce54cc9be080dba70f3a883d1c)
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Upstream-commit: 7f1c6bf5a745c8faeba695d3556dff4c4ff5f473
Component: engine
2019-06-05 11:50:50 -07:00
bb4beac3ed Windows: Don't attempt detach VHD for R/O layers
Signed-off-by: John Howard <jhoward@microsoft.com>
(cherry picked from commit 293c74ba79f0008f48073985507b34af59b45fa6)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Upstream-commit: 91f5be57af815df372371e1e989d00963ce4d02f
Component: engine
2019-06-05 14:45:21 +02:00
6f78462c1a UnmountIpcMount: simplify
As standard mount.Unmount does what we need, let's use it.

In addition, this adds ignoring "not mounted" condition, which
was previously implemented (see PR#33329, commit cfa2591d3f26)
via a very expensive call to mount.Mounted().

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 77bc327e24a60791fe7e87980faf704cf7273cf9)
Upstream-commit: 893b24b80db170279d5c9532ed508a81c328de5e
Component: engine
2019-06-04 15:21:22 -07:00
741575eb38 Increase max recv gRPC message size for nodes and secrets
Increases the max recieved gRPC message size for Node and Secret list
operations. This has already been done for the other swarm types, but
was not done for these.

Signed-off-by: Drew Erny <drew.erny@docker.com>
(cherry picked from commit a0903e1fa3eca32065c7dbfda8d1e0879cbfbb8f)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Upstream-commit: a987f31fbc42526bf2cccf1381066f075b18dfe0
Component: engine
2019-06-03 23:01:33 +02:00
7432dfa749 Add chroot for tar packing operations
Previously only unpack operations were supported with chroot.
This adds chroot support for packing operations.
This prevents potential breakouts when copying data from a container.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit 3029e765e241ea2b5249868705dbf9095bc4d529)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Upstream-commit: 61e0459053c359e322b8d5c017e855f616fd34c0
Component: engine
2019-06-03 18:55:00 +02:00
23dc194c07 Pass root to chroot to for chroot Untar
This is useful for preventing CVE-2018-15664 where a malicious container
process can take advantage of a race on symlink resolution/sanitization.

Before this change chrootarchive would chroot to the destination
directory which is attacker controlled. With this patch we always chroot
to the container's root which is not attacker controlled.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit d089b639372a8f9301747ea56eaf0a42df24016a)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Upstream-commit: 155939994f453559676656bc4b05635e83ebef56
Component: engine
2019-06-03 18:54:52 +02:00
8852e9cdc4 bugfix: fetch the right device number which great than 255
Signed-off-by: frankyang <yyb196@gmail.com>
(cherry picked from commit b9f31912deb511e732763e4fa5ecd0208b104eb2)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Upstream-commit: e4cf15b4f51b3b6ef380dbfe5061568bc84eece1
Component: engine
2019-05-21 12:12:48 +02:00
d2f16c6807 Use assert.NilError() instead of assert.Assert()
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 3449b12cc7eefa8ebd0de6ec8b9803c6ee823af0)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Upstream-commit: 93de0314c7e44d625d791fa14d7316da6d1fc9c7
Component: engine
2019-04-17 23:08:54 +02:00
c975557150 cli: fix images filter when use multi reference filter
Signed-off-by: zhangyue <zy675793960@yeah.net>
(cherry picked from commit 5007c36d71ac86f5b47b6ba10a5ed6e841807284)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Upstream-commit: 8a614d1b2494b6d2381e157a99ad2736a6038d2d
Component: engine
2019-04-17 21:32:23 +02:00
4c39df1955 Merge pull request #180 from thaJeztah/18.09_backport_fix_net_host_systemd_resolved
[18.09 backport] Fix: network=host using wrong resolv.conf with systemd-resolved
Upstream-commit: 80e10316cd01832a0fe04bc7970569bf86c3e391
Component: engine
2019-03-28 10:17:58 -07:00
bd877991d7 Merge pull request #178 from thaJeztah/18.09_backport_exec_spec
[18.09 backport] Use original process spec for execs
Upstream-commit: b4bf217633d7884999b675b8a7b9b81892b46086
Component: engine
2019-03-28 09:37:29 -07:00
705a3b6133 Fix stopped containers with restart-policy showing as "restarting"
When manually stopping a container with a restart-policy, the container
would show as "restarting" in `docker ps` whereas its actual state
is "exited".

Stopping a container with a restart policy shows the container as "restarting"

    docker run -d --name test --restart unless-stopped busybox false

    docker stop test

    docker ps
    CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS                       PORTS               NAMES
    7e07409fa1d3        busybox             "false"             5 minutes ago       Restarting (1) 4 minutes ago                     test

However, inspecting the same container shows that it's exited:

    docker inspect test --format '{{ json .State }}'
    {
      "Status": "exited",
      "Running": false,
      "Paused": false,
      "Restarting": false,
      "OOMKilled": false,
      "Dead": false,
      "Pid": 0,
      "ExitCode": 1,
      "Error": "",
      "StartedAt": "2019-02-14T13:26:27.6091648Z",
      "FinishedAt": "2019-02-14T13:26:27.689427Z"
    }

And killing the container confirms this;

    docker kill test
    Error response from daemon: Cannot kill container: test: Container 7e07409fa1d36dc8d8cb8f25cf12ee1168ad9040183b85fafa73ee2c1fcf9361 is not running

    docker run -d --name test --restart unless-stopped busybox false

    docker stop test

    docker ps
    CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS                PORTS               NAMES
    d0595237054a        busybox             "false"             5 minutes ago       Restarting (1)       4 minutes ago                       exit

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 8c0ecb638705e89746a81fa1320aafaa7ff701b2)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Upstream-commit: 00f0b9df0dde80b2c40023b12a29dd476f2a77e2
Component: engine
2019-03-26 14:17:52 +01:00
c992d819e3 Fix: network=host using wrong resolv.conf with systemd-resolved
When running a container in the host's network namespace, the container
gets a copy of the host's resolv.conf (copied to `/etc/resolv.conf` inside
the container).

The current code always used the default (`/etc/resolv.conf`) path on the
host, irregardless if `systemd-resolved` was used or not.

This patch uses the correct file if `systemd-resolved` was detected
to be running.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 8364d1c9d590d4266871cd820b76ef12e2b934ed)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Upstream-commit: 04ae160eca4a8bc134ca9abafa044191b893080b
Component: engine
2019-03-26 12:55:53 +01:00
bb938bc1ef Use original process spec for execs
Fixes #38865

Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
(cherry picked from commit 7603c22c7365d7d7150597fe396e0707d6e561da)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Upstream-commit: 3d3d7570714a8ab60b979eaba39309b6e8fcf75e
Component: engine
2019-03-26 12:37:59 +01:00
7e985fdbcc set bigger grpc limit for GetConfigs api
Signed-off-by: Dani Louca <dani.louca@docker.com>
(cherry picked from commit 3fbbeb703c1d04e9eb723459960fbfc7f3bbfc40)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Upstream-commit: 5f40e17cfd2474776088a97552fce7d5a9abe549
Component: engine
2019-02-27 19:01:12 +01:00
cdacf38efa Merge pull request #154 from thaJeztah/18.09_backport_fix_stale_container_on_start
[18.09 backport] Delete stale containerd object on start failure
Upstream-commit: ba8664cc2219c3ed75d3f4306569edf8b7b97a98
Component: engine
2019-02-22 13:52:47 -08:00
c835df1d14 Merge pull request #160 from thaJeztah/18.09_backport_add_missing_char_device_mode
[18.09 backport] Graphdriver: fix "device" mode not being detected if "character-device" bit is set
Upstream-commit: 2e4c5c57c30522dc4b33b5cd5371f294ae3fee82
Component: engine
2019-02-21 17:01:40 -08:00
a345ccfd0e Graphdriver: fix "device" mode not being detected if "character-device" bit is set
Due to a bug in Golang (github.com/golang#27640), the "character device"
bit was omitted when checking file-modes with `os.ModeType`.

This bug was resolved in Go 1.12, but as a result, graphdrivers
would no longer recognize "device" files, causing pulling of
images that have a file with this filemode to fail;

    failed to register layer:
    unknown file type for /var/lib/docker/vfs/dir/.../dev/console

The current code checked for an exact match of Modes to be set. The
`os.ModeCharDevice` and `os.ModeDevice` bits will always be set in
tandem, however, because the code was only looking for an exact
match, this detection broke now that `os.ModeCharDevice` was added.

This patch changes the code to be more defensive, and instead
check if the `os.ModeDevice` bit is set (either with, or without
the `os.ModeCharDevice` bit).

In addition, some information was added to the error-message if
no type was matched, to assist debugging in case additional types
are added in future.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit c7a38c2c06f7ab844a48c6c447942913131b83d6)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Upstream-commit: 3744b45ba8ad93f1a21cbc80420856b04efc4593
Component: engine
2019-02-20 14:31:18 +01:00
241a2913ec Delete stale containerd object on start failure
containerd has two objects with regard to containers.
There is a "container" object which is metadata and a "task" which is
manging the actual runtime state.

When docker starts a container, it creartes both the container metadata
and the task at the same time. So when a container exits, docker deletes
both of these objects as well.

This ensures that if, on start, when we go to create the container metadata object
in containerd, if there is an error due to a name conflict that we go
ahead and clean that up and try again.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit 5ba30cd1dc6000ee53b34f628cbff91d7f6d7231)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Upstream-commit: 1d0353548a21b0b60595ad44859a7072f90ea6c6
Component: engine
2019-02-15 01:01:52 +01:00
242fedd184 keep old network ids
for windows all networks are re-populated in the store during network controller initialization. In current version it also regenerate network Ids which may be referenced by other components and it may cause broken references to a networks. This commit avoids regeneration of network ids.

Signed-off-by: Andrey Kolomentsev <andrey.kolomentsev@docker.com>
(cherry picked from commit e017717d96540dd263d95f90fdb2457928909924)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Upstream-commit: 317e0acc4e602f978e4d9c0130a113d179026c8e
Component: engine
2019-02-11 22:12:52 +01:00
c9df8f4688 use empty string as cgroup path to grab first find
Signed-off-by: Andrew Hsu <andrewhsu@docker.com>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 78045a5419718ba8884ed5afc0b41a3a80afc3b4)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Upstream-commit: b66c7ad62ebff12112318db0d1ed48b14f817efa
Component: engine
2018-12-07 18:58:03 +01:00
7e016c5a84 Merge pull request #81 from thaJeztah/18.09_backport_processandiot
[18.09 backport] Windows:Allow process isolation
Upstream-commit: a1f6b04a8d373dbed6262f076d9aac834031d24a
Component: engine
2018-11-27 09:58:22 -08:00
06d25f9c17 Merge pull request #85 from thaJeztah/18.09_backport_deprecated_storagedrivers
[18.09 backport] deprecate legacy "overlay", and "devicemapper" storage drivers 
Upstream-commit: 7a566c0e4a98ec9960b5312e806ab546203dff15
Component: engine
2018-11-27 09:57:46 -08:00
1e216e2ec4 Merge pull request #112 from thaJeztah/18.09_backport_moby_37747
[18.09 backport] awslogs: account for UTF-8 normalization in limits
Upstream-commit: 08a77f11a63b2d25c00ca0e35012194bdcfbe917
Component: engine
2018-11-27 09:48:39 -08:00
bf3a1002a3 Merge pull request #113 from thaJeztah/18.09_backport_detach
[18.09 backport] Windows: DetachVhd attempt in cleanup
Upstream-commit: 4fd103ae26fc7784e79ed40f4e500e96477e14d2
Component: engine
2018-11-27 09:47:07 -08:00
f10a3615ef Merge pull request #121 from thaJeztah/18.09_backport_containerd_v1.2.1
[18.09 backport] Update containerd to v1.2.1-rc.0
Upstream-commit: c95cf2a5d36b5448e4771a29f2ad72214a2311dd
Component: engine
2018-11-27 09:15:48 -08:00
2aed2152c7 Merge pull request #84 from thaJeztah/18.09_backport_ovr2_index
[18.09 backport] overlay2: use index=off if possible (fix EBUSY on mount)
Upstream-commit: 27b0fee84688d0590cac13bf2f507a58bc841251
Component: engine
2018-11-21 15:46:01 -06:00
825b77191a Windows: DetachVhd attempt in cleanup
Signed-off-by: John Howard <jhoward@microsoft.com>

This is a fix for a few related scenarios where it's impossible to remove layers or containers
until the host is rebooted. Generally (or at least easiest to repro) through a forced daemon kill
while a container is running.

Possibly slightly worse than that, as following a host reboot, the scratch layer would possibly be leaked and
left on disk under the dataroot\windowsfilter directory after the container is removed.

One such example of a failure:

1. run a long running container with the --rm flag
docker run --rm -d --name test microsoft/windowsservercore powershell sleep 30
2. Force kill the daemon not allowing it to cleanup. Simulates a crash or a host power-cycle.
3. (re-)Start daemon
4. docker ps -a
PS C:\control> docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED             STATUS                PORTS               NAMES
7aff773d782b        malloc              "powershell start-sl…"   11 seconds ago      Removal In Progress                       malloc
5. Try to remove
PS C:\control> docker rm 7aff
Error response from daemon: container 7aff773d782bbf35d95095369ffcb170b7b8f0e6f8f65d5aff42abf61234855d: driver "windowsfilter" failed to remove root filesystem: rename C:\control\windowsfilter\7aff773d782bbf35d95095369ffcb170b7b8f0e6f8f65d5aff42abf61234855d C:\control\windowsfilter\7aff773d782bbf35d95095369ffcb170b7b8f0e6f8f65d5aff42abf61234855d-removing: Access is denied.
PS C:\control>

Step 5 fails.

(cherry picked from commit efdad5374465a2a889d7572834a2dcca147af4fb)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Upstream-commit: 02fe71843e0e45ddc986a6c8182370b042349a27
Component: engine
2018-11-09 23:31:49 +01:00
6ad9059d36 awslogs: account for UTF-8 normalization in limits
The CloudWatch Logs API defines its limits in terms of bytes, but its
inputs in terms of UTF-8 encoded strings.  Byte-sequences which are not
valid UTF-8 encodings are normalized to the Unicode replacement
character U+FFFD, which is a 3-byte sequence in UTF-8.  This replacement
can cause the input to grow, exceeding the API limit and causing failed
API calls.

This commit adds logic for counting the effective byte length after
normalization and splitting input without splitting valid UTF-8
byte-sequences into two invalid byte-sequences.

Fixes https://github.com/moby/moby/issues/37747

Signed-off-by: Samuel Karp <skarp@amazon.com>
(cherry picked from commit 1e8ef386279e2e28aff199047e798fad660efbdd)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Upstream-commit: 757650e8dcca87f95ba083a80639769a0b6ca1cc
Component: engine
2018-11-08 15:26:01 +01:00
97ead3de83 Merge pull request #74 from thaJeztah/18.09_backport_no_more_version_mismatch
[18.09 backport] remove version-checks for containerd and runc
Upstream-commit: f5749085e9cb0565afe342e73a67631f97547054
Component: engine
2018-11-06 11:31:40 -08:00
cd9d72e185 LCOW: ApplyDiff() use tar2ext4, not SVM
Signed-off-by: John Howard <jhoward@microsoft.com>

This removes the need for an SVM in the LCOW driver to ApplyDiff.

This change relates to a fix for https://github.com/moby/moby/issues/36353

However, it found another issue, tracked by https://github.com/moby/moby/issues/37955

(cherry picked from commit bde99960657818efabf313991867c42921882275)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Upstream-commit: 9cf6464b639aef09df9b16999cc340a1276d0bf8
Component: engine
2018-11-06 11:03:00 +01:00
a659f7a44e Merge pull request #99 from andrewhsu/grpc
[18.09] cluster: set bigger grpc limit for array requests
Upstream-commit: fb51c760c4fed545a83a6b6f88716e5d6bb8b1f0
Component: engine
2018-10-30 18:49:11 -07:00
00083e1aa8 cluster: set bigger grpc limit for array requests
4MB client side limit was introduced in vendoring go-grpc#1165 (v1.4.0)
making these requests likely to produce errors

Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
(cherry picked from commit 489b8eda6674523df8b82a210399b7d2954427d0)
Signed-off-by: Andrew Hsu <andrewhsu@docker.com>
Upstream-commit: 6ca0546f2571cf4acdc1f541bccfac23a78cb8d2
Component: engine
2018-10-30 23:04:27 +00:00
927264f5be Fix incorrect spelling in error message
Signed-off-by: Phil Estes <estesp@linux.vnet.ibm.com>
(cherry picked from commit f962bd06ed8824d1f75d8546b428965cd61bdf7f)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Upstream-commit: 5591f0b1ee7dec101b490228258613cd7caf64ee
Component: engine
2018-10-30 11:29:02 +01:00
d00ceca6f0 btrfs: ensure graphdriver home is bind mount
For some reason, shared mount propagation between the host
and a container does not work for btrfs, unless container
root directory (i.e. graphdriver home) is a bind mount.

The above issue was reproduced on SLES 12sp3 + btrfs using
the following script:

	#!/bin/bash
	set -eux -o pipefail

	# DIR should not be under a subvolume
	DIR=${DIR:-/lib}
	MNT=$DIR/my-mnt
	FILE=$MNT/file

	ID=$(docker run -d --privileged -v $DIR:$DIR:rshared ubuntu sleep 24h)
	docker exec $ID mkdir -p $MNT
	docker exec $ID mount -t tmpfs tmpfs $MNT
	docker exec $ID touch $FILE
	ls -l $FILE
	umount $MNT
	docker rm -f $ID

which fails this way:

	+ ls -l /lib/my-mnt/file
	ls: cannot access '/lib/my-mnt/file': No such file or directory

meaning the mount performed inside a priviledged container is not
propagated back to the host (even if all the mounts have "shared"
propagation mode).

The remedy to the above is to make graphdriver home a bind mount.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 16d822bba8ac5ab22c8697750f700403bca3dbf3)
Upstream-commit: fa8ac946165b8004a15e85744e774ed6ba99fd38
Component: engine
2018-10-12 09:29:38 -07:00
09de879e7a Deprecate legacy overlay storage driver, and add warning
The `overlay` storage driver is deprecated in favor of the `overlay2` storage
driver, which has all the benefits of `overlay`, without its limitations (excessive
inode consumption). The legacy `overlay` storage driver will be removed in a future
release. Users of the `overlay` storage driver should migrate to the `overlay2`
storage driver.

The legacy `overlay` storage driver allowed using overlayFS-backed filesystems
on pre 4.x kernels. Now that all supported distributions are able to run `overlay2`
(as they are either on kernel 4.x, or have support for multiple lowerdirs
backported), there is no reason to keep maintaining the `overlay` storage driver.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 31be4e0ba11532e5d6df8401f5d459e292c58f36)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Upstream-commit: c20e8dffbbb4dd328e4c00d18e04eecb80cf4d4e
Component: engine
2018-10-12 02:26:17 +02:00
b1f3714ba1 Deprecate "devicemapper" storage driver, and add warning
The `devicemapper` storage driver is deprecated in favor of `overlay2`, and will
be removed in a future release. Users of the `devicemapper` storage driver are
recommended to migrate to a different storage driver, such as `overlay2`, which
is now the default storage driver.

The `devicemapper` storage driver facilitates running Docker on older (3.x) kernels
that have no support for other storage drivers (such as overlay2, or AUFS).

Now that support for `overlay2` is added to all supported distros (as they are
either on kernel 4.x, or have support for multiple lowerdirs backported), there
is no reason to continue maintenance of the `devicemapper` storage driver.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 06fcabbaa0d4f25db0580d3604bbed708223fb06)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Upstream-commit: 734e7a8e55e41b064025273776dc5a76e45bf2e1
Component: engine
2018-10-12 02:25:39 +02:00
0746eb40ef Merge pull request #73 from thaJeztah/18.09_backport_addr_pool
[18.09] backport default-addr-pool-mask-length param max value check
Upstream-commit: b38d454861a8b4343142a62094cc3cbf4b157e00
Component: engine
2018-10-11 13:27:22 -07:00
921a9476d4 Merge pull request #70 from thaJeztah/18.09_backport_upstream_dos_fix
[18.09] backport fix denial of service with large numbers in cpuset-cpus and cpuset-mems
Upstream-commit: 4b8336f7cf091fd5c4742286bda1e34c45667d78
Component: engine
2018-10-11 13:25:55 -07:00