Commit Graph

860 Commits

Author SHA1 Message Date
d00ceca6f0 btrfs: ensure graphdriver home is bind mount
For some reason, shared mount propagation between the host
and a container does not work for btrfs, unless container
root directory (i.e. graphdriver home) is a bind mount.

The above issue was reproduced on SLES 12sp3 + btrfs using
the following script:

	#!/bin/bash
	set -eux -o pipefail

	# DIR should not be under a subvolume
	DIR=${DIR:-/lib}
	MNT=$DIR/my-mnt
	FILE=$MNT/file

	ID=$(docker run -d --privileged -v $DIR:$DIR:rshared ubuntu sleep 24h)
	docker exec $ID mkdir -p $MNT
	docker exec $ID mount -t tmpfs tmpfs $MNT
	docker exec $ID touch $FILE
	ls -l $FILE
	umount $MNT
	docker rm -f $ID

which fails this way:

	+ ls -l /lib/my-mnt/file
	ls: cannot access '/lib/my-mnt/file': No such file or directory

meaning the mount performed inside a priviledged container is not
propagated back to the host (even if all the mounts have "shared"
propagation mode).

The remedy to the above is to make graphdriver home a bind mount.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 16d822bba8ac5ab22c8697750f700403bca3dbf3)
Upstream-commit: fa8ac946165b8004a15e85744e774ed6ba99fd38
Component: engine
2018-10-12 09:29:38 -07:00
1fc8f05911 gd/dm: fix error message
The parameter name was wrong, which may mislead a user.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit c378fb774e413ba8bf5cadf655d2b67e9c94245a)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Upstream-commit: c9ddc6effc444c54def41d498b359a9a986ad79d
Component: engine
2018-10-03 14:01:13 +02:00
dc90119f4b Merge pull request #37659 from Microsoft/jjh/37356
LCOW: Mount to short container paths to avoid command-line length limit
Upstream-commit: 8613b34a7e363948555eb986fd6078d9f9c808d7
Component: engine
2018-08-17 15:48:35 -07:00
2db873cf38 LCOW: Capture stderr on external process. Log actual error rather than throwaway
Signed-off-by: John Howard <jhoward@microsoft.com>
Upstream-commit: dffc96616503936fc282c0e3a3eea8695ee11b67
Component: engine
2018-08-16 19:20:14 -07:00
cef8826091 LCOW: Mount to short container paths to avoid command-line length limit
Signed-off-by: John Howard <jhoward@microsoft.com>

Fixes #36764

@johnstep PTAL. @jterry75 FYI.

There are two commits in this PR. The first ensure that errors are actually returned to the caller - it was being thrown away.

The second commit changes the LCOW driver to map, on a per service VM basis, "long" container paths such as `/tmp/c8fa0ae1b348f505df2707060f6a49e63280d71b83b7936935c827e2e9bde16d` to much shorter paths, based on a per-service VM counter, so something more like /tmp/d3. This means that the root cause of the failure where the mount call to create the overlay was failing due to command line length becomes something much shorter such as below.

`mount -t overlay overlay -olowerdir=/tmp/d3:/tmp/d4:/tmp/d5:/tmp/d6:/tmp/d7:/tmp/d8:/tmp/d9:/tmp/d10:/tmp/d11:/tmp/d12:/tmp/d13:/tmp/d14:/tmp/d15:/tmp/d16:/tmp/d17:/tmp/d18:/tmp/d19:/tmp/d20:/tmp/d21:/tmp/d22:/tmp/d23:/tmp/d24:/tmp/d25:/tmp/d26:/tmp/d27:/tmp/d28:/tmp/d29:/tmp/d30:/tmp/d31:/tmp/d32:/tmp/d33:/tmp/d34:/tmp/d35:/tmp/d36:/tmp/d37:/tmp/d38:/tmp/d39:/tmp/d40:/tmp/d41:/tmp/d42:/tmp/d43:/tmp/d44:/tmp/d45:/tmp/d46:/tmp/d47:/tmp/d48:/tmp/d49:/tmp/d50:/tmp/d51:/tmp/d52:/tmp/d53:/tmp/d54:/tmp/d55:/tmp/d56:/tmp/d57:/tmp/d58:/tmp/d59:/tmp/d60:/tmp/d61:/tmp/d62,upperdir=/tmp/d2/upper,workdir=/tmp/d2/work /tmp/c8fa0ae1b348f505df2707060f6a49e63280d71b83b7936935c827e2e9bde16d-mount`

For those worrying about overflow (which I'm sure @thaJeztah will mention...): It's safe to use a counter here as SVMs are disposable in the default configuration. The exception is when running the daemon in unsafe LCOW "global" mode (ie `--storage-opt lcow.globalmode=1`) where the SVMs aren't disposed of, but a single one is reused. However, to overflow the command line length, it would require several hundred-thousand trillion (conservative, I should sit down and work it out accurately if I get -really- bored) of SCSI hot-add operations, and even to hit that would be hard as just running containers normally uses the VPMEM path for the containers UVM, not to the global SVM on SCSI. It gets incremented by one per build step (commit more accurately) as a general rule. Hence it would be necessary to have to be doing automated builds without restarting the daemon for literally years on end in unsafe mode. 😇

Note that in reality, the previous limit of ~47 layers before hitting the command line length limit is close to what is possible in the platform, at least as of RS5/Windows Server 2019 where, in the HCS v1 schema, a single SCSI controller is used, and that can only support 64 disks per controller per the Hyper-V VDEV. And remember we have one slot taken up for the SVMs scratch, and another for the containers scratch when committing a layer. So the best you can architecturally get on the platform is around the following (it's also different by 1 depending on whether in unsafe or default mode)

```
PS E:\docker\build\36764\short> docker build --no-cache .
Sending build context to Docker daemon  2.048kB
Step 1/4 : FROM alpine as first
 ---> 11cd0b38bc3c
Step 2/4 : RUN echo test > /test
 ---> Running in 8ddfe20e5bfb
Removing intermediate container 8ddfe20e5bfb
 ---> b0103a00b1c9
Step 3/4 : FROM alpine
 ---> 11cd0b38bc3c
Step 4/4 : COPY --from=first /test /test
 ---> 54bfae391eba
Successfully built 54bfae391eba
PS E:\docker\build\36764\short> cd ..
PS E:\docker\build\36764> docker build --no-cache .
Sending build context to Docker daemon  4.689MB
Step 1/61 : FROM alpine as first
 ---> 11cd0b38bc3c
Step 2/61 : RUN echo test > /test
 ---> Running in 02597ff870db
Removing intermediate container 02597ff870db
 ---> 3096de6fc454
Step 3/61 : RUN echo test > /test
 ---> Running in 9a8110f4ff19
Removing intermediate container 9a8110f4ff19
 ---> 7691808cf28e
Step 4/61 : RUN echo test > /test
 ---> Running in 9afb8f51510b
Removing intermediate container 9afb8f51510b
 ---> e42a0df2bb1c
Step 5/61 : RUN echo test > /test
 ---> Running in fe977ed6804e
Removing intermediate container fe977ed6804e
 ---> 55850c9b0479
Step 6/61 : RUN echo test > /test
 ---> Running in be65cbfad172
Removing intermediate container be65cbfad172
 ---> 0cf8acba70f0
Step 7/61 : RUN echo test > /test
 ---> Running in fd5b0907b6a9
Removing intermediate container fd5b0907b6a9
 ---> 257a4493d85d
Step 8/61 : RUN echo test > /test
 ---> Running in f7ca0ffd9076
Removing intermediate container f7ca0ffd9076
 ---> 3baa6f4fa2d5
Step 9/61 : RUN echo test > /test
 ---> Running in 5146814d4727
Removing intermediate container 5146814d4727
 ---> 485b9d5cf228
Step 10/61 : RUN echo test > /test
 ---> Running in a090eec1b743
Removing intermediate container a090eec1b743
 ---> a7eb10155b51
Step 11/61 : RUN echo test > /test
 ---> Running in 942660b288df
Removing intermediate container 942660b288df
 ---> 9d286a1e2133
Step 12/61 : RUN echo test > /test
 ---> Running in c3d369aa91df
Removing intermediate container c3d369aa91df
 ---> f78be4788992
Step 13/61 : RUN echo test > /test
 ---> Running in a03c3ac6888f
Removing intermediate container a03c3ac6888f
 ---> 6504363f61ab
Step 14/61 : RUN echo test > /test
 ---> Running in 0c3c2fca3f90
Removing intermediate container 0c3c2fca3f90
 ---> fe3448b8bb29
Step 15/61 : RUN echo test > /test
 ---> Running in 828d51c76d3b
Removing intermediate container 828d51c76d3b
 ---> 870684e3aea0
Step 16/61 : RUN echo test > /test
 ---> Running in 59a2f7c5f3ad
Removing intermediate container 59a2f7c5f3ad
 ---> cf84556ca5c0
Step 17/61 : RUN echo test > /test
 ---> Running in bfb4e088eeb3
Removing intermediate container bfb4e088eeb3
 ---> 9c8f9f652cef
Step 18/61 : RUN echo test > /test
 ---> Running in f1b88bb5a2d7
Removing intermediate container f1b88bb5a2d7
 ---> a6233ad21648
Step 19/61 : RUN echo test > /test
 ---> Running in 45f70577d709
Removing intermediate container 45f70577d709
 ---> 1b5cc52d370d
Step 20/61 : RUN echo test > /test
 ---> Running in 2ce231d5043d
Removing intermediate container 2ce231d5043d
 ---> 4a0e17cbebaa
Step 21/61 : RUN echo test > /test
 ---> Running in 52e4b0928f1f
Removing intermediate container 52e4b0928f1f
 ---> 99b50e989bcb
Step 22/61 : RUN echo test > /test
 ---> Running in f7ba3da7460d
Removing intermediate container f7ba3da7460d
 ---> bfa3cad88285
Step 23/61 : RUN echo test > /test
 ---> Running in 60180bf60f88
Removing intermediate container 60180bf60f88
 ---> fe7271988bcb
Step 24/61 : RUN echo test > /test
 ---> Running in 20324d396531
Removing intermediate container 20324d396531
 ---> e930bc039128
Step 25/61 : RUN echo test > /test
 ---> Running in b3ac70fd4404
Removing intermediate container b3ac70fd4404
 ---> 39d0a11ea6d8
Step 26/61 : RUN echo test > /test
 ---> Running in 0193267d3787
Removing intermediate container 0193267d3787
 ---> 8062d7aab0a5
Step 27/61 : RUN echo test > /test
 ---> Running in f41f45fb7985
Removing intermediate container f41f45fb7985
 ---> 1f5f18f2315b
Step 28/61 : RUN echo test > /test
 ---> Running in 90dd09c63d6e
Removing intermediate container 90dd09c63d6e
 ---> 02f0a1141f11
Step 29/61 : RUN echo test > /test
 ---> Running in c557e5386e0a
Removing intermediate container c557e5386e0a
 ---> dbcd6fb1f6f4
Step 30/61 : RUN echo test > /test
 ---> Running in 65369385d855
Removing intermediate container 65369385d855
 ---> e6e9058a0650
Step 31/61 : RUN echo test > /test
 ---> Running in d861fcc388fd
Removing intermediate container d861fcc388fd
 ---> 6e4c2c0f741f
Step 32/61 : RUN echo test > /test
 ---> Running in 1483962b7e1c
Removing intermediate container 1483962b7e1c
 ---> cf8f142aa055
Step 33/61 : RUN echo test > /test
 ---> Running in 5868934816c1
Removing intermediate container 5868934816c1
 ---> d5ff87cdc204
Step 34/61 : RUN echo test > /test
 ---> Running in e057f3201f3a
Removing intermediate container e057f3201f3a
 ---> b4031b7ab4ac
Step 35/61 : RUN echo test > /test
 ---> Running in 22b769b9079c
Removing intermediate container 22b769b9079c
 ---> 019d898510b6
Step 36/61 : RUN echo test > /test
 ---> Running in f1d364ef4ff8
Removing intermediate container f1d364ef4ff8
 ---> 9525cafdf04d
Step 37/61 : RUN echo test > /test
 ---> Running in 5bf505b8bdcc
Removing intermediate container 5bf505b8bdcc
 ---> cd5002b33bfd
Step 38/61 : RUN echo test > /test
 ---> Running in be24a921945c
Removing intermediate container be24a921945c
 ---> 8675db44d1b7
Step 39/61 : RUN echo test > /test
 ---> Running in 352dc6beef3d
Removing intermediate container 352dc6beef3d
 ---> 0ab0ece43c71
Step 40/61 : RUN echo test > /test
 ---> Running in eebde33e5d9b
Removing intermediate container eebde33e5d9b
 ---> 46ca4b0dfc03
Step 41/61 : RUN echo test > /test
 ---> Running in f920313a1e85
Removing intermediate container f920313a1e85
 ---> 7f3888414d58
Step 42/61 : RUN echo test > /test
 ---> Running in 10e2f4dc1ac7
Removing intermediate container 10e2f4dc1ac7
 ---> 14db9e15f2dc
Step 43/61 : RUN echo test > /test
 ---> Running in c849d6e89aa5
Removing intermediate container c849d6e89aa5
 ---> fdb770494dd6
Step 44/61 : RUN echo test > /test
 ---> Running in 419d1a8353db
Removing intermediate container 419d1a8353db
 ---> d12e9cf078be
Step 45/61 : RUN echo test > /test
 ---> Running in 0f1805263e4c
Removing intermediate container 0f1805263e4c
 ---> cd005e7b08a4
Step 46/61 : RUN echo test > /test
 ---> Running in 5bde05b46441
Removing intermediate container 5bde05b46441
 ---> 05aa426a3d4a
Step 47/61 : RUN echo test > /test
 ---> Running in 01ebc84bd1bc
Removing intermediate container 01ebc84bd1bc
 ---> 35d371fa4342
Step 48/61 : RUN echo test > /test
 ---> Running in 49f6c2f51dd4
Removing intermediate container 49f6c2f51dd4
 ---> 1090b5dfa130
Step 49/61 : RUN echo test > /test
 ---> Running in f8a9089cd725
Removing intermediate container f8a9089cd725
 ---> b2d0eec0716d
Step 50/61 : RUN echo test > /test
 ---> Running in a1697a0b2db0
Removing intermediate container a1697a0b2db0
 ---> 10d96ac8f497
Step 51/61 : RUN echo test > /test
 ---> Running in 33a2332c06eb
Removing intermediate container 33a2332c06eb
 ---> ba5bf5609c1c
Step 52/61 : RUN echo test > /test
 ---> Running in e8920392be0d
Removing intermediate container e8920392be0d
 ---> 5b3a95685c7e
Step 53/61 : RUN echo test > /test
 ---> Running in 4b9298587c65
Removing intermediate container 4b9298587c65
 ---> d4961a349141
Step 54/61 : RUN echo test > /test
 ---> Running in 8a0c960c2ba1
Removing intermediate container 8a0c960c2ba1
 ---> b413197fcfa2
Step 55/61 : RUN echo test > /test
 ---> Running in 536ee3b9596b
Removing intermediate container 536ee3b9596b
 ---> fc16b69b224a
Step 56/61 : RUN echo test > /test
 ---> Running in 8b817b8d7b59
Removing intermediate container 8b817b8d7b59
 ---> 2f0896400ff9
Step 57/61 : RUN echo test > /test
 ---> Running in ab0ed79ec3d4
Removing intermediate container ab0ed79ec3d4
 ---> b4fb420e736c
Step 58/61 : RUN echo test > /test
 ---> Running in 8548d7eead1f
Removing intermediate container 8548d7eead1f
 ---> 745103fd5a38
Step 59/61 : RUN echo test > /test
 ---> Running in 1980559ad5d6
Removing intermediate container 1980559ad5d6
 ---> 08c1c74a5618
Step 60/61 : FROM alpine
 ---> 11cd0b38bc3c
Step 61/61 : COPY --from=first /test /test
 ---> 67f053c66c27
Successfully built 67f053c66c27
PS E:\docker\build\36764>
```

Note also that subsequent error messages once you go beyond current platform limitations kind of suck (such as insufficient resources with a bunch of spew which is incomprehensible to most) and we could do better to detect this earlier in the daemon. That'll be for a (reasonably low-priority) follow-up though as and when I have time. Theoretically we *may*, if the platform doesn't require additional changes for RS5, be able to have bigger platform limits using the v2 schema with up to 127 VPMem devices, and the possibility to have multiple SCSI controllers per SVM/UVM. However, currently LCOW is using HCS v1 schema calls, and there's no plans to rewrite the graphdriver/libcontainerd components outside of the moving LCOW fully over to the containerd runtime/snapshotter using HCS v2 schema, which is still some time off fruition.

PS OK, while waiting for a full run to complete, I did get bored. Turns out it won't overflow line length as max(uint64) is 18446744073709551616 which would still be short enough at 127 layers, double the current platform limit. And I could always change it to hex or base36 to make it even shorter, or remove the 'd' from /tmp/dN. IOW, pretty sure no-one is going to hit the limit even if we could get the platform to 256 which is the current Hyper-V SCSI limit per VM (4x64), although PMEM at 127 would be the next immediate limit.
Upstream-commit: f586fe5637bbadd919ce4a126c6d95e4f3b1523b
Component: engine
2018-08-16 19:18:55 -07:00
7414934b6c Add ADD/COPY --chown flag support to Windows
This implements chown support on Windows. Built-in accounts as well
as accounts included in the SAM database of the container are supported.

NOTE: IDPair is now named Identity and IDMappings is now named
IdentityMapping.

The following are valid examples:
ADD --chown=Guest . <some directory>
COPY --chown=Administrator . <some directory>
COPY --chown=Guests . <some directory>
COPY --chown=ContainerUser . <some directory>

On Windows an owner is only granted the permission to read the security
descriptor and read/write the discretionary access control list. This
fix also grants read/write and execute permissions to the owner.

Signed-off-by: Salahuddin Khan <salah@docker.com>
Upstream-commit: 763d8392612942ff5c32a35f8bdafd7ae93d3321
Component: engine
2018-08-13 21:59:11 -07:00
8a9efc106f lcow: fix debug in startServiceVMIfNotRunning()
When go-1.11beta1 is used for building, the following error is
reported:

> 14:56:20 daemon\graphdriver\lcow\lcow.go:236: Debugf format %s reads
> arg #2, but call has 1 arg

While fixing this, let's also fix a few other things in this
very function (startServiceVMIfNotRunning):

1. Do not use fmt.Printf when not required.
2. Use `title` whenever possible.
3. Don't add `id` to messages as `title` already has it.
4. Remove duplicated colons.
5. Try to unify style of messages.
6. s/startservicevmifnotrunning/startServiceVMIfNotRunning/
...

In general, logging/debugging here is a mess and requires much more
love than I can give it at the moment. Areas for improvement:

1. Add a global var logger = logrus.WithField("storage-driver", "lcow")
and use it everywhere else in the code.
2. Use logger.WithField("id", id) whenever possible (same for "context"
and other similar fields).
3. Revise all the errors returned to be uniform.
4. Make use of errors.Wrap[f] whenever possible.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Upstream-commit: b7a95a3ce4c731c0fca204435be758ea89d6050f
Component: engine
2018-07-12 13:42:23 -07:00
d01a9704de aufs: fix Wrapf args
Fix the following go-1.11beta1 build error:

> daemon/graphdriver/aufs/aufs.go:376: Wrapf format %s reads arg #1, but call has 0 args

While at it, change '%s' to %q.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Upstream-commit: 2e30e9e6db42043cb2bd67d25a7152488c834f9f
Component: engine
2018-07-11 15:52:22 +02:00
775daf1b72 Update overlay2 to use naive diff for changes
The archive changes function is not implemented correctly
to handle opaque directories.

Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Upstream-commit: ac5d363e67e01c769b4d9e20ed76a278434ee4de
Component: engine
2018-06-20 11:07:36 -07:00
4cadaa03f8 Update tests to use gotest.tools 👼
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
Upstream-commit: 38457285242e57306c5b7ee652c7ccbb9fbd6713
Component: engine
2018-06-13 09:04:30 +02:00
2ca4487c26 When id is empty for overlay2/overlay, do not remove the directories.
Signed-off-by: fanjiyun <fan.jiyun@zte.com.cn>
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Upstream-commit: 0e8f96e31724a7bb49d0ade9acec116f68c85c74
Component: engine
2018-06-07 17:35:54 -07:00
285424548c Move plugin client to separate interface
This makes it a bit simpler to remove this interface for v2 plugins
and not break external projects (libnetwork and swarmkit).

Note that before we remove the `Client()` interface from `CompatPlugin`
libnetwork and swarmkit must be updated to explicitly check for the v1
client interface as is done int his PR.

This is just a minor tweak that I realized is needed after trying to
implement the needed changes on libnetwork.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Upstream-commit: 7c77df8acc597cd4f540d873de5fe53a3d414ba9
Component: engine
2018-05-30 15:22:10 -04:00
4c7a840b18 Move plugin client creation to the extension point
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Upstream-commit: f51a96c0165fdcbbe11f62b66b582b7e202f211b
Component: engine
2018-05-25 15:18:53 -04:00
0f96e98e12 Various code-cleanup
remove unnescessary import aliases, brackets, and so on.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Upstream-commit: f23c00d8701e4bd0f2372a586dacbf66a26f9a51
Component: engine
2018-05-23 17:50:54 +02:00
91ebfe260a aufs: log reason why aufs is not supported.
In case aufs driver is not supported because supportsAufs() said so,
it is not possible to get a real reason from the logs.

To fix, log the error returned.

Note we're not using WithError here as the error message itself is the
sole message we want to print (i.e. there's nothing to add to it).

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Upstream-commit: 91f85d1c784f3dc9d892b2af2f51d6b6f3b0be69
Component: engine
2018-05-20 12:02:41 +02:00
30e8f43743 aufs: use a single logger
Simplify the code by using a single logger instance.

While at it, use WithError in Umount.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Upstream-commit: c6e2af54256211b8ac757e9b25caa6fb6c9b3c6e
Component: engine
2018-05-20 12:02:12 +02:00
c03259c23d Merge pull request #37022 from cgxu519/overlayutils-addmsg
Add additional message when backendfs is extfs without d_type support
Upstream-commit: f9dd74deee364dd513615ea112e78008a946b3e5
Component: engine
2018-05-19 22:38:45 +02:00
c3521e7da6 Add additional message when backendfs is extfs without d_type support
ext4 support d_type by default, but filetype feature is a tunable so
there is still a chance to disable it for some reasons. In this case,
print additional message to explicitly tell how to support d_type.

Signed-off-by: Chengguang Xu <cgxu519@gmx.com>
Upstream-commit: 8a21b128d4deb874c05eb81ebbc1265175ad69ba
Component: engine
2018-05-18 10:32:47 +08:00
7f78f7fe15 Fix some linting issues
These showed locally when running `make validate`. CI doesn't seem to have the
same (possibly it's disabled in the configuration)

    builder/fscache/fscache.go:618::error: github.com/docker/docker/vendor/github.com/tonistiigi/fsutil.StatInfo composite literal uses unkeyed fields (vet)
    client/swarm_unlock_test.go:44::error: github.com/docker/docker/api/types/swarm.UnlockRequest composite literal uses unkeyed fields (vet)
    client/swarm_unlock_test.go:20::error: github.com/docker/docker/api/types/swarm.UnlockRequest composite literal uses unkeyed fields (vet)
    cmd/dockerd/daemon_unix.go:113::error: github.com/docker/docker/cmd/dockerd/hack.MalformedHostHeaderOverride composite literal uses unkeyed fields (vet)
    cmd/dockerd/daemon_unix.go:110::error: github.com/docker/docker/cmd/dockerd/hack.MalformedHostHeaderOverride composite literal uses unkeyed fields (vet)
    daemon/graphdriver/overlay/overlay.go:171::error: github.com/docker/docker/pkg/idtools.IDPair composite literal uses unkeyed fields (vet)
    daemon/graphdriver/overlay/overlay.go:413::error: github.com/docker/docker/pkg/idtools.IDPair composite literal uses unkeyed fields (vet)
    daemon/graphdriver/overlay2/overlay.go:203::error: github.com/docker/docker/pkg/idtools.IDPair composite literal uses unkeyed fields (vet)
    daemon/graphdriver/overlay2/overlay.go:584::error: github.com/docker/docker/pkg/idtools.IDPair composite literal uses unkeyed fields (vet)
    daemon/graphdriver/zfs/zfs.go:109::error: github.com/docker/docker/pkg/idtools.IDPair composite literal uses unkeyed fields (vet)
    daemon/graphdriver/zfs/zfs.go:388::error: github.com/docker/docker/pkg/idtools.IDPair composite literal uses unkeyed fields (vet)
    daemon/volumes_windows.go:27::error: github.com/docker/docker/pkg/idtools.IDPair composite literal uses unkeyed fields (vet)
    integration/service/network_test.go:31::error: github.com/docker/docker/api/types/network.NetworkingConfig composite literal uses unkeyed fields (vet)
    api/server/server.go:129:10⚠️ should not use basic type string as key in context.WithValue (golint)
    integration/service/network_test.go:54::error: github.com/docker/docker/api/types/network.NetworkingConfig composite literal uses unkeyed fields (vet)
    libcontainerd/client_daemon_linux.go:61::error: github.com/docker/docker/pkg/idtools.IDPair composite literal uses unkeyed fields (vet)
    libcontainerd/client_daemon_linux.go:74::error: github.com/docker/docker/pkg/idtools.IDPair composite literal uses unkeyed fields (vet)
    pkg/archive/archive_windows.go:76::error: github.com/docker/docker/pkg/idtools.IDPair composite literal uses unkeyed fields (vet)
    plugin/manager_linux.go:56::error: github.com/docker/docker/pkg/idtools.IDPair composite literal uses unkeyed fields (vet)

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Upstream-commit: 4f8c870d623c63b98f8ef8002448f07f4dda4aa9
Component: engine
2018-05-17 19:28:27 +02:00
9e6796b652 Fix typos
Signed-off-by: weipeng <weipeng@tuscloud.io>
Upstream-commit: f781ec45b5c1745279f0429c2b7f19c793efcb85
Component: engine
2018-05-16 09:15:43 +08:00
70cd643c8a Merge pull request #37040 from thaJeztah/error_on_unsupported_options
overlay: do not ignore invalid storage-driver options
Upstream-commit: a79d04ae55196f73d67eeb8a40cfbeb67fed95e9
Component: engine
2018-05-14 15:37:33 +02:00
0d708bdf61 overlay2: remove unused cdMountFrom() helper function
This function was added in 23e5c94cfb26eb72c097892712d3dbaa93ee9bc0 but never used

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Upstream-commit: 8a8ec00c1fece033b20548b7a5ec2f3a1f340834
Component: engine
2018-05-11 13:51:50 +02:00
5da69b8b39 overlay: do not ignore invalid storage-driver options
The overlay storage driver currently does not support any option, but was silently
ignoring any option that was passed.

This patch verifies that no options are passed, and if they are passed will produce
an error.

Before this change:

    dockerd --storage-driver=overlay --storage-opt dm.thinp_percent=95
    INFO[2018-05-11T11:40:40.996597152Z] libcontainerd: started new docker-containerd process  pid=256
    ....
    INFO[2018-05-11T11:40:41.135392535Z] Daemon has completed initialization
    INFO[2018-05-11T11:40:41.141035093Z] API listen on /var/run/docker.sock

After this change:

    dockerd --storage-driver=overlay --storage-opt dm.thinp_percent=95
    INFO[2018-05-11T11:39:21.632610319Z] libcontainerd: started new docker-containerd process  pid=233
    ....
    Error starting daemon: error initializing graphdriver: overlay: unknown option dm.thinp_percent

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Upstream-commit: 30f15d2bdc551870464d1cd024a92341cf1ae4aa
Component: engine
2018-05-11 13:40:46 +02:00
153889840e Merge pull request #36599 from Microsoft/jjh/dockersave
LCOW: Implement `docker save`
Upstream-commit: 65c575f592a247b3dd0d10dd707f1bbf3d459ae1
Component: engine
2018-04-24 11:14:32 -07:00
554b5427fa Merge pull request #36091 from kolyshkin/mount
pkg/mount improvements
Upstream-commit: 53982e3fc1642f2605189cea48bc6de7b9318611
Component: engine
2018-04-21 11:03:54 +02:00
57e3475a77 Merge pull request #36492 from alejgh/36478-log-standardization
36478 storage-drivers log format standardization
Upstream-commit: c4ba5be6af4aa960d2f6d2071b2218cf0b62bb20
Component: engine
2018-04-20 12:11:14 +02:00
a97e7a41c8 pkg/mount: implement/use filter for mountinfo parsing
Functions `GetMounts()` and `parseMountTable()` return all the entries
as read and parsed from /proc/self/mountinfo. In many cases the caller
is only interested only one or a few entries, not all of them.

One good example is `Mounted()` function, which looks for a specific
entry only. Another example is `RecursiveUnmount()` which is only
interested in mount under a specific path.

This commit adds `filter` argument to `GetMounts()` to implement
two things:
 1. filter out entries a caller is not interested in
 2. stop processing if a caller is found what it wanted

`nil` can be passed to get a backward-compatible behavior, i.e. return
all the entries.

A few filters are implemented:
 - `PrefixFilter`: filters out all entries not under `prefix`
 - `SingleEntryFilter`: looks for a specific entry

Finally, `Mounted()` is modified to use `SingleEntryFilter()`, and
`RecursiveUnmount()` is using `PrefixFilter()`.

Unit tests are added to check filters are working.

[v2: ditch NoFilter, use nil]
[v3: ditch GetMountsFiltered()]
[v4: add unit test for filters]
[v5: switch to gotestyourself]

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Upstream-commit: bb934c6aca3e77541dd4fd51b9ab2706294dadda
Component: engine
2018-04-19 14:48:09 -07:00
fe0deb4aa6 Merge pull request #36738 from Microsoft/jjh/getlayerpath
Windows: Add GetLayerPath implementation in graphdriver
Upstream-commit: 5c2e6064a1c5cb816081958e5a7079bc6b474940
Component: engine
2018-04-14 09:59:38 -07:00
7e2f77eef8 Merge pull request #36728 from darrenstahlmsft/LayerLeak
Fix Windows layer leak when write fails
Upstream-commit: a8260058e39f7014f811394f86ae2c0bc899e11e
Component: engine
2018-04-04 19:05:59 -07:00
28820945d1 Fix Windows layer leak when write fails
Signed-off-by: Darren Stahl <darst@microsoft.com>
Upstream-commit: 1f28844d7869609f371ab2a7881e4488a79a7e27
Component: engine
2018-04-04 12:31:23 -07:00
94036de956 Fix FIFO, sockets and device files when run in user NS
commit 617c352e9225 "Don't create devices if in a user namespace"

introduced check, which meant to skip mknod operation when run
in user namespace, but instread skipped FIFO and socket files
copy.

Signed-off-by: Maxim Ivanov <ivanov.maxim@gmail.com>
Upstream-commit: 6f084f292932c464a30b56edb9edbe238bdcf0aa
Component: engine
2018-04-01 12:33:59 +01:00
187be19082 Merge pull request #36736 from tonistiigi/remove-test-kernel-version
copy: remove kernel version test
Upstream-commit: 6c948b562d826fd95da5f972305d22c47a479494
Component: engine
2018-03-30 10:53:40 +02:00
9b41292c7f Support cancellation in directory.Size()
Makes sure that if the user cancels a request that the daemon stops
trying to traverse a directory.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Upstream-commit: 9d46c4c138d7b3f7778c13fe84857712bd6c97a9
Component: engine
2018-03-29 15:49:15 -04:00
60950b5248 Windows: GetLayerPath in graphdriver
Signed-off-by: John Howard (VM) <jhoward@ntdev.microsoft.com>
Upstream-commit: 3aab84803dec4a28484b319531685276435263cd
Component: engine
2018-03-29 11:45:45 -07:00
c26d6b4944 copy: remove kernel version test
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Upstream-commit: 4d6b8f73a89abf715fa33b3f553884c575a3b136
Component: engine
2018-03-29 10:45:43 -07:00
252f993dde Standardized log messages accross the different storage drivers.
Now all of the storage drivers use the field "storage-driver" in their log
messages, which is set to name of the respective driver.
Storage drivers changed:
- Aufs
- Btrfs
- Devicemapper
- Overlay
- Overlay 2
- Zfs

Signed-off-by: Alejandro GonzÃlez Hevia <alejandrgh11@gmail.com>
Upstream-commit: 9392838150f5495a63f33fed6570ae41f5a6d62e
Component: engine
2018-03-27 14:37:30 +02:00
0331f04e35 Post migration assertion fixes
Signed-off-by: Daniel Nephin <dnephin@docker.com>
Upstream-commit: c9e52bd0da0461e605a3678b85702f83081504a7
Component: engine
2018-03-16 11:03:46 -04:00
60daf5fa97 Automated migration using
gty-migrate-from-testify --ignore-build-tags

Signed-off-by: Daniel Nephin <dnephin@docker.com>
Upstream-commit: 6be0f709830113966f295401327b027ec2f0bbca
Component: engine
2018-03-16 11:03:43 -04:00
c113de75ae LCOW: Graphdriver DiffGetter implementation
Signed-off-by: John Howard <jhoward@microsoft.com>
Upstream-commit: 6b4f989bc2c19be58e10731b9b0504b5f94607bf
Component: engine
2018-03-14 11:55:06 -07:00
1158d96fc9 Merge pull request #36307 from kolyshkin/dm-misc
devmapper cleanup improvements
Upstream-commit: bc7424b4434443b036604277bfcdee3fa34a268a
Component: engine
2018-03-07 12:57:21 -08:00
ca48b9593b Merge pull request #36437 from kolyshkin/dm-unused
devmapper.Mounted: remove
Upstream-commit: a6b1d2ea297827d763de9e4b1940434e00a37883
Component: engine
2018-03-05 18:14:36 -08:00
8b5e7a334b devmapper.shutdown: optimize
Move the "unmount and deactivate" code into a separate method, and
optimize it a bit:

1. Do not use filepath.Walk() as there's no requirement to recursively
   go into every directory under home/mnt; a list of directories in mnt
   is sufficient. With filepath.Walk(), in case some container will fail
   to unmount, it'll go through the whole container filesystem which is
   excessive and useless.

2. Do not use GetMounts() and check if a directory is mounted; just
   unmount it and ignore "not mounted" error. Note the same error
   is returned in case of wrong flags set, but as flags are hardcoded
   we can safely ignore such case.

While at it, promote "can't unmount" log level from debug to warning.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Upstream-commit: f1a459229724f5e8e440b49f058167c2eeeb2dc6
Component: engine
2018-03-05 10:08:56 -08:00
8f7dd9a1c4 devmapper cleanup: improve error msg
1. Make sure it's clear the error is from unmount.

2. Simplify the code a bit to make it more readable.

[v2: use errors.Wrap]
[v3: use errors.Wrapf]
[v4: lowercase the error message]

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Upstream-commit: 9d00aedebc25507042c5afd4ab8fc6b333ca7c53
Component: engine
2018-03-05 10:08:56 -08:00
ce4da0689b devmapper/Remove(): use Rmdir, ignore errors
1. Replace EnsureRemoveAll() with Rmdir(), as here we are removing
   the container's mount point, which is already properly unmounted
   and is therefore an empty directory.

2. Ignore the Rmdir() error (but log it unless it's ENOENT). This
   is a mount point, currently unmounted (i.e. an empty directory),
   and an older kernel can return EBUSY if e.g. the mount was
   leaked to other mount namespaces.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Upstream-commit: 732dd9b848bec70a2ecb5b4998918886a0cec497
Component: engine
2018-03-02 18:10:57 -08:00
1a6f7f8e03 devmapper.Mounted: remove
It looks like no one uses this function.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Upstream-commit: 0450f61cb995c8fc2f41a6909526be6ed4093565
Component: engine
2018-02-27 19:13:46 -08:00
50ae06e05c Golint fix up
This fix fixes a golint issue.

Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
Upstream-commit: 742d4506bd5aa6f686070d0d5e984a8b9ee2cda6
Component: engine
2018-02-23 16:40:37 +00:00
ad01430349 Merge pull request #35829 from cpuguy83/no_private_mount_for_plugins
Perform plugin mounts in the runtime
Upstream-commit: 20028325daab4fcbee9c8e28f43dbfb2b1c5d568
Component: engine
2018-02-21 12:28:13 +01:00
39cad2aa10 Merge pull request #36256 from wcwxyz/fix-refcounter-memory-leak
graphdriver: Fix RefCounter memory leak
Upstream-commit: 733ed2ddd3c621dadafbb74feb7b80d20fd3fd6f
Component: engine
2018-02-19 10:32:14 -08:00
8bb8847f9c Merge pull request #36237 from cpuguy83/zfs_do_not_unmount
Do not recursive unmount on cleanup of zfs/btrfs
Upstream-commit: 68c3201626439d5be5c24d14d4fe7e27fe93954d
Component: engine
2018-02-14 09:49:17 -05:00
4adc380b90 Fix typos in daemon
Signed-off-by: bin liu <liubin0329@gmail.com>
Upstream-commit: b00a67be6e3d3f241879110bd342abaa8e23cbac
Component: engine
2018-02-10 19:42:54 +08:00