Commit Graph

835 Commits

Author SHA1 Message Date
554b5427fa Merge pull request #36091 from kolyshkin/mount
pkg/mount improvements
Upstream-commit: 53982e3fc1642f2605189cea48bc6de7b9318611
Component: engine
2018-04-21 11:03:54 +02:00
57e3475a77 Merge pull request #36492 from alejgh/36478-log-standardization
36478 storage-drivers log format standardization
Upstream-commit: c4ba5be6af4aa960d2f6d2071b2218cf0b62bb20
Component: engine
2018-04-20 12:11:14 +02:00
a97e7a41c8 pkg/mount: implement/use filter for mountinfo parsing
Functions `GetMounts()` and `parseMountTable()` return all the entries
as read and parsed from /proc/self/mountinfo. In many cases the caller
is only interested only one or a few entries, not all of them.

One good example is `Mounted()` function, which looks for a specific
entry only. Another example is `RecursiveUnmount()` which is only
interested in mount under a specific path.

This commit adds `filter` argument to `GetMounts()` to implement
two things:
 1. filter out entries a caller is not interested in
 2. stop processing if a caller is found what it wanted

`nil` can be passed to get a backward-compatible behavior, i.e. return
all the entries.

A few filters are implemented:
 - `PrefixFilter`: filters out all entries not under `prefix`
 - `SingleEntryFilter`: looks for a specific entry

Finally, `Mounted()` is modified to use `SingleEntryFilter()`, and
`RecursiveUnmount()` is using `PrefixFilter()`.

Unit tests are added to check filters are working.

[v2: ditch NoFilter, use nil]
[v3: ditch GetMountsFiltered()]
[v4: add unit test for filters]
[v5: switch to gotestyourself]

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Upstream-commit: bb934c6aca3e77541dd4fd51b9ab2706294dadda
Component: engine
2018-04-19 14:48:09 -07:00
fe0deb4aa6 Merge pull request #36738 from Microsoft/jjh/getlayerpath
Windows: Add GetLayerPath implementation in graphdriver
Upstream-commit: 5c2e6064a1c5cb816081958e5a7079bc6b474940
Component: engine
2018-04-14 09:59:38 -07:00
7e2f77eef8 Merge pull request #36728 from darrenstahlmsft/LayerLeak
Fix Windows layer leak when write fails
Upstream-commit: a8260058e39f7014f811394f86ae2c0bc899e11e
Component: engine
2018-04-04 19:05:59 -07:00
28820945d1 Fix Windows layer leak when write fails
Signed-off-by: Darren Stahl <darst@microsoft.com>
Upstream-commit: 1f28844d7869609f371ab2a7881e4488a79a7e27
Component: engine
2018-04-04 12:31:23 -07:00
94036de956 Fix FIFO, sockets and device files when run in user NS
commit 617c352e9225 "Don't create devices if in a user namespace"

introduced check, which meant to skip mknod operation when run
in user namespace, but instread skipped FIFO and socket files
copy.

Signed-off-by: Maxim Ivanov <ivanov.maxim@gmail.com>
Upstream-commit: 6f084f292932c464a30b56edb9edbe238bdcf0aa
Component: engine
2018-04-01 12:33:59 +01:00
187be19082 Merge pull request #36736 from tonistiigi/remove-test-kernel-version
copy: remove kernel version test
Upstream-commit: 6c948b562d826fd95da5f972305d22c47a479494
Component: engine
2018-03-30 10:53:40 +02:00
9b41292c7f Support cancellation in directory.Size()
Makes sure that if the user cancels a request that the daemon stops
trying to traverse a directory.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Upstream-commit: 9d46c4c138d7b3f7778c13fe84857712bd6c97a9
Component: engine
2018-03-29 15:49:15 -04:00
60950b5248 Windows: GetLayerPath in graphdriver
Signed-off-by: John Howard (VM) <jhoward@ntdev.microsoft.com>
Upstream-commit: 3aab84803dec4a28484b319531685276435263cd
Component: engine
2018-03-29 11:45:45 -07:00
c26d6b4944 copy: remove kernel version test
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Upstream-commit: 4d6b8f73a89abf715fa33b3f553884c575a3b136
Component: engine
2018-03-29 10:45:43 -07:00
252f993dde Standardized log messages accross the different storage drivers.
Now all of the storage drivers use the field "storage-driver" in their log
messages, which is set to name of the respective driver.
Storage drivers changed:
- Aufs
- Btrfs
- Devicemapper
- Overlay
- Overlay 2
- Zfs

Signed-off-by: Alejandro GonzÃlez Hevia <alejandrgh11@gmail.com>
Upstream-commit: 9392838150f5495a63f33fed6570ae41f5a6d62e
Component: engine
2018-03-27 14:37:30 +02:00
0331f04e35 Post migration assertion fixes
Signed-off-by: Daniel Nephin <dnephin@docker.com>
Upstream-commit: c9e52bd0da0461e605a3678b85702f83081504a7
Component: engine
2018-03-16 11:03:46 -04:00
60daf5fa97 Automated migration using
gty-migrate-from-testify --ignore-build-tags

Signed-off-by: Daniel Nephin <dnephin@docker.com>
Upstream-commit: 6be0f709830113966f295401327b027ec2f0bbca
Component: engine
2018-03-16 11:03:43 -04:00
1158d96fc9 Merge pull request #36307 from kolyshkin/dm-misc
devmapper cleanup improvements
Upstream-commit: bc7424b4434443b036604277bfcdee3fa34a268a
Component: engine
2018-03-07 12:57:21 -08:00
ca48b9593b Merge pull request #36437 from kolyshkin/dm-unused
devmapper.Mounted: remove
Upstream-commit: a6b1d2ea297827d763de9e4b1940434e00a37883
Component: engine
2018-03-05 18:14:36 -08:00
8b5e7a334b devmapper.shutdown: optimize
Move the "unmount and deactivate" code into a separate method, and
optimize it a bit:

1. Do not use filepath.Walk() as there's no requirement to recursively
   go into every directory under home/mnt; a list of directories in mnt
   is sufficient. With filepath.Walk(), in case some container will fail
   to unmount, it'll go through the whole container filesystem which is
   excessive and useless.

2. Do not use GetMounts() and check if a directory is mounted; just
   unmount it and ignore "not mounted" error. Note the same error
   is returned in case of wrong flags set, but as flags are hardcoded
   we can safely ignore such case.

While at it, promote "can't unmount" log level from debug to warning.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Upstream-commit: f1a459229724f5e8e440b49f058167c2eeeb2dc6
Component: engine
2018-03-05 10:08:56 -08:00
8f7dd9a1c4 devmapper cleanup: improve error msg
1. Make sure it's clear the error is from unmount.

2. Simplify the code a bit to make it more readable.

[v2: use errors.Wrap]
[v3: use errors.Wrapf]
[v4: lowercase the error message]

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Upstream-commit: 9d00aedebc25507042c5afd4ab8fc6b333ca7c53
Component: engine
2018-03-05 10:08:56 -08:00
ce4da0689b devmapper/Remove(): use Rmdir, ignore errors
1. Replace EnsureRemoveAll() with Rmdir(), as here we are removing
   the container's mount point, which is already properly unmounted
   and is therefore an empty directory.

2. Ignore the Rmdir() error (but log it unless it's ENOENT). This
   is a mount point, currently unmounted (i.e. an empty directory),
   and an older kernel can return EBUSY if e.g. the mount was
   leaked to other mount namespaces.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Upstream-commit: 732dd9b848bec70a2ecb5b4998918886a0cec497
Component: engine
2018-03-02 18:10:57 -08:00
1a6f7f8e03 devmapper.Mounted: remove
It looks like no one uses this function.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Upstream-commit: 0450f61cb995c8fc2f41a6909526be6ed4093565
Component: engine
2018-02-27 19:13:46 -08:00
50ae06e05c Golint fix up
This fix fixes a golint issue.

Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
Upstream-commit: 742d4506bd5aa6f686070d0d5e984a8b9ee2cda6
Component: engine
2018-02-23 16:40:37 +00:00
ad01430349 Merge pull request #35829 from cpuguy83/no_private_mount_for_plugins
Perform plugin mounts in the runtime
Upstream-commit: 20028325daab4fcbee9c8e28f43dbfb2b1c5d568
Component: engine
2018-02-21 12:28:13 +01:00
39cad2aa10 Merge pull request #36256 from wcwxyz/fix-refcounter-memory-leak
graphdriver: Fix RefCounter memory leak
Upstream-commit: 733ed2ddd3c621dadafbb74feb7b80d20fd3fd6f
Component: engine
2018-02-19 10:32:14 -08:00
8bb8847f9c Merge pull request #36237 from cpuguy83/zfs_do_not_unmount
Do not recursive unmount on cleanup of zfs/btrfs
Upstream-commit: 68c3201626439d5be5c24d14d4fe7e27fe93954d
Component: engine
2018-02-14 09:49:17 -05:00
4adc380b90 Fix typos in daemon
Signed-off-by: bin liu <liubin0329@gmail.com>
Upstream-commit: b00a67be6e3d3f241879110bd342abaa8e23cbac
Component: engine
2018-02-10 19:42:54 +08:00
f74751654b graphdriver: Fix RefCounter memory leak
Signed-off-by: WANG Chao <chao.wang@ucloud.cn>
Upstream-commit: 9015a05606a9bb80f0d8d2e3d43b0b682ca53db4
Component: engine
2018-02-09 10:26:06 +08:00
90450b2044 Ensure plugin returns correctly scoped paths
Before this change, volume management was relying on the fact that
everything the plugin mounts is visible on the host within the plugin's
rootfs. In practice this caused some issues with mount leaks, so we
changed the behavior such that mounts are not visible on the plugin's
rootfs, but available outside of it, which breaks volume management.

To fix the issue, allow the plugin to scope the path correctly rather
than assuming that everything is visible in `p.Rootfs`.
In practice this is just scoping the `PropagatedMount` paths to the
correct host path.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Upstream-commit: 0e5eaf8ee32662182147f5f62c1bfebef66f5c47
Component: engine
2018-02-07 15:48:27 -05:00
3928c278e5 Plugins perform propagated mount in runtime spec
Setting up the mounts on the host increases chances of mount leakage and
makes for more cleanup after the plugin has stopped.
With this change all mounts for the plugin are performed by the
container runtime and automatically cleaned up when the container exits.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Upstream-commit: a53930a04fa81b082aa78e66b342ff19cc63cc5f
Component: engine
2018-02-07 15:48:27 -05:00
33ddc6d172 Do not recursive unmount on cleanup of zfs/btrfs
This was added in #36047 just as a way to make sure the tree is fully
unmounted on shutdown.

For ZFS this could be a breaking change since there was no unmount before.
Someone could have setup the zfs tree themselves. It would be better, if
we really do want the cleanup to actually the unpacked layers checking
for mounts rather than a blind recursive unmount of the root.

BTRFS does not use mounts and does not need to unmount anyway.
These was only an unmount to begin with because for some reason the
btrfs tree was being moutned with `private` propagation.

For the other graphdrivers that still have a recursive unmount here...
these were already being unmounted and performing the recursive unmount
shouldn't break anything. If anyone had anything mounted at the
graphdriver location it would have been unmounted on shutdown anyway.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Upstream-commit: 2fe4f888bee52b1f256d6fa5e20f9b061d30221c
Component: engine
2018-02-07 15:08:17 -05:00
be83c11fb0 Add canonical import comment
Signed-off-by: Daniel Nephin <dnephin@docker.com>
Upstream-commit: 4f0d95fa6ee7f865597c03b9e63702cdcb0f7067
Component: engine
2018-02-05 16:51:57 -05:00
6c313c03fe Merge pull request #36114 from Microsoft/jjh/fixdeadlock-lcowdriver-hotremove
LCOW: Graphdriver fix deadlock in hotRemoveVHDs
Upstream-commit: 03a1df95369ddead968e48697038904c84578d00
Component: engine
2018-01-29 09:57:43 -08:00
db2a10168c Merge pull request #36047 from cpuguy83/graphdriver_improvements
Do not make graphdriver homes private mounts.
Upstream-commit: 2c05aefc99d33edde47b08e38978b6c2f4178648
Component: engine
2018-01-26 13:54:30 -05:00
a002f8068e LCOW: Graphdriver fix deadlock
Signed-off-by: John Howard <jhoward@microsoft.com>
Upstream-commit: a44fcd3d27c06aaa60d8d1cbce169f0d982e74b1
Component: engine
2018-01-26 08:57:52 -08:00
c49971b835 Merge pull request #36052 from Microsoft/jjh/no-overlay-off-only-one-disk
LCOW: Regular mount if only one layer
Upstream-commit: a8d0e36d0329063af9205b4848d1f5c09bd4c3be
Component: engine
2018-01-25 15:46:16 -08:00
12ceea25e6 Merge pull request #36051 from Microsoft/jjh/remotefs-read-return-error
LCOW remotefs - return error in Read() implementation
Upstream-commit: 3c9d023af3428f49241a2e2385dae43151185466
Component: engine
2018-01-19 11:27:13 -08:00
ebd586c561 LCOW remotefs - return error in Read() implementation
Signed-off-by: John Howard <jhoward@microsoft.com>
Upstream-commit: 6112ad6e7d5d7f5afc698447da80f91bdbf62720
Component: engine
2018-01-18 17:46:58 -08:00
942fd3c62c LCOW: Regular mount if only one layer
Signed-off-by: John Howard <jhoward@microsoft.com>
Upstream-commit: 420dc4eeb48b155e6b83fccf62f8727ce4bf5b21
Component: engine
2018-01-18 12:01:58 -08:00
852153685d LCOW: Refactor to multiple layer-stores based on feedback
Signed-off-by: John Howard <jhoward@microsoft.com>
Upstream-commit: afd305c4b5682fbc297e1685e2b7a49628b7c7f0
Component: engine
2018-01-18 08:31:05 -08:00
9b47a9d16f Do not make graphdriver homes private mounts.
The idea behind making the graphdrivers private is to prevent leaking
mounts into other namespaces.
Unfortunately this is not really what happens.

There is one case where this does work, and that is when the namespace
was created before the daemon's namespace.
However with systemd each system servie winds up with it's own mount
namespace. This causes a race betwen daemon startup and other system
services as to if the mount is actually private.

This also means there is a negative impact when other system services
are started while the daemon is running.

Basically there are too many things that the daemon does not have
control over (nor should it) to be able to protect against these kinds
of leakages. One thing is certain, setting the graphdriver roots to
private disconnects the mount ns heirarchy preventing propagation of
unmounts... new mounts are of course not propagated either, but the
behavior is racey (or just bad in the case of restarting services)... so
it's better to just be able to keep mount propagation in tact.

It also does not protect situations like `-v
/var/lib/docker:/var/lib/docker` where all mounts are recursively bound
into the container anyway.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Upstream-commit: 9803272f2db84df7955b16c0d847ad72cdc494d1
Component: engine
2018-01-18 09:34:00 -05:00
5a20e1240c LCOW: Fix OpenFile parameters
Signed-off-by: John Howard <jhoward@microsoft.com>
Upstream-commit: 141b9a74716c016029badf16aca21dc96975aaac
Component: engine
2018-01-17 07:58:18 -08:00
4a656e30d0 Improve zfs init log message for zfs
Signed-off-by: Drew Hubl <drew.hubl@gmail.com>
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Upstream-commit: 27b002f4a02e2d9f6eded9004b82cb81f121264f
Component: engine
2018-01-16 21:42:05 -05:00
d7084e8097 Merge pull request #35638 from cpuguy83/error_helpers2
Add helpers to create errdef errors
Upstream-commit: c36274da8326c59aaa12c48196671b41dcb89e5b
Component: engine
2018-01-15 10:56:46 -08:00
621388138c Golint: remove redundant ifs
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Upstream-commit: b4a63139696aea2c73ec361a9af8b36a118f0423
Component: engine
2018-01-15 00:42:25 +01:00
d4d0b5c268 Move api/errdefs to errdefs
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Upstream-commit: d453fe35b9b8b52d0677fe0c3cc8373f2f5d30d0
Component: engine
2018-01-11 21:21:43 -05:00
58f87a5e9a Cleanup graphdriver/quota test
Signed-off-by: Daniel Nephin <dnephin@docker.com>
Upstream-commit: e3a6323419c8b2a81a21134ed49a1624eb483c81
Component: engine
2018-01-08 13:41:45 -05:00
f1638ade2e Merge pull request #35674 from kolyshkin/zfs-rmdir
zfs: fix ebusy on umount etc
Upstream-commit: daded8da9178ec097e59a6b0b2bf12754b5472d8
Component: engine
2017-12-28 15:35:19 -08:00
0a947da27f Merge pull request #35827 from kolyshkin/vfs-quota
Fix VFS vs quota regression
Upstream-commit: 3e1df952b7d693ac3961f492310852bdf3106240
Component: engine
2017-12-22 12:32:03 -06:00
f0d368cb33 vfs gd: ignore quota setup errors
This is a fix to regression in vfs graph driver introduced by
commit 7a1618ced359a3ac92 ("add quota support to VFS graphdriver").

On some filesystems, vfs fails to init with the following error:

> Error starting daemon: error initializing graphdriver: Failed to mknod
> /go/src/github.com/docker/docker/bundles/test-integration/d6bcf6de610e9/root/vfs/backingFsBlockDev:
> function not implemented

As quota is not essential for vfs, let's ignore (but log as a warning) any error
from quota init.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Upstream-commit: 1e8a087850aa9f96c5000a3ad90757d2e9c0499f
Component: engine
2017-12-19 13:55:04 -08:00
c5f1cdf92c projectquota: treat ENOSYS as quota unsupported
If mknod() returns ENOSYS, it most probably means quota is not supported
here, so return the appropriate error.

This is a conservative* fix to regression in vfs graph driver introduced
by commit 7a1618ced359a3ac92 ("add quota support to VFS graphdriver").
On some filesystems, vfs fails to init with the following error:

> Error starting daemon: error initializing graphdriver: Failed to mknod
> /go/src/github.com/docker/docker/bundles/test-integration/d6bcf6de610e9/root/vfs/backingFsBlockDev:
> function not implemented

Reported-by: Brian Goff <cpuguy83@gmail.com>
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Upstream-commit: 2dd39b7841bdb9968884bbedc5db97ff77d4fe3e
Component: engine
2017-12-18 10:32:36 -08:00
fd08bae89c Remove redundant build-tags
Files that are suffixed with `_linux.go` or `_windows.go` are
already only built on Linux / Windows, so these build-tags
were redundant.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Upstream-commit: 6ed1163c98703f8dd0693cecbadc84d2cda811c3
Component: engine
2017-12-18 17:41:53 +01:00