We've seen some cases in the wild where waiting for unmount/deactivate
of devmapper devices taking a long time (several seconds). So, we increase
the sleeps to 10 seconds before we timeout. For instance:
https://github.com/dotcloud/docker/issues/4389
But, in order to not keep other processes blocked we unlock the global
dm lock while waiting to allow other devices to continue working.
Docker-DCO-1.1-Signed-off-by: Alexander Larsson <alexl@redhat.com> (github: alexlarsson)
Upstream-commit: 81f148be566ab2b17810ad4be61a5d8beac8330f
Component: engine
We currently use a global lock to protect global data (like the
Devices map) as well as device data itself and access to
(non-threadsafe) libdevmapper.
This commit also adds a per-device lock, which will allow per-device
operations to temporarily release the global lock while e.g. waiting.
The per-device lock will make sure that nothing else accesses that
device while we're operating on it.
Docker-DCO-1.1-Signed-off-by: Alexander Larsson <alexl@redhat.com> (github: alexlarsson)
Upstream-commit: 3e8a02a9399618917194b37435f5eed9ff86fe2f
Component: engine
We've seen situations where removal of "ID-init" failed during
container deletion (EBUSY), after removal of "ID" has succeeded. This
caused the container delete operation to fail, and on the next delete
attempt the removal of "ID" failed immediately with "does not exist".
Ideally we should not fail the ID-init removal, but its also non-ideal
to allow a state where the container is half-removed and we cannot
make progress deleting the container. So, we silently ignore not-exist
errors on device removal.
Docker-DCO-1.1-Signed-off-by: Alexander Larsson <alexl@redhat.com> (github: alexlarsson)
Upstream-commit: 1cdd775f5d95c4da2895da85b00ffa2917bbf9b0
Component: engine
This makes it possible to pass the rest of the testsuite without aufs in the kernel.
Docker-DCO-1.1-Signed-off-by: Alexander Larsson <alexl@redhat.com> (github: alexlarsson)
Upstream-commit: 9e28c3e3f86664ef8f219f11ef19c74a414346a9
Component: engine
All archive that are created from somewhere generally have to be closed, because
at some point there is a file or a pipe or something that backs them. So, we
make archive.Archive a ReadCloser. However, code consuming archives does not
typically close them so we add an archive.ArchiveReader and use that when we're
only reading.
We then change all the Tar/Archive places to create ReadClosers, and to properly
close them everywhere.
As an added bonus we can use ReadCloserWrapper rather than EofReader in several places,
which is good as EofReader doesn't always work right. For instance, many compression
schemes like gzip knows it is EOF before having read the EOF from the stream, so the
EofCloser never sees an EOF.
Docker-DCO-1.1-Signed-off-by: Alexander Larsson <alexl@redhat.com> (github: alexlarsson)
Upstream-commit: f198ee525ad6862dce3940e08c72e0a092380a7b
Component: engine
We no longer pass "pool" anywhere that uses byHash() per the last
commit, so we can now remove this hack.
Docker-DCO-1.1-Signed-off-by: Alexander Larsson <alexl@redhat.com> (github: alexlarsson)
Upstream-commit: 6128dcea4a9bbe808baba4e18c9c4fee3a265532
Component: engine
Shutdown contains debug warnings like:
[debug] deviceset.go:699 [deviceset docker-0:33-17945897] waitRemove(/dev/mapper/docker-0:33-17945897-pool)
[debug] deviceset.go:380 libdevmapper(3): libdm-common.c:552 (-1) Device /dev/mapper/docker-0:33-17945897-pool not found
This is because shutdown is using removeDeviceAndWait() to remove the pool device and the
wait part fails because the pool is gone.
We fix this by adding a pool specific removal function which avoids all the trickiness of the
normal remove.
Docker-DCO-1.1-Signed-off-by: Alexander Larsson <alexl@redhat.com> (github: alexlarsson)
Upstream-commit: eab270395e5b47b16a41c54ec6e1427f8144bffc
Component: engine
If /dev/loop-control exists on the system running the test then
ioctlLoopCtlGetFree() will be called, but if not it won't. It does
not exist in the standard docker build environment, so the tests
currently require this to not be called. This makes it instead
optional, allowing the tests to run on e.g. Fedora.
Docker-DCO-1.1-Signed-off-by: Alexander Larsson <alexl@redhat.com> (github: alexlarsson)
Upstream-commit: eb4578daee98561b16d11d2978b5f5e297d903e8
Component: engine
We're currently leaving around lots of empty directories in
/var/lib/docker/devicemapper/mnt/ for removed images and containers.
Fix this by removing the directory when the device is removed.
Docker-DCO-1.1-Signed-off-by: Alexander Larsson <alexl@redhat.com> (github: alexlarsson)
Upstream-commit: 2343fe44533f19ebae5e6127f4a2a19d1d8773fa
Component: engine
This makes the device mapper not zero out blocks allocated on the
thinp device. This is safe in our use case, as we access the device
via a filesystem that doesn't leak any uninitialized data to userspace.
This partially helps with https://github.com/dotcloud/docker/issues/3280
and should generally improve preformance on the devicemapper backend.
Docker-DCO-1.1-Signed-off-by: Alexander Larsson <alexl@redhat.com> (github: alexlarsson)
Upstream-commit: 695719b29a6311a56faae0c6ed3c985b7a75add0
Component: engine
Properly error out if passed an id that doesn't exist.
Docker-DCO-1.1-Signed-off-by: Alexander Larsson <alexl@redhat.com> (github: alexlarsson)
Upstream-commit: 304e33a2fe2d006cc1063c8060c6e22c9ade3d59
Component: engine
For some reason we seem to get transient EBUSY when removing
thinp devices, which prohibit removing containers. When
this happens we retry a few times which seems to fix the
issue for me.
Docker-DCO-1.1-Signed-off-by: Alexander Larsson <alexl@redhat.com> (github: alexlarsson)
Upstream-commit: 2c82fd93d8a01cc1f53fe861378e6d2dca0486c6
Component: engine
This makes sure the device is removed just like in deactivateDevice.
Docker-DCO-1.1-Signed-off-by: Alexander Larsson <alexl@redhat.com> (github: alexlarsson)
Upstream-commit: 7e25cd5891f84c2fab656c364002290e4b87f934
Component: engine
We already have some kind of refcounting in DeviceSet, this fleshes
it out to allow it to completely subsume the refcounting in
devmapper.Driver.
This allows us to drop the double refcounting, and the locking inside
devmapper.Driver. This, in particular the locking simplification will
make it easier in the future to parallelize the device mapper.
Docker-DCO-1.1-Signed-off-by: Alexander Larsson <alexl@redhat.com> (github: alexlarsson)
Upstream-commit: b95c560fdda4813319a2377e240592a3261e30ef
Component: engine
This adds a function that calls the lowlevel removeDevice
and then waits for it to finish.
Docker-DCO-1.1-Signed-off-by: Alexander Larsson <alexl@redhat.com> (github: alexlarsson)
Upstream-commit: dca21dfac77de2bbd69dd5a52b8a2b7816a54367
Component: engine
"Remove" is a bit overloaded, as it is also what deactivating a
device mapper device is called. Using "delete" is more clear here.
Docker-DCO-1.1-Signed-off-by: Alexander Larsson <alexl@redhat.com> (github: alexlarsson)
Upstream-commit: 63d4b293e7c91b8e2ed8da56f338f1bd0a4e1464
Component: engine
Several of the error messages were wrong, but not one of them was helpful, so this fixes them to include the important information (the actual error), and a correct description of which operation failed.
Docker-DCO-1.1-Signed-off-by: Andrew Page <admwiggin@gmail.com> (github: tianon)
Upstream-commit: f7f8e3c2b3818d351781f96ef68432eba0fc2fda
Component: engine
This fixes compilation issues when btrfs.h isn't available (because we just need the relevant structs, which for userspace programs are supposed to come from btrfs-progs instead of the kernel headers).
Docker-DCO-1.1-Signed-off-by: Andrew Page <admwiggin@gmail.com> (github: tianon)
Upstream-commit: 6922f1be08111d889b0585b763b08f92d7a55e05
Component: engine
Document why we don't need to do anything in Put().
Docker-DCO-1.1-Signed-off-by: Alexander Larsson <alexl@redhat.com> (github: alexlarsson)
Upstream-commit: a1851a6d3e69d67bfc4e6bfdec85ba58293351b9
Component: engine
Rather than creating a new directory and moving it there before
deleting that new directory, just move the directory we intend to
delete.
In the old way, the Mkdirall could fail, which meant that you
couldn't delete containers when the disk was full.
Tested.
Docker-DCO-1.1-Signed-off-by: Peter Waller <p@pwaller.net> (github: pwaller)
Upstream-commit: 6f3d8d39085e7dc31f757c0221eca7ab5f220224
Component: engine
This is an experimental btrfs driver. To use it you must have
/var/lib/docker mounted on a btrfs filesystem and explicitly
specify DOCKER_DRIVER=btrfs in the docker daemon environment.
It works by using subvolumes for the docker image/container layers.
Docker-DCO-1.1-Signed-off-by: Alexander Larsson <alexl@redhat.com> (github: alexlarsson)
Upstream-commit: e51af36a85126aca6bf6da5291eaf960fd82aa56
Component: engine
This implements the new Put() operation such that
Get()/Put() maintains a refcount for each ID, mounting
only on first Get() and unmounting on the last Get().
This means we avoid littering the system with lots of mounts
and free resources related to them.
Docker-DCO-1.1-Signed-off-by: Alexander Larsson <alexl@redhat.com> (github: alexlarsson)
Upstream-commit: 5fe26ee426b9d748605c538829f30885462ad932
Component: engine
This implements the new Put() operation such that
Get()/Put() maintains a refcount for each ID, mounting
only on first Get() and unmounting on the last Get().
This means we avoid littering the system with lots of mounts
and active devicemapper devices and free resources related
to them.
Docker-DCO-1.1-Signed-off-by: Alexander Larsson <alexl@redhat.com> (github: alexlarsson)
asdfljk
Upstream-commit: 886f650d9b2133fb61e7cc5b0bb7795246852026
Component: engine
This makes all users of Put() have a corresponding call
to Get() which means we will be able to track whether
any particular ID is in use and if not unmount it.
Docker-DCO-1.1-Signed-off-by: Alexander Larsson <alexl@redhat.com> (github: alexlarsson)
Upstream-commit: bcaf6c2359d83acd5da54f499e21f4a148f491c5
Component: engine
This simplifies that code that calls out to tar by removing support
for now unused features.
Docker-DCO-1.1-Signed-off-by: Alexander Larsson <alexl@redhat.com> (github: alexlarsson)
Upstream-commit: 4fb1db7f742fb34a7a06621d0698063de87a572c
Component: engine
Mount is self contained and generic, it should be in pkg, to allow other pkg modules to use it.
Docker-DCO-1.1-Signed-off-by: Paul Nasrat <pnasrat@gmail.com> (github: pnasrat)
Upstream-commit: 2e094db6390dde57263c07665290b1477bb6df7e
Component: engine
This works around the fact that deleting a device in a thin pool
doesn't discard the free space. Unfortunately even this is not perfect,
as it seems discards are respected only for blocks that has never been
shared in the thin device code. However, this has been fixed in the
upstream kernel device-mapper tree:
http://git.kernel.org/cgit/linux/kernel/git/device-mapper/linux-dm.git/commit/?h=for-next&id=0ab1c92ff748b745c1ed7cde31bb37ad2c5f901a
When this hits the kernel I belive this will fully return space
for removed images/containers to the host FS. For now it only
helps partially (which is better than nothing).
Docker-DCO-1.1-Signed-off-by: Alexander Larsson <alexl@redhat.com> (github: alexlarsson)
Upstream-commit: 93e120e7d67313086d8bdecbcb57ea68958f91e4
Component: engine
Add some missing defines which are needed for compiling on older systems
like RHEL 6.
Upstream-commit: a015f38f4a886518828fe3807ee7dc9ff8ab9585
Component: engine
