remove unnescessary import aliases, brackets, and so on.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Upstream-commit: f23c00d8701e4bd0f2372a586dacbf66a26f9a51
Component: engine
dm_task_deferred_remove is not supported by all distributions, due to
out-dated versions of devicemapper. However, in the case where the
devicemapper library was updated without rebuilding Docker (which can
happen in some distributions) then we should attempt to dynamically load
the relevant object rather than try to link to it.
This can only be done if Docker was built dynamically, for obvious
reasons.
In order to avoid having issues arise when dlsym(3) was unnecessary,
gate the whole dlsym(3) logic behind a buildflag that we disable by
default (libdm_dlsym_deferred_remove).
Signed-off-by: Aleksa Sarai <asarai@suse.de>
Upstream-commit: 98fe4bd8f1e35f8e498e268f653a43cbfa31e751
Component: engine
if thin device is deteled and the metadata exists, you can not
delete related containers. This patch ignore Nodata errors for
thin device deletion
Signed-off-by: Liu Hua <sdu.liu@huawei.com>
Upstream-commit: 8451d03d8ef7457f82112179cd3e300c05a08d3d
Component: engine
Static build with devmapper is impossible now since libudev is required
and no static version of libudev is available (as static libraries are
not supported by systemd which udev is part of).
This should not hurt anyone as "[t]he primary user of static builds
is the Editions, and docker in docker via the containers, and none
of those use device mapper".
Also, since the need for static libdevmapper is gone, there is no need
to self-compile libdevmapper -- let's use the one from Debian Stretch.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Upstream-commit: 84f1c054e66d3999aaf2751062cda8a77925b7ae
Component: engine
Let's use latest lvm2 sources to compile the libdevmapper library.
Initial reason for compiling devmapper lib from sources was a need to
have the static version of the library at hand, in order to build
the static dockerd, but note that the same headers/solib are used
for dynamic build (dynbinary) as well.
The reason for this patch is to enable the deferral removal feature.
The supplied devmapper library (and headers) are too old, lacking the
needed functions, so the daemon is built with 'libdm_no_deferred_remove'
build tag (see the check in hack/make.sh). Because of this, even if the
kernel dm driver is perfectly able to support the feature, it can not
be used. For more details and background story, see [1].
Surely, one can't just change the version number. While at it:
- improve the comments;
- remove obsoleted URLs;
- remove s390 and ppc configure updates that are no longer needed;
- use pkg-config instead of hardcoding the flags (newer lib added
some more dependencies);
[1] https://github.com/moby/moby/issues/34298
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Upstream-commit: c4fde49a5c0198890078e4c5f7566b387884fe0f
Component: engine
1. devmapper_wrapper_{,no_}deferred_remove.go:
Comments about LibraryDeferredRemovalSupport were very totally
misleading to me. This thing has nothing to do with either static
or dynamic linking (but with build tags). Fix the comment accordingly.
2. devmapper.go:
Reveal the source of those magic device* constants.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Upstream-commit: adce3ca48e306a6117ce5677b8d9437551e98a26
Component: engine
I am getting the following warning from gcc when compiling the daemon:
> # github.com/docker/docker/pkg/devicemapper
> pkg/devicemapper/devmapper_wrapper.go: In function ‘log_cb’:
> pkg/devicemapper/devmapper_wrapper.go:20:2: warning: ignoring return
> value of ‘vasprintf’, declared with attribute warn_unused_result
> [-Wunused-result]
> vasprintf(&buffer, f, ap);
> ^
vasprintf(3) man page says if the function returns -1, the buffer is
undefined, so we should not use it. In practice, I assume, this never
happens so we just return.
Introduced by https://github.com/moby/moby/pull/33845 that resulted in
commit 63328c6 ("devicemapper: remove 256 character limit of libdm logs")
Cc: Aleksa Sarai <asarai@suse.de>
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Upstream-commit: 7da12bcfa9db4d84b2c547bee93dafeaead15b16
Component: engine
devicemapper: Wait for device removal if deferredRemoval=true and deferredDeletion=…
Upstream-commit: e04dbe5ac287c2a856b7c96972d931ee5f0e288f
Component: engine
Changes most references of syscall to golang.org/x/sys/
Ones aren't changes include, Errno, Signal and SysProcAttr
as they haven't been implemented in /x/sys/.
Signed-off-by: Christopher Jones <tophj@linux.vnet.ibm.com>
[s390x] switch utsname from unsigned to signed
per 33267e036f
char in s390x in the /x/sys/unix package is now signed, so
change the buildtags
Signed-off-by: Christopher Jones <tophj@linux.vnet.ibm.com>
Upstream-commit: 069fdc8a083cb1663e4f86fe3fd9b9a1aebc3e54
Component: engine
LogInit used to act as a manual way of registering the *necessary*
pkg/devicemapper logging callbacks. In addition, it was used to split up
the logic of pkg/devicemapper into daemon/graphdriver/devmapper (such
that some things were logged from libdm).
The manual aspect of this API was completely non-sensical and was just
begging for incorrect usage of pkg/devicemapper, so remove that semantic
and always register our own libdm callbacks.
In addition, recombine the split out logging callbacks into
pkg/devicemapper so that the default logger is local to the library and
also shown to be the recommended logger. This makes the code
substantially easier to read. Also the new DefaultLogger now has
configurable upper-bound for the log level, which allows for dynamically
changing the logging level.
Signed-off-by: Aleksa Sarai <asarai@suse.de>
Upstream-commit: cfd39e8d6d79eee8a71fbe6820dd67babf05d951
Component: engine
e07d3cd9a ("devmapper: Fix libdm logging") removed all of the callers of
DmLogInitVerbose, but we still kept around the wrapper. However, the
libdm dm_log_init_verbose API changes the verbosity of the *default*
libdm logger. Because pkg/devicemapper internally *relies* on using
logging callbacks to understand what errors were encountered by libdm,
this wrapper is useless (it only makes sense for the default logger
which we do not user).
Any user not inside Docker of this function almost certainly was not
using this API correctly, because pkg/devicemapper will misbehave if our
logging callbacks were not registered.
Signed-off-by: Aleksa Sarai <asarai@suse.de>
Upstream-commit: c654147f0635736fff0035ecdfa53a41c7fad0d0
Component: engine
This limit is unecessary and can lead to the truncation of long libdm
logs (which is quite annoying).
Fixes: b440ec013 ("device-mapper: Move all devicemapper spew to log through utils.Debugf().")
Signed-off-by: Aleksa Sarai <asarai@suse.de>
Upstream-commit: 63328c6882c3d1f54c66499ef9963adfbf1883f0
Component: engine
There have been some cases where umount, a device can be busy for a very
short duration. Maybe its udev rules, or maybe it is runc related races
or probably it is something else. We don't know yet.
If deferred removal is enabled but deferred deletion is not, then for the
case of "docker run -ti --rm fedora bash", a container will exit, device
will be deferred removed and then immediately a call will come to delete
the device. It is possible that deletion will fail if device was busy
at that time.
A device can't be deleted if it can't be removed/deactivated first. There
is only one exception and that is when deferred deletion is on. In that
case graph driver will keep track of deleted device and try to delete it
later and return success to caller.
Always make sure that device deactivation is synchronous when device is
being deleted (except the case when deferred deletion is enabled).
This should also take care of small races when device is busy for a short
duration and it is being deleted.
Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
Upstream-commit: 36cb6efebc599900b691e206fb9e99d3aa2fb9a3
Component: engine
Recent changes to devmapper broke the implicit requirement that UdevWait be
called after every call to task.setCookie. Failure to do so results in leaks of
semaphores in the LVM code, eventually leading to semaphore exhaustion.
Previously this was handled by calling UdevWait in a ubiquitous defer function.
While there was initially some concern with deferring the UdevWait function
would cause some amount of race possibiliy, the fact that we never return the
cookie value or any value used to find it, makes that possibility seem unlikely,
so lets go back to that method
Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
Upstream-commit: 23dcfec1404411489e23fb9b76fc7096db139d04
Component: engine
DeviceMapper tasks in go use SetFinalizer to clean up C construct
counterparts in the C LVM library. While thats well and good, it relies
heavily on the exact interpretation of when the golang garbage collector
determines that an object is unreachable is subject to reclaimation.
While common sense would assert that for stack variables (which these DM
tasks always are), are unreachable when the stack frame in which they
are declared returns, thats not the case. According to this:
https://golang.org/pkg/runtime/#SetFinalizer
The garbage collector decides that, if a function calls into a
systemcall (which task.run() always will in LVM), and there are no
subsequent references to the task variable within that stack frame, then
it can be reclaimed. Those conditions are met in several devmapper.go
routines, and if the garbage collector runs in the middle of a
deviceMapper operation, then the task can be destroyed while the
operation is in progress, leading to crashes, failed operations and
other unpredictable behavior.
The fix is to use the KeepAlive interface:
https://golang.org/pkg/runtime/#KeepAlive
The KeepAlive method is effectively an empy reference that fools the
garbage collector into thinking that a variable is still reachable. By
adding a call to KeepAlive in the task.run() method, we can ensure that
the garbage collector won't reclaim a task object until its execution
within the deviceMapper C library is complete.
Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
Upstream-commit: d764d8b16624e4924b3949273089f851efa0f717
Component: engine
All LVM actions in the devicemapper library are asyncronous, involving a call to
a task enqueue function (dm_run_task) and a wait on a resultant udev event
(UdevWait). Currently devmapper.go defers all calls to UdevWait, which discards
the return value. While it still generates an error message in the log (if
debugging is enabled), the calling thread is still allowed to continue as if no
error has occured, leading to subsequent errors, and significant confusion when
debugging, due to those subsequent errors. Given that there is no risk of panic
between the task submission and the wait operation, it seems more reasonable to
preform the UdevWait inline at the end of any given lvm action so that errors
can be caught and returned before docker can continue and create additional
failures.
Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
Upstream-commit: 5206d45e70512f5fc06006047fb67b2f478b304d
Component: engine
Currently, the devicemapper library sets cookies to correlate wait operations,
which must be unique (as the lvm2 library doesn't detect duplicate cookies).
The current method for cookie generation is to take the address of a cookie
variable. However, because the variable is declared on the stack, execution
patterns can lead to the cookie variable being declared at the same stack
location, which results in a high likelyhood of duplicate cookie use, which in
turn can lead to various odd lvm behaviors, which can be hard to track down
(object use before create, duplicate completions, etc). Lets guarantee that the
cookie we generate is unique by declaring it on the heap instead. This
guarantees that the address of the variable won't be reused until such time as
the UdevWait operation completes, and drops its reference to it, at which time
the gc can reclaim it.
Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
Upstream-commit: edd1c9e3255ddd80c97313bd596812dbb3470a5b
Component: engine
If a wait event fails when preforming a devicemapper operation, it would be good
to know, in addition to the cookie that its waiting on, we reported the error
that was reported from the lvm2 library.
Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
Upstream-commit: 547510fb554bb77aa7f09efd46361306152b9d55
Component: engine
if there is no cookie set in dm task, or flag DM_UDEV_DISABLE_LIBRARY_FALLBACK
is cleared for a DM_DEV_REMOVE task, libdevmapper will fallback to clean up the
symlink under /dev/mapper by itself, no matter the device removal is executed
immediately or deferred by the kernel.In some cases, the removal is deferred by the
kernel, while the symlink is deleted directly by libdevmapper, when docker tries to
activate the device again, the deferred removal will be canceld, but the symlink will
not show up again, so docker's attempt to mount the device by the symlink will fail,
and it will eventually leads to a `docker start/diff` error.
Fixes#24671
Signed-off-by: Ji.Zhilong <zhilongji@gmail.com>
Upstream-commit: 5e505d101f0201a4d045510d0a9b0c66697dedfe
Component: engine
Problem Description:
An example scenario that involves deferred removal
1. A new base image gets created (e.g. 'docker load -i'). The base device is activated and
mounted at some point in time during image creation.
2. While image creation is in progress, a privileged container is started
from another image and the host's mount name space is shared with this
container ('docker run --privileged -v /:/host').
3. Image creation completes and the base device gets unmounted. However,
as the privileged container still holds a reference on the base image
mount point, the base device cannot be removed right away. So it gets
flagged for deferred removal.
4. Next, the privileged container terminates and thus its reference to the
base image mount point gets released. The base device (which is flagged
for deferred removal) may now be cleaned up by the device-mapper. This
opens up an opportunity for a race between a 'kworker' thread (executing
the do_deferred_remove() function) and the Docker daemon (executing the
CreateSnapDevice() function).
This PR cancel the deferred removal, if the device is marked for it. And reschedule the
deferred removal later after the device is resumed successfully.
Signed-off-by: Shishir Mahajan <shishir.mahajan@redhat.com>
Upstream-commit: 0e633ee14aca32480ac4735675222c35f4e11d8c
Component: engine
This fix tries to fix logrus formatting by removing `f` from
`logrus.[Error|Warn|Debug|Fatal|Panic|Info]f` when formatting string
is not present.
This fix fixes#23459.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
Upstream-commit: a72b45dbec3caeb3237d1af5aedd04adeb083571
Component: engine
The loopback logic is not technically exclusive to the devicemapper
driver. This reorganizes the code such that the loopback code is usable
outside of the devicemapper package and driver.
Signed-off-by: Vincent Batts <vbatts@redhat.com>
Upstream-commit: af597527121c0ad973e9d0ae4f9cf970da1513cc
Component: engine
Closes#16667
Uses the prefix "devicemapper:" for all the fmt and logrus error, debug, and info messages.
Signed-off-by: Chris Dituri <csdituri@gmail.com>
Upstream-commit: 9b584781ad9c810bc9060448f391a3ae6a7d7cbb
Component: engine
Finally here is the patch to implement deferred deletion functionality.
Deferred deleted devices are marked as "Deleted" in device meta file.
First we try to delete the device and only if deletion fails and user has
enabled deferred deletion, device is marked for deferred deletion.
When docker starts up again, we go through list of deleted devices and
try to delete these again.
Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
Upstream-commit: d929589c1fc4538dcd1b2a7a3dc7d4afbdfa72fd
Component: engine
Got merged with it, removing it as it doesn't add anything.
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
Upstream-commit: 1dcb7d9e40d0ee5f9ab068fed55d3072246ce8ec
Component: engine
6990b76a696dd265674f4c2973f25755a6485f05 introduced a typo in function
declaration, this patch fixes that.
Signed-off-by: Antonio Murdaca <runcom@linux.com>
Upstream-commit: d4a123b20b4f8fb1858f3cf411b3695353c0a973
Component: engine
… and fixes the last bits that were missing :3.
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
Upstream-commit: 0c70eb83813af6d5266769bc908593f9660b13f1
Component: engine
Fixes: #15279
Due to
7904946eeb
the devices field is dropped.
This solution works on go1.4 and go1.5
Signed-off-by: Vincent Batts <vbatts@redhat.com>
Upstream-commit: f83d05c3be3c3bcc84f6fa229504848ee8078321
Component: engine
Often it happens that docker is not able to shutdown/remove the thin
pool it created because some device has leaked into some mount name
space. That means device is in use and that means pool can't be removed.
Docker will leave pool as it is and exit. Later when user starts the
docker, it finds pool is already there and docker uses it. But docker
does not know it is same pool which is using the loop devices. Now
docker thinks loop devices are not being used. That means it does not
display the data correctly in "docker info", giving user wrong information.
This patch tries to detect if loop devices as created by docker are
being used for pool and fills in the right details in "docker info".
Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
Upstream-commit: bebf53443981c70a6a714ea518dc966a0e2b6558
Component: engine
Deferred reove functionality was added to library later. So in old version
of library it did not report deferred_remove field.
Create a new function which also gets deferred_remove field and it will be
called only on newer version of library.
Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
Upstream-commit: 4986ce7cfbe74610d4fa2c4e79ceefe49c1aa155
Component: engine
If a device has been scheduled for deferred deactivation and container
is started again and we need to activate device again, we need to cancel
the deferred deactivation which is already scheduled on the device.
Create a method for the same.
Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
Upstream-commit: 20b38f427aa05186bd09c8c4201dcc95ed56aa46
Component: engine
A lot of time device mapper devices leak across mount namespace which docker
does not know about and when docker tries to deactivate/delete device,
operation fails as device is open in some mount namespace.
Create a mechanism where one can defer the device deactivation/deletion
so that docker operation does not fail and device automatically goes
away when last reference to it is dropped.
Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
Upstream-commit: 6964ab94befd8723585556e560219e0eef48a488
Component: engine
