Recent changes to devmapper broke the implicit requirement that UdevWait be
called after every call to task.setCookie. Failure to do so results in leaks of
semaphores in the LVM code, eventually leading to semaphore exhaustion.
Previously this was handled by calling UdevWait in a ubiquitous defer function.
While there was initially some concern with deferring the UdevWait function
would cause some amount of race possibiliy, the fact that we never return the
cookie value or any value used to find it, makes that possibility seem unlikely,
so lets go back to that method
Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
Upstream-commit: 23dcfec1404411489e23fb9b76fc7096db139d04
Component: engine
DeviceMapper tasks in go use SetFinalizer to clean up C construct
counterparts in the C LVM library. While thats well and good, it relies
heavily on the exact interpretation of when the golang garbage collector
determines that an object is unreachable is subject to reclaimation.
While common sense would assert that for stack variables (which these DM
tasks always are), are unreachable when the stack frame in which they
are declared returns, thats not the case. According to this:
https://golang.org/pkg/runtime/#SetFinalizer
The garbage collector decides that, if a function calls into a
systemcall (which task.run() always will in LVM), and there are no
subsequent references to the task variable within that stack frame, then
it can be reclaimed. Those conditions are met in several devmapper.go
routines, and if the garbage collector runs in the middle of a
deviceMapper operation, then the task can be destroyed while the
operation is in progress, leading to crashes, failed operations and
other unpredictable behavior.
The fix is to use the KeepAlive interface:
https://golang.org/pkg/runtime/#KeepAlive
The KeepAlive method is effectively an empy reference that fools the
garbage collector into thinking that a variable is still reachable. By
adding a call to KeepAlive in the task.run() method, we can ensure that
the garbage collector won't reclaim a task object until its execution
within the deviceMapper C library is complete.
Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
Upstream-commit: d764d8b16624e4924b3949273089f851efa0f717
Component: engine
All LVM actions in the devicemapper library are asyncronous, involving a call to
a task enqueue function (dm_run_task) and a wait on a resultant udev event
(UdevWait). Currently devmapper.go defers all calls to UdevWait, which discards
the return value. While it still generates an error message in the log (if
debugging is enabled), the calling thread is still allowed to continue as if no
error has occured, leading to subsequent errors, and significant confusion when
debugging, due to those subsequent errors. Given that there is no risk of panic
between the task submission and the wait operation, it seems more reasonable to
preform the UdevWait inline at the end of any given lvm action so that errors
can be caught and returned before docker can continue and create additional
failures.
Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
Upstream-commit: 5206d45e70512f5fc06006047fb67b2f478b304d
Component: engine
Currently, the devicemapper library sets cookies to correlate wait operations,
which must be unique (as the lvm2 library doesn't detect duplicate cookies).
The current method for cookie generation is to take the address of a cookie
variable. However, because the variable is declared on the stack, execution
patterns can lead to the cookie variable being declared at the same stack
location, which results in a high likelyhood of duplicate cookie use, which in
turn can lead to various odd lvm behaviors, which can be hard to track down
(object use before create, duplicate completions, etc). Lets guarantee that the
cookie we generate is unique by declaring it on the heap instead. This
guarantees that the address of the variable won't be reused until such time as
the UdevWait operation completes, and drops its reference to it, at which time
the gc can reclaim it.
Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
Upstream-commit: edd1c9e3255ddd80c97313bd596812dbb3470a5b
Component: engine
If a wait event fails when preforming a devicemapper operation, it would be good
to know, in addition to the cookie that its waiting on, we reported the error
that was reported from the lvm2 library.
Signed-off-by: Neil Horman <nhorman@tuxdriver.com>
Upstream-commit: 547510fb554bb77aa7f09efd46361306152b9d55
Component: engine
if there is no cookie set in dm task, or flag DM_UDEV_DISABLE_LIBRARY_FALLBACK
is cleared for a DM_DEV_REMOVE task, libdevmapper will fallback to clean up the
symlink under /dev/mapper by itself, no matter the device removal is executed
immediately or deferred by the kernel.In some cases, the removal is deferred by the
kernel, while the symlink is deleted directly by libdevmapper, when docker tries to
activate the device again, the deferred removal will be canceld, but the symlink will
not show up again, so docker's attempt to mount the device by the symlink will fail,
and it will eventually leads to a `docker start/diff` error.
Fixes#24671
Signed-off-by: Ji.Zhilong <zhilongji@gmail.com>
Upstream-commit: 5e505d101f0201a4d045510d0a9b0c66697dedfe
Component: engine
Problem Description:
An example scenario that involves deferred removal
1. A new base image gets created (e.g. 'docker load -i'). The base device is activated and
mounted at some point in time during image creation.
2. While image creation is in progress, a privileged container is started
from another image and the host's mount name space is shared with this
container ('docker run --privileged -v /:/host').
3. Image creation completes and the base device gets unmounted. However,
as the privileged container still holds a reference on the base image
mount point, the base device cannot be removed right away. So it gets
flagged for deferred removal.
4. Next, the privileged container terminates and thus its reference to the
base image mount point gets released. The base device (which is flagged
for deferred removal) may now be cleaned up by the device-mapper. This
opens up an opportunity for a race between a 'kworker' thread (executing
the do_deferred_remove() function) and the Docker daemon (executing the
CreateSnapDevice() function).
This PR cancel the deferred removal, if the device is marked for it. And reschedule the
deferred removal later after the device is resumed successfully.
Signed-off-by: Shishir Mahajan <shishir.mahajan@redhat.com>
Upstream-commit: 0e633ee14aca32480ac4735675222c35f4e11d8c
Component: engine
This fix tries to fix logrus formatting by removing `f` from
`logrus.[Error|Warn|Debug|Fatal|Panic|Info]f` when formatting string
is not present.
This fix fixes#23459.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
Upstream-commit: a72b45dbec3caeb3237d1af5aedd04adeb083571
Component: engine
The loopback logic is not technically exclusive to the devicemapper
driver. This reorganizes the code such that the loopback code is usable
outside of the devicemapper package and driver.
Signed-off-by: Vincent Batts <vbatts@redhat.com>
Upstream-commit: af597527121c0ad973e9d0ae4f9cf970da1513cc
Component: engine
Closes#16667
Uses the prefix "devicemapper:" for all the fmt and logrus error, debug, and info messages.
Signed-off-by: Chris Dituri <csdituri@gmail.com>
Upstream-commit: 9b584781ad9c810bc9060448f391a3ae6a7d7cbb
Component: engine
Finally here is the patch to implement deferred deletion functionality.
Deferred deleted devices are marked as "Deleted" in device meta file.
First we try to delete the device and only if deletion fails and user has
enabled deferred deletion, device is marked for deferred deletion.
When docker starts up again, we go through list of deleted devices and
try to delete these again.
Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
Upstream-commit: d929589c1fc4538dcd1b2a7a3dc7d4afbdfa72fd
Component: engine
Got merged with it, removing it as it doesn't add anything.
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
Upstream-commit: 1dcb7d9e40d0ee5f9ab068fed55d3072246ce8ec
Component: engine
6990b76a696dd265674f4c2973f25755a6485f05 introduced a typo in function
declaration, this patch fixes that.
Signed-off-by: Antonio Murdaca <runcom@linux.com>
Upstream-commit: d4a123b20b4f8fb1858f3cf411b3695353c0a973
Component: engine
… and fixes the last bits that were missing :3.
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
Upstream-commit: 0c70eb83813af6d5266769bc908593f9660b13f1
Component: engine
Fixes: #15279
Due to
7904946eeb
the devices field is dropped.
This solution works on go1.4 and go1.5
Signed-off-by: Vincent Batts <vbatts@redhat.com>
Upstream-commit: f83d05c3be3c3bcc84f6fa229504848ee8078321
Component: engine
Often it happens that docker is not able to shutdown/remove the thin
pool it created because some device has leaked into some mount name
space. That means device is in use and that means pool can't be removed.
Docker will leave pool as it is and exit. Later when user starts the
docker, it finds pool is already there and docker uses it. But docker
does not know it is same pool which is using the loop devices. Now
docker thinks loop devices are not being used. That means it does not
display the data correctly in "docker info", giving user wrong information.
This patch tries to detect if loop devices as created by docker are
being used for pool and fills in the right details in "docker info".
Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
Upstream-commit: bebf53443981c70a6a714ea518dc966a0e2b6558
Component: engine
Deferred reove functionality was added to library later. So in old version
of library it did not report deferred_remove field.
Create a new function which also gets deferred_remove field and it will be
called only on newer version of library.
Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
Upstream-commit: 4986ce7cfbe74610d4fa2c4e79ceefe49c1aa155
Component: engine
If a device has been scheduled for deferred deactivation and container
is started again and we need to activate device again, we need to cancel
the deferred deactivation which is already scheduled on the device.
Create a method for the same.
Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
Upstream-commit: 20b38f427aa05186bd09c8c4201dcc95ed56aa46
Component: engine
A lot of time device mapper devices leak across mount namespace which docker
does not know about and when docker tries to deactivate/delete device,
operation fails as device is open in some mount namespace.
Create a mechanism where one can defer the device deactivation/deletion
so that docker operation does not fail and device automatically goes
away when last reference to it is dropped.
Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
Upstream-commit: 6964ab94befd8723585556e560219e0eef48a488
Component: engine
UdevWait() is deferred and takes uint cookie as an argument. As arguments
to deferred functions are calculated at the time of call, it is possible
that any update to cookie later by libdm are not taken into account when
UdevWait() is called. Hence use a pointer to uint as argument to UdevWait()
function.
Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
Upstream-commit: 665656afbb8932a11c69c0cd79e21a768aa46d38
Component: engine
devmapper graph driver retries device removal 1000 times in case of failure
and if this fills up console with 1000 messages (when daemon is running in
debug mode). So remove these debug messages.
Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
Upstream-commit: cb7c893275c32ddfa775c3f22869a9c211024c71
Component: engine
moar information for the information gods
Signed-off-by: Vincent Batts <vbatts@redhat.com>
Upstream-commit: 4cfe9df0a9c206c368a90f460fea8fab197265d9
Component: engine
DM_UDEV_DISABLE_LIBRARY_FALLBACK is disabled by most applications today
when using device-mapper, and ensuring that device-mapper is in sync
with udev. This flag instructs devicemapper to not fallback to creating
the device nodes itself. In the case of udev sync not being supported,
devicemapper will attempt to create the devices in a timely manner,
regardless of udev.
Signed-off-by: Vincent Batts <vbatts@redhat.com>
Upstream-commit: dbb642b7fbdf42d3b401d610403ba58ebbde38d7
Component: engine
expose an api to call dm_udev_get_sync_support/dm_udev_set_sync_support
Signed-off-by: Vincent Batts <vbatts@redhat.com>
Upstream-commit: 9c3380039e15f11fc07741edd4cbc0c7e5c490aa
Component: engine
Currently devicemapper CreateDevice and CreateSnapDevice keep on retrying
device creation till a suitable device id is found.
With new transaction mechanism we need to store device id in transaction
before it has been created.
So change the logic in such a way that caller decides the devices Id to
use. If that device Id is not available, caller bumps up the device Id
and retries.
That way caller can update transaciton too when it tries a new Id. Transaction
related patches will come later in the series.
Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
Upstream-commit: 7b2b15d3e9f9b7ad898a36bbe5ceb42c9ca58d47
Component: engine
Currently we set up a cookie and upon failure not call UdevWait(). This
does not cleanup the cookie and associated semaphore and system will
soon max out on total number of semaphores.
To avoid this, call UdevWait() even in failure path which in turn will
cleanup associated semaphore.
Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
Signed-off-by: Vincent Batts <vbatts@redhat.com>
Upstream-commit: edc6df256d21eb1d1aa36b241dcc6d4b83d58d75
Component: engine
Otherwise udev can unecessarily execute various rules (and issue
scanning IO, etc) against the thin-pool -- which can never be a
top-level device.
Docker-DCO-1.1-Signed-off-by: Mike Snitzer <snitzer@redhat.com> (github: snitm)
Upstream-commit: ad6467f9e17205fa76a3b916efe51ba5c1b37506
Component: engine
