Commit Graph

3244 Commits

Author SHA1 Message Date
1d7e574dd4 Tests: Clean up any old devmapper leftovers before starting tests
Upstream-commit: be6fef02544d9bc31321b2a21d039e261dc02bd8
Component: engine
2013-09-30 17:35:02 -06:00
1461717d2e Image.Changes: Deactivate image device after unmounting it
There is no need to keep the image device around if we were the
onces creating the device.
Upstream-commit: 071cc18b58408eaa71575e0233a056475196b199
Component: engine
2013-09-30 17:35:02 -06:00
64c3059ad8 Add DeviceSet.HasActivatedDevice()
This lets you see if a device has been activated
Upstream-commit: 6ec5585501c6029ef10a474597503511d5074e10
Component: engine
2013-09-30 17:35:02 -06:00
890bafe58d image: Unmount before removing device in error paths
The device remove fails unless we unmount first
Upstream-commit: 6f57e8025a0a03da8052edc30157f7677f65f208
Component: engine
2013-09-30 17:35:02 -06:00
5fc36fa5bf runtime test: Ensure all containers are unmounted at nuke()
Otherwise we may leave around e.g. devmapper mounts
Upstream-commit: b0a9147fd5f0197931b73e18163a12b04ad8e826
Component: engine
2013-09-30 17:35:02 -06:00
c5a71d7b11 ShellQuoteArguments: Fix quoting
This accidentally used two quotes to start/end each quoted string.
Upstream-commit: a7e876e3571965de64615d2012e63fae9c3ebbf8
Component: engine
2013-09-30 17:35:01 -06:00
17aec56f2a Image: unmount device before removing it on failures
If we don't do this the remove will fail due to EBUSY
Upstream-commit: ecdbdfdaea5497c34a0299e7efd97b8a44bd7314
Component: engine
2013-09-30 17:35:01 -06:00
6101a0a475 image: Handle systems that don't support O_PATH when updating timestamp
Older kernel can't handle O_PATH in open() so this will
fail on dirs and symlinks. For dirs wa can fallback to
the normal Utimes, but for symlinks there is not much to do
but ignore their timestamps.
Upstream-commit: 2c71710b74829787a0c78f7e02f45d31935a996f
Component: engine
2013-09-30 17:35:01 -06:00
2af8abef3d devmapper: Fix loopback mount code
Typo in the loop-control code made it always fall back to the
old method of opening loopback devices.
Upstream-commit: bbc72c85f7733fefda499cb0e96f1dda2ce90625
Component: engine
2013-09-30 17:35:01 -06:00
91d411052a applyLayer() use btrfs reflinks if availible
We use the new file copy helper which uses btrfs reflinks if availible.
Upstream-commit: 1a082ed245aaf55cc92e884c988865b195ea0cb7
Component: engine
2013-09-30 17:35:01 -06:00
08576d6eaf Add CopyFile that can use btrfs reflinks if availible
Upstream-commit: 86421e8b5e84452d66e6b16b5cb1a5a438f8fa1f
Component: engine
2013-09-30 17:35:01 -06:00
19c4006853 Remove accidental commit that enabled MountMethodFilesystem
Upstream-commit: 91c69fd3532eaa92cfc21c6331699a995fdbe2a6
Component: engine
2013-09-30 17:35:01 -06:00
9f8a6e47d8 Add trivial copy-based CoW backend
This creates a container by copying the corresponding files
from the layers into the containers. This is not gonna be very useful
on a developer setup, as there is no copy-on-write or general diskspace
sharing. It also makes container instantiation slower.

However, it may be useful in deployment where we don't always have a lot
of containers running (long-running daemons) and where we don't
do a lot of docker commits.
Upstream-commit: 43a7d3d0e9e2fafcdc90cb84855e1bb3869d2729
Component: engine
2013-09-30 17:35:01 -06:00
6dae654f9c Add Changes.ChangesLayers()
This calculates the difference between a set of layers and a
directory tree.
Upstream-commit: 60f552cac3d165dbe4c8d65c1ab82d31700dcc5d
Component: engine
2013-09-30 17:35:01 -06:00
3953036d64 Changes: Better metadata comparison
Change the comparison to better handle files that are copied during
container creation but not actually changed:

* Inode - this will change during a copy
* ctime - this will change during a copy (as we can't set it back)
* blocksize - this will change for sparse files during copy
* size for directories - this can change anytime but doesn't
  necessarily reflect an actual contents change
* Compare mtimes at microsecond precision (as this is what utimes has)
Upstream-commit: ad402763e160ded1924c9b154983d81614e90deb
Component: engine
2013-09-30 17:35:01 -06:00
84b4aeca21 Image.applyLayer: Be better at creating identical files
There are some changes here that make the file metadata better match
the layer files:

* Set the mode of the file after the chown, as otherwise the per-group/uid
  specific flags and e.g. sticky bit is lost
* Use lchown instead of chown
* Delay mtime updates to after all other changes so that later file
  creation doesn't change the mtime for the parent directory
* Use Futimes in combination with O_PATH|O_NOFOLLOW to set mtime on symlinks
Upstream-commit: 5d2ace3424516bd7cc8d4a57fcaddd00fa1c4b5d
Component: engine
2013-09-30 17:35:01 -06:00
ce9d889436 Change how ChangesDirs() works
Rather than scan the files in the old directory twice to detect the
deletions we now scan both directories twice and then do all the
diffing on the in-memory structure.

This is more efficient, but it also lets us diff more complex things
later that are not exact on-disk trees.
Upstream-commit: 727e7fcccadf1d3e286f5a3c8d1aa388f6b4dab8
Component: engine
2013-09-30 17:35:01 -06:00
ff5de9f609 RootIsShared() - Fix array out of bounds error
This happened for me on the last (empty) line, but better safe than sorry
so we make the check general.
Upstream-commit: 7d566b4f761e8942cf9679e96774b320b8496b2f
Component: engine
2013-09-30 17:35:01 -06:00
74b00e51f4 devmapper: Move init layer to top rather than bottom
The init layer needs to be topmost to make sure certain files
are always there (for instance, the ubuntu:12.10 image wrongly
has /dev/shm being a symlink to /run/shm, and we need to override
that). However, previously the devmapper code implemented the
init layer by putting it in the base devmapper device, which meant
layers above it could override these files (so that ubuntu:12.10
broke).

So, instead we put the base layer in *each* images devmapper device.
This is "safe" because we still have the pristine layer data
in the layer directory. Also, it means we diff the container
against the image with the init layer applied, so it won't show
up in diffs/commits.
Upstream-commit: fdbc2695fe00d522c5c1a962f9be2f802bf53943
Component: engine
2013-09-30 17:35:01 -06:00
1d1c891003 lxc: Work around lxc-start need for private mounts
lxc-start requires / to be mounted private, otherwise the changes
it does inside the container (both mounts and unmounts) will propagate
out to the host.

We work around this by starting up lxc-start in its own namespace where
we set / to rprivate.

Unfortunately go can't really execute any code between clone and exec,
so we can't do this in a nice way. Instead we have a horrible hack that
use the unshare command, the shell and the mount command...
Upstream-commit: 429587779a95a4d38ec9cd66202de9729c320ef8
Component: engine
2013-09-30 17:35:01 -06:00
95fd1df21e Utils: Add ShellQuoteArguments
Upstream-commit: 145024c6ccc7bff1eda632823702f91899667ed3
Component: engine
2013-09-30 17:35:01 -06:00
f3c281436d Container: Inject into the mount, not the rwPath
For device-mapper setups we can't just push the file into the rwPath.
Upstream-commit: c1388010731cac1c8ff5159bd72b926545a64e58
Component: engine
2013-09-30 17:35:01 -06:00
4cb2cd0d23 api_test: Fix PostContainersCreate
We can't look for the created file in the rwpath, because that
doesn't exist in the device-mapper world, instead look in the
RootfsPath.
Upstream-commit: 07227866006ea75a0c492814b808fdbb672431ee
Component: engine
2013-09-30 17:35:01 -06:00
0453049ffb Runtime: Only remove device on destroy if it exists
Upstream-commit: 5f8e24f8428bc5fb7076ec757ded44f965c48888
Component: engine
2013-09-30 17:35:01 -06:00
eb4c81b5cf graph test: Unmount image via image.Unmount()
This helps us track the unmount
Upstream-commit: 0d7ab8db03814bac19fe076a0d53b6d423616bc7
Component: engine
2013-09-30 17:35:01 -06:00
89fbc4edd2 deviceset: Cleanup device sets on test end
We unmount all mounts and deactivate all device mapper devices to
make sure we're left with no leftovers after the test.
Upstream-commit: ed741f7b27b1b1cf5b6f8917551ce86bc39e9c78
Component: engine
2013-09-30 17:35:01 -06:00
12fbeaaae6 DeviceSet: Add UnmountDevice()
Right now this does nothing but add a new layer, but it means
that all DeviceMounts are paired with DeviceUnmounts so that we
can track (and cleanup) active mounts.
Upstream-commit: 9e64ebb29549db19a84b8cb514bea60c26184779
Component: engine
2013-09-30 17:35:01 -06:00
16f68cf6d5 Always start tests from a clean set of loopback images
This way we don't get any issues with leftovers
Upstream-commit: d951911b23b26cda087e823f203414a0ad184276
Component: engine
2013-09-30 17:35:01 -06:00
7bed559f55 tests: Store the loopback images for test outside unit-tests
This directory is copied to each test prefix which is really
slow with the large loopback mounts.
Upstream-commit: 7fb60caa5d34f8fc0abeae2ce7841000665414b0
Component: engine
2013-09-30 17:35:01 -06:00
70d3d13622 Allow specifying the docker client path in _DOCKER_INIT_PATH
I currently need this to get the tests running, otherwise it will
mount the docker.test binary inside the containers, which doesn't
work due to the libdevmapper.so dependency.
Upstream-commit: 76a2ab6e34e6cdd66899815aeeaac2048e5bafce
Component: engine
2013-09-30 17:35:00 -06:00
7d0232a89a Limit the amount of prints during normal runs
This removes some Debugf() calls and chages some direct prints to
Debugf(). This means we don't get a bunch of spew when running the
tests.
Upstream-commit: 6094257b28f2e4b5e1a6616c77961b5cec0c9195
Component: engine
2013-09-30 17:35:00 -06:00
816bf23aaf Reuse a single DeviceSetDM for all the tests
We wrap the "real" DeviceSet for each test so that we get only
a single device-mapper pool and loopback mounts, but still
separate out the IDs in the tests. This makes the test run
much faster.
Upstream-commit: 52294192b2a008b624862701cbf8491ad19b0798
Component: engine
2013-09-30 17:35:00 -06:00
0a6a934279 Add DeviceSetWrapper
This wraps an existing DeviceSet and just adds a prefix to all ids in
it. This will be useful for reusing a single DeviceSet for all the tests
(but with separate ids)
Upstream-commit: 381ce94ef4fe2590b0afa16c5f35a508de12e7a3
Component: engine
2013-09-30 17:35:00 -06:00
348d75b3b5 Delete corresponding Devices when deleting Images
If an image is deleted and there is a corresponding device
for that image we also delete the image.
Upstream-commit: 99393cf3cfd08de769f0c37f06b912fb3771a080
Component: engine
2013-09-30 17:35:00 -06:00
fef827ba7b Runtime: Delete corresponding devices when deleting container
Upstream-commit: 30890c7763e775d38c82d3eac5c1650e149137ae
Component: engine
2013-09-30 17:35:00 -06:00
79ab6381a5 Implement container.ExportRW() on device-mapper
Upstream-commit: b0626f403b168b9020a802ec3bc4ad8c9bbc2486
Component: engine
2013-09-30 17:34:59 -06:00
06dfc21edb Make TarFilter more useful
There are a few changes:
* Callers can specify if they want recursive behaviour or not
* All file listings to tar are sent on stdin, to handle long lists better
* We can pass in a list of filenames which will be created as empty
  files in the tarball

This is exactly what we want for the creation of layer tarballs given
a container fs, a set of files to add and a set of whiteout files to create.
Upstream-commit: fda6ff9c2707efbd1c9d1f2bf151b9d1d082d0c6
Component: engine
2013-09-30 17:34:59 -06:00
53e75b5d60 Archive: Fix up tar commandline arguments in TarFilter()
There is no need to duplicate the compression flags for
every element in the filter.
Upstream-commit: b86f67126c86a07aac155f38aefd6c1ef538e24d
Component: engine
2013-09-30 17:34:59 -06:00
534a030a97 Implement docker diff for device-mapper
To do diffing we just compare file metadata, so this relies
on things like size and mtime/ctime to catch any changes.
Its *possible* to trick this by updating a file without
changing the size and setting back the mtime/ctime, but
that seems pretty unlikely to happen in reality, and lets
us avoid comparing the actual file data.
Upstream-commit: 1c5dc26a7c0a0abb7bc59174768ec309f6c5fd4f
Component: engine
2013-09-30 17:34:59 -06:00
8fb407ad51 devmapper: Base the device-mapper names on the root dir name
This means the default is "docker-*", but for tests we get separate
prefixes for each test.
Upstream-commit: 8e7cbbff504b1d5b0680d2a14821d1d7b0757ab0
Component: engine
2013-09-30 17:34:59 -06:00
11ba6503a7 Image: Always create a .docker-id file in the devices we create
Without this there is really no way to map back from the device-mapper
devices to the actual docker image/container ids in case the json file
somehow got lost
Upstream-commit: 074f38d49377411cf0b805095c0d9909d4859f3c
Component: engine
2013-09-30 17:34:59 -06:00
0134fab2c6 Image: Deactivate image device when unmounting container
There is no need to keep all the device-mapper devices active, we
can just activate them on demand if needed.
Upstream-commit: a9ec1dbc9bec91e1c0f1b751a06680570a04e915
Component: engine
2013-09-30 17:34:59 -06:00
f4fd5894a2 Image: Initial support for device-mapper mounts
This supports creating images from layers and mounting them
for running a container.

Not supported yet are:
* Creating diffs between images/containers
* Creating layers for new images from a device-mapper container
Upstream-commit: d2ba3e200576c2bcceb81d8d3d65b86fbf49313b
Component: engine
2013-09-30 17:34:59 -06:00
03b4e52ca7 Runtime: Add MountMethod to allow AUFS and device-mapper to coexist
Upstream-commit: 8f7361279c13660a617886a523c50ecbc6849129
Component: engine
2013-09-30 17:34:59 -06:00
4f27979b41 Runtime: Add DeviceSet singleton
This adds a DeviceSet singleton to the Runtime object which will be used for
any DeviceMapper dependent code.
Upstream-commit: ca2f7f955e697091f2b7bee9a33c6c4e106cecd0
Component: engine
2013-09-30 17:34:59 -06:00
6c1aee42e6 Server: Pass in device-mapper DeviceSet to server
This makes docker (but not docker-init) link to libdevmapper and will
allow it to use the DeviceSet
Upstream-commit: 1d36b8c7b7b0c943ccb7d69b7181b3e33566d77c
Component: engine
2013-09-30 17:34:59 -06:00
acf340a68b Add DeviceSet interface
This interface matches the device-mapper implementation (DeviceSetDM)
but is free from any dependencies. This allows core docker code
to refer to a DeviceSet without having an explicit dependency on
the devmapper package.

This is important, because the devmapper package has external
dependencies which are not wanted in the docker client app, as it
needs to run with minimal dependencies in the docker image.
Upstream-commit: e6216793d91a9b003814b535b0373902326753dd
Component: engine
2013-09-30 17:34:58 -06:00
3b80b99ef6 Image: Add runtime and container id args to Mount()
We will later need the runtime to get access to the VolumeSet
singleton, and the container id to have a name for the volume
for the container
Upstream-commit: e368c8bb01b3c52c8e4c334c3a7f32556af9d632
Component: engine
2013-09-30 17:34:58 -06:00
b25a04bd5f Runtime: Automatically use docker-init if it exists
In some builds the main docker binary is not statically linked,
and as such not usable in as the .dockerinit binary, for those
cases we look for a separately shipped docker-init binary and
use that instead.
Upstream-commit: 167601e85850aa58df96f9d0796f9c26ed2d6fa4
Component: engine
2013-09-30 17:34:58 -06:00
3681b7af0d Add a separate docker-init binary
This may be used for the .dockerinit case if the main binary is not
statically linked.
Upstream-commit: b8dc7b5f1a8111275e2b869dfb67a8689cecf9b9
Component: engine
2013-09-30 17:34:58 -06:00