Commit Graph

3452 Commits

Author SHA1 Message Date
4c426bcdc1 add a -mount-method flag
Upstream-commit: c1e25d7273f7f520a0dbac675db4e7e26c8a4a9b
Component: engine
2013-09-26 15:10:01 +00:00
54672e1985 RootIsShared: Fix root detection
Column 4 is the mount position, column 3 will not always be
"/" for the root. On one of my system its "/root".
Upstream-commit: d263aa6ca916ba9141f341447a2387e7a6316717
Component: engine
2013-09-26 15:09:33 +00:00
71899158fd Tests: Clean up any old devmapper leftovers before starting tests
Upstream-commit: 03320f0d1c5a687ec46cbf6d836cc91576b3b225
Component: engine
2013-09-26 15:09:33 +00:00
ceff235a4b Image.Changes: Deactivate image device after unmounting it
There is no need to keep the image device around if we were the
onces creating the device.
Upstream-commit: 6c7ae06435d6e288024691f1133d7a2a24fd8ef3
Component: engine
2013-09-26 15:09:33 +00:00
30395e320c Add DeviceSet.HasActivatedDevice()
This lets you see if a device has been activated
Upstream-commit: 395bce4c4174014cb3264c35a7c6f97a2cb0948f
Component: engine
2013-09-26 15:09:33 +00:00
9f649d336c image: Unmount before removing device in error paths
The device remove fails unless we unmount first
Upstream-commit: 41399ac005caa4de2dbb744e48f058f6a15e2d2b
Component: engine
2013-09-26 15:09:33 +00:00
f35010d2a6 runtime test: Ensure all containers are unmounted at nuke()
Otherwise we may leave around e.g. devmapper mounts
Upstream-commit: 67788723c99cda8b41e5a488b988e2a72732d684
Component: engine
2013-09-26 15:09:33 +00:00
d224af31e1 ShellQuoteArguments: Fix quoting
This accidentally used two quotes to start/end each quoted string.
Upstream-commit: f99f39abaa5a43e9abc19add5b9d1253a1d22485
Component: engine
2013-09-26 15:09:33 +00:00
f2940537fb Image: unmount device before removing it on failures
If we don't do this the remove will fail due to EBUSY
Upstream-commit: 009d0f9d81bbd5e130520986ce84e8f097d88a52
Component: engine
2013-09-26 15:09:32 +00:00
fbdbec9ab5 image: Handle systems that don't support O_PATH when updating timestamp
Older kernel can't handle O_PATH in open() so this will
fail on dirs and symlinks. For dirs wa can fallback to
the normal Utimes, but for symlinks there is not much to do
but ignore their timestamps.
Upstream-commit: ed658156133862b3f181c9d3061be24b91435095
Component: engine
2013-09-26 15:09:32 +00:00
63649a9435 devmapper: Fix loopback mount code
Typo in the loop-control code made it always fall back to the
old method of opening loopback devices.
Upstream-commit: cc28829429f5f11da287ecb75ee5b3e5f05d31ad
Component: engine
2013-09-26 15:09:32 +00:00
a3c699f6e9 applyLayer() use btrfs reflinks if availible
We use the new file copy helper which uses btrfs reflinks if availible.
Upstream-commit: 062a2b32e9500107841a52d3a63b9232a1cfde70
Component: engine
2013-09-26 15:09:32 +00:00
65124d19b3 Add CopyFile that can use btrfs reflinks if availible
Upstream-commit: cda87540135a85bd8d45d4eb4853d6962114ec55
Component: engine
2013-09-26 15:09:32 +00:00
468efce93c Remove accidental commit that enabled MountMethodFilesystem
Upstream-commit: 5415804c9d7af6c75a147252ac17f71b9c3f0069
Component: engine
2013-09-26 15:09:32 +00:00
9e1662b046 Add trivial copy-based CoW backend
This creates a container by copying the corresponding files
from the layers into the containers. This is not gonna be very useful
on a developer setup, as there is no copy-on-write or general diskspace
sharing. It also makes container instantiation slower.

However, it may be useful in deployment where we don't always have a lot
of containers running (long-running daemons) and where we don't
do a lot of docker commits.
Upstream-commit: adae6849871fad0d74945fa1731712ea784e9a88
Component: engine
2013-09-26 15:09:32 +00:00
ede74ca1b4 Add Changes.ChangesLayers()
This calculates the difference between a set of layers and a
directory tree.
Upstream-commit: ad0a6a03e3595aa04cf731cf17e90be87163389a
Component: engine
2013-09-26 15:09:32 +00:00
834ac3e088 Changes: Better metadata comparison
Change the comparison to better handle files that are copied during
container creation but not actually changed:

* Inode - this will change during a copy
* ctime - this will change during a copy (as we can't set it back)
* blocksize - this will change for sparse files during copy
* size for directories - this can change anytime but doesn't
  necessarily reflect an actual contents change
* Compare mtimes at microsecond precision (as this is what utimes has)
Upstream-commit: 36603e68e33dd5ab5c317c181e023f7ef7356434
Component: engine
2013-09-26 15:09:32 +00:00
5f068e31ed Image.applyLayer: Be better at creating identical files
There are some changes here that make the file metadata better match
the layer files:

* Set the mode of the file after the chown, as otherwise the per-group/uid
  specific flags and e.g. sticky bit is lost
* Use lchown instead of chown
* Delay mtime updates to after all other changes so that later file
  creation doesn't change the mtime for the parent directory
* Use Futimes in combination with O_PATH|O_NOFOLLOW to set mtime on symlinks
Upstream-commit: 99c7d129f422b488f478bc7887f37003dacc83e6
Component: engine
2013-09-26 15:09:32 +00:00
8f15bebd63 Change how ChangesDirs() works
Rather than scan the files in the old directory twice to detect the
deletions we now scan both directories twice and then do all the
diffing on the in-memory structure.

This is more efficient, but it also lets us diff more complex things
later that are not exact on-disk trees.
Upstream-commit: 02b5f1369ce09d597336e77df98e56d467b8d1ff
Component: engine
2013-09-26 15:09:32 +00:00
07b0e1a559 RootIsShared() - Fix array out of bounds error
This happened for me on the last (empty) line, but better safe than sorry
so we make the check general.
Upstream-commit: d478a4bb5401d7d657a2a100f98ee892a96fef2a
Component: engine
2013-09-26 15:09:32 +00:00
ee414671bb devmapper: Move init layer to top rather than bottom
The init layer needs to be topmost to make sure certain files
are always there (for instance, the ubuntu:12.10 image wrongly
has /dev/shm being a symlink to /run/shm, and we need to override
that). However, previously the devmapper code implemented the
init layer by putting it in the base devmapper device, which meant
layers above it could override these files (so that ubuntu:12.10
broke).

So, instead we put the base layer in *each* images devmapper device.
This is "safe" because we still have the pristine layer data
in the layer directory. Also, it means we diff the container
against the image with the init layer applied, so it won't show
up in diffs/commits.
Upstream-commit: c199ed228baf0e5d33b7739cc2442a32dece7020
Component: engine
2013-09-26 15:09:32 +00:00
40ab3d4138 lxc: Work around lxc-start need for private mounts
lxc-start requires / to be mounted private, otherwise the changes
it does inside the container (both mounts and unmounts) will propagate
out to the host.

We work around this by starting up lxc-start in its own namespace where
we set / to rprivate.

Unfortunately go can't really execute any code between clone and exec,
so we can't do this in a nice way. Instead we have a horrible hack that
use the unshare command, the shell and the mount command...
Upstream-commit: e40f5c7cb90fbc719241ace45b05c2c61aced658
Component: engine
2013-09-26 15:09:32 +00:00
9536e01f9b Utils: Add ShellQuoteArguments
Upstream-commit: d80be57c1538905a46f75c95d592acec49a498d6
Component: engine
2013-09-26 15:09:32 +00:00
0254852cc3 Container: Inject into the mount, not the rwPath
For device-mapper setups we can't just push the file into the rwPath.
Upstream-commit: 20bac716b525e1cbd2a778122a4ce41c0c2768c2
Component: engine
2013-09-26 15:09:32 +00:00
5e59d43e57 api_test: Fix PostContainersCreate
We can't look for the created file in the rwpath, because that
doesn't exist in the device-mapper world, instead look in the
RootfsPath.
Upstream-commit: 2566e2604c2e079f9597749a1da11f22bb39eb51
Component: engine
2013-09-26 15:09:32 +00:00
122f65c42a Runtime: Only remove device on destroy if it exists
Upstream-commit: e1c418cac3cf06d453c7ac10dd54b2b0f617f1f6
Component: engine
2013-09-26 15:09:32 +00:00
8b38bd9ddf graph test: Unmount image via image.Unmount()
This helps us track the unmount
Upstream-commit: 3343b3f8f8520c2a1a841675609b4b2c54f3533d
Component: engine
2013-09-26 15:09:32 +00:00
51abe008c8 deviceset: Cleanup device sets on test end
We unmount all mounts and deactivate all device mapper devices to
make sure we're left with no leftovers after the test.
Upstream-commit: c6e8813c979bbea8832f47dc6468e805a1a18c3e
Component: engine
2013-09-26 15:09:32 +00:00
f82807f434 DeviceSet: Add UnmountDevice()
Right now this does nothing but add a new layer, but it means
that all DeviceMounts are paired with DeviceUnmounts so that we
can track (and cleanup) active mounts.
Upstream-commit: 251a7ed437c17ecb66d33782f0b42633033198dd
Component: engine
2013-09-26 15:09:31 +00:00
a4196242f7 Always start tests from a clean set of loopback images
This way we don't get any issues with leftovers
Upstream-commit: 261b0b01dffde42595d987b609d38c3b6d8368f7
Component: engine
2013-09-26 15:09:31 +00:00
1c3fa571b9 tests: Store the loopback images for test outside unit-tests
This directory is copied to each test prefix which is really
slow with the large loopback mounts.
Upstream-commit: a7fd1fce5d6fb29a8c627022da7cbbf0f4b740c7
Component: engine
2013-09-26 15:09:31 +00:00
a7892eb98a Allow specifying the docker client path in _DOCKER_INIT_PATH
I currently need this to get the tests running, otherwise it will
mount the docker.test binary inside the containers, which doesn't
work due to the libdevmapper.so dependency.
Upstream-commit: 6938a36c6985336205f1db247baec5e56fdac466
Component: engine
2013-09-26 15:09:31 +00:00
5e59cd4e23 Limit the amount of prints during normal runs
This removes some Debugf() calls and chages some direct prints to
Debugf(). This means we don't get a bunch of spew when running the
tests.
Upstream-commit: bc7fa7b95773d638754eb72e7921ac328acb2ad6
Component: engine
2013-09-26 15:09:31 +00:00
1e89c8ec2f Reuse a single DeviceSetDM for all the tests
We wrap the "real" DeviceSet for each test so that we get only
a single device-mapper pool and loopback mounts, but still
separate out the IDs in the tests. This makes the test run
much faster.
Upstream-commit: d47c18c5fbe50a2ad6ec011704f86a3c27360ff9
Component: engine
2013-09-26 15:09:31 +00:00
dc25730336 Add DeviceSetWrapper
This wraps an existing DeviceSet and just adds a prefix to all ids in
it. This will be useful for reusing a single DeviceSet for all the tests
(but with separate ids)
Upstream-commit: 0e686fa2f4d38eb6253e92ad701dd4c9caebfdce
Component: engine
2013-09-26 15:09:31 +00:00
2a4e60c781 Delete corresponding Devices when deleting Images
If an image is deleted and there is a corresponding device
for that image we also delete the image.
Upstream-commit: 3f3f5f0bbaafc67d3f77b920194f8d7bfb7bf6ee
Component: engine
2013-09-26 15:09:31 +00:00
3ffe64603d Runtime: Delete corresponding devices when deleting container
Upstream-commit: 19ba0b851bb00248b62a40695a60fc534d0df2cb
Component: engine
2013-09-26 15:08:55 +00:00
5c4c83afbc Implement container.ExportRW() on device-mapper
Upstream-commit: 94fa3c7bb5cadc31b64630b0fe8abfaeba0644aa
Component: engine
2013-09-26 15:08:55 +00:00
ce8763b123 Make TarFilter more useful
There are a few changes:
* Callers can specify if they want recursive behaviour or not
* All file listings to tar are sent on stdin, to handle long lists better
* We can pass in a list of filenames which will be created as empty
  files in the tarball

This is exactly what we want for the creation of layer tarballs given
a container fs, a set of files to add and a set of whiteout files to create.
Upstream-commit: 223280f31967edb7b3eb325cce8a3b82939c0f2b
Component: engine
2013-09-26 15:08:55 +00:00
1fb4f4dbcf Archive: Fix up tar commandline arguments in TarFilter()
There is no need to duplicate the compression flags for
every element in the filter.
Upstream-commit: 8f23945f7f666e3151890de90ebb39db3c7f5ada
Component: engine
2013-09-26 15:08:55 +00:00
5f5630fd65 Implement docker diff for device-mapper
To do diffing we just compare file metadata, so this relies
on things like size and mtime/ctime to catch any changes.
Its *possible* to trick this by updating a file without
changing the size and setting back the mtime/ctime, but
that seems pretty unlikely to happen in reality, and lets
us avoid comparing the actual file data.
Upstream-commit: 8e8ef7cb5b208ac657af812ebc5ffa783664cf3b
Component: engine
2013-09-26 15:08:55 +00:00
eda48ebeaf devmapper: Base the device-mapper names on the root dir name
This means the default is "docker-*", but for tests we get separate
prefixes for each test.
Upstream-commit: 8f343ea65a05b374ffa64940a1946abf28402689
Component: engine
2013-09-26 15:08:55 +00:00
abe30d5012 Image: Always create a .docker-id file in the devices we create
Without this there is really no way to map back from the device-mapper
devices to the actual docker image/container ids in case the json file
somehow got lost
Upstream-commit: b125f2334c7b68f624ee1eee06cb9d68922d0314
Component: engine
2013-09-26 15:08:55 +00:00
3fad97e978 Image: Deactivate image device when unmounting container
There is no need to keep all the device-mapper devices active, we
can just activate them on demand if needed.
Upstream-commit: a89a51128ecdf2db02a333a406752416de8a1db6
Component: engine
2013-09-26 15:08:54 +00:00
f67c857087 Image: Initial support for device-mapper mounts
This supports creating images from layers and mounting them
for running a container.

Not supported yet are:
* Creating diffs between images/containers
* Creating layers for new images from a device-mapper container
Upstream-commit: fcd41fe51ae0418d583f3d33dfac7fc0879ca30e
Component: engine
2013-09-26 15:08:54 +00:00
d814a010c6 Runtime: Add MountMethod to allow AUFS and device-mapper to coexist
Upstream-commit: 53851474c0b8127442ce11ab38fa0ae8d5c694f0
Component: engine
2013-09-26 15:08:54 +00:00
19e1e13936 Runtime: Add DeviceSet singleton
This adds a DeviceSet singleton to the Runtime object which will be used for
any DeviceMapper dependent code.
Upstream-commit: f317a6b6fe31685445ac97a1475136c5ab7860b5
Component: engine
2013-09-26 15:08:54 +00:00
676dd4d044 Server: Pass in device-mapper DeviceSet to server
This makes docker (but not docker-init) link to libdevmapper and will
allow it to use the DeviceSet
Upstream-commit: 87e248f52458eca2a40ed6e54e36de89228f6790
Component: engine
2013-09-26 15:08:54 +00:00
056e5df9a6 Add DeviceSet interface
This interface matches the device-mapper implementation (DeviceSetDM)
but is free from any dependencies. This allows core docker code
to refer to a DeviceSet without having an explicit dependency on
the devmapper package.

This is important, because the devmapper package has external
dependencies which are not wanted in the docker client app, as it
needs to run with minimal dependencies in the docker image.
Upstream-commit: ac194fc696fb95045ee5b634d04a9f7093f45685
Component: engine
2013-09-26 15:08:54 +00:00
dc920bde90 Image: Add runtime and container id args to Mount()
We will later need the runtime to get access to the VolumeSet
singleton, and the container id to have a name for the volume
for the container
Upstream-commit: 8637ba710e8bd10ee36aa5c71f5ef54ab9037dfa
Component: engine
2013-09-26 15:08:54 +00:00