[pkg/archive] Update archive/copy path handling
- Remove unused TarOptions.Name field.
- Add new TarOptions.RebaseNames field.
- Update some of the logic around path dir/base splitting.
- Update some of the logic behind archive entry name rebasing.
[api/types] Add LinkTarget field to PathStat
[daemon] Fix stat, archive, extract of symlinks
These operations *should* resolve symlinks that are in the path but if the
resource itself is a symlink then it *should not* be resolved. This patch
puts this logic into a common function `resolvePath` which resolves symlinks
of the path's dir in scope of the container rootfs but does not resolve the
final element of the path. Now archive, extract, and stat operations will
return symlinks if the path is indeed a symlink.
[api/client] Update cp path hanling
[docs/reference/api] Update description of stat
Add the linkTarget field to the header of the archive endpoint.
Remove path field.
[integration-cli] Fix/Add cp symlink test cases
Copying a symlink should do just that: copy the symlink NOT
copy the target of the symlink. Also, the resulting file from
the copy should have the name of the symlink NOT the name of
the target file.
Copying to a symlink should copy to the symlink target and not
modify the symlink itself.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
Upstream-commit: 75f6929b449a59335572436862d644afacf55cdb
Component: engine
TL;DR: check for IsExist(err) after a failed MkdirAll() is both
redundant and wrong -- so two reasons to remove it.
Quoting MkdirAll documentation:
> MkdirAll creates a directory named path, along with any necessary
> parents, and returns nil, or else returns an error. If path
> is already a directory, MkdirAll does nothing and returns nil.
This means two things:
1. If a directory to be created already exists, no error is returned.
2. If the error returned is IsExist (EEXIST), it means there exists
a non-directory with the same name as MkdirAll need to use for
directory. Example: we want to MkdirAll("a/b"), but file "a"
(or "a/b") already exists, so MkdirAll fails.
The above is a theory, based on quoted documentation and my UNIX
knowledge.
3. In practice, though, current MkdirAll implementation [1] returns
ENOTDIR in most of cases described in #2, with the exception when
there is a race between MkdirAll and someone else creating the
last component of MkdirAll argument as a file. In this very case
MkdirAll() will indeed return EEXIST.
Because of #1, IsExist check after MkdirAll is not needed.
Because of #2 and #3, ignoring IsExist error is just plain wrong,
as directory we require is not created. It's cleaner to report
the error now.
Note this error is all over the tree, I guess due to copy-paste,
or trying to follow the same usage pattern as for Mkdir(),
or some not quite correct examples on the Internet.
[v2: a separate aufs commit is merged into this one]
[1] https://github.com/golang/go/blob/f9ed2f75/src/os/path.go
Signed-off-by: Kir Kolyshkin <kir@openvz.org>
Upstream-commit: a83a76934787a20e96389d33bd56a09369f9b808
Component: engine
In `ApplyLayer` and `Untar`, the stream is magically decompressed. Since
this is not able to be toggled, rather than break this ./pkg/ API, add
an `ApplyUncompressedLayer` and `UntarUncompressed` that does not
magically decompress the layer stream.
Signed-off-by: Vincent Batts <vbatts@redhat.com>
Upstream-commit: 56bf275e32fc27bd31c5dba6b715336b43981695
Component: engine
Adds TarResource and CopyTo functions to be used for creating
archives for use with the new `docker cp` behavior.
Adds multiple test cases for the CopyFrom and CopyTo
functions in the pkg/archive package.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
Upstream-commit: a74799b701f5a70e964ec528c6d731c8c93019e6
Component: engine
This makes the "Buffering to disk" part of `docker push` 70% faster in
my use-case (having already applied #12833).
fsync'ing here serves no valuable purpose: if the drive's operation is
interrupted, so it the program's, and this archive has no value other
than the immediate and transient one.
Signed-off-by: Burke Libbey <burke.libbey@shopify.com>
Upstream-commit: 236dbc2e59f5b665f9fa30f3f7ba1fe6c8483b24
Component: engine
- Trying to add or complete unit test to each ``func``
- Removing dead code (``escapeName``)
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
Upstream-commit: c4fe5dad1deb45ecde460d7627523dbf032dc205
Component: engine
This change modifies the chmod bits of build context archives built on
windows to preserve the execute bit and remove the r/w bits from
grp/others.
Also adjusted integ-cli tests to verify permissions based on the platform
the tests are running.
Fixes#11047.
Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
Upstream-commit: 1a22418f9f1573ca8521831d52c8e0562cb3ef8f
Component: engine
Currently TestBuildRenamedDockerfile fails since passing
custom dockerfile paths like:
docker build -f dir/file .
fails on windows because those are unix paths. Instead, on
windows accept windows style paths like:
docker build -f dir\file .
and convert them to unix style paths using the helper we
have in `pkg/archive` so that daemon can correctly locate
the path in the context.
Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
Upstream-commit: 5ecab9e831c6d39e649430013087a3e2a85fb949
Component: engine
Currently pkg/archive stores nested windows files with
backslashes (e.g. `dir\`, `dir\file.txt`) and this causes
tar not being correctly extracted on Linux daemon.
This change assures we canonicalize all paths to unix
paths and add them to tar with that name independent of platform.
Fixes the following test cases for Windows CI:
- TestBuildAddFileWithWhitespace
- TestBuildCopyFileWithWhitespace
- TestBuildAddDirContentToRoot
- TestBuildAddDirContentToExistingDir
- TestBuildCopyDirContentToRoot
- TestBuildCopyDirContentToExistDir
- TestBuildDockerignore
- TestBuildEnvUsage
- TestBuildEnvUsage2
Signed-off-by: Ahmet Alp Balkan <ahmetalpbalkan@gmail.com>
Upstream-commit: 28842d3f093e69cf62a090a532aacba1e6ff6d1c
Component: engine
sort changes found and exported.
Sorting the files before appending them to the tar archive
would mean a dependable ordering for types like hardlinks.
Also, combine sort logic used
Signed-off-by: Vincent Batts <vbatts@redhat.com>
Upstream-commit: 32d6d49539f5876e6ef75f00db917d8a3aaa0569
Component: engine
If .dockerignore mentions either then the client will send them to the
daemon but the daemon will erase them after the Dockerfile has been parsed
to simulate them never being sent in the first place.
an events test kept failing for me so I tried to fix that too
Closes#8330
Signed-off-by: Doug Davis <dug@us.ibm.com>
Upstream-commit: 6d801a3caa54ad7ef574bc426aa1ffc412c5af82
Component: engine
With 32ba6ab from #9261, TempArchive now closes the underlying file and
cleans it up as soon as the file's contents have been read. When pushing
an image, PushImageLayerRegistry attempts to call Close() on the layer,
which is a TempArchive that has already been closed. In this situation,
Close() returns an "invalid argument" error.
Add a Close method to TempArchive that does a no-op if the underlying
file has already been closed.
Signed-off-by: Andy Goldstein <agoldste@redhat.com>
Upstream-commit: 48ec176cd51da20e23564941da2d9906a7779d28
Component: engine
This fixes the removal of TempArchives which can read with only one
read. Such archives weren't getting removed because EOF wasn't being
triggered.
Docker-DCO-1.1-Signed-off-by: Cristian Staretu <cristian.staretu@gmail.com> (github: unclejack)
Upstream-commit: 32ba6ab83c7e47d627a2b971e7f6ca9b56e1be85
Component: engine
pkg/archive contains code both invoked from cli (cross platform) and
daemon (linux only) and Unix-specific dependencies break compilation on
Windows. We extracted those stat-related funcs into platform specific
implementations at pkg/system and added unit tests.
Signed-off-by: Ahmet Alp Balkan <ahmetb@microsoft.com>
Upstream-commit: 2180aa4f6f2ad4d8f284d63ee29e93547263976e
Component: engine
Some parts of pkg/archive is called on both client/daemon code. To get
it compiling on Windows, these funcs are extracted into files with
build tags.
Signed-off-by: Ahmet Alp Balkan <ahmetb@microsoft.com>
Upstream-commit: 3d2fae353f6ddc819d3a3c4db80887a40ac6f5f0
Component: engine
* integration test for preserving hardlinks
Signed-off-by: Vincent Batts <vbatts@redhat.com>
Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>
Upstream-commit: f9f80443638fc2d703ee6205c8ef3db8e38db9a3
Component: engine
In an effort to make layer content 'stable' between import
and export from two different graph drivers, we must resolve
an issue where AUFS produces metadata files in its layers
which other drivers explicitly ignore when importing.
The issue presents itself like this:
- Generate a layer using AUFS
- On commit of that container, the new stored layer contains
AUFS metadata files/dirs. The stored layer content has some
tarsum value: '1234567'
- `docker save` that image to a USB drive and `docker load`
into another docker engine instance which uses another
graph driver, say 'btrfs'
- On load, this graph driver explicitly ignores any AUFS metadata
that it encounters. The stored layer content now has some
different tarsum value: 'abcdefg'.
The only (apparent) useful aufs metadata to keep are the psuedo link
files located at `/.wh..wh.plink/`. Thes files hold information at the
RW layer about hard linked files between this layer and another layer.
The other graph drivers make sure to copy up these psuedo linked files
but I've tested out a few different situations and it seems that this
is unnecessary (In my test, AUFS already copies up the other hard linked
files to the RW layer).
This changeset adds explicit exclusion of the AUFS metadata files and
directories (NOTE: not the whiteout files!) on commit of a container
using the AUFS storage driver.
Also included is a change to the archive package. It now explicitly
ignores the root directory from being included in the resulting tar archive
for 2 reasons: 1) it's unnecessary. 2) It's another difference between
what other graph drivers produce when exporting a layer to a tar archive.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
Upstream-commit: 0d97e082c3b849a6901992e653b8963e8ef10f54
Component: engine
Fixes#1992
Right now when you `docker cp` a path which is in a volume, the cp
itself works, however you end up getting files that are in the
container's fs rather than the files in the volume (which is not in the
container's fs).
This makes it so when you `docker cp` a path that is in a volume it
follows the volume to the real path on the host.
archive.go has been modified so that when you do `docker cp mydata:/foo
.`, and /foo is the volume, the outputed folder is called "foo" instead
of the volume ID (because we are telling it to tar up
`/var/lib/docker/vfs/dir/<some id>` and not "foo", but the user would be
expecting "foo", not the ID
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Upstream-commit: ef98fe0763024abd90bd5a573fec816895ee92e4
Component: engine
Now that the archive package does not depend on any docker-specific
packages, only those in pkg and vendor, it can be safely moved into pkg.
Signed-off-by: Rafe Colton <rafael.colton@gmail.com>
Upstream-commit: 30d5a42c1f24e26f681b7330249f04fec891aee9
Component: engine
Lei Jitang
Tibor Vass
Josh Hawn
Kir Kolyshkin
Vincent Batts
Alexander Morozov
John Howard
Jessie Frazelle
Burke Libbey
Tianon Gravi
buddhamagnet
Vincent Demeester
Vincent Batts
Antonio Murdaca
Ahmet Alp Balkan
Iavael
Michael Crosby
Jason Sommer
unclejack
Doug Davis
Michael Crosby
Andy Goldstein
Tibor Vass
Ahmet Alp Balkan
Steven Burgess
Brian Goff
Rafe Colton