The Golang built-in gzip library is serialized, and fairly slow
at decompressing. It also only decompresses on demand, versus
pipelining decompression.
This change switches to using the pigz external command
for gzip decompression, as opposed to using the built-in
golang one. This code is not vendored, but will be used
if it autodetected as part of the OS.
This also switches to using context, versus a manually
managed channel to manage cancellations, and synchronization.
There is a little bit of weirdness around manually having
to cancel in the error cases.
Signed-off-by: Sargun Dhillon <sargun@sargun.me>
Upstream-commit: fd35494a251a497c359f706f61f33e689e2af678
Component: engine
For obvious reasons that it is not really supported now.
Signed-off-by: Michael Crosby <crosbymichael@gmail.com>
Upstream-commit: 5a9b5f10cf967f31f0856871ad08f9a0286b4a46
Component: engine
Also enable GC in linting to reduce memory usage.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
Upstream-commit: 09652bf8789142a5a5a1de2d41590300761b4954
Component: engine
With docker-17.06.0 some images pulled do not extract properly. Some files don't appear in correct directories. This may or may not cause the pull to fail. These images can't be pushed or saved. 17.06 is the first version of Docker built with go1.8.
Cause
There are multiple updates to the tar package in go1.8.
https://go-review.googlesource.com/c/32234/ disables using "prefix" field when new tar archives are being written. Prefix field was previously set when a record in the archive used a path longer than 100 bytes.
Another change https://go-review.googlesource.com/c/31444/ makes the reader ignore the "prefix" field value if the record is in GNU format. GNU format defines that same area should be used for access and modified times. If the "prefix" field is not read, a file will only be extracted by the basename.
The problem is that with a previous version of the golang archive package headers could be written, that use the prefix field while at the same time setting the header format to GNU. This happens when numeric fields are big enough that they can not be written as octal strings and need to be written in binary. Usually, this shouldn't happen: uid, gid, devmajor, devminor can use up to 7 bytes, size and timestamp can use 11. If one of the records does overflow it switches the whole writer to GNU mode and all next files will be saved in GNU format.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Upstream-commit: 4a3cfda45e37b81211fbfbf0c45dbe64860a3ad0
Component: engine
I noticed that we're using a homegrown package for assertions. The
functions are extremely similar to testify, but with enough slight
differences to be confusing (for example, Equal takes its arguments in a
different order). We already vendor testify, and it's used in a few
places by tests.
I also found some problems with pkg/testutil/assert. For example, the
NotNil function seems to be broken. It checks the argument against
"nil", which only works for an interface. If you pass in a nil map or
slice, the equality check will fail.
In the interest of avoiding NIH, I'm proposing replacing
pkg/testutil/assert with testify. The test code looks almost the same,
but we avoid the confusion of having two similar but slightly different
assertion packages, and having to maintain our own package instead of
using a commonly-used one.
In the process, I found a few places where the tests should halt if an
assertion fails, so I've made those cases (that I noticed) use "require"
instead of "assert", and I've vendored the "require" package from
testify alongside the already-present "assert" package.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Upstream-commit: 6052f2b3969feadb01662d8e2f30337d9c7f61af
Component: engine
Improve test coverage of ReplaceFileTarWrapper()
Signed-off-by: Daniel Nephin <dnephin@docker.com>
Upstream-commit: 8cd6c30a489f7a0210526b0f94469c525ba8e0ee
Component: engine
Heavily based on implementation by David Sheets
Signed-off-by: David Sheets <sheets@alum.mit.edu>
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Upstream-commit: 3f6dc81e10b8b813fffaa9b4167a60c5a507fa38
Component: engine
fix typo I found AMAP in integration-cli/*
fix typo mentioned by Allencloud
Signed-off-by: Aaron.L.Xu <likexu@harmonycloud.cn>
Upstream-commit: 40af5691648c5b9d07b1231e3ed3be29fd66521a
Component: engine
If we are running in a user namespace, don't try to mknod as
it won't be allowed. libcontainer will bind-mount the host's
devices over files in the container anyway, so it's not needed.
The chrootarchive package does a chroot (without mounting /proc) before
its work, so we cannot check /proc/self/uid_map when we need to. So
compute it in advance and pass it along with the tar options.
Signed-off-by: Serge Hallyn <serge.hallyn@ubuntu.com>
Upstream-commit: 617c352e9225b1d598e893aa5f89a8863808e4f2
Component: engine
Moved a defer up to a better spot.
Fixed TestUntarPathWithInvalidDest to actually fail for the right reason
Closes#18170
Signed-off-by: Doug Davis <dug@us.ibm.com>
Upstream-commit: 7bb9fc415ac50abf5f754005264bfe152eeca7d6
Component: engine
The race is between pools.Put which calls buf.Reset and exec.Cmd
doing io.Copy from the buffer; it caused a runtime crash, as
described in #16924:
``` docker-daemon cat the-tarball.xz | xz -d -c -q | docker-untar /path/to/... (aufs ) ```
When docker-untar side fails (like try to set xattr on aufs, or a broken
tar), invokeUnpack will be responsible to exhaust all input, otherwise
`xz` will be write pending for ever.
this change add a receive only channel to cmdStream, and will close it
to notify it's now safe to close the input stream;
in CmdStream the change to use Stdin / Stdout / Stderr keeps the
code simple, os/exec.Cmd will spawn goroutines and call io.Copy automatically.
the CmdStream is actually called in the same file only, change it
lowercase to mark as private.
[...]
INFO[0000] Docker daemon commit=0a8c2e3 execdriver=native-0.2 graphdriver=aufs version=1.8.2
DEBU[0006] Calling POST /build
INFO[0006] POST /v1.20/build?cgroupparent=&cpuperiod=0&cpuquota=0&cpusetcpus=&cpusetmems=&cpushares=0&dockerfile=Dockerfile&memory=0&memswap=0&rm=1&t=gentoo-x32&ulimits=null
DEBU[0008] [BUILDER] Cache miss
DEBU[0009] Couldn't untar /home/lib-docker-v1.8.2-tmp/tmp/docker-build316710953/stage3-x32-20151004.tar.xz to /home/lib-docker-v1.8.2-tmp/aufs/mnt/d909abb87150463939c13e8a349b889a72d9b14f0cfcab42a8711979be285537: Untar re-exec error: exit status 1: output: operation not supported
DEBU[0009] CopyFileWithTar(/home/lib-docker-v1.8.2-tmp/tmp/docker-build316710953/stage3-x32-20151004.tar.xz, /home/lib-docker-v1.8.2-tmp/aufs/mnt/d909abb87150463939c13e8a349b889a72d9b14f0cfcab42a8711979be285537/)
panic: runtime error: slice bounds out of range
goroutine 42 [running]:
bufio.(*Reader).fill(0xc208187800)
/usr/local/go/src/bufio/bufio.go:86 +0x2db
bufio.(*Reader).WriteTo(0xc208187800, 0x7ff39602d150, 0xc2083f11a0, 0x508000, 0x0, 0x0)
/usr/local/go/src/bufio/bufio.go:449 +0x27e
io.Copy(0x7ff39602d150, 0xc2083f11a0, 0x7ff3960261f8, 0xc208187800, 0x0, 0x0, 0x0)
/usr/local/go/src/io/io.go:354 +0xb2
github.com/docker/docker/pkg/archive.func·006()
/go/src/github.com/docker/docker/pkg/archive/archive.go:817 +0x71
created by github.com/docker/docker/pkg/archive.CmdStream
/go/src/github.com/docker/docker/pkg/archive/archive.go:819 +0x1ec
goroutine 1 [chan receive]:
main.(*DaemonCli).CmdDaemon(0xc20809da30, 0xc20800a020, 0xd, 0xd, 0x0, 0x0)
/go/src/github.com/docker/docker/docker/daemon.go:289 +0x1781
reflect.callMethod(0xc208140090, 0xc20828fce0)
/usr/local/go/src/reflect/value.go:605 +0x179
reflect.methodValueCall(0xc20800a020, 0xd, 0xd, 0x1, 0xc208140090, 0x0, 0x0, 0xc208140090, 0x0, 0x45343f, ...)
/usr/local/go/src/reflect/asm_amd64.s:29 +0x36
github.com/docker/docker/cli.(*Cli).Run(0xc208129fb0, 0xc20800a010, 0xe, 0xe, 0x0, 0x0)
/go/src/github.com/docker/docker/cli/cli.go:89 +0x38e
main.main()
/go/src/github.com/docker/docker/docker/docker.go:69 +0x428
goroutine 5 [syscall]:
os/signal.loop()
/usr/local/go/src/os/signal/signal_unix.go:21 +0x1f
created by os/signal.init·1
/usr/local/go/src/os/signal/signal_unix.go:27 +0x35
Signed-off-by: Derek Ch <denc716@gmail.com>
Upstream-commit: b48f4bf5239280b99c781a33000b4d4687967c34
Component: engine
[pkg/archive] Update archive/copy path handling
- Remove unused TarOptions.Name field.
- Add new TarOptions.RebaseNames field.
- Update some of the logic around path dir/base splitting.
- Update some of the logic behind archive entry name rebasing.
[api/types] Add LinkTarget field to PathStat
[daemon] Fix stat, archive, extract of symlinks
These operations *should* resolve symlinks that are in the path but if the
resource itself is a symlink then it *should not* be resolved. This patch
puts this logic into a common function `resolvePath` which resolves symlinks
of the path's dir in scope of the container rootfs but does not resolve the
final element of the path. Now archive, extract, and stat operations will
return symlinks if the path is indeed a symlink.
[api/client] Update cp path hanling
[docs/reference/api] Update description of stat
Add the linkTarget field to the header of the archive endpoint.
Remove path field.
[integration-cli] Fix/Add cp symlink test cases
Copying a symlink should do just that: copy the symlink NOT
copy the target of the symlink. Also, the resulting file from
the copy should have the name of the symlink NOT the name of
the target file.
Copying to a symlink should copy to the symlink target and not
modify the symlink itself.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
Upstream-commit: 75f6929b449a59335572436862d644afacf55cdb
Component: engine
Adds TarResource and CopyTo functions to be used for creating
archives for use with the new `docker cp` behavior.
Adds multiple test cases for the CopyFrom and CopyTo
functions in the pkg/archive package.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
Upstream-commit: a74799b701f5a70e964ec528c6d731c8c93019e6
Component: engine
Add tests on:
- changes.go
- archive.go
- wrap.go
Should fix#11603 as the coverage is now 81.2% on the ``pkg/archive``
package. There is still room for improvement though :).
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
Upstream-commit: c21d408ad24cf8e2b5bd761d562fae7e3ae1bc54
Component: engine
- Trying to add or complete unit test to each ``func``
- Removing dead code (``escapeName``)
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
Upstream-commit: c4fe5dad1deb45ecde460d7627523dbf032dc205
Component: engine
If .dockerignore mentions either then the client will send them to the
daemon but the daemon will erase them after the Dockerfile has been parsed
to simulate them never being sent in the first place.
an events test kept failing for me so I tried to fix that too
Closes#8330
Signed-off-by: Doug Davis <dug@us.ibm.com>
Upstream-commit: 6d801a3caa54ad7ef574bc426aa1ffc412c5af82
Component: engine
With 32ba6ab from #9261, TempArchive now closes the underlying file and
cleans it up as soon as the file's contents have been read. When pushing
an image, PushImageLayerRegistry attempts to call Close() on the layer,
which is a TempArchive that has already been closed. In this situation,
Close() returns an "invalid argument" error.
Add a Close method to TempArchive that does a no-op if the underlying
file has already been closed.
Signed-off-by: Andy Goldstein <agoldste@redhat.com>
Upstream-commit: 48ec176cd51da20e23564941da2d9906a7779d28
Component: engine
* integration test for preserving hardlinks
Signed-off-by: Vincent Batts <vbatts@redhat.com>
Signed-off-by: Vincent Batts <vbatts@hashbangbash.com>
Upstream-commit: f9f80443638fc2d703ee6205c8ef3db8e38db9a3
Component: engine
Adding moar information, so benchmark comparisons can be moar
comparative.
Signed-off-by: Vincent Batts <vbatts@redhat.com>
Upstream-commit: 0fdf7839a21390f5813589195639caf594a3ddc2
Component: engine
Now that the archive package does not depend on any docker-specific
packages, only those in pkg and vendor, it can be safely moved into pkg.
Signed-off-by: Rafe Colton <rafael.colton@gmail.com>
Upstream-commit: 30d5a42c1f24e26f681b7330249f04fec891aee9
Component: engine
