If a build context tar has path names of the form 'x/./y', they will be
stored in this unnormalized form internally by tarsum. When the builder
walks the untarred directory tree and queries hashes for each relative
path, it will query paths of the form 'x/y', and they will not be found.
To correct this, have tarsum normalize path names by calling Clean.
Add a test to detect this caching false positive.
Fixes#21715
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Upstream-commit: 8691a77e441996fef96019b94f299a11b7244080
Component: engine
Some structures use int for sizes and UNIX timestamps. On some
platforms, int is 32 bits, so this can lead to the year 2038 issues and
overflows when dealing with large containers or layers.
Consistently use int64 to store sizes and UNIX timestamps in
api/types/types.go. Update related to code accordingly (i.e.
strconv.FormatInt instead of strconv.Itoa).
Use int64 in progressreader package to avoid integer overflow when
dealing with large quantities. Update related code accordingly.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Upstream-commit: 1f61084d83aea37b212468aaa975020094b7f7c9
Component: engine
And removing unused code.
- tarsum.go :
NewTarSumHash could be non exported (for now)
NewTarSumForLabel is never used, except for the tests
- fileinfosums.go:
SortByPos is never used, except for the tests
- versionning.go:
GetVersions is never used, expect for the tests
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
Upstream-commit: 25f7d7822d2b858864b03ea5dfc534c32c67326b
Component: engine
Updates `image.StoreImage()` to always ensure that images
that are installed in Docker have a tarsum.v1 checksum.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
Upstream-commit: ba3bad66f92da057dba2a43ff9a512cf7ad72375
Component: engine
While the v2 pull operation is writing the body of the layer blob to disk
it now computes the tarsum checksum of the archive before extracting it to
the backend storage driver. If the checksum does not match that from the
image manifest an error is raised.
Also adds more debug logging to the pull operation and fixes existing test
cases which were failing. Adds a reverse lookup constructor to the tarsum
package so that you can get a tarsum object using a checksum label.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
Upstream-commit: 213e3d116642431adbe634d39740eddc5a81e063
Component: engine
If .dockerignore mentions either then the client will send them to the
daemon but the daemon will erase them after the Dockerfile has been parsed
to simulate them never being sent in the first place.
an events test kept failing for me so I tried to fix that too
Closes#8330
Signed-off-by: Doug Davis <dug@us.ibm.com>
Upstream-commit: 6d801a3caa54ad7ef574bc426aa1ffc412c5af82
Component: engine
Another update to TarSum tests, this patch fixes an issue where
the benchmarks were generating archives incorrectly by not closing
the tarWriter.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
Upstream-commit: 385917e22c5bd0d577a5a0be0de9e88a0499fe87
Component: engine
These two cases did not actually read the same content with each iteration
of the benchmark. After the first read, the buffer was consumed. This patch
corrects this by using a bytes.Reader and seeking to the beginning of the
buffer at the beginning of each iteration.
Unfortunately, this benchmark was not actually as fast as we believed. But
the new results do bring its results closer to those of the other benchmarks.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
Upstream-commit: 92fd49f7ca42cf5d97825853034eac685b90fc1d
Component: engine
I noticed that 3 of the tarsum test cases had expected a tarsum with
a sha256 hash of
e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
As I've been working with sha256 quite a bit lately, it struck me that
this is the initial digest value for sha256, which means that no data
was processed. However, these tests *do* process data. It turns out that
there was a bug in the test handling code which did not wait for tarsum
to end completely. This patch corrects these test cases.
I'm unaware of anywhere else in the code base where this would be an issue,
though we definitily need to look out in the future to ensure we are
completing tarsum reads (waiting for EOF).
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
Upstream-commit: 8d9e25dbddc189f4094e0f25a90f2b8a25deec9d
Component: engine
The current Dev version of TarSum includes hashing of extended
file attributes and omits inclusion of modified time headers.
I refactored the logic around the version differences to make it
more clear that the difference between versions is in how tar
headers are selected and ordered.
TarSum Version 1 is now declared with the new Dev version continuing
to track it.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
Upstream-commit: a7aa2c8ad26149e9be753bc08964f35cb09d313c
Component: engine
When read is called on a tarsum with a two different read sizes, specifically the second call larger than the first, the dynamic buffer does not get reallocated causing a slice read error.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Upstream-commit: 5cdf7f507771e159ddd335f3757cd33f7ba4e426
Component: engine
Not breaking the default cipher and algorithm for calculating checksums,
but allow for using alternate block ciphers during the checksum
calculation.
Docker-DCO-1.1-Signed-off-by: Vincent Batts <vbatts@redhat.com> (github: vbatts)
Upstream-commit: 4e9925d780665149b8bc940d5ba242ada1973c4e
Component: engine
If a tar were constructed with duplicate file names, then depending on
the order, it could result in same tarsum.
Signed-off-by: Vincent Batts <vbatts@redhat.com>
Upstream-commit: c5e6362c53cbbc09ddbabd5a7323e04438b57d31
Component: engine
Tarsum now correctly closes the internal TarWriter which appends
a block of 1024 zeros to the end of the returned archive.
Signed-off-by: Josh Hawn <josh.hawn@docker.com>
Upstream-commit: bc956c6b79dbe9cc7a1ab198f92b6843d072eb28
Component: engine
Now that TarSum can be versioned, move the addition of xattr headers in
to the next version of TarSum
Signed-off-by: Vincent Batts <vbatts@redhat.com>
Upstream-commit: 0ecafb8a6909f5d342246aefb9ac97350cef6558
Component: engine
this is to enhance the tarsum algorithm, but _MUST_ be done in lock step
with the same for docker-registry. (PR will be cited)
Docker-DCO-1.1-Signed-off-by: Vincent Batts <vbatts@redhat.com> (github: vbatts)
Upstream-commit: 448c8ecb0fb453bb29fcada5e7486355ccfadbe9
Component: engine
don't export the exsisting TarSum struct and call the interface 'TarSum'
instead.
Signed-off-by: Vincent Batts <vbatts@redhat.com>
Upstream-commit: ec01eb653db9d5e5b9291c1670dde57e7f742874
Component: engine
This introduces Versions for TarSum checksums.
Fixes: https://github.com/docker/docker/issues/7526
It preserves current functionality and abstracts the interface for
future flexibility of hashing algorithms. As a POC, the VersionDev
Tarsum does not include the mtime in the checksum calculation, and would
solve https://github.com/docker/docker/issues/7387 though this is not a
settled Version is subject to change until a version number is assigned.
Signed-off-by: Vincent Batts <vbatts@redhat.com>
Upstream-commit: 747f89cd327db9d50251b17797c4d825162226d0
Component: engine
This commit makes tarsum buffer allocation dynamic. This change
is required to avoid allocating memory excessively after the archive
buffering changes.
Docker-DCO-1.1-Signed-off-by: Cristian Staretu <cristian.staretu@gmail.com> (github: unclejack)
Upstream-commit: 7ef34407509fa76e3ead12a20c8b731f434e1971
Component: engine
This code is vital to the security of the project and it is important
we assure it is well-maintained and guarded.
I am vested in assuring this code maintains security and
provides as much compatibility as possible between releases.
Signed-off-by: Eric Windisch <eric@windisch.us>
Upstream-commit: cd71f92a7fda7a8bdf328a91c37a759463862aeb
Component: engine
