PR #15493 removed compression of layers when pushing them to a V2
registry. This this makes layer uploads larger than they should be.
This commit restores the compression. It uses an io.Pipe to turn the
gzip compressor output Writer into a Reader, so the ReadFrom method can
be used on the BlobWriter (which is very important for avoiding many
PATCH requests per layer).
Fixes#17209Fixes#17038
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Upstream-commit: 4dce280d96797b6a82e3eb54243e21bce7f55e5d
Component: engine
Also remove unused func `newTempFile` and prevent a possible deadlock
between pull_v2 `attemptIDReuse` and graph `register`
Signed-off-by: Antonio Murdaca <runcom@redhat.com>
Upstream-commit: f6577be1c93150149c291f9d18375d7bcae9ebb1
Component: engine
Lookup the graph parent reference to detect a builder cache miss before
looping the whole graph image index to build a parent-children tree.
Signed-off-by: Antonio Murdaca <runcom@redhat.com>
Upstream-commit: f9e81b40f4065e3d0851172759ef58fda6572cce
Component: engine
also fix a typo in pkg/truncindex package comment
Signed-off-by: Antonio Murdaca <runcom@redhat.com>
Upstream-commit: 56f5e3459f8d7477d2aa60dee02bc7cd8a8731ad
Component: engine
After v1.8.3 layer checksum is used for image ID
validation. Rewriting the checksums on push would
mean that next pulls will get different image IDs
and pulls may fail if its detected that same
manifest digest can now point to new image ID.
Fixes#17178
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Upstream-commit: fb4a7256921b9863bd27079ec821b9612368fee8
Component: engine
Ensure v2 registries are given more than 5 seconds to return a ping and avoid an unnecessary fallback to v1.
Elevates log level about failed v2 ping to a warning to match the warning related to using v1 registries.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Upstream-commit: f8ea4ad16d51f6f062be2c250be26d731b92bd3c
Component: engine
Remove all unneeded disk operations (reload TagStore, umarshal image)
for checking if image still points to same ID. Now slowest part is
queries to sqlite which hopefuly will be removed soon.
Signed-off-by: Alexander Morozov <lk4d4@docker.com>
Upstream-commit: f256d8ad2d1fbd9407d2b7a8eb69c96665c4015e
Component: engine
Tags and digests are kept in the same storage. We want to make sure that they are completely separated - tags are something users set and digests can only be set by pull-by-digest code path.
Reverts #14664
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Upstream-commit: d08ca5c2b65d442f37424957e53d81f96356d12c
Component: engine
Add a unit test for validateManifest which ensures extra data can't be
injected by adding data to the JSON object outside the payload area.
This also removes validation of legacy signatures at pull time. This
starts the path of deprecating legacy signatures, whose presence in the
very JSON document they attempt to sign is problematic. These
signatures were only checked for official images, and since they only
caused a weakly-worded message to be printed, removing the verification
should not cause impact.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Upstream-commit: 011bfd666eeb21a111ca450c42a3893ad03c9324
Component: engine
Generate a hash chain involving the image configuration, layer digests,
and parent image hashes. Use the digests to compute IDs for each image
in a manifest, instead of using the remotely specified IDs.
To avoid breaking users' caches, check for images already in the graph
under old IDs, and avoid repulling an image if the version on disk under
the legacy ID ends up with the same digest that was computed from the
manifest for that image.
When a calculated ID already exists in the graph but can't be verified,
continue trying SHA256(digest) until a suitable ID is found.
"save" and "load" are not changed to use a similar scheme. "load" will
preserve the IDs present in the tar file.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Upstream-commit: 504e67b867865a2835e8002c01087a2cfd7bfd0e
Component: engine
Adds support for the daemon to handle user namespace maps as a
per-daemon setting.
Support for handling uid/gid mapping is added to the builder,
archive/unarchive packages and functions, all graphdrivers (except
Windows), and the test suite is updated to handle user namespace daemon
rootgraph changes.
Docker-DCO-1.1-Signed-off-by: Phil Estes <estesp@linux.vnet.ibm.com> (github: estesp)
Upstream-commit: 442b45628ee12ebd8e8bd08497896d5fa8eec4bd
Component: engine
also fix a typo in pkg/truncindex package comment
Signed-off-by: Antonio Murdaca <amurdaca@redhat.com>
Upstream-commit: 292a1564dca2f32b9158a6886fadee1cc184f987
Component: engine
progressreader.Broadcaster becomes broadcaster.Buffered and
broadcastwriter.Writer becomes broadcaster.Unbuffered.
The package broadcastwriter is thus renamed to broadcaster.
Signed-off-by: Tibor Vass <tibor@docker.com>
Upstream-commit: 2391233404e2e6892c79a24f31cc99715c086b21
Component: engine
Although having a request ID available throughout the codebase is very
valuable, the impact of requiring a Context as an argument to every
function in the codepath of an API request, is too significant and was
not properly understood at the time of the review.
Furthermore, mixing API-layer code with non-API-layer code makes the
latter usable only by API-layer code (one that has a notion of Context).
This reverts commit de4164043546d2b9ee3bf323dbc41f4979c84480, reversing
changes made to 7daeecd42d7bb112bfe01532c8c9a962bb0c7967.
Signed-off-by: Tibor Vass <tibor@docker.com>
Conflicts:
api/server/container.go
builder/internals.go
daemon/container_unix.go
daemon/create.go
Upstream-commit: b08f071e18043abe8ce15f56826d38dd26bedb78
Component: engine
Add a daemon flag to control this behaviour. Add a warning message when pulling
an image from a v1 registry. The default order of pull is slightly altered
with this changset.
Previously it was:
https v2, https v1, http v2, http v1
now it is:
https v2, http v2, https v1, http v1
Prevent login to v1 registries by explicitly setting the version before ping to
prevent fallback to v1.
Add unit tests for v2 only mode. Create a mock server that can register
handlers for various endpoints. Assert no v1 endpoints are hit with legacy
registries disabled for the following commands: pull, push, build, run and
login. Assert the opposite when legacy registries are not disabled.
Signed-off-by: Richard Scothern <richard.scothern@gmail.com>
Upstream-commit: 39f2f15a35121f348e1a87db4203bcb72e632b16
Component: engine
This PR adds a "request ID" to each event generated, the 'docker events'
stream now looks like this:
```
2015-09-10T15:02:50.000000000-07:00 [reqid: c01e3534ddca] de7c5d4ca927253cf4e978ee9c4545161e406e9b5a14617efb52c658b249174a: (from ubuntu) create
```
Note the `[reqID: c01e3534ddca]` part, that's new.
Each HTTP request will generate its own unique ID. So, if you do a
`docker build` you'll see a series of events all with the same reqID.
This allow for log processing tools to determine which events are all related
to the same http request.
I didn't propigate the context to all possible funcs in the daemon,
I decided to just do the ones that needed it in order to get the reqID
into the events. I'd like to have people review this direction first, and
if we're ok with it then I'll make sure we're consistent about when
we pass around the context - IOW, make sure that all funcs at the same level
have a context passed in even if they don't call the log funcs - this will
ensure we're consistent w/o passing it around for all calls unnecessarily.
ping @icecrime @calavera @crosbymichael
Signed-off-by: Doug Davis <dug@us.ibm.com>
Upstream-commit: 26b1064967d9fcefd4c35f60e96bf6d7c9a3b5f8
Component: engine
Fix: Docker Daemon didn't send actual actions client ask for to Token Server
Upstream-commit: d25dada639fe475ff6496b1d357d2250a0e014e3
Component: engine
The original purpose of this was to cancel downloads if pullV2Tag
returns an error, preventing an associated crash (see #15353). The
broadcaster now accomplishes the same thing that the pipe does, making
the pipe redundant. When pullV2Tag returns, all broadcasters are closed,
which means all further writes to those broadcasters will return errors.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Upstream-commit: 50c355777e31c08d3a97ec70b67bbe5a3bc883af
Component: engine
Before, this only waited for the download to complete. There was no
guarantee that the layer had been registered in the graph and was ready
use. This is especially problematic with v2 pulls, which wait for all
downloads before extracting layers.
Change Broadcaster to allow an error value to be propagated from Close
to the waiters.
Make the wait stop when the extraction is finished, rather than just the
download.
This also fixes v2 layer downloads to prefix the pool key with "layer:"
instead of "img:". "img:" is the wrong prefix, because this is what v1
uses for entire images. A v1 pull waiting for one of these operations to
finish would only wait for that particular layer, not all its
dependencies.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Upstream-commit: 23e68679f080fee7ceb25cf791832f523a3a024a
Component: engine
Ensure that layers are not excluded from manifests based on previous pushes.
Continue skipping pushes on layers which were pushed by a previous tag.
Update push multiple tag tests.
Ensure that each tag pushed exists on the registry and is pullable.
Add output comparison on multiple tag push check.
fixes#15536
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Upstream-commit: a0d9ffd6f70fdbbc392b31fe1a230bec3ce58df0
Component: engine
It seems the author foget to change the variable name after copying
from the valid testcase.
Signed-off-by: Hu Keping <hukeping@huawei.com>
Upstream-commit: fd5c33194ca6ff0851925bf1efe386a5375362a5
Component: engine
Close the pipeWriter even if there was no error.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Upstream-commit: 3037e4f7c4eec06cf3a5afca6f58ec58b4bdd63e
Component: engine
The Docker Daemon should send actual actions client ask for to issue tokens,
not all the permissions that client is guaranteed.
Signed-off-by: xiekeyang <xiekeyang@huawei.com>
Upstream-commit: e451ddbf4657cbe72f539e1f9513a5bc3a31f3d6
Component: engine
This file was not well documented and had very high cyclomatic complexity.
This patch completely rearranges this file and the ImageDelete method to
be easier to follow and more maintainable in the future.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
Upstream-commit: 111d2f348767692a34366ff5cd01f607cf54ad57
Component: engine
These don't get seen, so they are unnecessary.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Upstream-commit: 9b9d70ad271b46a67bde57b8a807121f6e85b75f
Component: engine
- Rename to Broadcaster
- Document exported types
- Change Wait function to just wait. Writing a message to the writer and
adding the writer to the observers list are now handled by separate
function calls.
- Avoid importing logrus (the condition where it was used should never
happen, anyway).
- Make writes non-blocking
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Upstream-commit: 26c9b585042ac7dce8db83478a69fd01a4b003d7
Component: engine
