This commit adds a transfer manager which deduplicates and schedules
transfers, and also an upload manager and download manager that build on
top of the transfer manager to provide high-level interfaces for uploads
and downloads. The push and pull code is modified to use these building
blocks.
Some benefits of the changes:
- Simplification of push/pull code
- Pushes can upload layers concurrently
- Failed downloads and uploads are retried after backoff delays
- Cancellation is supported, but individual transfers will only be
cancelled if all pushes or pulls using them are cancelled.
- The distribution code is decoupled from Docker Engine packages and API
conventions (i.e. streamformatter), which will make it easier to split
out.
This commit also includes unit tests for the new distribution/xfer
package. The tests cover 87.8% of the statements in the package.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Upstream-commit: 572ce802306a4e919802e5b77cbeca94acda7c0a
Component: engine
This test was directly comparing lines of output from "docker images".
Sometimes, when busybox had been pushed to the hub recently, the
relative creation times would differ like this:
... obtained []string = []string{"busybox", "latest", "d9551b4026f0", "27", "minutes", "ago", "1.113", "MB"}
... expected []string = []string{"busybox", "latest", "d9551b4026f0", "26", "minutes", "ago", "1.113", "MB"}
Fixing by removing the time-since-creation fields from the comparison.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Upstream-commit: d17669999fff55219f5ed236f00fdf8a08d09304
Component: engine
Add distribution package for managing pulls and pushes. This is based on
the old code in the graph package, with major changes to work with the
new image/layer model.
Add v1 migration code.
Update registry, api/*, and daemon packages to use the reference
package's types where applicable.
Update daemon package to use image/layer/tag stores instead of the graph
package
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Upstream-commit: 4352da7803d182a6013a5238ce20a7c749db979a
Component: engine
Generate a hash chain involving the image configuration, layer digests,
and parent image hashes. Use the digests to compute IDs for each image
in a manifest, instead of using the remotely specified IDs.
To avoid breaking users' caches, check for images already in the graph
under old IDs, and avoid repulling an image if the version on disk under
the legacy ID ends up with the same digest that was computed from the
manifest for that image.
When a calculated ID already exists in the graph but can't be verified,
continue trying SHA256(digest) until a suitable ID is found.
"save" and "load" are not changed to use a similar scheme. "load" will
preserve the IDs present in the tar file.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Upstream-commit: 504e67b867865a2835e8002c01087a2cfd7bfd0e
Component: engine
- utils_test.go and docker_utils_test.go
- Moved docker related function to docker_utils.go
- add a test for integration-cli/checker
Signed-off-by: Vincent Demeester <vincent@sbr.pm>
Upstream-commit: 5109071706cab06257b21172627ce51e5fa3a88c
Component: engine
Update and migrate existing tests to the `DockerHubPullSuite`. Most
tests were preserved, but refactored and made more exhaustive. One test
was deliberately removed (`TestPullVerified`) as it is unreliable and
that the feature was obsoleted by content trust.
Move all trust related tests to `docker_cli_pull_trusted_test.go`.
Move tests depending on a local registry to `docker_cli_pull_local_test.go`.
Signed-off-by: Arnaud Porterie <arnaud.porterie@docker.com>
Upstream-commit: f324f4851f28edfd8dab82cd624a4ec1f01cd207
Component: engine
Currently some notary tests change the system clock to check for expiration.
Skip these tests until the code can be refactored to not rely on updating the system clock.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Upstream-commit: bf3c1e6a3afc951e41643b8d55d4ac25fa9cb06d
Component: engine
Update help line to allow 90 characters instead of 80
The trust flag pushes out the help description column wider, requiring more room to display help messages.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Upstream-commit: 259cadb0b19a9049302e78fec529b0105efabb1d
Component: engine
Clean up tests to remove duplicate code
Add tests which run pull and create in an isolated configuration directory.
Add build test for untrusted tag
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Upstream-commit: 871d2b96ed5cf234c41a5e731a34fc9deda4e9f1
Component: engine
Created date util function
Signed-off-by: Nathan McCauley <nathan.mccauley@docker.com>
Upstream-commit: 1406cb35fd814a987b91f92ed1a4b61a21205736
Component: engine
Added notary server to docker base image.
Created trust suite which runs trust server for running trusted commands.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Upstream-commit: 58a1de9b59594948df152f0003e759b77bcaa56a
Component: engine
To avoid manually creating and destroying registrys in tests.
Signed-off-by: Alexander Morozov <lk4d4@docker.com>
Upstream-commit: f696b1071a296bee1f4ac7cafa807ce337fb9f2c
Component: engine
It prints test name and duration for each test.
Also performs deleteAllContainers after each test.
Signed-off-by: Alexander Morozov <lk4d4@docker.com>
Upstream-commit: dc944ea7e48d11a2906e751d3e61daf08faee054
Component: engine
Only show the verification message if all the tarsum checks pass and the image manifest is verified.
No longer return an error when a tarsum verification fails, just reset the verification flag.
Tarsum verification is less meaningful without a verified manifest and therefore it should not cause an error.
Updated the verified image test to pull an image which expected to have a verified manifest and contents.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Upstream-commit: 6088df20c33afafa523b9dd9b0acdd3b987c4534
Component: engine
Passing RepositoryInfo to ResolveAuthConfig, pullRepository, and pushRepository
Moving --registry-mirror configuration to registry config
Created resolve_repository job
Repo names with 'index.docker.io' or 'docker.io' are now synonymous with omitting an index name.
Adding test for RepositoryInfo
Adding tests for opts.StringSetOpts and registry.ValidateMirror
Fixing search term use of repoInfo
Adding integration tests for registry mirror configuration
Normalizing LookupImage image name to match LocalName parsing rules
Normalizing repository LocalName to avoid multiple references to an official image
Removing errorOut use in tests
Removing TODO comment
gofmt changes
golint comments cleanup. renaming RegistryOptions => registry.Options, and RegistryServiceConfig => registry.ServiceConfig
Splitting out builtins.Registry and registry.NewService calls
Stray whitespace cleanup
Moving integration tests for Mirrors and InsecureRegistries into TestNewIndexInfo unit test
Factoring out ValidateRepositoryName from NewRepositoryInfo
Removing unused IndexServerURL
Allowing json marshaling of ServiceConfig. Exposing ServiceConfig in /info
Switching to CamelCase for json marshaling
PR cleanup; removing 'Is' prefix from boolean members. Removing unneeded json tags.
Removing non-cleanup related fix for 'localhost:[port]' in splitReposName
Merge fixes for gh9735
Fixing integration test
Reapplying #9754
Adding comment on config.IndexConfigs use from isSecureIndex
Remove unused error return value from isSecureIndex
Signed-off-by: Don Kjer <don.kjer@gmail.com>
Adding back comment in isSecureIndex
Signed-off-by: Don Kjer <don.kjer@gmail.com>
Upstream-commit: 568f86eb186731b907b659e4ec64bda21c2fe31d
Component: engine
There has been a lot of discussion (issues 4242 and 5262) about making
`FROM scratch` either a special case or making `FROM` optional, implying
starting from an empty file system.
This patch makes the build command `FROM scratch` special cased from now on
and if used does not pull/set the the initial layer of the build to the ancient
image ID (511136ea..) but instead marks the build as having no base image. The
next command in the dockerfile will create an image with a parent image ID of "".
This means every image ever can now use one fewer layer!
This also makes the image name `scratch` a reserved name by the TagStore. You
will not be able to tag an image with this name from now on. If any users
currently have an image tagged as `scratch`, they will still be able to use that
image, but will not be able to tag a new image with that name.
Goodbye '511136ea3c5a64f264b78b5433614aec563103b4d4702f3ba7d4d2698e22c158',
it was nice knowing you.
Fixes#4242
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
Upstream-commit: 8936789919c5c8004f346f44a3452d1521818b60
Component: engine
