Removed images were not cleaned up from the
digest-set that is used for the search index.
Fixes#18437
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Upstream-commit: fcb083c6ac0fdb71e8db7a58f07a561342d631d3
Component: engine
Moved a defer up to a better spot.
Fixed TestUntarPathWithInvalidDest to actually fail for the right reason
Closes#18170
Signed-off-by: Doug Davis <dug@us.ibm.com>
Upstream-commit: 7bb9fc415ac50abf5f754005264bfe152eeca7d6
Component: engine
Currently, the resources associated with the io.Reader returned by
TarStream are only freed when it is read until EOF. This means that
partial uploads or exports (for example, in the case of a full disk or
severed connection) can leak a goroutine and open file. This commit
changes TarStream to return an io.ReadCloser. Resources are freed when
Close is called.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Upstream-commit: 21278efaee563b356851a530b08b0537fee095d7
Component: engine
The image store abstracts image handling. It keeps track of the
available images, and makes it possible to delete existing images or
register new ones. The image store holds references to the underlying
layers for each image.
The image/v1 package provides compatibility functions for interoperating
with older (non-content-addressable) image structures.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Upstream-commit: 01ba0a935b9227583ec05a96e85273cc2276bb93
Component: engine
Generate a hash chain involving the image configuration, layer digests,
and parent image hashes. Use the digests to compute IDs for each image
in a manifest, instead of using the remotely specified IDs.
To avoid breaking users' caches, check for images already in the graph
under old IDs, and avoid repulling an image if the version on disk under
the legacy ID ends up with the same digest that was computed from the
manifest for that image.
When a calculated ID already exists in the graph but can't be verified,
continue trying SHA256(digest) until a suitable ID is found.
"save" and "load" are not changed to use a similar scheme. "load" will
preserve the IDs present in the tar file.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Upstream-commit: 504e67b867865a2835e8002c01087a2cfd7bfd0e
Component: engine
Move graph related functions in image to graph package.
Consolidating graph functionality is the first step in refactoring graph into an image store model.
Subsequent refactors will involve breaking up graph into multiple types with a strongly defined interface.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Upstream-commit: 2b58b677a54950bc78c13760ef79fe6284154847
Component: engine
While reading some of the docs I noticed a few errors, so I ran
misspellings (https://pypi.python.org/pypi/misspellings) on markdown files
Signed-off-by: Joe Gordon <joe.gordon0@gmail.com>
Upstream-commit: 3d87b80b46d45c2ca7c0fbcbcbb3fcf97a48444b
Component: engine
The checksum is now being stored in a separate file beside the image
JSON file.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
Upstream-commit: 3414307306dc1780f80c5ca5e9dd7a8822e27eec
Component: engine
While checksums are verified when a layer is pulled from v2 registries,
there are known issues where the checksum may change when the layer diff
is computed again. To avoid these issues, the checksum should no longer
be computed and stored until after it has been extracted to the docker
storage driver. The checksums are instead computed lazily before they
are pushed to a v2 registry.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
Upstream-commit: e636df8d9637514e125afe7418a3b98e5270858f
Component: engine
The title `Image JSON Schema` was used as a header in the section
which describes the layout and fields of the image metadata JSON
file. It was pointed out that `JSON Schema` is its own term for
describing JSON in a machine-and-human-readable format, while the
word "Schema" in this context was used more generically to say that
the section is meant to be an example and outline of the Image JSON.
http://spacetelescope.github.io/understanding-json-schema/
This section now has the title `Image JSON Description` in order
to not cause this confusion.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
Upstream-commit: 71763636f232477b0c890e0506b7b858505f1b33
Component: engine
Updates `image.StoreImage()` to always ensure that images
that are installed in Docker have a tarsum.v1 checksum.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
Upstream-commit: ba3bad66f92da057dba2a43ff9a512cf7ad72375
Component: engine
While the v2 pull operation is writing the body of the layer blob to disk
it now computes the tarsum checksum of the archive before extracting it to
the backend storage driver. If the checksum does not match that from the
image manifest an error is raised.
Also adds more debug logging to the pull operation and fixes existing test
cases which were failing. Adds a reverse lookup constructor to the tarsum
package so that you can get a tarsum object using a checksum label.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
Upstream-commit: 213e3d116642431adbe634d39740eddc5a81e063
Component: engine
Many iterations have gone into documenting a v1 specification of Docker's Image
format.
v1 Image spec: clarify parent field
- metalivedev pointed out that the description was ambiguous, so I've removed
mention that it was randomly generated. It IS the ID of the parent image.
Updated v1 image specificatino documentation
- More complete details and deprication notifications for each field
in the JSON metadata of an image.
- Details on the format for packaging combined Image JSON + Filesystem
Changeset archives for all layers of an image.
Clarify description of an image "Layer" in v1 spec
Updated intro of image v1 spec
Updated image v1 spec after more review
- Removed description of "Image" from the terminology section. The entire
document is meant to serve this purpose.
- Updated the definition of "Image Filesystem Changeset".
- Clarified the level of randomness needed for generating image IDs.
- Updated the description of "Image Checksum".
- Added term descriptions for "Repository" and "Tag"
- Removed extraneous/implementation-specific fields from the Image JSON
example file and field descriptions:
- removed "container_config" and "docker_version" fields.
- Added missing "author" field example and description.
- Removed extraneous/implementation-specific fields from the "config" struct
example and description:
- removed "Hostname", "Domainname", "Cpuset", "AttachStdin", "AttachStdout",
"AttachStderr", "PortSpecs", "Tty", "OpenStdin", "StdinOnce", "Image",
"NetworkDisabled", and "OnBuild".
- Updated example Image JSON config with better example values for "Env",
"Cmd", "Volumes", "WorkingDir", "Entrypoint", "CpuShares", "Memory",
"MemorySwap", and "User".
- Added notices that any fields not specified are to be considered as
implementation specific and should be ignored my implementations which
are unable to interpret them.
- Updated example of creating layer filesystem changesets to use less formal
language.
- Listed more details in the section regarding extraction of a bundle of image
layers into the root filesystem of a container.
- Updated the closing mention of Docker as an evolving implementation.
More updates to the v1 image spec
- Added line wrapping after 80 columns per line to adhere to documentation
style guides, as pointed out by @jamtur01
- Removed references to any specific docker commands, updated a few descriptions
or drop repeated statements, as pointed out by @cpuguy83
Cleanup image v1 spec draft after fredlf comments
Address comments by mmdriley on v1 image spec
Improve description of image v1 spec 'config.User`
- Improves description of image v1 specification for the 'User' runtime
parameter after recomendations by tianon.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
Upstream-commit: 79910625f0a7aa76590e4362817dda22f28343aa
Component: engine
Now, newly created/imported layers will have the checksum of
the layer diff computed and stored in the image json file.
For now, it is not an error if the computed checksum does not
match an existing checksum, only a warning message is logged. The
eventual goal is to use the checksums in the image JSON to verify
the integrity of the layer contents when doing `docker load` or
`docker pull`, and error out if it does not match.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
Upstream-commit: 059e589c3427b1afa1b112acc931f59b5b063e16
Component: engine
The argument specified the json data to save to disk when registering
a new image into the image graph. If it is nil, then the given image
is serialized to json and that is written by default. This default
behavior is sufficient if the given image was originally deserialzed
from this jsonData to begin with which has always been the case.
Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
Upstream-commit: 26184de8ab1dfe812094c55c9becd8ebb60ed7be
Component: engine
Now that the archive package does not depend on any docker-specific
packages, only those in pkg and vendor, it can be safely moved into pkg.
Signed-off-by: Rafe Colton <rafael.colton@gmail.com>
Upstream-commit: 30d5a42c1f24e26f681b7330249f04fec891aee9
Component: engine
Some graphdrivers are Differs and type assertions are made
in various places throughout the project. Differ offers some
convenience in generating/applying diffs of filesystem layers
but for most graphdrivers another code path is taken.
This patch brings all of the logic related to filesystem
diffs in one place, and simplifies the implementation of some
common types like Image, Daemon, and Container.
Signed-off-by: Josh Hawn <josh.hawn@docker.com>
Upstream-commit: dee6b481fe0da1d845261ffff2e610fb05898d3c
Component: engine
