Replace use of foreign sources with descriptors and describable
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Upstream-commit: 2c60430a3d1431e0879aa1c66ca23143de987b35
Component: engine
This is used to support downloading Windows base images from Microsoft
servers.
Signed-off-by: John Starks <jostarks@microsoft.com>
Upstream-commit: 05bd04350b8348b3c3bbe3156420257313e4e804
Component: engine
Since the layer store was introduced, the level above the graphdriver
now differentiates between read/write and read-only layers. This
distinction is useful for graphdrivers that need to take special steps
when creating a layer based on whether it is read-only or not.
Adding this parameter allows the graphdrivers to differentiate, which
in the case of the Windows graphdriver, removes our dependence on parsing
the id of the parent for "-init" in order to infer this information.
This will also set the stage for unblocking some of the layer store
unit tests in the next preview build of Windows.
Signed-off-by: Stefan J. Wernli <swernli@microsoft.com>
Upstream-commit: ef5bfad3210a9e9c8b761f2c11c0c6289490ebff
Component: engine
Get was calling getReference without layerL held. This meant writes to
the references map could race. Such races are dangerous because they can
corrupt the map and crash the process.
Fixes#21616Fixes#21674
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Upstream-commit: 0538981c31a714a4183e846a4b512deb7879cc29
Component: engine
Fix unmount issues in the daemon crash and restart lifecycle, w.r.t
graph drivers. This change sets a live container RWLayer's activity
count to 1, so that the RWLayer is aware of the mount. Note that
containerd has experimental support for restore live containers.
Added/updated corresponding tests.
Signed-off-by: Anusha Ragunathan <anusha@docker.com>
Upstream-commit: 511a70583fbb901f57acb44d501cca8e6dcbce2c
Component: engine
Instead of implementing refcounts at each graphdriver, implement this in
the layer package which is what the engine actually interacts with now.
This means interacting directly with the graphdriver is no longer
explicitly safe with regard to Get/Put calls being refcounted.
In addition, with the containerd, layers may still be mounted after
a daemon restart since we will no longer explicitly kill containers when
we shutdown or startup engine.
Because of this ref counts would need to be repopulated.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Upstream-commit: 65d79e3e5e537039b244afd7eda29e721a93d84f
Component: engine
Instead of implementing refcounts at each graphdriver, implement this in
the layer package which is what the engine actually interacts with now.
This means interacting directly with the graphdriver is no longer
explicitly safe with regard to Get/Put calls being refcounted.
In addition, with the containerd, layers may still be mounted after
a daemon restart since we will no longer explicitly kill containers when
we shutdown or startup engine.
Because of this ref counts would need to be repopulated.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Upstream-commit: 563d0711f83952e561a0d7d5c48fef9810b4f010
Component: engine
This allows a graph driver to provide a custom FileGetter for tar-split
to use. Windows will use this to provide a more efficient implementation
in a follow-up change.
Signed-off-by: John Starks <jostarks@microsoft.com>
Upstream-commit: 58bec40d16265362fd4e41dbd652e6fba903794d
Component: engine
Close tarsplit gzip writer when creating tar-split.json.gz files during layer migration
Upstream-commit: 70687226781545499c45414286a5b864bfc4eecd
Component: engine
There is a missing call to Close on the gzip.Writer that is used to
compress newly created tar-split files during layer migration. This can
result in corrupt tar-split files that later cause docker push and
docker save to fail. The Close call is necessary to flush buffered data
to the stream.
Fixes: #20104
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Upstream-commit: 1c05c65f6fbb5ea35608da259dfe4a6d211dbf82
Component: engine
This adds verification for getting layer data out
of layerstore. These failures should only be possible
if layer metadata files have been manually changed
of if something is wrong with tar-split algorithm.
Failing early makes sure we don’t upload invalid data
to the registries where it would fail after someone
tries to pull it.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
(cherry picked from commit e29e580f7fe628e936925681a4885d0b655bb151)
Upstream-commit: 50a498ea1c49cc3caa81bb9fb1417de387117e89
Component: engine
Add continue when layer fails on store creation
Trim whitespace from layerstore files to keep trailing space from failing a layer load
Fixes#19449
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Upstream-commit: caef48f4e25eb56ae9d63aa58bea588891bed944
Component: engine
Support restoreCustomImage for windows with a new interface to extract
the graph driver from the LayerStore.
Signed-off-by: Daniel Nephin <dnephin@docker.com>
Upstream-commit: f5916b10ae02c7db83052a97205ac345a3d96300
Component: engine
RWLayer will now have more operations and be protected through a referenced type rather than always looked up by string in the layer store.
Separates creation of RWLayer (write capture layer) from mounting of the layer.
This allows mount labels to be applied after creation and allowing RWLayer objects to have the same lifespan as a container without performance regressions from requiring mount.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Upstream-commit: d04fa49a0dec89d2f71a813ce8eaa182184139c5
Component: engine
Add function to get metadata from layer store for a mutable layer
fixes#18614
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Upstream-commit: a7e096832123280d26df3c121ecad8dd012060b9
Component: engine
Currently, the resources associated with the io.Reader returned by
TarStream are only freed when it is read until EOF. This means that
partial uploads or exports (for example, in the case of a full disk or
severed connection) can leak a goroutine and open file. This commit
changes TarStream to return an io.ReadCloser. Resources are freed when
Close is called.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Upstream-commit: 21278efaee563b356851a530b08b0537fee095d7
Component: engine
Layer store manages read-only and read-write layers on a union file system.
Read only layers are always referenced by content addresses.
Read-write layer identifiers are handled by the caller but upon registering
its difference, the committed read-only layer will be referenced by content
hash.
Signed-off-by: Derek McGowan <derek@mcgstyle.net> (github: dmcgowan)
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Upstream-commit: 500e77bad0b19b3b1c8e6ac195485adcb70daef1
Component: engine
