User may still enable discards by setting dm.blkdiscard=true
Docker-DCO-1.1-Signed-off-by: Mike Snitzer <snitzer@redhat.com> (github: snitm)
Upstream-commit: e49567ba729001c31fe71e4b715eed8f50d7ded9
Component: engine
Ideally lvm2 would be used to create/manage the thin-pool volume that is
then handed to docker to exclusively create/manage the thin and thin
snapshot volumes needed for it's containers. Managing the thin-pool
outside of docker makes for the most feature-rich method of having
docker utilize device mapper thin provisioning as the backing storage
for docker's containers. lvm2-based thin-pool management feature
highlights include: automatic or interactive thin-pool resize support,
dynamically change thin-pool features, automatic thinp metadata checking
when lvm2 activates the thin-pool, etc.
Docker will not activate/deactivate the specified thin-pool device but
it will exclusively manage/create thin and thin snapshot volumes in it.
Docker will not take ownership of the specified thin-pool device unless
it has 0 data blocks used and a transaction id of 0. This should help
guard against using a thin-pool that is already in use.
Also fix typos in setupBaseImage() relative to the thin volume type of
the base image.
Docker-DCO-1.1-Signed-off-by: Mike Snitzer <snitzer@redhat.com> (github: snitm)
Upstream-commit: 2b10749cdd0939e4b9e6e18e160984129d733663
Component: engine
Took care of some review comments from crosbymichael.
v2:
- Return "err = nil" if file deviceset-metadata file does not exist.
- Use json.Decoder() interface for loading deviceset metadata.
v3:
- Reverted back to json marshal interface in loadDeviceSetMetaData().
Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
Upstream-commit: 15c74bebc1ea2d51612b5809b4477551547a8b3d
Component: engine
In previous patch I had introduce json:"-" tags to be on safer side to make
sure certain fields are not marshalled/unmarshalled. But struct fields
starting with small letter are not exported so they will not be marshalled
anyway. So remove json:"-" tags from there.
Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
Upstream-commit: 0f57c902450b1d4f7a676dc693689debca002e98
Component: engine
This is a first pass at splitting out devicemapper into separate, usable
bindings.
Signed-off-by: Vincent Batts <vbatts@redhat.com>
Upstream-commit: e2f8fbfbcc450432536e387777b1ff080c94a948
Component: engine
My pull request failed the build due to gofmat issues. I have run gofmt
on specified files and this commit fixes it.
Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
Upstream-commit: ff56531de47c08157b2a37e6c6b6189a5006dba2
Component: engine
The way thin-pool right now is designed, user space is supposed to keep
track of what device ids have already been used. If user space tries to
create a new thin/snap device and device id has already been used, thin
pool retuns -EEXIST.
Upon receiving -EEXIST, current docker implementation simply tries the
NextDeviceId++ and keeps on doing this till it finds a free device id.
This approach has two issues.
- It is little suboptimal.
- If device id already exists, current kenrel implementation spits out
a messsage on console.
[17991.140135] device-mapper: thin: Creation of new snapshot 33 of device 3 failed.
Here kenrel is trying to tell user that device id 33 has already been used.
And this shows up for every device id docker tries till it reaches a point
where device ids are not used. So if there are thousands of container and
one is trying to create a new container after fresh docker start, expect
thousands of such warnings to flood console.
This patch saves the NextDeviceId in a file in
/var/lib/docker/devmapper/metadata/deviceset-metadata and reads it back
when docker starts. This way we don't retry lots of device ids which
have already been used.
There might be some device ids which are free but we will get back to them
once device numbers wrap around (24bit limit on device ids).
This patch should cut down on number of kernel warnings.
Notice that I am creating a deviceset metadata file which is a global file
for this pool. So down the line if we need to save more data we should be
able to do that.
Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
Upstream-commit: 8c9e5e5e05f8ddfcf8cd5218edb83d9fe8238d81
Component: engine
I was trying to save nextDeviceId to a file but it would not work and
json.Marshal() will do nothing. Then some search showed that I need to
make first letter of struct field capital, exporting this field and
now json.Marshal() works.
This is a preparatory patch for the next one.
Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
Upstream-commit: 8e9a18039be6ade0b8db65f7f298959055d86192
Component: engine
Currently we save device metadata and have a helper function saveMetadata()
which converts data in json format as well as saves it to file. For
converting data in json format, one needs to know what is being saved.
Break this function down in two functions. One function only has file
write capability and takes in argument about byte array of json data.
Now this function does not have to know what data is being saved. It
only knows about a stream of json data is being saved to a file.
This allows me to reuse this function to save a different type of
metadata. In this case I am planning to save NextDeviceId so that
docker can use this device Id upon next restart. Otherwise docker
starts from 0 which is suboptimal.
Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
Upstream-commit: 67fbd34d8379a1b8232aea5d126a389f64bdc59a
Component: engine
The logic is unrelated to graphdriver.
Signed-off-by: Vincent Batts <vbatts@redhat.com>
Upstream-commit: 930a756ad55ad5f4e5e6391b41673743d7254c2b
Component: engine
If `--storage-opt dm.datadev=/dev/loop0 --storage-opt
dm.metadatadev=/dev/loop1 ` were provided, the information was not
reflected in the information output.
Closes: #7137
Signed-off-by: Vincent Batts <vbatts@redhat.com>
Upstream-commit: 7e9ba22dc358196ac0d13524aa05677327f19ff9
Component: engine
Some graphdrivers are Differs and type assertions are made
in various places throughout the project. Differ offers some
convenience in generating/applying diffs of filesystem layers
but for most graphdrivers another code path is taken.
This patch brings all of the logic related to filesystem
diffs in one place, and simplifies the implementation of some
common types like Image, Daemon, and Container.
Signed-off-by: Josh Hawn <josh.hawn@docker.com>
Upstream-commit: dee6b481fe0da1d845261ffff2e610fb05898d3c
Component: engine
functions to pkg/parsers/kernel, and parsing filters to
pkg/parsers/filter. Adjust imports and package references.
Docker-DCO-1.1-Signed-off-by: Erik Hollensbe <github@hollensbe.org> (github: erikh)
Upstream-commit: 4398108433121ce2ac9942e607da20fa1680871a
Component: engine
Commit 09ee269d ("devmapper: Add option for specifying the thin pool
blocksize") also switched the default dm-thin-pool blocksize from 64K to
512K. That change unfortunately breaks the activation of dm-thin-pool
devices that were previously created using a 64K blocksize. Here is an
example of the dm-thin-pool activation failure users may experience:
device-mapper: thin: 253:4: pool target (204800 blocks) too small: expected 1638400
device-mapper: table: 253:4: thin-pool: preresume failed, error = -22
The reason for this is docker is passing 512K as the blocksize for a
dm-thin-pool that was previously created using a 64K blocksize. Docker
doesn't record the blocksize the is used when it creates a dm-thin-pool.
Until now it never had a need to do so because the blocksize was always
hardcoded. The dm-thin-pool blocksize must be the same every time a
dm-thin-pool is activated.
As a stop-gap fix, revert to using 64K for the default blocksize.
But we do need a proper fix for this now that 'dm.blocksize' is exposed
as a proper storage option. One possible fix would be to record the
blocksize for each dm-thin-pool that docker creates and to pass that
recorded blocksize down in the dmsetup table load each time the
dm-thin-pool is activated (this would be comparable to what lvm2 does).
Docker-DCO-1.1-Signed-off-by: Mike Snitzer <snitzer@redhat.com> (github: snitm)
Upstream-commit: d715803d45d41bfbbd5faf1f4a8239c08d6d8c90
Component: engine
Add dm.blocksize option that you can use with --storage-opt to set a
specific blocksize for the thin provisioning pool.
Also change the default dm-thin-pool blocksize from 64K to 512K. This
strikes a balance between the desire to have smaller blocksize given
docker's use of snapshots versus the desire to have more performance
that comes with using a larger blocksize. But if very small files will
be used on average the user is encouraged to override this default.
Docker-DCO-1.1-Signed-off-by: Mike Snitzer <snitzer@redhat.com> (github: snitm)
Upstream-commit: 09ee269d998ad04733ef577739fa051df9d3f12e
Component: engine
Device Mapper needs device sizes in binary (1024) multiples. Otherwise
kernel checks can find that the specified thin-pool device sizes aren't
a multiple of the specified thin-pool blocksize.
The name for "RAMInBytes" is likely too narrow given the new consumers
but... Also add "tebibyte" support to RAMInBytes.
Docker-DCO-1.1-Signed-off-by: Mike Snitzer <snitzer@redhat.com> (github: snitm)
Upstream-commit: 2470a5ed999d9dc8d4b9bd250727ac13964db5b3
Component: engine
