Very soon we will have the notion of an open transaction and keep its
details in a metafile.
When a new transaction is opened, we allocate a new transaction Id,
do the device creation/deletion and then we will close the transaction.
I thought that OpenTransactionId better represents the semantics of
transaction Id associated with an open transaction instead of NewtransactionId.
This patch just does the renaming. No functionality change.
I have also introduced a structure "Transaction" which will keep all
the details associated with a transaction. Later patches will add more
fields in this structure.
Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
Upstream-commit: f078bcd8e50913fd8b05022ebd047c5a1f2e3d52
Component: engine
Currently new transaction Id is created using allocateTransactionId()
function. This function takes NewTransactionId and bumps up by one
to create NewTransactionId.
I think ideally we should be bumping up devices.TransactionId by 1
to come up with NewTransactionId. Because idea is that devices.TransactionId
contains the current pool transaction Id and to come up with a new
transaction Id bump it up by one.
Current code is not wrong as we are keeping NewTransactionId and
TransactionId in sync. But it will be more direct if we look at
devices.TransactionId to come up with NewTransactionId. That way
we don't have to even initialize NewTransactionId during startup
as first time somebody wants to do a transaction, it will be
allocated fresh.
So simplify the code a bit. No functionality change.
Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
Upstream-commit: 7b0a1b814b8f13e30df466dd66c3fdc2114eac28
Component: engine
Currently updatePoolTransactionId() checks if NewTransactionId and
TransactionId are not same only then update the transaction Id in pool. This
check is redundant. Currently we call updatePoolTransactionId() only from
two places and both of these first allocate a new transaction Id.
Also updatePoolTransactionId() should only be called after allocating
new transaction Id otherwise it does not make any sense.
Remove the redundant check and reduce confusion.
Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
Upstream-commit: 6d347aeb6984ebdcb1051212ab3103880ef69ab0
Component: engine
Create two new helper functions for device and snap device creation. These
functions will not only create the device and also register the device.
Again, makes the code structure better and keeps all transaction logic
contained to functions instead of spilling over into functions like
setupBaseImage or AddDevice().
Just the code reorganization. No functionality change.
Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
Upstream-commit: ad9118c696c0953ec48eec15ea4b7546296d7c20
Component: engine
Currently registerDevice() adds a device to in-memory table, saves metadata
and also updates the pool transaction ID.
Now move transaciton Id update out of registerDevice() and provide a new
function unregisterDevice() which does the reverse of registerDevice().
This will simplify some code down the line and make it more structured.
This is just code reorganization and should not change functionality.
Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
Upstream-commit: 442247927b8e6c102ce1f94de58c7f93aab3d271
Component: engine
Currently devicemapper CreateDevice and CreateSnapDevice keep on retrying
device creation till a suitable device id is found.
With new transaction mechanism we need to store device id in transaction
before it has been created.
So change the logic in such a way that caller decides the devices Id to
use. If that device Id is not available, caller bumps up the device Id
and retries.
That way caller can update transaciton too when it tries a new Id. Transaction
related patches will come later in the series.
Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
Upstream-commit: 7b2b15d3e9f9b7ad898a36bbe5ceb42c9ca58d47
Component: engine
When we are deleting a device, we also delete associated metadata file. If
that file removal fails, we are adding back the device in in-memory
table. I really can't see what's the point. When next lookup takes place
it will be automatically loaded if need be. Remove that code.
Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
Upstream-commit: 5be77901cd505aad002b912b5febe2ba6baa23fd
Component: engine
Right now initMetaData() first queries the pool for current transaciton Id
and then it migrates the old metafile.
Move pool transaction Id query and file migration in separate functions
for better code reuse and organization.
Given we have removed device transaction Id dependency from saveMetaData(),
we don't have to query pool transaction Id before migrating files.
Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
Upstream-commit: 824a87f7efb94e4f307d920c3c3689156d6e633f
Component: engine
Right now saveMetaData() is kind of little overloaded function. It is
supposed to save file metadata to disk. But in addition if user has
bumped up NewTransactionId before calling saveMetaData(), then it will
also update the transaction ID in pool.
Keep saveMetaData() simple and let it just save the file. Any update
of pool transaction ID is done inline in the code which needs it.
Also create an helper function updatePoolTransactionId() to update pool
transaction Id.
Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
Upstream-commit: 0db6cc85edfccb16ce5308eea767530e1a3f6906
Component: engine
Remove call to allocateTransactionId() during device removal. This seems to
be unnecessary and it is not clear what this call is doing.
Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
Upstream-commit: 004d8b9b337f4a6cf68c124e89e02e673c6320fc
Component: engine
Again, just because device transaction id is greater than pool transaction
id, it does not guarantee that device is in the pool. So do not check
of this during loading of device metadata.
Docker needs to deal with it. And device activation will fail when we try
to activate a device for whom metafile is present but there is no device
in the pool.
Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
Upstream-commit: b721d6d8d0313fbb4b80e12318c96fc4004ee96b
Component: engine
Current code is associating a transaction id with each device and if pool
transaction id is greater that value, then current code assumes that device
is there in pool.
Transaction id of pool is a mechanism so that during device creation and
removal one can define a transaction and during startup figure out if
transaction was complete or not. I think we are using transaction id
throughout the code little inappropriately.
For example, if a device is being deleted, it is possible that we deleted
the device from pool but before we could delete metafile docker crashed.
When docker comes back it will think that device is in the pool (due to
device transaction id being less than pool transaction id) but device
is not in the pool.
Similary, it could happen that some data in the pool is corrupted and
during pool repair some devices are lost (without docker knowing about
it). In that case tool pool transaction id will be higher than device
transaction id and there are no guaratees that device is actually in
the pool.
So move away from this model where we think that a device is in pool if pool
transaction id is greater than device transaction Id. Per device
transaction Id just says that after device creation this should be pool's
transaction Id and nothing more.
Transaction id is per pool property (as opposed to per device property) and
will be used internally to figure out if last transaction was complete or
not and recover from failure during docker startup.
If for some reason metafile is present but device is not in pool, then
device activation will fail later.
Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
Upstream-commit: bb00453e58a86b9787ac4b3e7df3c48d8ddc3f87
Component: engine
TreeSize uses syscall.Stat_t which is not available on Windows.
It's called only on daemon path, therefore extracting it to daemon
with build tag 'daemon'
Signed-off-by: Ahmet Alp Balkan <ahmetb@microsoft.com>
Upstream-commit: b64c9b521ab4e4082ed874a23a493f4a266304d5
Component: engine
be default it is on, with build tags to disable the version info
Signed-off-by: Vincent Batts <vbatts@redhat.com>
Upstream-commit: 25154682a5cd57aa4fc3ef88baeee3ce1f204060
Component: engine
User may still enable discards by setting dm.blkdiscard=true
Docker-DCO-1.1-Signed-off-by: Mike Snitzer <snitzer@redhat.com> (github: snitm)
Upstream-commit: e49567ba729001c31fe71e4b715eed8f50d7ded9
Component: engine
Fixes#1171Fixes#6465
Data passed to mount(2) is clipped to PAGE_SIZE if its bigger. Previous
implementation checked if error was returned and then started to append layers
one by one. But if the PAGE_SIZE clipping appeared in between the paths, in the
permission sections or in xino definition the call would not error and
remaining layers would just be skipped(or some other unknown situation).
This also optimizes system calls as it tries to mount as much as possible with
the first mount.
Signed-off-by: Tõnis Tiigi <tonistiigi@gmail.com> (github: tonistiigi)
Upstream-commit: 6d97339ca23ada27812572016ad4ff9ccffa8b09
Component: engine
Ideally lvm2 would be used to create/manage the thin-pool volume that is
then handed to docker to exclusively create/manage the thin and thin
snapshot volumes needed for it's containers. Managing the thin-pool
outside of docker makes for the most feature-rich method of having
docker utilize device mapper thin provisioning as the backing storage
for docker's containers. lvm2-based thin-pool management feature
highlights include: automatic or interactive thin-pool resize support,
dynamically change thin-pool features, automatic thinp metadata checking
when lvm2 activates the thin-pool, etc.
Docker will not activate/deactivate the specified thin-pool device but
it will exclusively manage/create thin and thin snapshot volumes in it.
Docker will not take ownership of the specified thin-pool device unless
it has 0 data blocks used and a transaction id of 0. This should help
guard against using a thin-pool that is already in use.
Also fix typos in setupBaseImage() relative to the thin volume type of
the base image.
Docker-DCO-1.1-Signed-off-by: Mike Snitzer <snitzer@redhat.com> (github: snitm)
Upstream-commit: 2b10749cdd0939e4b9e6e18e160984129d733663
Component: engine
Took care of some review comments from crosbymichael.
v2:
- Return "err = nil" if file deviceset-metadata file does not exist.
- Use json.Decoder() interface for loading deviceset metadata.
v3:
- Reverted back to json marshal interface in loadDeviceSetMetaData().
Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
Upstream-commit: 15c74bebc1ea2d51612b5809b4477551547a8b3d
Component: engine
In previous patch I had introduce json:"-" tags to be on safer side to make
sure certain fields are not marshalled/unmarshalled. But struct fields
starting with small letter are not exported so they will not be marshalled
anyway. So remove json:"-" tags from there.
Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
Upstream-commit: 0f57c902450b1d4f7a676dc693689debca002e98
Component: engine
This is a first pass at splitting out devicemapper into separate, usable
bindings.
Signed-off-by: Vincent Batts <vbatts@redhat.com>
Upstream-commit: e2f8fbfbcc450432536e387777b1ff080c94a948
Component: engine
My pull request failed the build due to gofmat issues. I have run gofmt
on specified files and this commit fixes it.
Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
Upstream-commit: ff56531de47c08157b2a37e6c6b6189a5006dba2
Component: engine
The way thin-pool right now is designed, user space is supposed to keep
track of what device ids have already been used. If user space tries to
create a new thin/snap device and device id has already been used, thin
pool retuns -EEXIST.
Upon receiving -EEXIST, current docker implementation simply tries the
NextDeviceId++ and keeps on doing this till it finds a free device id.
This approach has two issues.
- It is little suboptimal.
- If device id already exists, current kenrel implementation spits out
a messsage on console.
[17991.140135] device-mapper: thin: Creation of new snapshot 33 of device 3 failed.
Here kenrel is trying to tell user that device id 33 has already been used.
And this shows up for every device id docker tries till it reaches a point
where device ids are not used. So if there are thousands of container and
one is trying to create a new container after fresh docker start, expect
thousands of such warnings to flood console.
This patch saves the NextDeviceId in a file in
/var/lib/docker/devmapper/metadata/deviceset-metadata and reads it back
when docker starts. This way we don't retry lots of device ids which
have already been used.
There might be some device ids which are free but we will get back to them
once device numbers wrap around (24bit limit on device ids).
This patch should cut down on number of kernel warnings.
Notice that I am creating a deviceset metadata file which is a global file
for this pool. So down the line if we need to save more data we should be
able to do that.
Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
Upstream-commit: 8c9e5e5e05f8ddfcf8cd5218edb83d9fe8238d81
Component: engine
I was trying to save nextDeviceId to a file but it would not work and
json.Marshal() will do nothing. Then some search showed that I need to
make first letter of struct field capital, exporting this field and
now json.Marshal() works.
This is a preparatory patch for the next one.
Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
Upstream-commit: 8e9a18039be6ade0b8db65f7f298959055d86192
Component: engine
Currently we save device metadata and have a helper function saveMetadata()
which converts data in json format as well as saves it to file. For
converting data in json format, one needs to know what is being saved.
Break this function down in two functions. One function only has file
write capability and takes in argument about byte array of json data.
Now this function does not have to know what data is being saved. It
only knows about a stream of json data is being saved to a file.
This allows me to reuse this function to save a different type of
metadata. In this case I am planning to save NextDeviceId so that
docker can use this device Id upon next restart. Otherwise docker
starts from 0 which is suboptimal.
Signed-off-by: Vivek Goyal <vgoyal@redhat.com>
Upstream-commit: 67fbd34d8379a1b8232aea5d126a389f64bdc59a
Component: engine
The logic is unrelated to graphdriver.
Signed-off-by: Vincent Batts <vbatts@redhat.com>
Upstream-commit: 930a756ad55ad5f4e5e6391b41673743d7254c2b
Component: engine
In an effort to make layer content 'stable' between import
and export from two different graph drivers, we must resolve
an issue where AUFS produces metadata files in its layers
which other drivers explicitly ignore when importing.
The issue presents itself like this:
- Generate a layer using AUFS
- On commit of that container, the new stored layer contains
AUFS metadata files/dirs. The stored layer content has some
tarsum value: '1234567'
- `docker save` that image to a USB drive and `docker load`
into another docker engine instance which uses another
graph driver, say 'btrfs'
- On load, this graph driver explicitly ignores any AUFS metadata
that it encounters. The stored layer content now has some
different tarsum value: 'abcdefg'.
The only (apparent) useful aufs metadata to keep are the psuedo link
files located at `/.wh..wh.plink/`. Thes files hold information at the
RW layer about hard linked files between this layer and another layer.
The other graph drivers make sure to copy up these psuedo linked files
but I've tested out a few different situations and it seems that this
is unnecessary (In my test, AUFS already copies up the other hard linked
files to the RW layer).
This changeset adds explicit exclusion of the AUFS metadata files and
directories (NOTE: not the whiteout files!) on commit of a container
using the AUFS storage driver.
Also included is a change to the archive package. It now explicitly
ignores the root directory from being included in the resulting tar archive
for 2 reasons: 1) it's unnecessary. 2) It's another difference between
what other graph drivers produce when exporting a layer to a tar archive.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
Upstream-commit: 0d97e082c3b849a6901992e653b8963e8ef10f54
Component: engine
This backend uses the overlayfs union filesystem for containers
plus hard link file sharing for images.
Each container/image can have a "root" subdirectory which is a plain
filesystem hierarchy, or they can use overlayfs.
If they use overlayfs there is a "upper" directory and a "lower-id"
file, as well as "merged" and "work" directories. The "upper"
directory has the upper layer of the overlay, and "lower-id" contains
the id of the parent whose "root" directory shall be used as the lower
layer in the overlay. The overlay itself is mounted in the "merged"
directory, and the "work" dir is needed for overlayfs to work.
When a overlay layer is created there are two cases, either the
parent has a "root" dir, then we start out with a empty "upper"
directory overlaid on the parents root. This is typically the
case with the init layer of a container which is based on an image.
If there is no "root" in the parent, we inherit the lower-id from
the parent and start by making a copy if the parents "upper" dir.
This is typically the case for a container layer which copies
its parent -init upper layer.
Additionally we also have a custom implementation of ApplyLayer
which makes a recursive copy of the parent "root" layer using
hardlinks to share file data, and then applies the layer on top
of that. This means all chile images share file (but not directory)
data with the parent.
Docker-DCO-1.1-Signed-off-by: Alexander Larsson <alexl@redhat.com> (github: alexlarsson)
Upstream-commit: 453552c8384929d8ae04dcf1c6954435c0111da0
Component: engine
The vfs storage driver currently shells out to the `cp` binary on the host
system to perform an 'archive' copy of the base image to a new directory.
The archive option preserves the modified time of the files which are created
but there was an issue where it was unable to preserve the modified time of
copied symbolic links on some host systems with an outdated version of `cp`.
This change no longer relies on the host system implementation and instead
utilizes the `CopyWithTar` function found in `pkg/archive` which is used
to copy from source to destination directory using a Tar archive, which
should correctly preserve file attributes.
Docker-DCO-1.1-Signed-off-by: Josh Hawn <josh.hawn@docker.com> (github: jlhawn)
Upstream-commit: b6db23cffe942b8d94c80d1e9b3f1f6fca87d139
Component: engine
Vivek Goyal
Vincent Batts
Mike Snitzer
unclejack
Jessie Frazelle
Michael Crosby
Tonis Tiigi
Ahmet Alp Balkan
Michael Crosby
unclejack
Vincent Batts
Tibor Vass
Josh Hawn
Alexander Larsson
Alexandr Morozov