Move plugins to shared distribution stack with images.
Create immutable plugin config that matches schema2 requirements.
Ensure data being pushed is same as pulled/created.
Store distribution artifacts in a blobstore.
Run init layer setup for every plugin start.
Fix breakouts from unsafe file accesses.
Add support for `docker plugin install --alias`
Uses normalized references for default names to avoid collisions when using default hosts/tags.
Some refactoring of the plugin manager to support the change, like removing the singleton manager and adding manager config struct.
Signed-off-by: Tonis Tiigi <tonistiigi@gmail.com>
Signed-off-by: Derek McGowan <derek@mcgstyle.net>
Upstream-commit: 3d86b0c79b16334ce5836c0315e4c310b84c2e17
Component: engine
bolt k/v pairs are only valid for the life of a transaction.
This means the memory that the k/v pair is referencing may be invalid if
it is accessed outside of the transaction.
This can potentially cause a panic.
For reference: https://godoc.org/github.com/boltdb/bolt#hdr-Caveats
To fix this issue, unmarshal the stored data into volume meta before
closing the transaction.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Upstream-commit: 4876a9047ebfd66294d88482a1b4b24634a632e6
Component: engine
Previously, it was comparing against the driver name passed in by the
caller. This could lead to subtle issues when using plugins, like
"plugin" vs. "plugin:latest".
Also, remove "conflict:" prefix to improve the error message.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Upstream-commit: 53d447c5d5c85d5595d5170411189c88a135a789
Component: engine
Ensures all known volumes (known b/c they are persisted to disk) have
their volume drivers refcounted properly.
In testing this, I found an issue with `--live-restore` (required since
currently the provided volume plugin doesn't keep state on restart)
where restorted plugins did not have a plugin client loaded causing a
panic when trying to use the plugin.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Upstream-commit: 6ef1060cd0acb847e06db890abb335faa837a9e2
Component: engine
Adds 2 new methods to v2 plugin `Acquire` and `Release` which allow
refcounting directly at the plugin level instead of just the store.
Since a graphdriver is initialized exactly once, and is really managed
by a separate object, it didn't really seem right to call
`getter.Get()` to refcount graphdriver plugins.
On shutdown it was particularly weird where we'd either need to keep a
driver reference in daemon, or keep a reference to the pluggin getter in
the layer store, and even then still store extra details on if the
graphdriver is a plugin or not.
Instead the plugin proxy itself will handle calling the neccessary
refcounting methods directly on the plugin object.
Also adds a new interface in `plugingetter` to account for these new
functions which are not going to be implemented by v1 plugins.
Changes terms `plugingetter.CREATE` and `plugingetter.REMOVE` to
`ACQUIRE` and `RELEASE` respectively, which seems to be better
adjectives for what we're doing.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Upstream-commit: f29bbd16f5d2bb82d815ea59f8ef85fe59384c89
Component: engine
Legacy plugins expect host-relative paths (such as for Volume.Mount).
However, a containerized plugin cannot respond with a host-relative
path. Therefore, this commit modifies new volume plugins' paths in Mount
and List to prepend the container's rootfs path.
This introduces a new PropagatedMount field in the Plugin Config.
When it is set for volume plugins, RootfsPropagation is set to rshared
and the path specified by PropagatedMount is bind-mounted with rshared
prior to launching the container. This is so that the daemon code can
access the paths returned by the plugin from the host mount namespace.
Signed-off-by: Tibor Vass <tibor@docker.com>
Upstream-commit: c54b717caf1a55e525ce180bfcb42addd59c6633
Component: engine
The current implementation of getRefs is a bit fragile. It returns a
slice to callers without copying its contents, and assumes the contents
will not be modified elsewhere.
Also, the current implementation of Dereference requires copying the
slice of references, excluding the one we wish to remove.
To improve both of these things, change refs to be a map of maps.
Deleting an item becomes trivial, and returning a slice of references
necessitates copying from the map.
Signed-off-by: Aaron Lehmann <aaron.lehmann@docker.com>
Upstream-commit: a762222396d21990b2c0772300660312e7a58b6c
Component: engine
Fix issue where out-of-band deletions and then a `docker volume create`
on the same driver caused volume to not be re-created in the driver but
return as created since it was stored in the cache.
Previous fix only worked if the driver names did not match.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Upstream-commit: d8ce4a6e108f4f870228912f105eed8218e087e4
Component: engine
This fix tries to address the issue raised in 28769 where
checkpoint name was not checked before passing to containerd.
As a result, it was possible to use a special checkpoint name
to get outside of the container's directory.
This fix add restriction `[a-zA-Z0-9][a-zA-Z0-9_.-]+` (`RestrictedNamePattern`).
This is the same as container name restriction.
This fix fixes 28769.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
Upstream-commit: c90ec0517544e7d054d79f71f4d24d9ebbad7408
Component: engine
Instead of converting nicely typed service mounts into untyped `Binds`
when creating containers, use the new `Mounts` API which is a 1-1
mapping between service mounts and container mounts.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Upstream-commit: 821aeb6a6f72f070ae045cb7813759cbbaba7bda
Component: engine
For some reason, `go vet` and `go fmt` validate does not capture
several issues.
The following was the output of `go vet`:
```
ubuntu@ubuntu:~/docker$ go vet ./... 2>&1 | grep -v ^vendor | grep -v '^exit status 1$'
cli/command/formatter/container_test.go:393: possible formatting directive in Log call
volume/volume_test.go:257: arg mp.RW for printf verb %s of wrong type: bool
```
The following was the output of `go fmt -s`:
```
ubuntu@ubuntu:~/docker$ gofmt -s -l . | grep -v ^vendor
cli/command/stack/list.go
daemon/commit.go
```
Fixed above issues with `go vet` and `go fmt -s`
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
Upstream-commit: ace786e9d517777473bd431e65d6c464d82e4f65
Component: engine
Fixes#22564
When an error occurs on mount, there should not be any call later to
unmount. This can throw off refcounting in the underlying driver
unexpectedly.
Consider these two cases:
```
$ docker run -v foo:/bar busybox true
```
```
$ docker run -v foo:/bar -w /foo busybox true
```
In the first case, if mounting `foo` fails, the volume driver will not
get a call to unmount (this is the incorrect behavior).
In the second case, the volume driver will not get a call to unmount
(correct behavior).
This occurs because in the first case, `/bar` does not exist in the
container, and as such there is no call to `volume.Mount()` during the
`create` phase. It will error out during the `start` phase.
In the second case `/bar` is created before dealing with the volume
because of the `-w`. Because of this, when the volume is being setup
docker will try to copy the image path contents in the volume, in which
case it will attempt to mount the volume and fail. This happens during
the `create` phase. This makes it so the container will not be created
(or at least fully created) and the user gets the error on `create`
instead of `start`. The error handling is different in these two phases.
Changed to only send `unmount` if the volume is mounted.
While investigating the cause of the reported issue I found some odd
behavior in unmount calls so I've cleaned those up a bit here as well.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Upstream-commit: 9a2d0bc3adc0c21c82cd1974be45ea0449f9f224
Component: engine
For `--mount type=tmpfs,target=/foo,readonly`, the `readonly` flag was just ignored.
Signed-off-by: Akihiro Suda <suda.akihiro@lab.ntt.co.jp>
Upstream-commit: 3e3d3c8086237f4938e3e13561dbc1d41b5c617b
Component: engine
When a conflict is found in the volume cache, check with the driver if
that volume still actually exists.
If the volume doesn't exist, purge it from the cache and allow the
create to happen.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Upstream-commit: 6a0bdffc1adf9225c6cca92f741ccafd2f90a618
Component: engine
This fix tries to address the issue raised in 25545 where
volume options at the creation time is not showed up
in `docker volume inspect`.
This fix adds the field `Options` in `Volume` type and
persist the options in volume db so that `volume inspect`
could display the options.
This fix adds a couple of test cases to cover the changes.
This fix fixes 25545.
Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
Upstream-commit: 9ce8aac55e6df65bbf49c682374871a94d379bf3
Component: engine
Found a couple of places where pretty low level errors were never being
wrapped with any sort of context.
For example, if you try to create a local volume using some bad mount
options, the kernel will return `invalid argument` when we try to mount
it at container start.
What would happen is a user would `docker run` with this volume and get
an error like `Error response from daemon: invalid argument`.
This uses github.com/pkg/errors to provide some context to the error
message without masking the original error.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Upstream-commit: 2a5e85e2e871bf037c976471cf8a14da22ddd9a9
Component: engine
As part of making graphdrivers support pluginv2, a PluginGetter
interface was necessary for cleaner separation and avoiding import
cycles.
This commit creates a PluginGetter interface and makes pluginStore
implement it. Then the pluginStore object is created in the daemon
(rather than by the plugin manager) and passed to plugin init as
well as to the different subsystems (eg. graphdrivers, volumedrivers).
A side effect of this change was that some code was moved out of
experimental. This is good, since plugin support will be stable soon.
Signed-off-by: Anusha Ragunathan <anusha@docker.com>
Upstream-commit: fefea805e930a67fb6327f8e59415932861358cb
Component: engine
`Mounts` allows users to specify in a much safer way the volumes they
want to use in the container.
This replaces `Binds` and `Volumes`, which both still exist, but
`Mounts` and `Binds`/`Volumes` are exclussive.
The CLI will continue to use `Binds` and `Volumes` due to concerns with
parsing the volume specs on the client side and cross-platform support
(for now).
The new API follows exactly the services mount API.
Example usage of `Mounts`:
```
$ curl -XPOST localhost:2375/containers/create -d '{
"Image": "alpine:latest",
"HostConfig": {
"Mounts": [{
"Type": "Volume",
"Target": "/foo"
},{
"Type": "bind",
"Source": "/var/run/docker.sock",
"Target": "/var/run/docker.sock",
},{
"Type": "volume",
"Name": "important_data",
"Target": "/var/data",
"ReadOnly": true,
"VolumeOptions": {
"DriverConfig": {
Name: "awesomeStorage",
Options: {"size": "10m"},
Labels: {"some":"label"}
}
}]
}
}'
```
There are currently 2 types of mounts:
- **bind**: Paths on the host that get mounted into the
container. Paths must exist prior to creating the container.
- **volume**: Volumes that persist after the
container is removed.
Not all fields are available in each type, and validation is done to
ensure these fields aren't mixed up between types.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Upstream-commit: fc7b904dced4d18d49c8a6c47ae3f415d16d0c43
Component: engine
