Merge pull request #421 from docker/v12

[17.12] bump version to 17.12.1-ce-rc1

CHANGELOG.md

@@ -5,8 +5,44 @@ information on the list of deprecated flags and APIs please have a look at
 https://docs.docker.com/engine/deprecated/ where target removal dates can also
 be found.
 
-## 17.12.0-ce (2017-12-DD)
+**IMPORTANT**: You must stop all containers and plugins **BEFORE** upgrading to Docker CE 17.12.
+See related PR: [moby/moby#35812](https://github.com/moby/moby/pull/35812) 
 
+## 17.12.1-ce (2017-02-DD)
+
+### Client
+- Fix `node-generic-resource` typo [moby/moby#35970](https://github.com/moby/moby/pull/35970) and [moby/moby#36125](https://github.com/moby/moby/pull/36125)
+* Return errors from daemon on stack deploy configs create/update [docker/cli#757](https://github.com/docker/cli/pull/757)
+
+### Logging
+- awslogs: fix batch size calculation for large logs [moby/moby#35726](https://github.com/moby/moby/pull/35726)
+
+### Networking
+- Fix ingress network when upgrading from 17.09 to 17.12 [moby/moby#36003](https://github.com/moby/moby/pull/36003)
+* Add verbose info to partial overlay ID [moby/moby#35989](https://github.com/moby/moby/pull/35989)
+
+### Packaging
+- Set TasksMax in docker.service [docker/docker-ce-packaging#78](https://github.com/docker/docker-ce-packaging/pull/78)
+
+### Runtime
+* Bump Golang to 1.9.4
+* Bump containerd to 1.0.1
+- Fix dockerd not being able to reconnect to containerd when it is restarted [moby/moby#36173](https://github.com/moby/moby/pull/36173)
+- Fix containerd events from being processed twice [moby/moby#35891](https://github.com/moby/moby/issues/35891)
+- Fix vfs graph driver failure to initialize because of failure to setup fs quota [moby/moby#35827](https://github.com/moby/moby/pull/35827)
+- Fix regression of health check not using container's working directory [moby/moby#35845](https://github.com/moby/moby/pull/35845)
+- Honor `DOCKER_RAMDISK` with containerd 1.0 [moby/moby#35957](https://github.com/moby/moby/pull/35957)
+- Update runc to fix hang during start and exec [moby/moby#36097](https://github.com/moby/moby/pull/36097) 
+- Windows: Vendor of Microsoft/hcsshim @v.0.6.8 partial fix for import layer failing [moby/moby#35924](https://github.com/moby/moby/pull/35924)
+
+## 17.12.0-ce (2017-12-27)
+
+## Known Issues
+* AWS logs batch size calculation [moby/moby#35726](https://github.com/moby/moby/pull/35726)
+* Health check no longer uses the container's working directory [moby/moby#35843](https://github.com/moby/moby/issues/35843) 
+* Errors not returned from client in stack deploy configs [moby/moby#757](https://github.com/docker/cli/pull/757)
+* Daemon aborts when project quota fails [moby/moby#35827](https://github.com/moby/moby/pull/35827) 
+* Docker cannot use memory limit when using systemd options [moby/moby#35123](https://github.com/moby/moby/issues/35123)
 
 ### Builder
 
@@ -25,6 +61,8 @@ be found.
 - Fix behaviour of `rmi -f` with unexpected errors [docker/cli#654](https://github.com/docker/cli/pull/654)
 * Integrated Generic resource in service create [docker/cli#429](https://github.com/docker/cli/pull/429)
 - Fix external networks in stacks [docker/cli#743](https://github.com/docker/cli/pull/743)
+* Remove support for referencing images by image shortid [docker/cli#753](https://github.com/docker/cli/pull/753) and [moby/moby#35790](https://github.com/moby/moby/pull/35790)
+* Use commit-sha instead of tag for containerd [moby/moby#35770](https://github.com/moby/moby/pull/35770)
 
 ### Documentation
 
@@ -49,6 +87,7 @@ be found.
 - Fix timeout on netlink sockets and watchmiss leak [moby/moby#35677](https://github.com/moby/moby/pull/35677)
 + New daemon config for networking diagnosis [moby/moby#35677](https://github.com/moby/moby/pull/35677)
 - Clean up node management logic [docker/libnetwork#2036](https://github.com/docker/libnetwork/pull/2036)
+- Allocate VIPs when endpoints are restored [docker/swarmkit#2474](https://github.com/docker/swarmkit/pull/2474)
 
 ### Runtime
 
@@ -81,13 +120,19 @@ be found.
 * Create labels when volume exists only remotely [moby/moby#34896](https://github.com/moby/moby/pull/34896)
 - Fix leaking container/exec state [moby/moby#35484](https://github.com/moby/moby/pull/35484)
 * Disallow using legacy (v1) registries [moby/moby#35751](https://github.com/moby/moby/pull/35751) and [docker/cli#747](https://github.com/docker/cli/pull/747)
+- Windows: Fix case insensitive filename matching against builder cache [moby/moby#35793](https://github.com/moby/moby/pull/35793)
+- Fix race conditions around process handling and error checks [moby/moby#35809](https://github.com/moby/moby/pull/35809)
+* Ensure containers are stopped on daemon startup [moby/moby#35805](https://github.com/moby/moby/pull/35805)
+* Follow containerd namespace conventions [moby/moby#35812](https://github.com/moby/moby/pull/35812)
 
 ### Swarm Mode
 
 + Added support for swarm service isolation mode [moby/moby#34424](https://github.com/moby/moby/pull/34424)
+- Fix task clean up for tasks that are complete [docker/swarmkit#2477](https://github.com/docker/swarmkit/pull/2477)
 
 ### Packaging
 
 + Add Packaging for Fedora 27 [docker/docker-ce-packaging#59](https://github.com/docker/docker-ce-packaging/pull/59)
 * Change default versioning scheme to 0.0.0-dev unless specified for packaging [docker/docker-ce-packaging#67](https://github.com/docker/docker-ce-packaging/pull/67)
 * Pass Version to engine static builds [docker/docker-ce-packaging#70](https://github.com/docker/docker-ce-packaging/pull/70)
++ Added support for aarch64 on Debian (stretch/jessie) and Ubuntu Zesty or newer [docker/docker-ce-packaging#35](https://github.com/docker/docker-ce-packaging/pull/35)

VERSION

@@ -1 +1 @@
-17.12.0-ce-rc3
+17.12.1-ce-rc1

components/cli/VERSION

@@ -1 +1 @@
-17.12.0-ce-rc3
+17.12.1-ce-rc1

components/cli/cli/command/stack/deploy_composefile.go

@@ -248,13 +248,13 @@ func createConfigs(
 		case err == nil:
 			// config already exists, then we update that
 			if err := client.ConfigUpdate(ctx, config.ID, config.Meta.Version, configSpec); err != nil {
-				errors.Wrapf(err, "failed to update config %s", configSpec.Name)
+				return errors.Wrapf(err, "failed to update config %s", configSpec.Name)
 			}
 		case apiclient.IsErrNotFound(err):
 			// config does not exist, then we create a new one.
 			fmt.Fprintf(dockerCli.Out(), "Creating config %s\n", configSpec.Name)
 			if _, err := client.ConfigCreate(ctx, configSpec); err != nil {
-				errors.Wrapf(err, "failed to create config %s", configSpec.Name)
+				return errors.Wrapf(err, "failed to create config %s", configSpec.Name)
 			}
 		default:
 			return err

components/cli/contrib/completion/bash/docker

@@ -2220,6 +2220,7 @@ _docker_daemon() {
 		--metrics-addr
 		--mtu
 		--network-control-plane-mtu
+		--node-generic-resource
 		--oom-score-adjust
 		--pidfile -p
 		--registry-mirror
@@ -2865,6 +2866,7 @@ _docker_inspect() {
 						$(__docker_services)
 						$(__docker_volumes)
 					" -- "$cur" ) )
+					__ltrim_colon_completions "$cur"
 					;;
 				container)
 					__docker_complete_containers_all
@@ -3395,6 +3397,7 @@ _docker_service_update_and_create() {
 			--dns-option
 			--dns-search
 			--env-file
+			--generic-resource
 			--group
 			--host
 			--mode
@@ -3456,6 +3459,8 @@ _docker_service_update_and_create() {
 			--dns-rm
 			--dns-search-add
 			--dns-search-rm
+			--generic-resource-add
+			--generic-resource-rm
 			--group-add
 			--group-rm
 			--host-add

components/cli/contrib/completion/zsh/_docker

@@ -745,7 +745,7 @@ __docker_container_subcommand() {
                 "($help)--privileged[Give extended Linux capabilities to the command]" \
                 "($help -t --tty)"{-t,--tty}"[Allocate a pseudo-tty]" \
                 "($help -u --user)"{-u=,--user=}"[Username or UID]:user:_users" \
-                "($help -w --workdir)"{-w=,--workdir=}"[Working directory inside the container]:directory:_directories"
+                "($help -w --workdir)"{-w=,--workdir=}"[Working directory inside the container]:directory:_directories" \
                 "($help -):containers:__docker_complete_running_containers" \
                 "($help -)*::command:->anycommand" && ret=0
             case $state in

components/cli/dockerfiles/Dockerfile.cross

@@ -1,3 +1,3 @@
-FROM    dockercore/golang-cross@sha256:2e843a0e4d82b6bab34d2cb7abe26d1a6cda23226ecc3869100c8db553603f9b
+FROM    dockercore/golang-cross:1.9.4@sha256:b8d43ef11ccaa15bec63a1f1fd0c28a0e729074aa62fcfa51f0a5888f3571315
 ENV     DISABLE_WARN_OUTSIDE_CONTAINER=1
 WORKDIR /go/src/github.com/docker/cli

components/cli/dockerfiles/Dockerfile.dev

@@ -1,5 +1,5 @@
 
-FROM    golang:1.9.2-alpine3.6
+FROM    golang:1.9.4-alpine3.6
 
 RUN     apk add -U git make bash coreutils ca-certificates
 

components/cli/dockerfiles/Dockerfile.lint

@@ -1,4 +1,4 @@
-FROM    golang:1.9.2-alpine3.6
+FROM    golang:1.9.4-alpine3.6
 
 RUN     apk add -U git
 

components/cli/docs/deprecated.md

@@ -74,9 +74,13 @@ The `filter` param to filter the list of image by reference (name or name:tag) i
 ### `repository:shortid` image references
 **Deprecated In Release: [v1.13.0](https://github.com/docker/docker/releases/tag/v1.13.0)**
 
-**Target For Removal In Release: v17.12**
+**Removed In Release: v17.12**
 
-`repository:shortid` syntax for referencing images is very little used, collides with tag references can be confused with digest references.
+The `repository:shortid` syntax for referencing images is very little used,
+collides with tag references, and can be confused with digest references.
+
+Support for the `repository:shortid` notation to reference images was removed
+in Docker 17.12.
 
 ### `docker daemon` subcommand
 **Deprecated In Release: [v1.13.0](https://github.com/docker/docker/releases/tag/v1.13.0)**

components/cli/docs/reference/commandline/run.md

@@ -588,11 +588,11 @@ Use Docker's `--restart` to specify a container's *restart policy*. A restart
 policy controls whether the Docker daemon restarts a container after exit.
 Docker supports the following restart policies:
 
-| Policy    | Result                                                                                                                                                                                                                                                           |
-|:----------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| `no`      | Do not automatically restart the container when it exits. This is the default.                                                                                                                                                                                   |
-| `failure` | Restart only if the container exits with a non-zero exit status. Optionally, limit the number of restart retries the Docker daemon attempts.                                                                                                                     |
-| `always`  | Always restart the container regardless of the exit status. When you specify always, the Docker daemon will try to restart the container indefinitely. The container will also always start on daemon startup, regardless of the current state of the container. |
+| Policy                     | Result                                                                                                                                                                                                                                                           |
+|:---------------------------|:-----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| `no`                       | Do not automatically restart the container when it exits. This is the default.                                                                                                                                                                                   |
+| `on-failure[:max-retries]` | Restart only if the container exits with a non-zero exit status. Optionally, limit the number of restart retries the Docker daemon attempts.                                                                                                                     |
+| `always`                   | Always restart the container regardless of the exit status. When you specify always, the Docker daemon will try to restart the container indefinitely. The container will also always start on daemon startup, regardless of the current state of the container. |
 
 ```bash
 $ docker run --restart=always redis

components/engine/Dockerfile

@@ -87,7 +87,7 @@ RUN apt-get update && apt-get install -y \
 #            will need updating, to avoid errors. Ping #docker-maintainers on IRC
 #            with a heads-up.
 # IMPORTANT: When updating this please note that stdlib archive/tar pkg is vendored
-ENV GO_VERSION 1.9.2
+ENV GO_VERSION 1.9.4
 RUN curl -fsSL "https://golang.org/dl/go${GO_VERSION}.linux-amd64.tar.gz" \
 	| tar -xzC /usr/local
 

components/engine/Dockerfile.aarch64

@@ -73,7 +73,7 @@ RUN apt-get update && apt-get install -y \
 
 # Install Go
 # IMPORTANT: When updating this please note that stdlib archive/tar pkg is vendored
-ENV GO_VERSION 1.9.2
+ENV GO_VERSION 1.9.4
 RUN curl -fsSL "https://golang.org/dl/go${GO_VERSION}.linux-arm64.tar.gz" \
 	| tar -xzC /usr/local
 

components/engine/Dockerfile.armhf

@@ -63,7 +63,7 @@ RUN apt-get update && apt-get install -y \
 
 # Install Go
 # IMPORTANT: When updating this please note that stdlib archive/tar pkg is vendored
-ENV GO_VERSION 1.9.2
+ENV GO_VERSION 1.9.4
 RUN curl -fsSL "https://golang.org/dl/go${GO_VERSION}.linux-armv6l.tar.gz" \
 	| tar -xzC /usr/local
 ENV PATH /go/bin:/usr/local/go/bin:$PATH

components/engine/Dockerfile.e2e

@@ -1,5 +1,5 @@
 ## Step 1: Build tests
-FROM golang:1.9.2-alpine3.6 as builder
+FROM golang:1.9.4-alpine3.6 as builder
 
 RUN apk add --update \
     bash \

components/engine/Dockerfile.ppc64le

@@ -64,7 +64,7 @@ RUN apt-get update && apt-get install -y \
 # Install Go
 # NOTE: official ppc64le go binaries weren't available until go 1.6.4 and 1.7.4
 # IMPORTANT: When updating this please note that stdlib archive/tar pkg is vendored
-ENV GO_VERSION 1.9.2
+ENV GO_VERSION 1.9.4
 RUN curl -fsSL "https://golang.org/dl/go${GO_VERSION}.linux-ppc64le.tar.gz" \
 	| tar -xzC /usr/local
 

components/engine/Dockerfile.s390x

@@ -58,7 +58,7 @@ RUN apt-get update && apt-get install -y \
 	--no-install-recommends
 
 # IMPORTANT: When updating this please note that stdlib archive/tar pkg is vendored
-ENV GO_VERSION 1.9.2
+ENV GO_VERSION 1.9.4
 RUN curl -fsSL "https://golang.org/dl/go${GO_VERSION}.linux-s390x.tar.gz" \
 	| tar -xzC /usr/local
 

components/engine/Dockerfile.simple

@@ -40,7 +40,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
 #            will need updating, to avoid errors. Ping #docker-maintainers on IRC
 #            with a heads-up.
 # IMPORTANT: When updating this please note that stdlib archive/tar pkg is vendored
-ENV GO_VERSION 1.9.2
+ENV GO_VERSION 1.9.4
 RUN curl -fsSL "https://golang.org/dl/go${GO_VERSION}.linux-amd64.tar.gz" \
 	| tar -xzC /usr/local
 ENV PATH /go/bin:/usr/local/go/bin:$PATH

components/engine/Dockerfile.windows

@@ -161,7 +161,7 @@ SHELL ["powershell", "-Command", "$ErrorActionPreference = 'Stop'; $ProgressPref
 # Environment variable notes:
 #  - GO_VERSION must be consistent with 'Dockerfile' used by Linux.
 #  - FROM_DOCKERFILE is used for detection of building within a container.
-ENV GO_VERSION=1.9.2 `
+ENV GO_VERSION=1.9.4 `
     GIT_VERSION=2.11.1 `
     GOPATH=C:\go `
     FROM_DOCKERFILE=1

components/engine/api/server/router/network/network_routes.go

@@ -177,6 +177,13 @@ func (n *networkRouter) getNetwork(ctx context.Context, w http.ResponseWriter, r
 		// return the network. Skipped using isMatchingScope because it is true if the scope
 		// is not set which would be case if the client API v1.30
 		if strings.HasPrefix(nwk.ID, term) || (netconst.SwarmScope == scope) {
+			// If we have a previous match "backend", return it, we need verbose when enabled
+			// ex: overlay/partial_ID or name/swarm_scope
+			if nwv, ok := listByPartialID[nwk.ID]; ok {
+				nwk = nwv
+			} else if nwv, ok := listByFullName[nwk.ID]; ok {
+				nwk = nwv
+			}
 			return httputils.WriteJSON(w, http.StatusOK, nwk)
 		}
 	}

components/engine/builder/dockerfile/internals.go

@@ -11,7 +11,6 @@ import (
 	"os"
 	"path"
 	"path/filepath"
-	"strconv"
 	"strings"
 
 	"github.com/docker/docker/api/types"
@@ -23,10 +22,8 @@ import (
 	"github.com/docker/docker/pkg/containerfs"
 	"github.com/docker/docker/pkg/idtools"
 	"github.com/docker/docker/pkg/stringid"
-	"github.com/docker/docker/pkg/symlink"
 	"github.com/docker/docker/pkg/system"
 	"github.com/docker/go-connections/nat"
-	lcUser "github.com/opencontainers/runc/libcontainer/user"
 	"github.com/pkg/errors"
 )
 
@@ -216,82 +213,6 @@ func (b *Builder) performCopy(state *dispatchState, inst copyInstruction) error
 	return b.exportImage(state, imageMount, runConfigWithCommentCmd)
 }
 
-func parseChownFlag(chown, ctrRootPath string, idMappings *idtools.IDMappings) (idtools.IDPair, error) {
-	var userStr, grpStr string
-	parts := strings.Split(chown, ":")
-	if len(parts) > 2 {
-		return idtools.IDPair{}, errors.New("invalid chown string format: " + chown)
-	}
-	if len(parts) == 1 {
-		// if no group specified, use the user spec as group as well
-		userStr, grpStr = parts[0], parts[0]
-	} else {
-		userStr, grpStr = parts[0], parts[1]
-	}
-
-	passwdPath, err := symlink.FollowSymlinkInScope(filepath.Join(ctrRootPath, "etc", "passwd"), ctrRootPath)
-	if err != nil {
-		return idtools.IDPair{}, errors.Wrapf(err, "can't resolve /etc/passwd path in container rootfs")
-	}
-	groupPath, err := symlink.FollowSymlinkInScope(filepath.Join(ctrRootPath, "etc", "group"), ctrRootPath)
-	if err != nil {
-		return idtools.IDPair{}, errors.Wrapf(err, "can't resolve /etc/group path in container rootfs")
-	}
-	uid, err := lookupUser(userStr, passwdPath)
-	if err != nil {
-		return idtools.IDPair{}, errors.Wrapf(err, "can't find uid for user "+userStr)
-	}
-	gid, err := lookupGroup(grpStr, groupPath)
-	if err != nil {
-		return idtools.IDPair{}, errors.Wrapf(err, "can't find gid for group "+grpStr)
-	}
-
-	// convert as necessary because of user namespaces
-	chownPair, err := idMappings.ToHost(idtools.IDPair{UID: uid, GID: gid})
-	if err != nil {
-		return idtools.IDPair{}, errors.Wrapf(err, "unable to convert uid/gid to host mapping")
-	}
-	return chownPair, nil
-}
-
-func lookupUser(userStr, filepath string) (int, error) {
-	// if the string is actually a uid integer, parse to int and return
-	// as we don't need to translate with the help of files
-	uid, err := strconv.Atoi(userStr)
-	if err == nil {
-		return uid, nil
-	}
-	users, err := lcUser.ParsePasswdFileFilter(filepath, func(u lcUser.User) bool {
-		return u.Name == userStr
-	})
-	if err != nil {
-		return 0, err
-	}
-	if len(users) == 0 {
-		return 0, errors.New("no such user: " + userStr)
-	}
-	return users[0].Uid, nil
-}
-
-func lookupGroup(groupStr, filepath string) (int, error) {
-	// if the string is actually a gid integer, parse to int and return
-	// as we don't need to translate with the help of files
-	gid, err := strconv.Atoi(groupStr)
-	if err == nil {
-		return gid, nil
-	}
-	groups, err := lcUser.ParseGroupFileFilter(filepath, func(g lcUser.Group) bool {
-		return g.Name == groupStr
-	})
-	if err != nil {
-		return 0, err
-	}
-	if len(groups) == 0 {
-		return 0, errors.New("no such group: " + groupStr)
-	}
-	return groups[0].Gid, nil
-}
-
 func createDestInfo(workingDir string, inst copyInstruction, imageMount *imageMount, platform string) (copyInfo, error) {
 	// Twiddle the destination when it's a relative path - meaning, make it
 	// relative to the WORKINGDIR

components/engine/builder/dockerfile/internals_linux.go

@@ -0,0 +1,88 @@
+package dockerfile
+
+import (
+	"path/filepath"
+	"strconv"
+	"strings"
+
+	"github.com/docker/docker/pkg/idtools"
+	"github.com/docker/docker/pkg/symlink"
+	lcUser "github.com/opencontainers/runc/libcontainer/user"
+	"github.com/pkg/errors"
+)
+
+func parseChownFlag(chown, ctrRootPath string, idMappings *idtools.IDMappings) (idtools.IDPair, error) {
+	var userStr, grpStr string
+	parts := strings.Split(chown, ":")
+	if len(parts) > 2 {
+		return idtools.IDPair{}, errors.New("invalid chown string format: " + chown)
+	}
+	if len(parts) == 1 {
+		// if no group specified, use the user spec as group as well
+		userStr, grpStr = parts[0], parts[0]
+	} else {
+		userStr, grpStr = parts[0], parts[1]
+	}
+
+	passwdPath, err := symlink.FollowSymlinkInScope(filepath.Join(ctrRootPath, "etc", "passwd"), ctrRootPath)
+	if err != nil {
+		return idtools.IDPair{}, errors.Wrapf(err, "can't resolve /etc/passwd path in container rootfs")
+	}
+	groupPath, err := symlink.FollowSymlinkInScope(filepath.Join(ctrRootPath, "etc", "group"), ctrRootPath)
+	if err != nil {
+		return idtools.IDPair{}, errors.Wrapf(err, "can't resolve /etc/group path in container rootfs")
+	}
+	uid, err := lookupUser(userStr, passwdPath)
+	if err != nil {
+		return idtools.IDPair{}, errors.Wrapf(err, "can't find uid for user "+userStr)
+	}
+	gid, err := lookupGroup(grpStr, groupPath)
+	if err != nil {
+		return idtools.IDPair{}, errors.Wrapf(err, "can't find gid for group "+grpStr)
+	}
+
+	// convert as necessary because of user namespaces
+	chownPair, err := idMappings.ToHost(idtools.IDPair{UID: uid, GID: gid})
+	if err != nil {
+		return idtools.IDPair{}, errors.Wrapf(err, "unable to convert uid/gid to host mapping")
+	}
+	return chownPair, nil
+}
+
+func lookupUser(userStr, filepath string) (int, error) {
+	// if the string is actually a uid integer, parse to int and return
+	// as we don't need to translate with the help of files
+	uid, err := strconv.Atoi(userStr)
+	if err == nil {
+		return uid, nil
+	}
+	users, err := lcUser.ParsePasswdFileFilter(filepath, func(u lcUser.User) bool {
+		return u.Name == userStr
+	})
+	if err != nil {
+		return 0, err
+	}
+	if len(users) == 0 {
+		return 0, errors.New("no such user: " + userStr)
+	}
+	return users[0].Uid, nil
+}
+
+func lookupGroup(groupStr, filepath string) (int, error) {
+	// if the string is actually a gid integer, parse to int and return
+	// as we don't need to translate with the help of files
+	gid, err := strconv.Atoi(groupStr)
+	if err == nil {
+		return gid, nil
+	}
+	groups, err := lcUser.ParseGroupFileFilter(filepath, func(g lcUser.Group) bool {
+		return g.Name == groupStr
+	})
+	if err != nil {
+		return 0, err
+	}
+	if len(groups) == 0 {
+		return 0, errors.New("no such group: " + groupStr)
+	}
+	return groups[0].Gid, nil
+}

components/engine/builder/dockerfile/internals_linux_test.go

@@ -0,0 +1,138 @@
+package dockerfile
+
+import (
+	"os"
+	"path/filepath"
+	"testing"
+
+	"github.com/docker/docker/pkg/idtools"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+func TestChownFlagParsing(t *testing.T) {
+	testFiles := map[string]string{
+		"passwd": `root:x:0:0::/bin:/bin/false
+bin:x:1:1::/bin:/bin/false
+wwwwww:x:21:33::/bin:/bin/false
+unicorn:x:1001:1002::/bin:/bin/false
+		`,
+		"group": `root:x:0:
+bin:x:1:
+wwwwww:x:33:
+unicorn:x:1002:
+somegrp:x:5555:
+othergrp:x:6666:
+		`,
+	}
+	// test mappings for validating use of maps
+	idMaps := []idtools.IDMap{
+		{
+			ContainerID: 0,
+			HostID:      100000,
+			Size:        65536,
+		},
+	}
+	remapped := idtools.NewIDMappingsFromMaps(idMaps, idMaps)
+	unmapped := &idtools.IDMappings{}
+
+	contextDir, cleanup := createTestTempDir(t, "", "builder-chown-parse-test")
+	defer cleanup()
+
+	if err := os.Mkdir(filepath.Join(contextDir, "etc"), 0755); err != nil {
+		t.Fatalf("error creating test directory: %v", err)
+	}
+
+	for filename, content := range testFiles {
+		createTestTempFile(t, filepath.Join(contextDir, "etc"), filename, content, 0644)
+	}
+
+	// positive tests
+	for _, testcase := range []struct {
+		name      string
+		chownStr  string
+		idMapping *idtools.IDMappings
+		expected  idtools.IDPair
+	}{
+		{
+			name:      "UIDNoMap",
+			chownStr:  "1",
+			idMapping: unmapped,
+			expected:  idtools.IDPair{UID: 1, GID: 1},
+		},
+		{
+			name:      "UIDGIDNoMap",
+			chownStr:  "0:1",
+			idMapping: unmapped,
+			expected:  idtools.IDPair{UID: 0, GID: 1},
+		},
+		{
+			name:      "UIDWithMap",
+			chownStr:  "0",
+			idMapping: remapped,
+			expected:  idtools.IDPair{UID: 100000, GID: 100000},
+		},
+		{
+			name:      "UIDGIDWithMap",
+			chownStr:  "1:33",
+			idMapping: remapped,
+			expected:  idtools.IDPair{UID: 100001, GID: 100033},
+		},
+		{
+			name:      "UserNoMap",
+			chownStr:  "bin:5555",
+			idMapping: unmapped,
+			expected:  idtools.IDPair{UID: 1, GID: 5555},
+		},
+		{
+			name:      "GroupWithMap",
+			chownStr:  "0:unicorn",
+			idMapping: remapped,
+			expected:  idtools.IDPair{UID: 100000, GID: 101002},
+		},
+		{
+			name:      "UserOnlyWithMap",
+			chownStr:  "unicorn",
+			idMapping: remapped,
+			expected:  idtools.IDPair{UID: 101001, GID: 101002},
+		},
+	} {
+		t.Run(testcase.name, func(t *testing.T) {
+			idPair, err := parseChownFlag(testcase.chownStr, contextDir, testcase.idMapping)
+			require.NoError(t, err, "Failed to parse chown flag: %q", testcase.chownStr)
+			assert.Equal(t, testcase.expected, idPair, "chown flag mapping failure")
+		})
+	}
+
+	// error tests
+	for _, testcase := range []struct {
+		name      string
+		chownStr  string
+		idMapping *idtools.IDMappings
+		descr     string
+	}{
+		{
+			name:      "BadChownFlagFormat",
+			chownStr:  "bob:1:555",
+			idMapping: unmapped,
+			descr:     "invalid chown string format: bob:1:555",
+		},
+		{
+			name:      "UserNoExist",
+			chownStr:  "bob",
+			idMapping: unmapped,
+			descr:     "can't find uid for user bob: no such user: bob",
+		},
+		{
+			name:      "GroupNoExist",
+			chownStr:  "root:bob",
+			idMapping: unmapped,
+			descr:     "can't find gid for group bob: no such group: bob",
+		},
+	} {
+		t.Run(testcase.name, func(t *testing.T) {
+			_, err := parseChownFlag(testcase.chownStr, contextDir, testcase.idMapping)
+			assert.EqualError(t, err, testcase.descr, "Expected error string doesn't match")
+		})
+	}
+}

components/engine/builder/dockerfile/internals_test.go

@@ -2,8 +2,6 @@ package dockerfile
 
 import (
 	"fmt"
-	"os"
-	"path/filepath"
 	"runtime"
 	"testing"
 
@@ -13,7 +11,6 @@ import (
 	"github.com/docker/docker/builder"
 	"github.com/docker/docker/builder/remotecontext"
 	"github.com/docker/docker/pkg/archive"
-	"github.com/docker/docker/pkg/idtools"
 	"github.com/docker/go-connections/nat"
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
@@ -171,130 +168,3 @@ func TestDeepCopyRunConfig(t *testing.T) {
 	copy.Shell[0] = "sh"
 	assert.Equal(t, fullMutableRunConfig(), runConfig)
 }
-
-func TestChownFlagParsing(t *testing.T) {
-	testFiles := map[string]string{
-		"passwd": `root:x:0:0::/bin:/bin/false
-bin:x:1:1::/bin:/bin/false
-wwwwww:x:21:33::/bin:/bin/false
-unicorn:x:1001:1002::/bin:/bin/false
-		`,
-		"group": `root:x:0:
-bin:x:1:
-wwwwww:x:33:
-unicorn:x:1002:
-somegrp:x:5555:
-othergrp:x:6666:
-		`,
-	}
-	// test mappings for validating use of maps
-	idMaps := []idtools.IDMap{
-		{
-			ContainerID: 0,
-			HostID:      100000,
-			Size:        65536,
-		},
-	}
-	remapped := idtools.NewIDMappingsFromMaps(idMaps, idMaps)
-	unmapped := &idtools.IDMappings{}
-
-	contextDir, cleanup := createTestTempDir(t, "", "builder-chown-parse-test")
-	defer cleanup()
-
-	if err := os.Mkdir(filepath.Join(contextDir, "etc"), 0755); err != nil {
-		t.Fatalf("error creating test directory: %v", err)
-	}
-
-	for filename, content := range testFiles {
-		createTestTempFile(t, filepath.Join(contextDir, "etc"), filename, content, 0644)
-	}
-
-	// positive tests
-	for _, testcase := range []struct {
-		name      string
-		chownStr  string
-		idMapping *idtools.IDMappings
-		expected  idtools.IDPair
-	}{
-		{
-			name:      "UIDNoMap",
-			chownStr:  "1",
-			idMapping: unmapped,
-			expected:  idtools.IDPair{UID: 1, GID: 1},
-		},
-		{
-			name:      "UIDGIDNoMap",
-			chownStr:  "0:1",
-			idMapping: unmapped,
-			expected:  idtools.IDPair{UID: 0, GID: 1},
-		},
-		{
-			name:      "UIDWithMap",
-			chownStr:  "0",
-			idMapping: remapped,
-			expected:  idtools.IDPair{UID: 100000, GID: 100000},
-		},
-		{
-			name:      "UIDGIDWithMap",
-			chownStr:  "1:33",
-			idMapping: remapped,
-			expected:  idtools.IDPair{UID: 100001, GID: 100033},
-		},
-		{
-			name:      "UserNoMap",
-			chownStr:  "bin:5555",
-			idMapping: unmapped,
-			expected:  idtools.IDPair{UID: 1, GID: 5555},
-		},
-		{
-			name:      "GroupWithMap",
-			chownStr:  "0:unicorn",
-			idMapping: remapped,
-			expected:  idtools.IDPair{UID: 100000, GID: 101002},
-		},
-		{
-			name:      "UserOnlyWithMap",
-			chownStr:  "unicorn",
-			idMapping: remapped,
-			expected:  idtools.IDPair{UID: 101001, GID: 101002},
-		},
-	} {
-		t.Run(testcase.name, func(t *testing.T) {
-			idPair, err := parseChownFlag(testcase.chownStr, contextDir, testcase.idMapping)
-			require.NoError(t, err, "Failed to parse chown flag: %q", testcase.chownStr)
-			assert.Equal(t, testcase.expected, idPair, "chown flag mapping failure")
-		})
-	}
-
-	// error tests
-	for _, testcase := range []struct {
-		name      string
-		chownStr  string
-		idMapping *idtools.IDMappings
-		descr     string
-	}{
-		{
-			name:      "BadChownFlagFormat",
-			chownStr:  "bob:1:555",
-			idMapping: unmapped,
-			descr:     "invalid chown string format: bob:1:555",
-		},
-		{
-			name:      "UserNoExist",
-			chownStr:  "bob",
-			idMapping: unmapped,
-			descr:     "can't find uid for user bob: no such user: bob",
-		},
-		{
-			name:      "GroupNoExist",
-			chownStr:  "root:bob",
-			idMapping: unmapped,
-			descr:     "can't find gid for group bob: no such group: bob",
-		},
-	} {
-		t.Run(testcase.name, func(t *testing.T) {
-			_, err := parseChownFlag(testcase.chownStr, contextDir, testcase.idMapping)
-			assert.EqualError(t, err, testcase.descr, "Expected error string doesn't match")
-		})
-	}
-}

components/engine/builder/dockerfile/internals_windows.go

@@ -0,0 +1,7 @@
+package dockerfile
+
+import "github.com/docker/docker/pkg/idtools"
+
+func parseChownFlag(chown, ctrRootPath string, idMappings *idtools.IDMappings) (idtools.IDPair, error) {
+	return idMappings.RootPair(), nil
+}

components/engine/cmd/dockerd/config.go

@@ -67,7 +67,7 @@ func installCommonConfigFlags(conf *config.Config, flags *pflag.FlagSet) {
 
 	flags.StringVar(&conf.MetricsAddress, "metrics-addr", "", "Set default address and port to serve the metrics api on")
 
-	flags.Var(opts.NewListOptsRef(&conf.NodeGenericResources, opts.ValidateSingleGenericResource), "node-generic-resource", "Advertise user-defined resource")
+	flags.Var(opts.NewNamedListOptsRef("node-generic-resources", &conf.NodeGenericResources, opts.ValidateSingleGenericResource), "node-generic-resource", "Advertise user-defined resource")
 
 	flags.IntVar(&conf.NetworkControlPlaneMTU, "network-control-plane-mtu", config.DefaultNetworkMtu, "Network Control plane MTU")
 

components/engine/cmd/dockerd/daemon_test.go

@@ -61,6 +61,22 @@ func TestLoadDaemonCliConfigWithConflicts(t *testing.T) {
 	testutil.ErrorContains(t, err, "as a flag and in the configuration file: labels")
 }
 
+func TestLoadDaemonCliWithConflictingNodeGenericResources(t *testing.T) {
+	tempFile := fs.NewFile(t, "config", fs.WithContent(`{"node-generic-resources": ["foo=bar", "bar=baz"]}`))
+	defer tempFile.Remove()
+	configFile := tempFile.Path()
+
+	opts := defaultOptions(configFile)
+	flags := opts.flags
+
+	assert.NoError(t, flags.Set("config-file", configFile))
+	assert.NoError(t, flags.Set("node-generic-resource", "r1=bar"))
+	assert.NoError(t, flags.Set("node-generic-resource", "r2=baz"))
+
+	_, err := loadDaemonCliConfig(opts)
+	testutil.ErrorContains(t, err, "as a flag and in the configuration file: node-generic-resources")
+}
+
 func TestLoadDaemonCliWithConflictingLabels(t *testing.T) {
 	opts := defaultOptions("")
 	flags := opts.flags

components/engine/cmd/dockerd/docker_windows.go

@@ -1,18 +1,5 @@
 package main
 
 import (
-	"sync/atomic"
-
 	_ "github.com/docker/docker/autogen/winresources/dockerd"
 )
-
-//go:cgo_import_dynamic main.dummy CommandLineToArgvW%2 "shell32.dll"
-
-var dummy uintptr
-
-func init() {
-	// Ensure that this import is not removed by the linker. This is used to
-	// ensure that shell32.dll is loaded by the system loader, preventing
-	// go#15286 from triggering on Nano Server TP5.
-	atomic.LoadUintptr(&dummy)
-}

components/engine/contrib/builder/deb/aarch64/debian-jessie/Dockerfile

@@ -7,7 +7,7 @@ FROM aarch64/debian:jessie
 RUN echo deb http://ftp.debian.org/debian jessie-backports main > /etc/apt/sources.list.d/backports.list
 RUN apt-get update && apt-get install -y apparmor bash-completion btrfs-tools build-essential cmake curl ca-certificates debhelper dh-apparmor dh-systemd git libapparmor-dev libdevmapper-dev pkg-config vim-common libsystemd-journal-dev libseccomp-dev --no-install-recommends && rm -rf /var/lib/apt/lists/*
 
-ENV GO_VERSION 1.9.2
+ENV GO_VERSION 1.9.4
 RUN curl -fSL "https://golang.org/dl/go${GO_VERSION}.linux-arm64.tar.gz" | tar xzC /usr/local
 ENV PATH $PATH:/usr/local/go/bin
 

components/engine/contrib/builder/deb/aarch64/debian-stretch/Dockerfile

@@ -6,7 +6,7 @@ FROM aarch64/debian:stretch
 
 RUN apt-get update && apt-get install -y apparmor bash-completion btrfs-tools build-essential cmake curl ca-certificates debhelper dh-apparmor dh-systemd git libapparmor-dev libdevmapper-dev pkg-config vim-common libsystemd-dev libseccomp-dev --no-install-recommends && rm -rf /var/lib/apt/lists/*
 
-ENV GO_VERSION 1.9.2
+ENV GO_VERSION 1.9.4
 RUN curl -fSL "https://golang.org/dl/go${GO_VERSION}.linux-arm64.tar.gz" | tar xzC /usr/local
 ENV PATH $PATH:/usr/local/go/bin
 

components/engine/contrib/builder/deb/aarch64/ubuntu-trusty/Dockerfile

@@ -6,7 +6,7 @@ FROM aarch64/ubuntu:trusty
 
 RUN apt-get update && apt-get install -y apparmor bash-completion btrfs-tools build-essential cmake curl ca-certificates debhelper dh-apparmor dh-systemd git libapparmor-dev libdevmapper-dev pkg-config vim-common libsystemd-journal-dev --no-install-recommends && rm -rf /var/lib/apt/lists/*
 
-ENV GO_VERSION 1.9.2
+ENV GO_VERSION 1.9.4
 RUN curl -fSL "https://golang.org/dl/go${GO_VERSION}.linux-arm64.tar.gz" | tar xzC /usr/local
 ENV PATH $PATH:/usr/local/go/bin
 

components/engine/contrib/builder/deb/aarch64/ubuntu-xenial/Dockerfile

@@ -6,7 +6,7 @@ FROM aarch64/ubuntu:xenial
 
 RUN apt-get update && apt-get install -y apparmor bash-completion btrfs-tools build-essential cmake curl ca-certificates debhelper dh-apparmor dh-systemd git libapparmor-dev libdevmapper-dev pkg-config vim-common libsystemd-dev libseccomp-dev --no-install-recommends && rm -rf /var/lib/apt/lists/*
 
-ENV GO_VERSION 1.9.2
+ENV GO_VERSION 1.9.4
 RUN curl -fSL "https://golang.org/dl/go${GO_VERSION}.linux-arm64.tar.gz" | tar xzC /usr/local
 ENV PATH $PATH:/usr/local/go/bin
 

components/engine/contrib/builder/deb/amd64/debian-jessie/Dockerfile

@@ -10,7 +10,7 @@ RUN sed -ri "s/(httpredir|deb).debian.org/$APT_MIRROR/g" /etc/apt/sources.list
 
 RUN apt-get update && apt-get install -y apparmor bash-completion btrfs-tools build-essential cmake curl ca-certificates debhelper dh-apparmor dh-systemd git libapparmor-dev libdevmapper-dev  pkg-config vim-common libsystemd-journal-dev --no-install-recommends && rm -rf /var/lib/apt/lists/*
 
-ENV GO_VERSION 1.9.2
+ENV GO_VERSION 1.9.4
 RUN curl -fSL "https://golang.org/dl/go${GO_VERSION}.linux-amd64.tar.gz" | tar xzC /usr/local
 ENV PATH $PATH:/usr/local/go/bin
 

components/engine/contrib/builder/deb/amd64/debian-stretch/Dockerfile

@@ -10,7 +10,7 @@ RUN sed -ri "s/(httpredir|deb).debian.org/$APT_MIRROR/g" /etc/apt/sources.list
 
 RUN apt-get update && apt-get install -y apparmor bash-completion btrfs-tools build-essential cmake curl ca-certificates debhelper dh-apparmor dh-systemd git libapparmor-dev libdevmapper-dev libseccomp-dev pkg-config vim-common libsystemd-dev --no-install-recommends && rm -rf /var/lib/apt/lists/*
 
-ENV GO_VERSION 1.9.2
+ENV GO_VERSION 1.9.4
 RUN curl -fSL "https://golang.org/dl/go${GO_VERSION}.linux-amd64.tar.gz" | tar xzC /usr/local
 ENV PATH $PATH:/usr/local/go/bin
 

components/engine/contrib/builder/deb/amd64/debian-wheezy/Dockerfile

@@ -12,7 +12,7 @@ RUN sed -ri "s/(httpredir|deb).debian.org/$APT_MIRROR/g" /etc/apt/sources.list.d
 RUN apt-get update && apt-get install -y -t wheezy-backports btrfs-tools --no-install-recommends && rm -rf /var/lib/apt/lists/*
 RUN apt-get update && apt-get install -y apparmor bash-completion  build-essential cmake curl ca-certificates debhelper dh-apparmor dh-systemd git libapparmor-dev libdevmapper-dev  pkg-config vim-common --no-install-recommends && rm -rf /var/lib/apt/lists/*
 
-ENV GO_VERSION 1.9.2
+ENV GO_VERSION 1.9.4
 RUN curl -fSL "https://golang.org/dl/go${GO_VERSION}.linux-amd64.tar.gz" | tar xzC /usr/local
 ENV PATH $PATH:/usr/local/go/bin
 

components/engine/contrib/builder/deb/amd64/ubuntu-trusty/Dockerfile

@@ -6,7 +6,7 @@ FROM ubuntu:trusty
 
 RUN apt-get update && apt-get install -y apparmor bash-completion btrfs-tools build-essential cmake curl ca-certificates debhelper dh-apparmor dh-systemd git libapparmor-dev libdevmapper-dev  pkg-config vim-common libsystemd-journal-dev --no-install-recommends && rm -rf /var/lib/apt/lists/*
 
-ENV GO_VERSION 1.9.2
+ENV GO_VERSION 1.9.4
 RUN curl -fSL "https://golang.org/dl/go${GO_VERSION}.linux-amd64.tar.gz" | tar xzC /usr/local
 ENV PATH $PATH:/usr/local/go/bin
 

components/engine/contrib/builder/deb/amd64/ubuntu-xenial/Dockerfile

@@ -6,7 +6,7 @@ FROM ubuntu:xenial
 
 RUN apt-get update && apt-get install -y apparmor bash-completion btrfs-tools build-essential cmake curl ca-certificates debhelper dh-apparmor dh-systemd git libapparmor-dev libdevmapper-dev libseccomp-dev pkg-config vim-common libsystemd-dev --no-install-recommends && rm -rf /var/lib/apt/lists/*
 
-ENV GO_VERSION 1.9.2
+ENV GO_VERSION 1.9.4
 RUN curl -fSL "https://golang.org/dl/go${GO_VERSION}.linux-amd64.tar.gz" | tar xzC /usr/local
 ENV PATH $PATH:/usr/local/go/bin
 

components/engine/contrib/builder/deb/amd64/ubuntu-yakkety/Dockerfile

@@ -6,7 +6,7 @@ FROM ubuntu:yakkety
 
 RUN apt-get update && apt-get install -y apparmor bash-completion btrfs-tools build-essential cmake curl ca-certificates debhelper dh-apparmor dh-systemd git libapparmor-dev libdevmapper-dev libseccomp-dev pkg-config vim-common libsystemd-dev --no-install-recommends && rm -rf /var/lib/apt/lists/*
 
-ENV GO_VERSION 1.9.2
+ENV GO_VERSION 1.9.4
 RUN curl -fSL "https://golang.org/dl/go${GO_VERSION}.linux-amd64.tar.gz" | tar xzC /usr/local
 ENV PATH $PATH:/usr/local/go/bin
 

components/engine/contrib/builder/deb/amd64/ubuntu-zesty/Dockerfile

@@ -6,7 +6,7 @@ FROM ubuntu:zesty
 
 RUN apt-get update && apt-get install -y apparmor bash-completion btrfs-tools build-essential cmake curl ca-certificates debhelper dh-apparmor dh-systemd git libapparmor-dev libdevmapper-dev libseccomp-dev pkg-config vim-common libsystemd-dev --no-install-recommends && rm -rf /var/lib/apt/lists/*
 
-ENV GO_VERSION 1.9.2
+ENV GO_VERSION 1.9.4
 RUN curl -fSL "https://golang.org/dl/go${GO_VERSION}.linux-amd64.tar.gz" | tar xzC /usr/local
 ENV PATH $PATH:/usr/local/go/bin
 

components/engine/contrib/builder/deb/armhf/debian-jessie/Dockerfile

@@ -10,7 +10,7 @@ RUN sed -ri "s/(httpredir|deb).debian.org/$APT_MIRROR/g" /etc/apt/sources.list
 
 RUN apt-get update && apt-get install -y apparmor bash-completion btrfs-tools build-essential cmake curl ca-certificates debhelper dh-apparmor dh-systemd git libapparmor-dev libdevmapper-dev  pkg-config vim-common libsystemd-journal-dev --no-install-recommends && rm -rf /var/lib/apt/lists/*
 
-ENV GO_VERSION 1.9.2
+ENV GO_VERSION 1.9.4
 RUN curl -fSL "https://golang.org/dl/go${GO_VERSION}.linux-armv6l.tar.gz" | tar xzC /usr/local
 ENV PATH $PATH:/usr/local/go/bin
 

components/engine/contrib/builder/deb/armhf/raspbian-jessie/Dockerfile

@@ -10,7 +10,7 @@ RUN sed -ri "s/(httpredir|deb).debian.org/$APT_MIRROR/g" /etc/apt/sources.list
 
 RUN apt-get update && apt-get install -y apparmor bash-completion btrfs-tools build-essential cmake curl ca-certificates debhelper dh-apparmor dh-systemd git libapparmor-dev libdevmapper-dev  pkg-config vim-common libsystemd-journal-dev --no-install-recommends && rm -rf /var/lib/apt/lists/*
 
-ENV GO_VERSION 1.9.2
+ENV GO_VERSION 1.9.4
 # GOARM is the ARM architecture version which is unrelated to the above Golang version
 ENV GOARM 6
 RUN curl -fSL "https://golang.org/dl/go${GO_VERSION}.linux-armv6l.tar.gz" | tar xzC /usr/local

components/engine/contrib/builder/deb/armhf/ubuntu-trusty/Dockerfile

@@ -6,7 +6,7 @@ FROM armhf/ubuntu:trusty
 
 RUN apt-get update && apt-get install -y apparmor bash-completion btrfs-tools build-essential cmake curl ca-certificates debhelper dh-apparmor dh-systemd git libapparmor-dev libdevmapper-dev  pkg-config vim-common libsystemd-journal-dev --no-install-recommends && rm -rf /var/lib/apt/lists/*
 
-ENV GO_VERSION 1.9.2
+ENV GO_VERSION 1.9.4
 RUN curl -fSL "https://golang.org/dl/go${GO_VERSION}.linux-armv6l.tar.gz" | tar xzC /usr/local
 ENV PATH $PATH:/usr/local/go/bin
 

components/engine/contrib/builder/deb/armhf/ubuntu-xenial/Dockerfile

@@ -6,7 +6,7 @@ FROM armhf/ubuntu:xenial
 
 RUN apt-get update && apt-get install -y apparmor bash-completion btrfs-tools build-essential cmake curl ca-certificates debhelper dh-apparmor dh-systemd git libapparmor-dev libdevmapper-dev libseccomp-dev pkg-config vim-common libsystemd-dev --no-install-recommends && rm -rf /var/lib/apt/lists/*
 
-ENV GO_VERSION 1.9.2
+ENV GO_VERSION 1.9.4
 RUN curl -fSL "https://golang.org/dl/go${GO_VERSION}.linux-armv6l.tar.gz" | tar xzC /usr/local
 ENV PATH $PATH:/usr/local/go/bin
 

components/engine/contrib/builder/deb/armhf/ubuntu-yakkety/Dockerfile

@@ -6,7 +6,7 @@ FROM armhf/ubuntu:yakkety
 
 RUN apt-get update && apt-get install -y apparmor bash-completion btrfs-tools build-essential cmake curl ca-certificates debhelper dh-apparmor dh-systemd git libapparmor-dev libdevmapper-dev libseccomp-dev pkg-config vim-common libsystemd-dev --no-install-recommends && rm -rf /var/lib/apt/lists/*
 
-ENV GO_VERSION 1.9.2
+ENV GO_VERSION 1.9.4
 RUN curl -fSL "https://golang.org/dl/go${GO_VERSION}.linux-armv6l.tar.gz" | tar xzC /usr/local
 ENV PATH $PATH:/usr/local/go/bin
 

components/engine/contrib/builder/deb/ppc64le/ubuntu-trusty/Dockerfile

@@ -6,7 +6,7 @@ FROM ppc64le/ubuntu:trusty
 
 RUN apt-get update && apt-get install -y apparmor bash-completion btrfs-tools build-essential cmake curl ca-certificates debhelper dh-apparmor dh-systemd git libapparmor-dev libdevmapper-dev pkg-config vim-common libsystemd-journal-dev --no-install-recommends && rm -rf /var/lib/apt/lists/*
 
-ENV GO_VERSION 1.9.2
+ENV GO_VERSION 1.9.4
 RUN curl -fsSL "https://golang.org/dl/go${GO_VERSION}.linux-ppc64le.tar.gz" | tar xzC /usr/local
 ENV PATH $PATH:/usr/local/go/bin
 

components/engine/contrib/builder/deb/ppc64le/ubuntu-xenial/Dockerfile

@@ -6,7 +6,7 @@ FROM ppc64le/ubuntu:xenial
 
 RUN apt-get update && apt-get install -y apparmor bash-completion btrfs-tools build-essential cmake curl ca-certificates debhelper dh-apparmor dh-systemd git libapparmor-dev libdevmapper-dev pkg-config vim-common libseccomp-dev libsystemd-dev --no-install-recommends && rm -rf /var/lib/apt/lists/*
 
-ENV GO_VERSION 1.9.2
+ENV GO_VERSION 1.9.4
 RUN curl -fsSL "https://golang.org/dl/go${GO_VERSION}.linux-ppc64le.tar.gz" | tar xzC /usr/local
 ENV PATH $PATH:/usr/local/go/bin
 

components/engine/contrib/builder/deb/ppc64le/ubuntu-yakkety/Dockerfile

@@ -6,7 +6,7 @@ FROM ppc64le/ubuntu:yakkety
 
 RUN apt-get update && apt-get install -y apparmor bash-completion btrfs-tools build-essential cmake curl ca-certificates debhelper dh-apparmor dh-systemd git libapparmor-dev libdevmapper-dev pkg-config vim-common libseccomp-dev libsystemd-dev --no-install-recommends && rm -rf /var/lib/apt/lists/*
 
-ENV GO_VERSION 1.9.2
+ENV GO_VERSION 1.9.4
 RUN curl -fsSL "https://golang.org/dl/go${GO_VERSION}.linux-ppc64le.tar.gz" | tar xzC /usr/local
 ENV PATH $PATH:/usr/local/go/bin
 

components/engine/contrib/builder/deb/s390x/ubuntu-xenial/Dockerfile

@@ -6,7 +6,7 @@ FROM s390x/ubuntu:xenial
 
 RUN apt-get update && apt-get install -y apparmor bash-completion btrfs-tools build-essential cmake curl ca-certificates debhelper dh-apparmor dh-systemd git libapparmor-dev libdevmapper-dev libseccomp-dev pkg-config libsystemd-dev vim-common --no-install-recommends && rm -rf /var/lib/apt/lists/*
 
-ENV GO_VERSION 1.9.2
+ENV GO_VERSION 1.9.4
 RUN curl -fSL "https://golang.org/dl/go${GO_VERSION}.linux-s390x.tar.gz" | tar xzC /usr/local
 ENV PATH $PATH:/usr/local/go/bin
 

components/engine/contrib/builder/deb/s390x/ubuntu-yakkety/Dockerfile

@@ -6,7 +6,7 @@ FROM s390x/ubuntu:yakkety
 
 RUN apt-get update && apt-get install -y apparmor bash-completion btrfs-tools build-essential cmake curl ca-certificates debhelper dh-apparmor dh-systemd git libapparmor-dev libdevmapper-dev libseccomp-dev pkg-config libsystemd-dev vim-common --no-install-recommends && rm -rf /var/lib/apt/lists/*
 
-ENV GO_VERSION 1.9.2
+ENV GO_VERSION 1.9.4
 RUN curl -fSL "https://golang.org/dl/go${GO_VERSION}.linux-s390x.tar.gz" | tar xzC /usr/local
 ENV PATH $PATH:/usr/local/go/bin
 

components/engine/contrib/builder/rpm/amd64/amazonlinux-latest/Dockerfile

@@ -7,7 +7,7 @@ FROM amazonlinux:latest
 RUN yum groupinstall -y "Development Tools"
 RUN yum install -y btrfs-progs-devel device-mapper-devel glibc-static libseccomp-devel libselinux-devel pkgconfig selinux-policy selinux-policy-devel  tar git cmake vim-common
 
-ENV GO_VERSION 1.9.2
+ENV GO_VERSION 1.9.4
 RUN curl -fSL "https://golang.org/dl/go${GO_VERSION}.linux-amd64.tar.gz" | tar xzC /usr/local
 ENV PATH $PATH:/usr/local/go/bin
 

components/engine/contrib/builder/rpm/amd64/centos-7/Dockerfile

@@ -8,7 +8,7 @@ RUN yum groupinstall -y "Development Tools"
 RUN yum -y swap -- remove systemd-container systemd-container-libs -- install systemd systemd-libs
 RUN yum install -y btrfs-progs-devel device-mapper-devel glibc-static libseccomp-devel libselinux-devel pkgconfig selinux-policy selinux-policy-devel systemd-devel tar git cmake vim-common
 
-ENV GO_VERSION 1.9.2
+ENV GO_VERSION 1.9.4
 RUN curl -fSL "https://golang.org/dl/go${GO_VERSION}.linux-amd64.tar.gz" | tar xzC /usr/local
 ENV PATH $PATH:/usr/local/go/bin
 

components/engine/contrib/builder/rpm/amd64/fedora-24/Dockerfile

@@ -8,7 +8,7 @@ RUN dnf -y upgrade
 RUN dnf install -y @development-tools fedora-packager
 RUN dnf install -y btrfs-progs-devel device-mapper-devel glibc-static libseccomp-devel libselinux-devel pkgconfig selinux-policy selinux-policy-devel systemd-devel tar git cmake vim-common
 
-ENV GO_VERSION 1.9.2
+ENV GO_VERSION 1.9.4
 RUN curl -fSL "https://golang.org/dl/go${GO_VERSION}.linux-amd64.tar.gz" | tar xzC /usr/local
 ENV PATH $PATH:/usr/local/go/bin
 

components/engine/contrib/builder/rpm/amd64/fedora-25/Dockerfile

@@ -8,7 +8,7 @@ RUN dnf -y upgrade
 RUN dnf install -y @development-tools fedora-packager
 RUN dnf install -y btrfs-progs-devel device-mapper-devel glibc-static libseccomp-devel libselinux-devel pkgconfig selinux-policy selinux-policy-devel systemd-devel tar git cmake vim-common
 
-ENV GO_VERSION 1.9.2
+ENV GO_VERSION 1.9.4
 RUN curl -fSL "https://golang.org/dl/go${GO_VERSION}.linux-amd64.tar.gz" | tar xzC /usr/local
 ENV PATH $PATH:/usr/local/go/bin
 

components/engine/contrib/builder/rpm/amd64/opensuse-13.2/Dockerfile

@@ -7,7 +7,7 @@ FROM opensuse:13.2
 RUN zypper --non-interactive install ca-certificates* curl gzip rpm-build
 RUN zypper --non-interactive install libbtrfs-devel device-mapper-devel glibc-static  libselinux-devel pkg-config selinux-policy selinux-policy-devel systemd-devel tar git cmake vim systemd-rpm-macros
 
-ENV GO_VERSION 1.9.2
+ENV GO_VERSION 1.9.4
 RUN curl -fSL "https://golang.org/dl/go${GO_VERSION}.linux-amd64.tar.gz" | tar xzC /usr/local
 ENV PATH $PATH:/usr/local/go/bin
 

components/engine/contrib/builder/rpm/amd64/oraclelinux-6/Dockerfile

@@ -10,7 +10,7 @@ RUN yum install -y kernel-uek-devel-4.1.12-32.el6uek
 RUN yum groupinstall -y "Development Tools"
 RUN yum install -y btrfs-progs-devel device-mapper-devel glibc-static  libselinux-devel pkgconfig selinux-policy selinux-policy-devel  tar git cmake vim-common
 
-ENV GO_VERSION 1.9.2
+ENV GO_VERSION 1.9.4
 RUN curl -fSL "https://golang.org/dl/go${GO_VERSION}.linux-amd64.tar.gz" | tar xzC /usr/local
 ENV PATH $PATH:/usr/local/go/bin
 

components/engine/contrib/builder/rpm/amd64/oraclelinux-7/Dockerfile

@@ -7,7 +7,7 @@ FROM oraclelinux:7
 RUN yum groupinstall -y "Development Tools"
 RUN yum install -y --enablerepo=ol7_optional_latest btrfs-progs-devel device-mapper-devel glibc-static libseccomp-devel libselinux-devel pkgconfig selinux-policy selinux-policy-devel systemd-devel tar git cmake vim-common
 
-ENV GO_VERSION 1.9.2
+ENV GO_VERSION 1.9.4
 RUN curl -fSL "https://golang.org/dl/go${GO_VERSION}.linux-amd64.tar.gz" | tar xzC /usr/local
 ENV PATH $PATH:/usr/local/go/bin
 

components/engine/contrib/builder/rpm/amd64/photon-1.0/Dockerfile

@@ -7,7 +7,7 @@ FROM photon:1.0
 RUN tdnf install -y wget curl ca-certificates gzip make rpm-build sed gcc linux-api-headers glibc-devel binutils libseccomp elfutils
 RUN tdnf install -y btrfs-progs-devel device-mapper-devel glibc-static libseccomp-devel libselinux-devel pkg-config selinux-policy selinux-policy-devel systemd-devel tar git cmake vim-common
 
-ENV GO_VERSION 1.9.2
+ENV GO_VERSION 1.9.4
 RUN curl -fSL "https://golang.org/dl/go${GO_VERSION}.linux-amd64.tar.gz" | tar xzC /usr/local
 ENV PATH $PATH:/usr/local/go/bin
 

components/engine/contrib/builder/rpm/armhf/centos-7/Dockerfile

@@ -9,7 +9,7 @@ RUN yum groupinstall --skip-broken -y "Development Tools"
 RUN yum -y swap -- remove systemd-container systemd-container-libs -- install systemd systemd-libs
 RUN yum install -y btrfs-progs-devel device-mapper-devel glibc-static libseccomp-devel libselinux-devel pkgconfig selinux-policy selinux-policy-devel sqlite-devel systemd-devel tar git cmake vim-common
 
-ENV GO_VERSION 1.9.2
+ENV GO_VERSION 1.9.4
 RUN curl -fSL "https://golang.org/dl/go${GO_VERSION}.linux-armv6l.tar.gz" | tar xzC /usr/local
 ENV PATH $PATH:/usr/local/go/bin
 

components/engine/contrib/builder/rpm/ppc64le/centos-7/Dockerfile

@@ -8,7 +8,7 @@ RUN yum groupinstall -y "Development Tools"
 RUN yum -y swap -- remove systemd-container systemd-container-libs -- install systemd systemd-libs
 RUN yum install -y btrfs-progs-devel device-mapper-devel glibc-static libseccomp-devel libselinux-devel pkgconfig selinux-policy selinux-policy-devel sqlite-devel systemd-devel tar git cmake vim-common
 
-ENV GO_VERSION 1.9.2
+ENV GO_VERSION 1.9.4
 RUN curl -fsSL "https://golang.org/dl/go${GO_VERSION}.linux-ppc64le.tar.gz" | tar xzC /usr/local
 ENV PATH $PATH:/usr/local/go/bin
 

components/engine/contrib/builder/rpm/ppc64le/fedora-24/Dockerfile

@@ -8,7 +8,7 @@ RUN dnf -y upgrade
 RUN dnf install -y @development-tools fedora-packager
 RUN dnf install -y btrfs-progs-devel device-mapper-devel glibc-static libseccomp-devel libselinux-devel pkgconfig selinux-policy selinux-policy-devel sqlite-devel systemd-devel tar git cmake vim-common
 
-ENV GO_VERSION 1.9.2
+ENV GO_VERSION 1.9.4
 RUN curl -fsSL "https://golang.org/dl/go${GO_VERSION}.linux-ppc64le.tar.gz" | tar xzC /usr/local
 ENV PATH $PATH:/usr/local/go/bin
 

components/engine/contrib/builder/rpm/ppc64le/opensuse-42.1/Dockerfile

@@ -9,7 +9,7 @@ RUN zypper addrepo -n ppc64le-updates -f https://download.opensuse.org/ports/upd
 RUN zypper --non-interactive install ca-certificates* curl gzip rpm-build
 RUN zypper --non-interactive install libbtrfs-devel device-mapper-devel glibc-static  libselinux-devel pkg-config selinux-policy selinux-policy-devel sqlite-devel systemd-devel tar git cmake vim
 
-ENV GO_VERSION 1.9.2
+ENV GO_VERSION 1.9.4
 RUN curl -fsSL "https://golang.org/dl/go${GO_VERSION}.linux-ppc64le.tar.gz" | tar xzC /usr/local
 ENV PATH $PATH:/usr/local/go/bin
 

components/engine/contrib/builder/rpm/s390x/clefos-base-s390x-7/Dockerfile

@@ -8,7 +8,7 @@ FROM sinenomine/clefos-base-s390x
 RUN touch /var/lib/rpm/* && yum groupinstall -y "Development Tools"
 RUN touch /var/lib/rpm/* && yum install -y btrfs-progs-devel device-mapper-devel glibc-static libseccomp-devel libselinux-devel pkgconfig selinux-policy selinux-policy-devel sqlite-devel systemd-devel tar git cmake vim-common
 
-ENV GO_VERSION 1.9.2
+ENV GO_VERSION 1.9.4
 RUN curl -fsSL "https://golang.org/dl/go${GO_VERSION}.linux-s390x.tar.gz" | tar xzC /usr/local
 ENV PATH $PATH:/usr/local/go/bin
 

components/engine/contrib/builder/rpm/s390x/opensuse-tumbleweed-1/Dockerfile

@@ -9,7 +9,7 @@ RUN zypper ar https://download.opensuse.org/ports/zsystems/tumbleweed/repo/oss/
 RUN zypper --non-interactive install ca-certificates* curl gzip rpm-build
 RUN zypper --non-interactive install libbtrfs-devel device-mapper-devel glibc-static  libselinux-devel pkg-config selinux-policy selinux-policy-devel sqlite-devel systemd-devel tar git cmake vim systemd-rpm-macros
 
-ENV GO_VERSION 1.9.2
+ENV GO_VERSION 1.9.4
 RUN curl -fsSL "https://golang.org/dl/go${GO_VERSION}.linux-s390x.tar.gz" | tar xzC /usr/local
 ENV PATH $PATH:/usr/local/go/bin
 

components/engine/daemon/cluster/executor/container/executor.go

@@ -146,6 +146,11 @@ func (e *executor) Configure(ctx context.Context, node *api.Node) error {
 		attachments[na.Network.ID] = na.Addresses[0]
 	}
 
+	if (ingressNA == nil) && (node.Attachment != nil) {
+		ingressNA = node.Attachment
+		attachments[ingressNA.Network.ID] = ingressNA.Addresses[0]
+	}
+
 	if ingressNA == nil {
 		e.backend.ReleaseIngress()
 		return e.backend.GetAttachmentStore().ResetAttachments(attachments)

components/engine/daemon/daemon.go

@@ -62,8 +62,8 @@ import (
 	"github.com/pkg/errors"
 )
 
-// MainNamespace is the name of the namespace used for users containers
-const MainNamespace = "moby"
+// ContainersNamespace is the name of the namespace used for users containers
+const ContainersNamespace = "moby"
 
 var (
 	errSystemNotSupported = errors.New("the Docker daemon is not supported on this platform")
@@ -247,6 +247,11 @@ func (daemon *Daemon) restore() error {
 					logrus.WithError(err).Errorf("Failed to delete container %s from containerd", c.ID)
 					return
 				}
+			} else if !daemon.configStore.LiveRestoreEnabled {
+				if err := daemon.kill(c, c.StopSignal()); err != nil && !errdefs.IsNotFound(err) {
+					logrus.WithError(err).WithField("container", c.ID).Error("error shutting down container")
+					return
+				}
 			}
 
 			if c.IsRunning() || c.IsPaused() {
@@ -317,24 +322,24 @@ func (daemon *Daemon) restore() error {
 					activeSandboxes[c.NetworkSettings.SandboxID] = options
 					mapLock.Unlock()
 				}
-			} else {
-				// get list of containers we need to restart
+			}
 
-				// Do not autostart containers which
-				// has endpoints in a swarm scope
-				// network yet since the cluster is
-				// not initialized yet. We will start
-				// it after the cluster is
-				// initialized.
-				if daemon.configStore.AutoRestart && c.ShouldRestart() && !c.NetworkSettings.HasSwarmEndpoint {
-					mapLock.Lock()
-					restartContainers[c] = make(chan struct{})
-					mapLock.Unlock()
-				} else if c.HostConfig != nil && c.HostConfig.AutoRemove {
-					mapLock.Lock()
-					removeContainers[c.ID] = c
-					mapLock.Unlock()
-				}
+			// get list of containers we need to restart
+
+			// Do not autostart containers which
+			// has endpoints in a swarm scope
+			// network yet since the cluster is
+			// not initialized yet. We will start
+			// it after the cluster is
+			// initialized.
+			if daemon.configStore.AutoRestart && c.ShouldRestart() && !c.NetworkSettings.HasSwarmEndpoint {
+				mapLock.Lock()
+				restartContainers[c] = make(chan struct{})
+				mapLock.Unlock()
+			} else if c.HostConfig != nil && c.HostConfig.AutoRemove {
+				mapLock.Lock()
+				removeContainers[c.ID] = c
+				mapLock.Unlock()
 			}
 
 			c.Lock()
@@ -890,7 +895,7 @@ func NewDaemon(config *config.Config, registryService registry.Service, containe
 
 	go d.execCommandGC()
 
-	d.containerd, err = containerdRemote.NewClient(MainNamespace, d)
+	d.containerd, err = containerdRemote.NewClient(ContainersNamespace, d)
 	if err != nil {
 		return nil, err
 	}

components/engine/daemon/graphdriver/quota/projectquota.go

@@ -350,11 +350,17 @@ func makeBackingFsDev(home string) (string, error) {
 	backingFsBlockDev := path.Join(home, "backingFsBlockDev")
 	// Re-create just in case someone copied the home directory over to a new device
 	unix.Unlink(backingFsBlockDev)
-	if err := unix.Mknod(backingFsBlockDev, unix.S_IFBLK|0600, int(stat.Dev)); err != nil {
+	err := unix.Mknod(backingFsBlockDev, unix.S_IFBLK|0600, int(stat.Dev))
+	switch err {
+	case nil:
+		return backingFsBlockDev, nil
+
+	case unix.ENOSYS:
+		return "", ErrQuotaNotSupported
+
+	default:
 		return "", fmt.Errorf("Failed to mknod %s: %v", backingFsBlockDev, err)
 	}
-
-	return backingFsBlockDev, nil
 }
 
 func hasQuotaSupport(backingFsBlockDev string) (bool, error) {

components/engine/daemon/graphdriver/vfs/driver.go

@@ -35,9 +35,7 @@ func Init(home string, options []string, uidMaps, gidMaps []idtools.IDMap) (grap
 		return nil, err
 	}
 
-	if err := setupDriverQuota(d); err != nil {
-		return nil, err
-	}
+	setupDriverQuota(d)
 
 	return graphdriver.NewNaiveDiffDriver(d, uidMaps, gidMaps), nil
 }

components/engine/daemon/graphdriver/vfs/quota_linux.go

@@ -2,20 +2,21 @@
 
 package vfs
 
-import "github.com/docker/docker/daemon/graphdriver/quota"
+import (
+	"github.com/docker/docker/daemon/graphdriver/quota"
+	"github.com/sirupsen/logrus"
+)
 
 type driverQuota struct {
 	quotaCtl *quota.Control
 }
 
-func setupDriverQuota(driver *Driver) error {
+func setupDriverQuota(driver *Driver) {
 	if quotaCtl, err := quota.NewControl(driver.home); err == nil {
 		driver.quotaCtl = quotaCtl
 	} else if err != quota.ErrQuotaNotSupported {
-		return err
+		logrus.Warnf("Unable to setup quota: %v\n", err)
 	}
-
-	return nil
 }
 
 func (d *Driver) setupQuota(dir string, size uint64) error {

components/engine/daemon/health.go

@@ -80,6 +80,7 @@ func (p *cmdProbe) run(ctx context.Context, d *Daemon, cntr *container.Container
 	execConfig.Tty = false
 	execConfig.Privileged = false
 	execConfig.User = cntr.Config.User
+	execConfig.WorkingDir = cntr.Config.WorkingDir
 
 	linkedEnv, err := d.setupLinkedContainers(cntr)
 	if err != nil {

components/engine/daemon/image.go

@@ -6,7 +6,6 @@ import (
 
 	"github.com/docker/distribution/reference"
 	"github.com/docker/docker/image"
-	"github.com/docker/docker/pkg/stringid"
 )
 
 // errImageDoesNotExist is error returned when no image can be found for a reference.
@@ -59,21 +58,6 @@ func (daemon *Daemon) GetImageIDAndOS(refOrID string) (image.ID, string, error)
 		return id, imageOS, nil
 	}
 
-	// deprecated: repo:shortid https://github.com/docker/docker/pull/799
-	if tagged, ok := namedRef.(reference.Tagged); ok {
-		if tag := tagged.Tag(); stringid.IsShortID(stringid.TruncateID(tag)) {
-			for platform := range daemon.stores {
-				if id, err := daemon.stores[platform].imageStore.Search(tag); err == nil {
-					for _, storeRef := range daemon.referenceStore.References(id.Digest()) {
-						if storeRef.Name() == namedRef.Name() {
-							return id, platform, nil
-						}
-					}
-				}
-			}
-		}
-	}
-
 	// Search based on ID
 	for os := range daemon.stores {
 		if id, err := daemon.stores[os].imageStore.Search(refOrID); err == nil {

components/engine/daemon/kill.go

@@ -4,10 +4,10 @@ import (
 	"context"
 	"fmt"
 	"runtime"
-	"strings"
 	"syscall"
 	"time"
 
+	"github.com/docker/docker/api/errdefs"
 	containerpkg "github.com/docker/docker/container"
 	"github.com/docker/docker/libcontainerd"
 	"github.com/docker/docker/pkg/signal"
@@ -97,15 +97,11 @@ func (daemon *Daemon) killWithSignal(container *containerpkg.Container, sig int)
 	}
 
 	if err := daemon.kill(container, sig); err != nil {
-		err = errors.Wrapf(err, "Cannot kill container %s", container.ID)
-		// if container or process not exists, ignore the error
-		// TODO: we shouldn't have to parse error strings from containerd
-		if strings.Contains(err.Error(), "container not found") ||
-			strings.Contains(err.Error(), "no such process") {
-			logrus.Warnf("container kill failed because of 'container not found' or 'no such process': %s", err.Error())
+		if errdefs.IsNotFound(err) {
 			unpause = false
+			logrus.WithError(err).WithField("container", container.ID).WithField("action", "kill").Debug("container kill failed because of 'container not found' or 'no such process'")
 		} else {
-			return err
+			return errors.Wrapf(err, "Cannot kill container %s", container.ID)
 		}
 	}
 
@@ -171,7 +167,7 @@ func (daemon *Daemon) Kill(container *containerpkg.Container) error {
 // killPossibleDeadProcess is a wrapper around killSig() suppressing "no such process" error.
 func (daemon *Daemon) killPossiblyDeadProcess(container *containerpkg.Container, sig int) error {
 	err := daemon.killWithSignal(container, sig)
-	if err == syscall.ESRCH {
+	if errdefs.IsNotFound(err) {
 		e := errNoSuchProcess{container.GetPID(), sig}
 		logrus.Debug(e)
 		return e

components/engine/daemon/logger/awslogs/cloudwatchlogs.go

@@ -95,6 +95,17 @@ func init() {
 	}
 }
 
+// eventBatch holds the events that are batched for submission and the
+// associated data about it.
+//
+// Warning: this type is not threadsafe and must not be used
+// concurrently. This type is expected to be consumed in a single go
+// routine and never concurrently.
+type eventBatch struct {
+	batch []wrappedEvent
+	bytes int
+}
+
 // New creates an awslogs logger using the configuration passed in on the
 // context.  Supported context configuration variables are awslogs-region,
 // awslogs-group, awslogs-stream, awslogs-create-group, awslogs-multiline-pattern
@@ -389,32 +400,32 @@ var newTicker = func(freq time.Duration) *time.Ticker {
 // Logs, the processEvents method is called.  If a multiline pattern is not
 // configured, log events are submitted to the processEvents method immediately.
 func (l *logStream) collectBatch() {
-	timer := newTicker(batchPublishFrequency)
-	var events []wrappedEvent
+	ticker := newTicker(batchPublishFrequency)
 	var eventBuffer []byte
 	var eventBufferTimestamp int64
+	var batch = newEventBatch()
 	for {
 		select {
-		case t := <-timer.C:
+		case t := <-ticker.C:
 			// If event buffer is older than batch publish frequency flush the event buffer
 			if eventBufferTimestamp > 0 && len(eventBuffer) > 0 {
 				eventBufferAge := t.UnixNano()/int64(time.Millisecond) - eventBufferTimestamp
 				eventBufferExpired := eventBufferAge > int64(batchPublishFrequency)/int64(time.Millisecond)
 				eventBufferNegative := eventBufferAge < 0
 				if eventBufferExpired || eventBufferNegative {
-					events = l.processEvent(events, eventBuffer, eventBufferTimestamp)
+					l.processEvent(batch, eventBuffer, eventBufferTimestamp)
 					eventBuffer = eventBuffer[:0]
 				}
 			}
-			l.publishBatch(events)
-			events = events[:0]
+			l.publishBatch(batch)
+			batch.reset()
 		case msg, more := <-l.messages:
 			if !more {
 				// Flush event buffer and release resources
-				events = l.processEvent(events, eventBuffer, eventBufferTimestamp)
+				l.processEvent(batch, eventBuffer, eventBufferTimestamp)
 				eventBuffer = eventBuffer[:0]
-				l.publishBatch(events)
-				events = events[:0]
+				l.publishBatch(batch)
+				batch.reset()
 				return
 			}
 			if eventBufferTimestamp == 0 {
@@ -425,7 +436,7 @@ func (l *logStream) collectBatch() {
 				if l.multilinePattern.Match(unprocessedLine) || len(eventBuffer)+len(unprocessedLine) > maximumBytesPerEvent {
 					// This is a new log event or we will exceed max bytes per event
 					// so flush the current eventBuffer to events and reset timestamp
-					events = l.processEvent(events, eventBuffer, eventBufferTimestamp)
+					l.processEvent(batch, eventBuffer, eventBufferTimestamp)
 					eventBufferTimestamp = msg.Timestamp.UnixNano() / int64(time.Millisecond)
 					eventBuffer = eventBuffer[:0]
 				}
@@ -434,7 +445,7 @@ func (l *logStream) collectBatch() {
 				eventBuffer = append(eventBuffer, processedLine...)
 				logger.PutMessage(msg)
 			} else {
-				events = l.processEvent(events, unprocessedLine, msg.Timestamp.UnixNano()/int64(time.Millisecond))
+				l.processEvent(batch, unprocessedLine, msg.Timestamp.UnixNano()/int64(time.Millisecond))
 				logger.PutMessage(msg)
 			}
 		}
@@ -450,8 +461,7 @@ func (l *logStream) collectBatch() {
 // bytes per event (defined in maximumBytesPerEvent).  There is a fixed per-event
 // byte overhead (defined in perEventBytes) which is accounted for in split- and
 // batch-calculations.
-func (l *logStream) processEvent(events []wrappedEvent, unprocessedLine []byte, timestamp int64) []wrappedEvent {
-	bytes := 0
+func (l *logStream) processEvent(batch *eventBatch, unprocessedLine []byte, timestamp int64) {
 	for len(unprocessedLine) > 0 {
 		// Split line length so it does not exceed the maximum
 		lineBytes := len(unprocessedLine)
@@ -459,38 +469,33 @@ func (l *logStream) processEvent(events []wrappedEvent, unprocessedLine []byte,
 			lineBytes = maximumBytesPerEvent
 		}
 		line := unprocessedLine[:lineBytes]
-		unprocessedLine = unprocessedLine[lineBytes:]
-		if (len(events) >= maximumLogEventsPerPut) || (bytes+lineBytes+perEventBytes > maximumBytesPerPut) {
-			// Publish an existing batch if it's already over the maximum number of events or if adding this
-			// event would push it over the maximum number of total bytes.
-			l.publishBatch(events)
-			events = events[:0]
-			bytes = 0
-		}
-		events = append(events, wrappedEvent{
+
+		event := wrappedEvent{
 			inputLogEvent: &cloudwatchlogs.InputLogEvent{
 				Message:   aws.String(string(line)),
 				Timestamp: aws.Int64(timestamp),
 			},
-			insertOrder: len(events),
-		})
-		bytes += (lineBytes + perEventBytes)
+			insertOrder: batch.count(),
+		}
+
+		added := batch.add(event, lineBytes)
+		if added {
+			unprocessedLine = unprocessedLine[lineBytes:]
+		} else {
+			l.publishBatch(batch)
+			batch.reset()
+		}
 	}
-	return events
 }
 
 // publishBatch calls PutLogEvents for a given set of InputLogEvents,
 // accounting for sequencing requirements (each request must reference the
 // sequence token returned by the previous request).
-func (l *logStream) publishBatch(events []wrappedEvent) {
-	if len(events) == 0 {
+func (l *logStream) publishBatch(batch *eventBatch) {
+	if batch.isEmpty() {
 		return
 	}
-
-	// events in a batch must be sorted by timestamp
-	// see http://docs.aws.amazon.com/AmazonCloudWatchLogs/latest/APIReference/API_PutLogEvents.html
-	sort.Sort(byTimestamp(events))
-	cwEvents := unwrapEvents(events)
+	cwEvents := unwrapEvents(batch.events())
 
 	nextSequenceToken, err := l.putLogEvents(cwEvents, l.sequenceToken)
 
@@ -615,3 +620,70 @@ func unwrapEvents(events []wrappedEvent) []*cloudwatchlogs.InputLogEvent {
 	}
 	return cwEvents
 }
+
+func newEventBatch() *eventBatch {
+	return &eventBatch{
+		batch: make([]wrappedEvent, 0),
+		bytes: 0,
+	}
+}
+
+// events returns a slice of wrappedEvents sorted in order of their
+// timestamps and then by their insertion order (see `byTimestamp`).
+//
+// Warning: this method is not threadsafe and must not be used
+// concurrently.
+func (b *eventBatch) events() []wrappedEvent {
+	sort.Sort(byTimestamp(b.batch))
+	return b.batch
+}
+
+// add adds an event to the batch of events accounting for the
+// necessary overhead for an event to be logged. An error will be
+// returned if the event cannot be added to the batch due to service
+// limits.
+//
+// Warning: this method is not threadsafe and must not be used
+// concurrently.
+func (b *eventBatch) add(event wrappedEvent, size int) bool {
+	addBytes := size + perEventBytes
+
+	// verify we are still within service limits
+	switch {
+	case len(b.batch)+1 > maximumLogEventsPerPut:
+		return false
+	case b.bytes+addBytes > maximumBytesPerPut:
+		return false
+	}
+
+	b.bytes += addBytes
+	b.batch = append(b.batch, event)
+
+	return true
+}
+
+// count is the number of batched events.  Warning: this method
+// is not threadsafe and must not be used concurrently.
+func (b *eventBatch) count() int {
+	return len(b.batch)
+}
+
+// size is the total number of bytes that the batch represents.
+//
+// Warning: this method is not threadsafe and must not be used
+// concurrently.
+func (b *eventBatch) size() int {
+	return b.bytes
+}
+
+func (b *eventBatch) isEmpty() bool {
+	zeroEvents := b.count() == 0
+	zeroSize := b.size() == 0
+	return zeroEvents && zeroSize
+}
+
+// reset prepares the batch for reuse.
+func (b *eventBatch) reset() {
+	b.bytes = 0
+	b.batch = b.batch[:0]
+}

components/engine/daemon/logger/awslogs/cloudwatchlogs_test.go

@@ -49,6 +49,15 @@ func (l *logStream) logGenerator(lineCount int, multilineCount int) {
 	}
 }
 
+func testEventBatch(events []wrappedEvent) *eventBatch {
+	batch := newEventBatch()
+	for _, event := range events {
+		eventlen := len([]byte(*event.inputLogEvent.Message))
+		batch.add(event, eventlen)
+	}
+	return batch
+}
+
 func TestNewAWSLogsClientUserAgentHandler(t *testing.T) {
 	info := logger.Info{
 		Config: map[string]string{
@@ -212,7 +221,7 @@ func TestPublishBatchSuccess(t *testing.T) {
 		},
 	}
 
-	stream.publishBatch(events)
+	stream.publishBatch(testEventBatch(events))
 	if stream.sequenceToken == nil {
 		t.Fatal("Expected non-nil sequenceToken")
 	}
@@ -257,7 +266,7 @@ func TestPublishBatchError(t *testing.T) {
 		},
 	}
 
-	stream.publishBatch(events)
+	stream.publishBatch(testEventBatch(events))
 	if stream.sequenceToken == nil {
 		t.Fatal("Expected non-nil sequenceToken")
 	}
@@ -291,7 +300,7 @@ func TestPublishBatchInvalidSeqSuccess(t *testing.T) {
 		},
 	}
 
-	stream.publishBatch(events)
+	stream.publishBatch(testEventBatch(events))
 	if stream.sequenceToken == nil {
 		t.Fatal("Expected non-nil sequenceToken")
 	}
@@ -354,7 +363,7 @@ func TestPublishBatchAlreadyAccepted(t *testing.T) {
 		},
 	}
 
-	stream.publishBatch(events)
+	stream.publishBatch(testEventBatch(events))
 	if stream.sequenceToken == nil {
 		t.Fatal("Expected non-nil sequenceToken")
 	}
@@ -859,7 +868,8 @@ func TestCollectBatchMaxEvents(t *testing.T) {
 }
 
 func TestCollectBatchMaxTotalBytes(t *testing.T) {
-	mockClient := newMockClientBuffered(1)
+	expectedPuts := 2
+	mockClient := newMockClientBuffered(expectedPuts)
 	stream := &logStream{
 		client:        mockClient,
 		logGroupName:  groupName,
@@ -867,11 +877,14 @@ func TestCollectBatchMaxTotalBytes(t *testing.T) {
 		sequenceToken: aws.String(sequenceToken),
 		messages:      make(chan *logger.Message),
 	}
-	mockClient.putLogEventsResult <- &putLogEventsResult{
-		successResult: &cloudwatchlogs.PutLogEventsOutput{
-			NextSequenceToken: aws.String(nextSequenceToken),
-		},
+	for i := 0; i < expectedPuts; i++ {
+		mockClient.putLogEventsResult <- &putLogEventsResult{
+			successResult: &cloudwatchlogs.PutLogEventsOutput{
+				NextSequenceToken: aws.String(nextSequenceToken),
+			},
+		}
 	}
+
 	var ticks = make(chan time.Time)
 	newTicker = func(_ time.Duration) *time.Ticker {
 		return &time.Ticker{
@@ -881,32 +894,57 @@ func TestCollectBatchMaxTotalBytes(t *testing.T) {
 
 	go stream.collectBatch()
 
-	longline := strings.Repeat("A", maximumBytesPerPut)
+	numPayloads := maximumBytesPerPut / (maximumBytesPerEvent + perEventBytes)
+	// maxline is the maximum line that could be submitted after
+	// accounting for its overhead.
+	maxline := strings.Repeat("A", maximumBytesPerPut-(perEventBytes*numPayloads))
+	// This will be split and batched up to the `maximumBytesPerPut'
+	// (+/- `maximumBytesPerEvent'). This /should/ be aligned, but
+	// should also tolerate an offset within that range.
 	stream.Log(&logger.Message{
-		Line:      []byte(longline + "B"),
+		Line:      []byte(maxline[:len(maxline)/2]),
+		Timestamp: time.Time{},
+	})
+	stream.Log(&logger.Message{
+		Line:      []byte(maxline[len(maxline)/2:]),
+		Timestamp: time.Time{},
+	})
+	stream.Log(&logger.Message{
+		Line:      []byte("B"),
 		Timestamp: time.Time{},
 	})
 
-	// no ticks
+	// no ticks, guarantee batch by size (and chan close)
 	stream.Close()
 
 	argument := <-mockClient.putLogEventsArgument
 	if argument == nil {
 		t.Fatal("Expected non-nil PutLogEventsInput")
 	}
-	bytes := 0
+
+	// Should total to the maximum allowed bytes.
+	eventBytes := 0
 	for _, event := range argument.LogEvents {
-		bytes += len(*event.Message)
+		eventBytes += len(*event.Message)
 	}
-	if bytes > maximumBytesPerPut {
-		t.Errorf("Expected <= %d bytes but was %d", maximumBytesPerPut, bytes)
+	eventsOverhead := len(argument.LogEvents) * perEventBytes
+	payloadTotal := eventBytes + eventsOverhead
+	// lowestMaxBatch allows the payload to be offset if the messages
+	// don't lend themselves to align with the maximum event size.
+	lowestMaxBatch := maximumBytesPerPut - maximumBytesPerEvent
+
+	if payloadTotal > maximumBytesPerPut {
+		t.Errorf("Expected <= %d bytes but was %d", maximumBytesPerPut, payloadTotal)
+	}
+	if payloadTotal < lowestMaxBatch {
+		t.Errorf("Batch to be no less than %d but was %d", lowestMaxBatch, payloadTotal)
 	}
 
 	argument = <-mockClient.putLogEventsArgument
 	if len(argument.LogEvents) != 1 {
 		t.Errorf("Expected LogEvents to contain 1 elements, but contains %d", len(argument.LogEvents))
 	}
-	message := *argument.LogEvents[0].Message
+	message := *argument.LogEvents[len(argument.LogEvents)-1].Message
 	if message[len(message)-1:] != "B" {
 		t.Errorf("Expected message to be %s but was %s", "B", message[len(message)-1:])
 	}

components/engine/daemon/logger/awslogs/cwlogsiface_mock_test.go

@@ -1,6 +1,10 @@
 package awslogs
 
-import "github.com/aws/aws-sdk-go/service/cloudwatchlogs"
+import (
+	"fmt"
+
+	"github.com/aws/aws-sdk-go/service/cloudwatchlogs"
+)
 
 type mockcwlogsclient struct {
 	createLogGroupArgument  chan *cloudwatchlogs.CreateLogGroupInput
@@ -67,7 +71,30 @@ func (m *mockcwlogsclient) PutLogEvents(input *cloudwatchlogs.PutLogEventsInput)
 		LogGroupName:  input.LogGroupName,
 		LogStreamName: input.LogStreamName,
 	}
+
+	// Intended mock output
 	output := <-m.putLogEventsResult
+
+	// Checked enforced limits in mock
+	totalBytes := 0
+	for _, evt := range events {
+		if evt.Message == nil {
+			continue
+		}
+		eventBytes := len([]byte(*evt.Message))
+		if eventBytes > maximumBytesPerEvent {
+			// exceeded per event message size limits
+			return nil, fmt.Errorf("maximum bytes per event exceeded: Event too large %d, max allowed: %d", eventBytes, maximumBytesPerEvent)
+		}
+		// total event bytes including overhead
+		totalBytes += eventBytes + perEventBytes
+	}
+
+	if totalBytes > maximumBytesPerPut {
+		// exceeded per put maximum size limit
+		return nil, fmt.Errorf("maximum bytes per put exceeded: Upload too large %d, max allowed: %d", totalBytes, maximumBytesPerPut)
+	}
+
 	return output.successResult, output.errorResult
 }
 

components/engine/hack/dockerfile/binaries-commits

@@ -3,8 +3,8 @@
 TOMLV_COMMIT=9baf8a8a9f2ed20a8e54160840c492f937eeaf9a
 
 # When updating RUNC_COMMIT, also update runc in vendor.conf accordingly
-RUNC_COMMIT=b2567b37d7b75eb4cf325b77297b140ea686ce8f
-CONTAINERD_COMMIT=89623f28b87a6004d4b785663257362d1658a729 # v1.0.0
+RUNC_COMMIT=9f9c96235cc97674e935002fc3d78361b696a69e
+CONTAINERD_COMMIT=9b55aab90508bd389d7654c4baf173a981477d55 # v1.0.1
 TINI_COMMIT=949e6facb77383876aeff8a6944dde66b3089574
 LIBNETWORK_COMMIT=7b2b1feb1de4817d522cc372af149ff48d25028e
 VNDR_COMMIT=a6e196d8b4b0cbbdc29aebdb20c59ac6926bb384

components/engine/integration-cli/docker_cli_create_test.go

@@ -268,7 +268,6 @@ func (s *DockerSuite) TestCreateByImageID(c *check.C) {
 
 	dockerCmd(c, "create", imageID)
 	dockerCmd(c, "create", truncatedImageID)
-	dockerCmd(c, "create", fmt.Sprintf("%s:%s", imageName, truncatedImageID))
 
 	// Ensure this fails
 	out, exit, _ := dockerCmdWithError("create", fmt.Sprintf("%s:%s", imageName, imageID))
@@ -280,7 +279,10 @@ func (s *DockerSuite) TestCreateByImageID(c *check.C) {
 		c.Fatalf(`Expected %q in output; got: %s`, expected, out)
 	}
 
-	out, exit, _ = dockerCmdWithError("create", fmt.Sprintf("%s:%s", "wrongimage", truncatedImageID))
+	if i := strings.IndexRune(imageID, ':'); i >= 0 {
+		imageID = imageID[i+1:]
+	}
+	out, exit, _ = dockerCmdWithError("create", fmt.Sprintf("%s:%s", "wrongimage", imageID))
 	if exit == 0 {
 		c.Fatalf("expected non-zero exit code; received %d", exit)
 	}

components/engine/integration-cli/docker_cli_daemon_test.go

@@ -1451,7 +1451,7 @@ func (s *DockerDaemonSuite) TestCleanupMountsAfterDaemonAndContainerKill(c *chec
 
 	// kill the container
 	icmd.RunCommand(ctrBinary, "--address", "/var/run/docker/containerd/docker-containerd.sock",
-		"--namespace", moby_daemon.MainNamespace, "tasks", "kill", id).Assert(c, icmd.Success)
+		"--namespace", moby_daemon.ContainersNamespace, "tasks", "kill", id).Assert(c, icmd.Success)
 
 	// restart daemon.
 	d.Restart(c)
@@ -2011,7 +2011,7 @@ func (s *DockerDaemonSuite) TestDaemonRestartWithKilledRunningContainer(t *check
 
 	// kill the container
 	icmd.RunCommand(ctrBinary, "--address", "/var/run/docker/containerd/docker-containerd.sock",
-		"--namespace", moby_daemon.MainNamespace, "tasks", "kill", cid).Assert(t, icmd.Success)
+		"--namespace", moby_daemon.ContainersNamespace, "tasks", "kill", cid).Assert(t, icmd.Success)
 
 	// Give time to containerd to process the command if we don't
 	// the exit event might be received after we do the inspect
@@ -2106,7 +2106,7 @@ func (s *DockerDaemonSuite) TestDaemonRestartWithUnpausedRunningContainer(t *che
 	result := icmd.RunCommand(
 		ctrBinary,
 		"--address", "/var/run/docker/containerd/docker-containerd.sock",
-		"--namespace", moby_daemon.MainNamespace,
+		"--namespace", moby_daemon.ContainersNamespace,
 		"tasks", "resume", cid)
 	result.Assert(t, icmd.Success)
 

components/engine/integration-cli/docker_cli_run_test.go

@@ -36,7 +36,6 @@ import (
 	"github.com/docker/libnetwork/types"
 	"github.com/go-check/check"
 	"github.com/gotestyourself/gotestyourself/icmd"
-	libcontainerUser "github.com/opencontainers/runc/libcontainer/user"
 	"golang.org/x/net/context"
 )
 
@@ -751,7 +750,7 @@ func (s *DockerSuite) TestRunUserByIDBig(c *check.C) {
 	if err == nil {
 		c.Fatal("No error, but must be.", out)
 	}
-	if !strings.Contains(strings.ToUpper(out), strings.ToUpper(libcontainerUser.ErrRange.Error())) {
+	if !strings.Contains(strings.ToLower(out), "uids and gids must be in range") {
 		c.Fatalf("expected error about uids range, got %s", out)
 	}
 }
@@ -764,7 +763,7 @@ func (s *DockerSuite) TestRunUserByIDNegative(c *check.C) {
 	if err == nil {
 		c.Fatal("No error, but must be.", out)
 	}
-	if !strings.Contains(strings.ToUpper(out), strings.ToUpper(libcontainerUser.ErrRange.Error())) {
+	if !strings.Contains(strings.ToLower(out), "uids and gids must be in range") {
 		c.Fatalf("expected error about uids range, got %s", out)
 	}
 }

components/engine/integration-cli/docker_cli_tag_test.go

@@ -1,13 +1,10 @@
 package main
 
 import (
-	"fmt"
 	"strings"
 
 	"github.com/docker/docker/integration-cli/checker"
-	"github.com/docker/docker/integration-cli/cli/build"
 	"github.com/docker/docker/internal/testutil"
-	"github.com/docker/docker/pkg/stringid"
 	"github.com/go-check/check"
 )
 
@@ -140,29 +137,3 @@ func (s *DockerSuite) TestTagInvalidRepoName(c *check.C) {
 		c.Fatal("tagging with image named \"sha256\" should have failed")
 	}
 }
-
-// ensure tags cannot create ambiguity with image ids
-func (s *DockerSuite) TestTagTruncationAmbiguity(c *check.C) {
-	buildImageSuccessfully(c, "notbusybox:latest", build.WithDockerfile(`FROM busybox
-		MAINTAINER dockerio`))
-	imageID := getIDByName(c, "notbusybox:latest")
-	truncatedImageID := stringid.TruncateID(imageID)
-	truncatedTag := fmt.Sprintf("notbusybox:%s", truncatedImageID)
-
-	id := inspectField(c, truncatedTag, "Id")
-
-	// Ensure inspect by image id returns image for image id
-	c.Assert(id, checker.Equals, imageID)
-	c.Logf("Built image: %s", imageID)
-
-	// test setting tag fails
-	_, _, err := dockerCmdWithError("tag", "busybox:latest", truncatedTag)
-	if err != nil {
-		c.Fatalf("Error tagging with an image id: %s", err)
-	}
-
-	id = inspectField(c, truncatedTag, "Id")
-
-	// Ensure id is imageID and not busybox:latest
-	c.Assert(id, checker.Not(checker.Equals), imageID)
-}

components/engine/integration/container/health_test.go

@@ -0,0 +1,61 @@
+package container
+
+import (
+	"context"
+	"testing"
+	"time"
+
+	"github.com/docker/docker/api/types"
+	"github.com/docker/docker/api/types/container"
+	"github.com/docker/docker/api/types/network"
+	"github.com/docker/docker/api/types/strslice"
+	"github.com/docker/docker/client"
+	"github.com/docker/docker/integration/util/request"
+	"github.com/gotestyourself/gotestyourself/poll"
+	"github.com/stretchr/testify/require"
+)
+
+// TestHealthCheckWorkdir verifies that health-checks inherit the containers'
+// working-dir.
+func TestHealthCheckWorkdir(t *testing.T) {
+	defer setupTest(t)()
+	ctx := context.Background()
+	client := request.NewAPIClient(t)
+
+	c, err := client.ContainerCreate(ctx,
+		&container.Config{
+			Image:      "busybox",
+			Tty:        true,
+			WorkingDir: "/foo",
+			Cmd:        strslice.StrSlice([]string{"top"}),
+			Healthcheck: &container.HealthConfig{
+				Test:     []string{"CMD-SHELL", "if [ \"$PWD\" = \"/foo\" ]; then exit 0; else exit 1; fi;"},
+				Interval: 50 * time.Millisecond,
+				Retries:  3,
+			},
+		},
+		&container.HostConfig{},
+		&network.NetworkingConfig{},
+		"healthtest",
+	)
+	require.NoError(t, err)
+	err = client.ContainerStart(ctx, c.ID, types.ContainerStartOptions{})
+	require.NoError(t, err)
+
+	poll.WaitOn(t, pollForHealthStatus(ctx, client, c.ID, types.Healthy), poll.WithDelay(100*time.Millisecond))
+}
+
+func pollForHealthStatus(ctx context.Context, client client.APIClient, containerID string, healthStatus string) func(log poll.LogT) poll.Result {
+	return func(log poll.LogT) poll.Result {
+		inspect, err := client.ContainerInspect(ctx, containerID)
+
+		switch {
+		case err != nil:
+			return poll.Error(err)
+		case inspect.State.Health.Status == healthStatus:
+			return poll.Success()
+		default:
+			return poll.Continue("waiting for container to become %s", healthStatus)
+		}
+	}
+}

components/engine/integration/container/restart_test.go

@@ -0,0 +1,112 @@
+package container
+
+import (
+	"context"
+	"fmt"
+	"testing"
+	"time"
+
+	"github.com/docker/docker/api/types"
+	"github.com/docker/docker/api/types/container"
+	"github.com/docker/docker/integration-cli/daemon"
+)
+
+func TestDaemonRestartKillContainers(t *testing.T) {
+	type testCase struct {
+		desc       string
+		config     *container.Config
+		hostConfig *container.HostConfig
+
+		xRunning            bool
+		xRunningLiveRestore bool
+	}
+
+	for _, c := range []testCase{
+		{
+			desc:                "container without restart policy",
+			config:              &container.Config{Image: "busybox", Cmd: []string{"top"}},
+			xRunningLiveRestore: true,
+		},
+		{
+			desc:                "container with restart=always",
+			config:              &container.Config{Image: "busybox", Cmd: []string{"top"}},
+			hostConfig:          &container.HostConfig{RestartPolicy: container.RestartPolicy{Name: "always"}},
+			xRunning:            true,
+			xRunningLiveRestore: true,
+		},
+	} {
+		for _, liveRestoreEnabled := range []bool{false, true} {
+			for fnName, stopDaemon := range map[string]func(*testing.T, *daemon.Daemon){
+				"kill-daemon": func(t *testing.T, d *daemon.Daemon) {
+					if err := d.Kill(); err != nil {
+						t.Fatal(err)
+					}
+				},
+				"stop-daemon": func(t *testing.T, d *daemon.Daemon) {
+					d.Stop(t)
+				},
+			} {
+				t.Run(fmt.Sprintf("live-restore=%v/%s/%s", liveRestoreEnabled, c.desc, fnName), func(t *testing.T) {
+					c := c
+					liveRestoreEnabled := liveRestoreEnabled
+					stopDaemon := stopDaemon
+
+					t.Parallel()
+
+					d := daemon.New(t, "", "dockerd", daemon.Config{})
+					client, err := d.NewClient()
+					if err != nil {
+						t.Fatal(err)
+					}
+
+					var args []string
+					if liveRestoreEnabled {
+						args = []string{"--live-restore"}
+					}
+
+					d.StartWithBusybox(t, args...)
+					defer d.Stop(t)
+					ctx := context.Background()
+
+					resp, err := client.ContainerCreate(ctx, c.config, c.hostConfig, nil, "")
+					if err != nil {
+						t.Fatal(err)
+					}
+					defer client.ContainerRemove(ctx, resp.ID, types.ContainerRemoveOptions{Force: true})
+
+					if err := client.ContainerStart(ctx, resp.ID, types.ContainerStartOptions{}); err != nil {
+						t.Fatal(err)
+					}
+
+					stopDaemon(t, d)
+					d.Start(t, args...)
+
+					expected := c.xRunning
+					if liveRestoreEnabled {
+						expected = c.xRunningLiveRestore
+					}
+
+					var running bool
+					for i := 0; i < 30; i++ {
+						inspect, err := client.ContainerInspect(ctx, resp.ID)
+						if err != nil {
+							t.Fatal(err)
+						}
+
+						running = inspect.State.Running
+						if running == expected {
+							break
+						}
+						time.Sleep(2 * time.Second)
+
+					}
+
+					if running != expected {
+						t.Fatalf("got unexpected running state, expected %v, got: %v", expected, running)
+					}
+					// TODO(cpuguy83): test pause states... this seems to be rather undefined currently
+				})
+			}
+		}
+	}
+}

components/engine/integration/network/inspect_test.go

@@ -0,0 +1,221 @@
+package network
+
+import (
+	"fmt"
+	"runtime"
+	"testing"
+	"time"
+
+	"github.com/docker/docker/api/types"
+	"github.com/docker/docker/api/types/filters"
+	"github.com/docker/docker/api/types/swarm"
+	"github.com/docker/docker/client"
+	"github.com/docker/docker/integration-cli/daemon"
+	"github.com/docker/docker/integration-cli/request"
+	"github.com/gotestyourself/gotestyourself/poll"
+	"github.com/stretchr/testify/require"
+	"golang.org/x/net/context"
+)
+
+const defaultSwarmPort = 2477
+const dockerdBinary = "dockerd"
+
+func TestInspectNetwork(t *testing.T) {
+	defer setupTest(t)()
+	d := newSwarm(t)
+	defer d.Stop(t)
+	client, err := request.NewClientForHost(d.Sock())
+	require.NoError(t, err)
+
+	overlayName := "overlay1"
+	networkCreate := types.NetworkCreate{
+		CheckDuplicate: true,
+		Driver:         "overlay",
+	}
+
+	netResp, err := client.NetworkCreate(context.Background(), overlayName, networkCreate)
+	require.NoError(t, err)
+	overlayID := netResp.ID
+
+	var instances uint64 = 4
+	serviceName := "TestService"
+	serviceSpec := swarmServiceSpec(serviceName, instances)
+	serviceSpec.TaskTemplate.Networks = append(serviceSpec.TaskTemplate.Networks, swarm.NetworkAttachmentConfig{Target: overlayName})
+
+	serviceResp, err := client.ServiceCreate(context.Background(), serviceSpec, types.ServiceCreateOptions{
+		QueryRegistry: false,
+	})
+	require.NoError(t, err)
+
+	pollSettings := func(config *poll.Settings) {
+		if runtime.GOARCH == "arm" {
+			config.Timeout = 30 * time.Second
+			config.Delay = 100 * time.Millisecond
+		}
+	}
+
+	serviceID := serviceResp.ID
+	poll.WaitOn(t, serviceRunningTasksCount(client, serviceID, instances), pollSettings)
+
+	_, _, err = client.ServiceInspectWithRaw(context.Background(), serviceID, types.ServiceInspectOptions{})
+	require.NoError(t, err)
+
+	// Test inspect verbose with full NetworkID
+	networkVerbose, err := client.NetworkInspect(context.Background(), overlayID, types.NetworkInspectOptions{
+		Verbose: true,
+	})
+	require.NoError(t, err)
+	require.True(t, validNetworkVerbose(networkVerbose, serviceName, instances))
+
+	// Test inspect verbose with partial NetworkID
+	networkVerbose, err = client.NetworkInspect(context.Background(), overlayID[0:11], types.NetworkInspectOptions{
+		Verbose: true,
+	})
+	require.NoError(t, err)
+	require.True(t, validNetworkVerbose(networkVerbose, serviceName, instances))
+
+	// Test inspect verbose with Network name and swarm scope
+	networkVerbose, err = client.NetworkInspect(context.Background(), overlayName, types.NetworkInspectOptions{
+		Verbose: true,
+		Scope:   "swarm",
+	})
+	require.NoError(t, err)
+	require.True(t, validNetworkVerbose(networkVerbose, serviceName, instances))
+
+	err = client.ServiceRemove(context.Background(), serviceID)
+	require.NoError(t, err)
+
+	poll.WaitOn(t, serviceIsRemoved(client, serviceID), pollSettings)
+	poll.WaitOn(t, noTasks(client), pollSettings)
+
+	serviceResp, err = client.ServiceCreate(context.Background(), serviceSpec, types.ServiceCreateOptions{
+		QueryRegistry: false,
+	})
+	require.NoError(t, err)
+
+	serviceID2 := serviceResp.ID
+	poll.WaitOn(t, serviceRunningTasksCount(client, serviceID2, instances), pollSettings)
+
+	err = client.ServiceRemove(context.Background(), serviceID2)
+	require.NoError(t, err)
+
+	poll.WaitOn(t, serviceIsRemoved(client, serviceID2), pollSettings)
+	poll.WaitOn(t, noTasks(client), pollSettings)
+
+	err = client.NetworkRemove(context.Background(), overlayID)
+	require.NoError(t, err)
+
+	poll.WaitOn(t, networkIsRemoved(client, overlayID), poll.WithTimeout(1*time.Minute), poll.WithDelay(10*time.Second))
+}
+
+func newSwarm(t *testing.T) *daemon.Swarm {
+	d := &daemon.Swarm{
+		Daemon: daemon.New(t, "", dockerdBinary, daemon.Config{
+			Experimental: testEnv.DaemonInfo.ExperimentalBuild,
+		}),
+		// TODO: better method of finding an unused port
+		Port: defaultSwarmPort,
+	}
+	// TODO: move to a NewSwarm constructor
+	d.ListenAddr = fmt.Sprintf("0.0.0.0:%d", d.Port)
+
+	// avoid networking conflicts
+	args := []string{"--iptables=false", "--swarm-default-advertise-addr=lo"}
+	d.StartWithBusybox(t, args...)
+
+	require.NoError(t, d.Init(swarm.InitRequest{}))
+	return d
+}
+
+func swarmServiceSpec(name string, replicas uint64) swarm.ServiceSpec {
+	return swarm.ServiceSpec{
+		Annotations: swarm.Annotations{
+			Name: name,
+		},
+		TaskTemplate: swarm.TaskSpec{
+			ContainerSpec: &swarm.ContainerSpec{
+				Image:   "busybox:latest",
+				Command: []string{"/bin/top"},
+			},
+		},
+		Mode: swarm.ServiceMode{
+			Replicated: &swarm.ReplicatedService{
+				Replicas: &replicas,
+			},
+		},
+	}
+}
+
+func serviceRunningTasksCount(client client.ServiceAPIClient, serviceID string, instances uint64) func(log poll.LogT) poll.Result {
+	return func(log poll.LogT) poll.Result {
+		filter := filters.NewArgs()
+		filter.Add("service", serviceID)
+		tasks, err := client.TaskList(context.Background(), types.TaskListOptions{
+			Filters: filter,
+		})
+		switch {
+		case err != nil:
+			return poll.Error(err)
+		case len(tasks) == int(instances):
+			for _, task := range tasks {
+				if task.Status.State != swarm.TaskStateRunning {
+					return poll.Continue("waiting for tasks to enter run state")
+				}
+			}
+			return poll.Success()
+		default:
+			return poll.Continue("task count at %d waiting for %d", len(tasks), instances)
+		}
+	}
+}
+
+func networkIsRemoved(client client.NetworkAPIClient, networkID string) func(log poll.LogT) poll.Result {
+	return func(log poll.LogT) poll.Result {
+		_, err := client.NetworkInspect(context.Background(), networkID, types.NetworkInspectOptions{})
+		if err == nil {
+			return poll.Continue("waiting for network %s to be removed", networkID)
+		}
+		return poll.Success()
+	}
+}
+
+func serviceIsRemoved(client client.ServiceAPIClient, serviceID string) func(log poll.LogT) poll.Result {
+	return func(log poll.LogT) poll.Result {
+		filter := filters.NewArgs()
+		filter.Add("service", serviceID)
+		_, err := client.TaskList(context.Background(), types.TaskListOptions{
+			Filters: filter,
+		})
+		if err == nil {
+			return poll.Continue("waiting for service %s to be deleted", serviceID)
+		}
+		return poll.Success()
+	}
+}
+
+func noTasks(client client.ServiceAPIClient) func(log poll.LogT) poll.Result {
+	return func(log poll.LogT) poll.Result {
+		filter := filters.NewArgs()
+		tasks, err := client.TaskList(context.Background(), types.TaskListOptions{
+			Filters: filter,
+		})
+		switch {
+		case err != nil:
+			return poll.Error(err)
+		case len(tasks) == 0:
+			return poll.Success()
+		default:
+			return poll.Continue("task count at %d waiting for 0", len(tasks))
+		}
+	}
+}
+
+// Check to see if Service and Tasks info are part of the inspect verbose response
+func validNetworkVerbose(network types.NetworkResource, service string, instances uint64) bool {
+	if service, ok := network.Services[service]; ok {
+		if len(service.Tasks) == int(instances) {
+			return true
+		}
+	}
+	return false
+}

components/engine/libcontainerd/client_daemon.go

@@ -27,6 +27,7 @@ import (
 	"github.com/containerd/containerd/archive"
 	"github.com/containerd/containerd/cio"
 	"github.com/containerd/containerd/content"
+	"github.com/containerd/containerd/errdefs"
 	"github.com/containerd/containerd/images"
 	"github.com/containerd/containerd/linux/runctypes"
 	"github.com/containerd/typeurl"
@@ -42,7 +43,7 @@ import (
 const InitProcessName = "init"
 
 type container struct {
-	sync.Mutex
+	mu sync.Mutex
 
 	bundleDir string
 	ctr       containerd.Container
@@ -51,6 +52,54 @@ type container struct {
 	oomKilled bool
 }
 
+func (c *container) setTask(t containerd.Task) {
+	c.mu.Lock()
+	c.task = t
+	c.mu.Unlock()
+}
+
+func (c *container) getTask() containerd.Task {
+	c.mu.Lock()
+	t := c.task
+	c.mu.Unlock()
+	return t
+}
+
+func (c *container) addProcess(id string, p containerd.Process) {
+	c.mu.Lock()
+	if c.execs == nil {
+		c.execs = make(map[string]containerd.Process)
+	}
+	c.execs[id] = p
+	c.mu.Unlock()
+}
+
+func (c *container) deleteProcess(id string) {
+	c.mu.Lock()
+	delete(c.execs, id)
+	c.mu.Unlock()
+}
+
+func (c *container) getProcess(id string) containerd.Process {
+	c.mu.Lock()
+	p := c.execs[id]
+	c.mu.Unlock()
+	return p
+}
+
+func (c *container) setOOMKilled(killed bool) {
+	c.mu.Lock()
+	c.oomKilled = killed
+	c.mu.Unlock()
+}
+
+func (c *container) getOOMKilled() bool {
+	c.mu.Lock()
+	killed := c.oomKilled
+	c.mu.Unlock()
+	return killed
+}
+
 type client struct {
 	sync.RWMutex // protects containers map
 
@@ -64,8 +113,21 @@ type client struct {
 	containers map[string]*container
 }
 
+func (c *client) setRemote(remote *containerd.Client) {
+	c.Lock()
+	c.remote = remote
+	c.Unlock()
+}
+
+func (c *client) getRemote() *containerd.Client {
+	c.RLock()
+	remote := c.remote
+	c.RUnlock()
+	return remote
+}
+
 func (c *client) Version(ctx context.Context) (containerd.Version, error) {
-	return c.remote.Version(ctx)
+	return c.getRemote().Version(ctx)
 }
 
 func (c *client) Restore(ctx context.Context, id string, attachStdio StdioCallback) (alive bool, pid int, err error) {
@@ -139,7 +201,7 @@ func (c *client) Create(ctx context.Context, id string, ociSpec *specs.Spec, run
 
 	c.logger.WithField("bundle", bdir).WithField("root", ociSpec.Root.Path).Debug("bundle dir created")
 
-	cdCtr, err := c.remote.NewContainer(ctx, id,
+	cdCtr, err := c.getRemote().NewContainer(ctx, id,
 		containerd.WithSpec(ociSpec),
 		// TODO(mlaventure): when containerd support lcow, revisit runtime value
 		containerd.WithRuntime(fmt.Sprintf("io.containerd.runtime.v1.%s", runtime.GOOS), runtimeOptions))
@@ -160,10 +222,10 @@ func (c *client) Create(ctx context.Context, id string, ociSpec *specs.Spec, run
 // Start create and start a task for the specified containerd id
 func (c *client) Start(ctx context.Context, id, checkpointDir string, withStdin bool, attachStdio StdioCallback) (int, error) {
 	ctr := c.getContainer(id)
-	switch {
-	case ctr == nil:
+	if ctr == nil {
 		return -1, errors.WithStack(newNotFoundError("no such container"))
-	case ctr.task != nil:
+	}
+	if t := ctr.getTask(); t != nil {
 		return -1, errors.WithStack(newConflictError("container already started"))
 	}
 
@@ -182,7 +244,7 @@ func (c *client) Start(ctx context.Context, id, checkpointDir string, withStdin
 		// remove the checkpoint when we're done
 		defer func() {
 			if cp != nil {
-				err := c.remote.ContentStore().Delete(context.Background(), cp.Digest)
+				err := c.getRemote().ContentStore().Delete(context.Background(), cp.Digest)
 				if err != nil {
 					c.logger.WithError(err).WithFields(logrus.Fields{
 						"ref":    checkpointDir,
@@ -213,8 +275,9 @@ func (c *client) Start(ctx context.Context, id, checkpointDir string, withStdin
 		func(_ context.Context, _ *containerd.Client, info *containerd.TaskInfo) error {
 			info.Checkpoint = cp
 			info.Options = &runctypes.CreateOptions{
-				IoUid: uint32(uid),
-				IoGid: uint32(gid),
+				IoUid:       uint32(uid),
+				IoGid:       uint32(gid),
+				NoPivotRoot: os.Getenv("DOCKER_RAMDISK") != "",
 			}
 			return nil
 		})
@@ -227,9 +290,7 @@ func (c *client) Start(ctx context.Context, id, checkpointDir string, withStdin
 		return -1, err
 	}
 
-	c.Lock()
-	c.containers[id].task = t
-	c.Unlock()
+	ctr.setTask(t)
 
 	// Signal c.createIO that it can call CloseIO
 	close(stdinCloseSync)
@@ -239,9 +300,7 @@ func (c *client) Start(ctx context.Context, id, checkpointDir string, withStdin
 			c.logger.WithError(err).WithField("container", id).
 				Error("failed to delete task after fail start")
 		}
-		c.Lock()
-		c.containers[id].task = nil
-		c.Unlock()
+		ctr.setTask(nil)
 		return -1, err
 	}
 
@@ -250,12 +309,15 @@ func (c *client) Start(ctx context.Context, id, checkpointDir string, withStdin
 
 func (c *client) Exec(ctx context.Context, containerID, processID string, spec *specs.Process, withStdin bool, attachStdio StdioCallback) (int, error) {
 	ctr := c.getContainer(containerID)
-	switch {
-	case ctr == nil:
+	if ctr == nil {
 		return -1, errors.WithStack(newNotFoundError("no such container"))
-	case ctr.task == nil:
+	}
+	t := ctr.getTask()
+	if t == nil {
 		return -1, errors.WithStack(newInvalidParameterError("container is not running"))
-	case ctr.execs != nil && ctr.execs[processID] != nil:
+	}
+
+	if p := ctr.getProcess(processID); p != nil {
 		return -1, errors.WithStack(newConflictError("id already in use"))
 	}
 
@@ -278,7 +340,7 @@ func (c *client) Exec(ctx context.Context, containerID, processID string, spec *
 		}
 	}()
 
-	p, err = ctr.task.Exec(ctx, processID, spec, func(id string) (cio.IO, error) {
+	p, err = t.Exec(ctx, processID, spec, func(id string) (cio.IO, error) {
 		rio, err = c.createIO(fifos, containerID, processID, stdinCloseSync, attachStdio)
 		return rio, err
 	})
@@ -291,21 +353,14 @@ func (c *client) Exec(ctx context.Context, containerID, processID string, spec *
 		return -1, err
 	}
 
-	ctr.Lock()
-	if ctr.execs == nil {
-		ctr.execs = make(map[string]containerd.Process)
-	}
-	ctr.execs[processID] = p
-	ctr.Unlock()
+	ctr.addProcess(processID, p)
 
 	// Signal c.createIO that it can call CloseIO
 	close(stdinCloseSync)
 
 	if err = p.Start(ctx); err != nil {
 		p.Delete(context.Background())
-		ctr.Lock()
-		delete(ctr.execs, processID)
-		ctr.Unlock()
+		ctr.deleteProcess(processID)
 		return -1, err
 	}
 
@@ -317,7 +372,7 @@ func (c *client) SignalProcess(ctx context.Context, containerID, processID strin
 	if err != nil {
 		return err
 	}
-	return p.Kill(ctx, syscall.Signal(signal))
+	return wrapError(p.Kill(ctx, syscall.Signal(signal)))
 }
 
 func (c *client) ResizeTerminal(ctx context.Context, containerID, processID string, width, height int) error {
@@ -431,12 +486,9 @@ func (c *client) DeleteTask(ctx context.Context, containerID string) (uint32, ti
 		return 255, time.Now(), nil
 	}
 
-	c.Lock()
-	if ctr, ok := c.containers[containerID]; ok {
-		ctr.task = nil
+	if ctr := c.getContainer(containerID); ctr != nil {
+		ctr.setTask(nil)
 	}
-	c.Unlock()
-
 	return status.ExitCode(), status.ExitTime(), nil
 }
 
@@ -470,7 +522,12 @@ func (c *client) Status(ctx context.Context, containerID string) (Status, error)
 		return StatusUnknown, errors.WithStack(newNotFoundError("no such container"))
 	}
 
-	s, err := ctr.task.Status(ctx)
+	t := ctr.getTask()
+	if t == nil {
+		return StatusUnknown, errors.WithStack(newNotFoundError("no such task"))
+	}
+
+	s, err := t.Status(ctx)
 	if err != nil {
 		return StatusUnknown, err
 	}
@@ -490,14 +547,14 @@ func (c *client) CreateCheckpoint(ctx context.Context, containerID, checkpointDi
 	}
 	// Whatever happens, delete the checkpoint from containerd
 	defer func() {
-		err := c.remote.ImageService().Delete(context.Background(), img.Name())
+		err := c.getRemote().ImageService().Delete(context.Background(), img.Name())
 		if err != nil {
 			c.logger.WithError(err).WithField("digest", img.Target().Digest).
 				Warnf("failed to delete checkpoint image")
 		}
 	}()
 
-	b, err := content.ReadBlob(ctx, c.remote.ContentStore(), img.Target().Digest)
+	b, err := content.ReadBlob(ctx, c.getRemote().ContentStore(), img.Target().Digest)
 	if err != nil {
 		return wrapSystemError(errors.Wrapf(err, "failed to retrieve checkpoint data"))
 	}
@@ -517,7 +574,7 @@ func (c *client) CreateCheckpoint(ctx context.Context, containerID, checkpointDi
 		return wrapSystemError(errors.Wrapf(err, "invalid checkpoint"))
 	}
 
-	rat, err := c.remote.ContentStore().ReaderAt(ctx, cpDesc.Digest)
+	rat, err := c.getRemote().ContentStore().ReaderAt(ctx, cpDesc.Digest)
 	if err != nil {
 		return wrapSystemError(errors.Wrapf(err, "failed to get checkpoint reader"))
 	}
@@ -546,26 +603,22 @@ func (c *client) removeContainer(id string) {
 
 func (c *client) getProcess(containerID, processID string) (containerd.Process, error) {
 	ctr := c.getContainer(containerID)
-	switch {
-	case ctr == nil:
+	if ctr == nil {
 		return nil, errors.WithStack(newNotFoundError("no such container"))
-	case ctr.task == nil:
-		return nil, errors.WithStack(newNotFoundError("container is not running"))
-	case processID == InitProcessName:
-		return ctr.task, nil
-	default:
-		ctr.Lock()
-		defer ctr.Unlock()
-		if ctr.execs == nil {
-			return nil, errors.WithStack(newNotFoundError("no execs"))
-		}
 	}
 
-	p := ctr.execs[processID]
+	t := ctr.getTask()
+	if t == nil {
+		return nil, errors.WithStack(newNotFoundError("container is not running"))
+	}
+	if processID == InitProcessName {
+		return t, nil
+	}
+
+	p := ctr.getProcess(processID)
 	if p == nil {
 		return nil, errors.WithStack(newNotFoundError("no such exec"))
 	}
-
 	return p, nil
 }
 
@@ -623,12 +676,7 @@ func (c *client) processEvent(ctr *container, et EventType, ei EventInfo) {
 		}
 
 		if et == EventExit && ei.ProcessID != ei.ContainerID {
-			var p containerd.Process
-			ctr.Lock()
-			if ctr.execs != nil {
-				p = ctr.execs[ei.ProcessID]
-			}
-			ctr.Unlock()
+			p := ctr.getProcess(ei.ProcessID)
 			if p == nil {
 				c.logger.WithError(errors.New("no such process")).
 					WithFields(logrus.Fields{
@@ -644,9 +692,8 @@ func (c *client) processEvent(ctr *container, et EventType, ei EventInfo) {
 					"process":   ei.ProcessID,
 				}).Warn("failed to delete process")
 			}
-			c.Lock()
-			delete(ctr.execs, ei.ProcessID)
-			c.Unlock()
+			ctr.deleteProcess(ei.ProcessID)
+
 			ctr := c.getContainer(ei.ContainerID)
 			if ctr == nil {
 				c.logger.WithFields(logrus.Fields{
@@ -680,16 +727,19 @@ func (c *client) processEventStream(ctx context.Context) {
 		}
 	}()
 
-	eventStream, err = c.remote.EventService().Subscribe(ctx, &eventsapi.SubscribeRequest{
+	eventStream, err = c.getRemote().EventService().Subscribe(ctx, &eventsapi.SubscribeRequest{
 		Filters: []string{
-			"namespace==" + c.namespace,
-			"topic~=/tasks/",
+			// Filter on both namespace *and* topic. To create an "and" filter,
+			// this must be a single, comma-separated string
+			"namespace==" + c.namespace + ",topic~=|^/tasks/|",
 		},
 	}, grpc.FailFast(false))
 	if err != nil {
 		return
 	}
 
+	c.logger.WithField("namespace", c.namespace).Debug("processing event stream")
+
 	var oomKilled bool
 	for {
 		ev, err = eventStream.Recv()
@@ -783,17 +833,17 @@ func (c *client) processEventStream(ctx context.Context) {
 		}
 
 		if oomKilled {
-			ctr.oomKilled = true
+			ctr.setOOMKilled(true)
 			oomKilled = false
 		}
-		ei.OOMKilled = ctr.oomKilled
+		ei.OOMKilled = ctr.getOOMKilled()
 
 		c.processEvent(ctr, et, ei)
 	}
 }
 
 func (c *client) writeContent(ctx context.Context, mediaType, ref string, r io.Reader) (*types.Descriptor, error) {
-	writer, err := c.remote.ContentStore().Writer(ctx, ref, 0, "")
+	writer, err := c.getRemote().ContentStore().Writer(ctx, ref, 0, "")
 	if err != nil {
 		return nil, err
 	}
@@ -816,12 +866,19 @@ func (c *client) writeContent(ctx context.Context, mediaType, ref string, r io.R
 }
 
 func wrapError(err error) error {
-	if err != nil {
-		msg := err.Error()
-		for _, s := range []string{"container does not exist", "not found", "no such container"} {
-			if strings.Contains(msg, s) {
-				return wrapNotFoundError(err)
-			}
+	if err == nil {
+		return nil
+	}
+
+	switch {
+	case errdefs.IsNotFound(err):
+		return wrapNotFoundError(err)
+	}
+
+	msg := err.Error()
+	for _, s := range []string{"container does not exist", "not found", "no such container"} {
+		if strings.Contains(msg, s) {
+			return wrapNotFoundError(err)
 		}
 	}
 	return err

components/engine/libcontainerd/remote_daemon.go

@@ -260,7 +260,7 @@ func (r *remote) startContainerd() error {
 	return nil
 }
 
-func (r *remote) monitorConnection(client *containerd.Client) {
+func (r *remote) monitorConnection(monitor *containerd.Client) {
 	var transientFailureCount = 0
 
 	ticker := time.NewTicker(500 * time.Millisecond)
@@ -269,7 +269,7 @@ func (r *remote) monitorConnection(client *containerd.Client) {
 	for {
 		<-ticker.C
 		ctx, cancel := context.WithTimeout(r.shutdownContext, healthCheckTimeout)
-		_, err := client.IsServing(ctx)
+		_, err := monitor.IsServing(ctx)
 		cancel()
 		if err == nil {
 			transientFailureCount = 0
@@ -278,40 +278,70 @@ func (r *remote) monitorConnection(client *containerd.Client) {
 
 		select {
 		case <-r.shutdownContext.Done():
-			r.logger.Info("stopping healtcheck following graceful shutdown")
-			client.Close()
+			r.logger.Info("stopping healthcheck following graceful shutdown")
+			monitor.Close()
 			return
 		default:
 		}
 
 		r.logger.WithError(err).WithField("binary", binaryName).Debug("daemon is not responding")
 
-		if r.daemonPid != -1 {
-			transientFailureCount++
-			if transientFailureCount >= maxConnectionRetryCount || !system.IsProcessAlive(r.daemonPid) {
-				transientFailureCount = 0
-				if system.IsProcessAlive(r.daemonPid) {
-					r.logger.WithField("pid", r.daemonPid).Info("killing and restarting containerd")
-					// Try to get a stack trace
-					syscall.Kill(r.daemonPid, syscall.SIGUSR1)
-					<-time.After(100 * time.Millisecond)
-					system.KillProcess(r.daemonPid)
+		if r.daemonPid == -1 {
+			continue
+		}
+
+		transientFailureCount++
+		if transientFailureCount < maxConnectionRetryCount || system.IsProcessAlive(r.daemonPid) {
+			continue
+		}
+
+		transientFailureCount = 0
+		if system.IsProcessAlive(r.daemonPid) {
+			r.logger.WithField("pid", r.daemonPid).Info("killing and restarting containerd")
+			// Try to get a stack trace
+			syscall.Kill(r.daemonPid, syscall.SIGUSR1)
+			<-time.After(100 * time.Millisecond)
+			system.KillProcess(r.daemonPid)
+		}
+		<-r.daemonWaitCh
+
+		monitor.Close()
+		os.Remove(r.GRPC.Address)
+		if err := r.startContainerd(); err != nil {
+			r.logger.WithError(err).Error("failed restarting containerd")
+			continue
+		}
+
+		newMonitor, err := containerd.New(r.GRPC.Address)
+		if err != nil {
+			r.logger.WithError(err).Error("failed connect to containerd")
+			continue
+		}
+
+		monitor = newMonitor
+		var wg sync.WaitGroup
+
+		for _, c := range r.clients {
+			wg.Add(1)
+
+			go func(c *client) {
+				defer wg.Done()
+				c.logger.WithField("namespace", c.namespace).Debug("creating new containerd remote client")
+				c.remote.Close()
+
+				remote, err := containerd.New(r.GRPC.Address, containerd.WithDefaultNamespace(c.namespace))
+				if err != nil {
+					r.logger.WithError(err).Error("failed to connect to containerd")
+					// TODO: Better way to handle this?
+					// This *shouldn't* happen, but this could wind up where the daemon
+					// is not able to communicate with an eventually up containerd
+					return
 				}
-				<-r.daemonWaitCh
-				var err error
-				client.Close()
-				os.Remove(r.GRPC.Address)
-				if err = r.startContainerd(); err != nil {
-					r.logger.WithError(err).Error("failed restarting containerd")
-				} else {
-					newClient, err := containerd.New(r.GRPC.Address)
-					if err != nil {
-						r.logger.WithError(err).Error("failed connect to containerd")
-					} else {
-						client = newClient
-					}
-				}
-			}
+
+				c.setRemote(remote)
+			}(c)
+
+			wg.Wait()
 		}
 	}
 }

components/engine/pkg/tarsum/fileinfosums.go

@@ -1,6 +1,10 @@
 package tarsum
 
-import "sort"
+import (
+	"runtime"
+	"sort"
+	"strings"
+)
 
 // FileInfoSumInterface provides an interface for accessing file checksum
 // information within a tar file. This info is accessed through interface
@@ -35,8 +39,11 @@ type FileInfoSums []FileInfoSumInterface
 
 // GetFile returns the first FileInfoSumInterface with a matching name.
 func (fis FileInfoSums) GetFile(name string) FileInfoSumInterface {
+	// We do case insensitive matching on Windows as c:\APP and c:\app are
+	// the same. See issue #33107.
 	for i := range fis {
-		if fis[i].Name() == name {
+		if (runtime.GOOS == "windows" && strings.EqualFold(fis[i].Name(), name)) ||
+			(runtime.GOOS != "windows" && fis[i].Name() == name) {
 			return fis[i]
 		}
 	}

components/engine/plugin/executor/containerd/containerd.go

@@ -16,7 +16,7 @@ import (
 )
 
 // PluginNamespace is the name used for the plugins namespace
-var PluginNamespace = "moby-plugins"
+var PluginNamespace = "plugins.moby"
 
 // ExitHandler represents an object that is called when the exit event is received from containerd
 type ExitHandler interface {

components/engine/vendor.conf

@@ -1,6 +1,6 @@
 # the following lines are in sorted order, FYI
 github.com/Azure/go-ansiterm d6e3b3328b783f23731bc4d058875b0371ff8109
-github.com/Microsoft/hcsshim v0.6.7
+github.com/Microsoft/hcsshim v0.6.8
 github.com/Microsoft/go-winio v0.4.5
 github.com/davecgh/go-spew 346938d642f2ec3594ed81d874461961cd0faa76
 github.com/docker/libtrust 9cbd2a1374f46905c68a4eb3694a130610adc62a
@@ -65,7 +65,7 @@ github.com/pborman/uuid v1.0
 google.golang.org/grpc v1.3.0
 
 # When updating, also update RUNC_COMMIT in hack/dockerfile/binaries-commits accordingly
-github.com/opencontainers/runc b2567b37d7b75eb4cf325b77297b140ea686ce8f
+github.com/opencontainers/runc 9f9c96235cc97674e935002fc3d78361b696a69e
 github.com/opencontainers/runtime-spec v1.0.0
 github.com/opencontainers/image-spec v1.0.0
 github.com/seccomp/libseccomp-golang 32f571b70023028bd57d9288c20efbcb237f3ce0
@@ -114,7 +114,7 @@ github.com/dmcgowan/go-tar go1.10
 github.com/stevvooe/ttrpc 76e68349ad9ab4d03d764c713826d31216715e4f
 
 # cluster
-github.com/docker/swarmkit 4429c763170d9ca96929249353c3270c19e7d39e
+github.com/docker/swarmkit 7598f7a937de4ad0a856012bd548009ceeb0d10e
 github.com/gogo/protobuf v0.4
 github.com/cloudflare/cfssl 7fb22c8cba7ecaf98e4082d22d65800cf45e042a
 github.com/google/certificate-transparency d90e65c3a07988180c5b1ece71791c0b6506826e

components/engine/vendor/github.com/Microsoft/hcsshim/legacy.go

@@ -121,6 +121,16 @@ func (r *legacyLayerReader) walkUntilCancelled() error {
 		if err != nil {
 			return err
 		}
+
+		// Indirect fix for https://github.com/moby/moby/issues/32838#issuecomment-343610048.
+		// Handle failure from what may be a golang bug in the conversion of
+		// UTF16 to UTF8 in files which are left in the recycle bin. Os.Lstat
+		// which is called by filepath.Walk will fail when a filename contains
+		// unicode characters. Skip the recycle bin regardless which is goodness.
+		if strings.HasPrefix(path, filepath.Join(r.root, `Files\$Recycle.Bin`)) {
+			return filepath.SkipDir
+		}
+
 		if path == r.root || path == filepath.Join(r.root, "tombstones.txt") || strings.HasSuffix(path, ".$wcidirs$") {
 			return nil
 		}

components/engine/vendor/github.com/docker/swarmkit/manager/allocator/cnmallocator/networkallocator.go

@@ -404,6 +404,11 @@ func (na *cnmNetworkAllocator) IsServiceAllocated(s *api.Service, flags ...func(
 	vipLoop:
 		for _, vip := range s.Endpoint.VirtualIPs {
 			if na.IsVIPOnIngressNetwork(vip) && networkallocator.IsIngressNetworkNeeded(s) {
+				// This checks the condition when ingress network is needed
+				// but allocation has not been done.
+				if _, ok := na.services[s.ID]; !ok {
+					return false
+				}
 				continue vipLoop
 			}
 			for _, net := range specNetworks {

components/engine/vendor/github.com/docker/swarmkit/manager/orchestrator/taskreaper/task_reaper.go

@@ -96,10 +96,10 @@ func (tr *TaskReaper) Run(ctx context.Context) {
 			// Serviceless tasks can be cleaned up right away since they are not attached to a service.
 			tr.cleanup = append(tr.cleanup, t.ID)
 		}
-		// tasks with desired state REMOVE that have progressed beyond SHUTDOWN can be cleaned up
+		// tasks with desired state REMOVE that have progressed beyond COMPLETE can be cleaned up
 		// right away
 		for _, t := range removeTasks {
-			if t.Status.State >= api.TaskStateShutdown {
+			if t.Status.State >= api.TaskStateCompleted {
 				tr.cleanup = append(tr.cleanup, t.ID)
 			}
 		}
@@ -138,10 +138,10 @@ func (tr *TaskReaper) Run(ctx context.Context) {
 				if t.Status.State >= api.TaskStateOrphaned && t.ServiceID == "" {
 					tr.cleanup = append(tr.cleanup, t.ID)
 				}
-				// add tasks that have progressed beyond SHUTDOWN and have desired state REMOVE. These
+				// add tasks that have progressed beyond COMPLETE and have desired state REMOVE. These
 				// tasks are associated with slots that were removed as part of a service scale down
 				// or service removal.
-				if t.DesiredState == api.TaskStateRemove && t.Status.State >= api.TaskStateShutdown {
+				if t.DesiredState == api.TaskStateRemove && t.Status.State >= api.TaskStateCompleted {
 					tr.cleanup = append(tr.cleanup, t.ID)
 				}
 			case api.EventUpdateCluster:
@@ -282,6 +282,8 @@ func (tr *TaskReaper) tick() {
 
 // Stop stops the TaskReaper and waits for the main loop to exit.
 func (tr *TaskReaper) Stop() {
+	// TODO(dperny) calling stop on the task reaper twice will cause a panic
+	// because we try to close a channel that will already have been closed.
 	close(tr.stopChan)
 	<-tr.doneChan
 }

components/engine/vendor/github.com/opencontainers/runc/README.md

@@ -56,7 +56,7 @@ make BUILDTAGS='seccomp apparmor'
 |-----------|------------------------------------|-------------|
 | seccomp   | Syscall filtering                  | libseccomp  |
 | selinux   | selinux process and mount labeling | <none>      |
-| apparmor  | apparmor profile support           | libapparmor |
+| apparmor  | apparmor profile support           | <none>      |
 | ambient   | ambient capability support         | kernel 4.3  |
 
 

components/engine/vendor/github.com/opencontainers/runc/libcontainer/apparmor/apparmor.go

@@ -2,15 +2,10 @@
 
 package apparmor
 
-// #cgo LDFLAGS: -lapparmor
-// #include <sys/apparmor.h>
-// #include <stdlib.h>
-import "C"
 import (
 	"fmt"
 	"io/ioutil"
 	"os"
-	"unsafe"
 )
 
 // IsEnabled returns true if apparmor is enabled for the host.
@@ -24,16 +19,36 @@ func IsEnabled() bool {
 	return false
 }
 
+func setprocattr(attr, value string) error {
+	// Under AppArmor you can only change your own attr, so use /proc/self/
+	// instead of /proc/<tid>/ like libapparmor does
+	path := fmt.Sprintf("/proc/self/attr/%s", attr)
+
+	f, err := os.OpenFile(path, os.O_WRONLY, 0)
+	if err != nil {
+		return err
+	}
+	defer f.Close()
+
+	_, err = fmt.Fprintf(f, "%s", value)
+	return err
+}
+
+// changeOnExec reimplements aa_change_onexec from libapparmor in Go
+func changeOnExec(name string) error {
+	value := "exec " + name
+	if err := setprocattr("exec", value); err != nil {
+		return fmt.Errorf("apparmor failed to apply profile: %s", err)
+	}
+	return nil
+}
+
 // ApplyProfile will apply the profile with the specified name to the process after
 // the next exec.
 func ApplyProfile(name string) error {
 	if name == "" {
 		return nil
 	}
-	cName := C.CString(name)
-	defer C.free(unsafe.Pointer(cName))
-	if _, err := C.aa_change_onexec(cName); err != nil {
-		return fmt.Errorf("apparmor failed to apply profile: %s", err)
-	}
-	return nil
+
+	return changeOnExec(name)
 }

components/engine/vendor/github.com/opencontainers/runc/libcontainer/configs/cgroup_unsupported.go

@@ -1,6 +0,0 @@
-// +build !windows,!linux,!freebsd
-
-package configs
-
-type Cgroup struct {
-}

components/engine/vendor/github.com/opencontainers/runc/libcontainer/configs/device_defaults.go

@@ -1,4 +1,4 @@
-// +build linux freebsd
+// +build linux
 
 package configs
 

components/engine/vendor/github.com/opencontainers/runc/libcontainer/devices/devices_unsupported.go

@@ -1,3 +0,0 @@
-// +build !linux
-
-package devices

components/engine/vendor/github.com/opencontainers/runc/libcontainer/system/linux.go

@@ -134,3 +134,14 @@ func RunningInUserNS() bool {
 func SetSubreaper(i int) error {
 	return unix.Prctl(PR_SET_CHILD_SUBREAPER, uintptr(i), 0, 0, 0)
 }
+
+// GetSubreaper returns the subreaper setting for the calling process
+func GetSubreaper() (int, error) {
+	var i uintptr
+
+	if err := unix.Prctl(unix.PR_GET_CHILD_SUBREAPER, uintptr(unsafe.Pointer(&i)), 0, 0, 0); err != nil {
+		return -1, err
+	}
+
+	return int(i), nil
+}

components/engine/vendor/github.com/opencontainers/runc/libcontainer/system/sysconfig.go

@@ -1,4 +1,4 @@
-// +build cgo,linux cgo,freebsd
+// +build cgo,linux
 
 package system
 

components/engine/vendor/github.com/opencontainers/runc/libcontainer/user/lookup_unsupported.go

@@ -1,38 +0,0 @@
-// +build !darwin,!dragonfly,!freebsd,!linux,!netbsd,!openbsd,!solaris
-
-package user
-
-import (
-	"io"
-	"syscall"
-)
-
-func GetPasswdPath() (string, error) {
-	return "", ErrUnsupported
-}
-
-func GetPasswd() (io.ReadCloser, error) {
-	return nil, ErrUnsupported
-}
-
-func GetGroupPath() (string, error) {
-	return "", ErrUnsupported
-}
-
-func GetGroup() (io.ReadCloser, error) {
-	return nil, ErrUnsupported
-}
-
-// CurrentUser looks up the current user by their user id in /etc/passwd. If the
-// user cannot be found (or there is no /etc/passwd file on the filesystem),
-// then CurrentUser returns an error.
-func CurrentUser() (User, error) {
-	return LookupUid(syscall.Getuid())
-}
-
-// CurrentGroup looks up the current user's group by their primary group id's
-// entry in /etc/passwd. If the group cannot be found (or there is no
-// /etc/group file on the filesystem), then CurrentGroup returns an error.
-func CurrentGroup() (Group, error) {
-	return LookupGid(syscall.Getgid())
-}

components/packaging/deb/common/rules

@@ -1,7 +1,7 @@
 #!/usr/bin/make -f
 
 VERSION ?= $(shell cat engine/VERSION)
-SYSTEMD_VERSION := $(shell dpkg-query -W -f='$${Version}\n' systemd | cut -d- -f1)
+SYSTEMD_VERSION := $(shell dpkg-query -W -f='$${Version}\n' systemd libsystemd-dev | head -1 | cut -d- -f1)
 SYSTEMD_GT_227 := $(shell [ '$(SYSTEMD_VERSION)' ] && [ '$(SYSTEMD_VERSION)' -gt 227 ] && echo true )
 
 override_dh_gencontrol: