Compare commits

..

29 Commits

Author SHA1 Message Date
b0992d220f Merge pull request #429 from andrewhsu/v
[17.12] bump version to 17.12.1-ce-rc2
2018-02-21 15:24:57 -08:00
5152611926 Merge pull request #431 from jose-bigio/17.12-changelog
[17.12] update changelog for 17.12.1-ce-rc2
2018-02-21 15:20:22 -08:00
893276b2e8 Changelog update for 17.12.1
Signed-off-by: jose-bigio <jose.bigio@docker.com>
2018-02-21 15:12:04 -08:00
319872e09a Merge pull request #433 from kolyshkin/17.12-backport-layer-not-retained
[17.12] backport layer not retained
2018-02-21 09:50:26 -08:00
71498a13be c.RWLayer: check for nil before use
Since commit e9b9e4ace294230c6b8eb has landed, there is a chance that
container.RWLayer is nil (due to some half-removed container). Let's
check the pointer before use to avoid any potential nil pointer
dereferences, resulting in a daemon crash.

Note that even without the abovementioned commit, it's better to perform
an extra check (even it's totally redundant) rather than to have a
possibility of a daemon crash. In other words, better be safe than
sorry.

[v2: add a test case for daemon.getInspectData]
[v3: add a check for container.Dead and a special error for the case]

Fixes: e9b9e4ace294230c6b8eb
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit 195893d38160c0893e326b8674e05ef6714aeaa4)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
2018-02-20 15:43:30 -08:00
deff200c90 daemon.cleanupContainer: nullify container RWLayer upon release
ReleaseRWLayer can and should only be called once (unless it returns
an error), but might be called twice in case of a failure from
`system.EnsureRemoveAll(container.Root)`. This results in the
following error:

> Error response from daemon: driver "XXX" failed to remove root filesystem for YYY: layer not retained

The obvious fix is to set container.RWLayer to nil as soon as
ReleaseRWLayer() succeeds.

Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
(cherry picked from commit e9b9e4ace294230c6b8eb010eda564a2541c4564)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2018-02-20 15:41:42 -08:00
b24bd87f20 bump version to 17.12.1-ce-rc2
Signed-off-by: Andrew Hsu <andrewhsu@docker.com>
2018-02-20 07:55:44 -08:00
e634ffeb95 Merge pull request #422 from thaJeztah/17.12-backport-36142-TaskState
[17.12] Add `REMOVE` and `ORPHANED` to TaskState
2018-02-20 07:55:11 -08:00
80c6fa7153 Merge pull request #430 from thaJeztah/17.12-backport-cleanup_daemon_root_mount
[17.12] Ensure daemon root is unmounted on shutdown
2018-02-20 07:53:20 -08:00
731f1c37f0 Merge pull request #428 from cpuguy83/backport_36055_slave_mounts_for_root
[17.12] Use rslave propagation for mounts from daemon root
2018-02-20 07:53:09 -08:00
1251f23e0d Merge pull request #416 from cpuguy83/17.12_backport_36096_use_rshared_prop_for_daemon_root
[17.12] Set daemon root to use shared propagation
2018-02-20 07:52:46 -08:00
1c24f4566e Merge pull request #415 from cpuguy83/17.12_backport_36047_graphdriver_improvements
[17.12] Do not make graphdriver homes private mounts.
2018-02-20 07:52:23 -08:00
a27f508a6c Ensure daemon root is unmounted on shutdown
This is only for the case when dockerd has had to re-mount the daemon
root as shared.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit 487c6c7e73dbb7871e80d75f176dd2a3539a2947)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2018-02-20 13:11:57 +01:00
31e4ca26a3 Merge pull request #425 from thaJeztah/17.12-backport-36083-network-inspect-created-time
[17.12] Fix issue where network inspect does not show Created time in swarm scope
2018-02-20 00:39:11 -08:00
5bc96351ab Merge pull request #391 from thaJeztah/17.12-backport-fix-mount-creation-on-start
[17.12] Re-validate Mounts on container start
2018-02-20 00:38:14 -08:00
ee633783d6 Merge pull request #393 from thaJeztah/17.12-bump-libnetwork
[17.12] bump libnetwork to 0ef1f8c94fa85acacf88b9be75ae07de91d0b2a5
2018-02-20 00:22:44 -08:00
72f6b9a3d7 Merge pull request #420 from thaJeztah/17.12-backport-support-proxy-in-splunk-driver
[17.12] Support a proxy in splunk log driver
2018-02-19 13:27:49 -08:00
7cb8389345 Use rslave propagation for mounts from daemon root
By default, if a user requests a bind mount it uses private propagation.
When the source path is a path within the daemon root this, along with
some other propagation values that the user can use, causes issues when
the daemon tries to remove a mountpoint because a container will then
have a private reference to that mount which prevents removal.

Unmouting with MNT_DETATCH can help this scenario on newer kernels, but
ultimately this is just covering up the problem and doesn't actually
free up the underlying resources until all references are destroyed.

This change does essentially 2 things:

1. Change the default propagation when unspecified to `rslave` when the
source path is within the daemon root path or a parent of the daemon
root (because everything is using rbinds).
2. Creates a validation error on create when the user tries to specify
an unacceptable propagation mode for these paths...
basically the only two acceptable modes are `rslave` and `rshared`.

In cases where we have used the new default propagation but the
underlying filesystem is not setup to handle it (fs must hvae at least
rshared propagation) instead of erroring out like we normally would,
this falls back to the old default mode of `private`, which preserves
backwards compatibility.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit 589a0afa8cbe39b6512662fd1705873e2d236dd0)
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2018-02-15 10:20:07 -05:00
a22eabf65d Do not recursive unmount on cleanup of zfs/btrfs
This was added in #36047 just as a way to make sure the tree is fully
unmounted on shutdown.

For ZFS this could be a breaking change since there was no unmount before.
Someone could have setup the zfs tree themselves. It would be better, if
we really do want the cleanup to actually the unpacked layers checking
for mounts rather than a blind recursive unmount of the root.

BTRFS does not use mounts and does not need to unmount anyway.
These was only an unmount to begin with because for some reason the
btrfs tree was being moutned with `private` propagation.

For the other graphdrivers that still have a recursive unmount here...
these were already being unmounted and performing the recursive unmount
shouldn't break anything. If anyone had anything mounted at the
graphdriver location it would have been unmounted on shutdown anyway.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit 2fe4f888bee52b1f256d6fa5e20f9b061d30221c)
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2018-02-14 09:56:00 -05:00
68dbbc33ca Fix issue where network inspect does not show Created time in swarm scope
This fix tries to address the issue raised in 36083 where
`network inspect` does not show Created time if the network is
created in swarm scope.

The issue was that Created was not converted from swarm api.
This fix addresses the issue.

An unit test has been added.

This fix fixes 36083.

Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
(cherry picked from commit 090c439fb8a863731cc80fcb9932ce5958d8166d)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2018-02-13 16:05:03 +01:00
246956ffb1 Add REMOVE and ORPHANED to TaskState
This fix tries to address the issue raised in 36142 where
there are discrepancies between Swarm API and swagger.yaml.

This fix adds two recently added state `REMOVE` and `ORPHANED` to TaskState.

This fix fixes 36142.

Signed-off-by: Yong Tang <yong.tang.github@outlook.com>
(cherry picked from commit a40687f5ac7df27bc6c6c3a6f69513a397a1a05a)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2018-02-13 14:07:02 +01:00
5fcd931d5f update vendor
Signed-off-by: Daniel Nephin <dnephin@docker.com>
(cherry picked from commit 7d296522f68a80d540b20753ea0648171ee12825)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2018-02-10 13:10:36 -08:00
38ba0c6ef7 Support a proxy in splunk log driver
Signed-off-by: Daniel Nephin <dnephin@docker.com>
(cherry picked from commit 3c4537d5b33d951237ea5e4cc123953eda7a37e7)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2018-02-10 13:10:06 -08:00
b1267c5341 update gotestyourself
pickup changes which use t.Helper()

Signed-off-by: Daniel Nephin <dnephin@docker.com>
(cherry picked from commit 4ac4b690f78a645cc50030b81077fd5319b53501)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2018-02-10 13:09:57 -08:00
b8eb1db1d3 Set daemon root to use shared propagation
This change sets an explicit mount propagation for the daemon root.
This is useful for people who need to bind mount the docker daemon root
into a container.

Since bind mounting the daemon root should only ever happen with at
least `rlsave` propagation (to prevent the container from holding
references to mounts making it impossible for the daemon to clean up its
resources), we should make sure the user is actually able to this.

Most modern systems have shared root (`/`) propagation by default
already, however there are some cases where this may not be so
(e.g. potentially docker-in-docker scenarios, but also other cases).
So this just gives the daemon a little more control here and provides
a more uniform experience across different systems.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit a510192b86e7eb1e1112f3f625d80687fdec6578)
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2018-02-07 15:23:33 -05:00
8136093f88 Do not make graphdriver homes private mounts.
The idea behind making the graphdrivers private is to prevent leaking
mounts into other namespaces.
Unfortunately this is not really what happens.

There is one case where this does work, and that is when the namespace
was created before the daemon's namespace.
However with systemd each system servie winds up with it's own mount
namespace. This causes a race betwen daemon startup and other system
services as to if the mount is actually private.

This also means there is a negative impact when other system services
are started while the daemon is running.

Basically there are too many things that the daemon does not have
control over (nor should it) to be able to protect against these kinds
of leakages. One thing is certain, setting the graphdriver roots to
private disconnects the mount ns heirarchy preventing propagation of
unmounts... new mounts are of course not propagated either, but the
behavior is racey (or just bad in the case of restarting services)... so
it's better to just be able to keep mount propagation in tact.

It also does not protect situations like `-v
/var/lib/docker:/var/lib/docker` where all mounts are recursively bound
into the container anyway.

Signed-off-by: Brian Goff <cpuguy83@gmail.com>
(cherry picked from commit 9803272f2db84df7955b16c0d847ad72cdc494d1)
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
2018-02-07 14:34:47 -05:00
4822efab31 bump libnetwork to 0ef1f8c94fa85acacf88b9be75ae07de91d0b2a5
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2018-01-26 16:39:26 -08:00
e3eb25711d [17.12] bump libnetwork to cb462375d91926adfce98bd8a1f47a325ed6f8f2
Brings in a cherry pick of 76bfcb0eb1

In sandbox creation we disable IPV6 config. But this causes problem in live-restore case
where all IPV6 configs are wiped out on running container. Hence extra check has been added
take care of this issue.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2018-01-22 18:27:18 -08:00
7d2c7004af Re-validate Mounts on container start
Validation of Mounts was only performed on container _creation_, not on
container _start_. As a result, if the host-path no longer existed
when the container was started, a directory was created in the given
location.

This is the wrong behavior, because when using the `Mounts` API, host paths
should never be created, and an error should be produced instead.

This patch adds a validation step on container start, and produces an
error if the host path is not found.

Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
(cherry picked from commit 7cb96ba308dc53824d2203fd343a4a297d17976e)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
2018-01-20 01:52:05 +01:00
64 changed files with 3724 additions and 240 deletions

View File

@ -8,7 +8,7 @@ be found.
**IMPORTANT**: You must stop all containers and plugins **BEFORE** upgrading to Docker CE 17.12.
See related PR: [moby/moby#35812](https://github.com/moby/moby/pull/35812)
## 17.12.1-ce (2017-02-DD)
## 17.12.1-ce (2018-02-DD)
### Client
- Fix `node-generic-resource` typo [moby/moby#35970](https://github.com/moby/moby/pull/35970) and [moby/moby#36125](https://github.com/moby/moby/pull/36125)
@ -16,10 +16,13 @@ See related PR: [moby/moby#35812](https://github.com/moby/moby/pull/35812)
### Logging
- awslogs: fix batch size calculation for large logs [moby/moby#35726](https://github.com/moby/moby/pull/35726)
* Support a proxy in splunk log driver [moby/moby#36220](https://github.com/moby/moby/pull/36220)
### Networking
- Fix ingress network when upgrading from 17.09 to 17.12 [moby/moby#36003](https://github.com/moby/moby/pull/36003)
* Add verbose info to partial overlay ID [moby/moby#35989](https://github.com/moby/moby/pull/35989)
- Fix IPv6 networking being deconfigured if live-restore is being enabled [docker/libnetwork#2043](https://github.com/docker/libnetwork/pull/2043)
- Fix watchMiss thread context [docker/libnetwork#2051](https://github.com/docker/libnetwork/pull/2051)
### Packaging
- Set TasksMax in docker.service [docker/docker-ce-packaging#78](https://github.com/docker/docker-ce-packaging/pull/78)
@ -34,6 +37,16 @@ See related PR: [moby/moby#35812](https://github.com/moby/moby/pull/35812)
- Honor `DOCKER_RAMDISK` with containerd 1.0 [moby/moby#35957](https://github.com/moby/moby/pull/35957)
- Update runc to fix hang during start and exec [moby/moby#36097](https://github.com/moby/moby/pull/36097)
- Windows: Vendor of Microsoft/hcsshim @v.0.6.8 partial fix for import layer failing [moby/moby#35924](https://github.com/moby/moby/pull/35924)
* Do not make graphdriver homes private mounts [moby/moby#36047](https://github.com/moby/moby/pull/36047)
* Use rslave propogation for mounts from daemon root [moby/moby#36055](https://github.com/moby/moby/pull/36055)
* Set daemon root to use shared mount propagation [moby/moby#36096](https://github.com/moby/moby/pull/36096)
* Validate that mounted paths exist when container is started, not just during creation [moby/moby#35833](https://github.com/moby/moby/pull/35833)
* Add `REMOVE` and `ORPHANED` to TaskState [moby/moby#36146](https://github.com/moby/moby/pull/36146)
- Fix issue where network inspect does not show Created time for networks in swarm scope [moby/moby#36095](https://github.com/moby/moby/pull/36095)
* Nullify container read write layer upon release [moby/moby#36130](https://github.com/moby/moby/pull/36160) and [moby/moby#36343](https://github.com/moby/moby/pull/36242)
### Swarm
* Remove watchMiss from swarm mode [docker/libnetwork#2047](https://github.com/docker/libnetwork/pull/2047)
## 17.12.0-ce (2017-12-27)
@ -74,7 +87,7 @@ See related PR: [moby/moby#35812](https://github.com/moby/moby/pull/35812)
* Logentries line-only logopt fix to maintain backwards compatibility [moby/moby#35628](https://github.com/moby/moby/pull/35628)
+ Add `--until` flag for docker logs [moby/moby#32914](https://github.com/moby/moby/pull/32914)
+ Add gelf log driver plugin to Windows build [moby/moby#35073](https://github.com/moby/moby/pull/35073)
* Set timeout on splunk batch send [moby/moby#35496](https://github.com/moby/moby/pull/35496)
* Set timeout on Splunk batch send [moby/moby#35496](https://github.com/moby/moby/pull/35496)
* Update Graylog2/go-gelf [moby/moby#35765](https://github.com/moby/moby/pull/35765)
### Networking

View File

@ -1 +1 @@
17.12.1-ce-rc1
17.12.1-ce-rc2

View File

@ -1 +1 @@
17.12.1-ce-rc1
17.12.1-ce-rc2

View File

@ -2788,6 +2788,8 @@ definitions:
- "shutdown"
- "failed"
- "rejected"
- "remove"
- "orphaned"
Task:
type: "object"

View File

@ -36,6 +36,10 @@ const (
TaskStateFailed TaskState = "failed"
// TaskStateRejected REJECTED
TaskStateRejected TaskState = "rejected"
// TaskStateRemove REMOVE
TaskStateRemove TaskState = "remove"
// TaskStateOrphaned ORPHANED
TaskStateOrphaned TaskState = "orphaned"
)
// Task represents a task.

View File

@ -22,6 +22,9 @@ func (daemon *Daemon) ContainerChanges(name string) ([]archive.Change, error) {
container.Lock()
defer container.Unlock()
if container.RWLayer == nil {
return nil, errors.New("RWLayer of container " + name + " is unexpectedly nil")
}
c, err := container.RWLayer.Changes()
if err != nil {
return nil, err

View File

@ -168,6 +168,7 @@ func BasicNetworkFromGRPC(n swarmapi.Network) basictypes.NetworkResource {
Ingress: IsIngressNetwork(&n),
Labels: n.Spec.Annotations.Labels,
}
nr.Created, _ = gogotypes.TimestampFromProto(n.Meta.CreatedAt)
if n.Spec.GetNetwork() != "" {
nr.ConfigFrom = networktypes.ConfigReference{

View File

@ -0,0 +1,34 @@
package convert
import (
"testing"
"time"
swarmapi "github.com/docker/swarmkit/api"
gogotypes "github.com/gogo/protobuf/types"
)
func TestNetworkConvertBasicNetworkFromGRPCCreatedAt(t *testing.T) {
expected, err := time.Parse("Jan 2, 2006 at 3:04pm (MST)", "Jan 10, 2018 at 7:54pm (PST)")
if err != nil {
t.Fatal(err)
}
createdAt, err := gogotypes.TimestampProto(expected)
if err != nil {
t.Fatal(err)
}
nw := swarmapi.Network{
Meta: swarmapi.Meta{
Version: swarmapi.Version{
Index: 1,
},
CreatedAt: createdAt,
},
}
n := BasicNetworkFromGRPC(nw)
if !n.Created.Equal(expected) {
t.Fatalf("expected time %s; received %s", expected, n.Created)
}
}

View File

@ -19,6 +19,7 @@ import (
"github.com/docker/docker/pkg/system"
"github.com/docker/docker/pkg/truncindex"
"github.com/docker/docker/runconfig"
"github.com/docker/docker/volume"
"github.com/docker/go-connections/nat"
"github.com/opencontainers/selinux/go-selinux/label"
"github.com/pkg/errors"
@ -293,6 +294,14 @@ func (daemon *Daemon) verifyContainerSettings(platform string, hostConfig *conta
return nil, errors.Errorf("can't create 'AutoRemove' container with restart policy")
}
// Validate mounts; check if host directories still exist
parser := volume.NewParser(platform)
for _, cfg := range hostConfig.Mounts {
if err := parser.ValidateMountConfig(&cfg); err != nil {
return nil, err
}
}
for _, extraHost := range hostConfig.ExtraHosts {
if _, err := opts.ValidateExtraHost(extraHost); err != nil {
return nil, err

View File

@ -1056,6 +1056,9 @@ func (daemon *Daemon) Shutdown() error {
// Mount sets container.BaseFS
// (is it not set coming in? why is it unset?)
func (daemon *Daemon) Mount(container *container.Container) error {
if container.RWLayer == nil {
return errors.New("RWLayer of container " + container.ID + " is unexpectedly nil")
}
dir, err := container.RWLayer.Mount(container.GetMountLabel())
if err != nil {
return err
@ -1078,6 +1081,9 @@ func (daemon *Daemon) Mount(container *container.Container) error {
// Unmount unsets the container base filesystem
func (daemon *Daemon) Unmount(container *container.Container) error {
if container.RWLayer == nil {
return errors.New("RWLayer of container " + container.ID + " is unexpectedly nil")
}
if err := container.RWLayer.Unmount(); err != nil {
logrus.Errorf("Error unmounting container %s: %s", container.ID, err)
return err

View File

@ -10,6 +10,7 @@ import (
"github.com/docker/docker/pkg/fileutils"
"github.com/docker/docker/pkg/mount"
"github.com/pkg/errors"
"github.com/sirupsen/logrus"
)
@ -67,9 +68,29 @@ func (daemon *Daemon) cleanupMountsFromReaderByID(reader io.Reader, id string, u
return nil
}
// cleanupMounts umounts shm/mqueue mounts for old containers
// cleanupMounts umounts used by container resources and the daemon root mount
func (daemon *Daemon) cleanupMounts() error {
return daemon.cleanupMountsByID("")
if err := daemon.cleanupMountsByID(""); err != nil {
return err
}
infos, err := mount.GetMounts()
if err != nil {
return errors.Wrap(err, "error reading mount table for cleanup")
}
info := getMountInfo(infos, daemon.root)
// `info.Root` here is the root mountpoint of the passed in path (`daemon.root`).
// The ony cases that need to be cleaned up is when the daemon has performed a
// `mount --bind /daemon/root /daemon/root && mount --make-shared /daemon/root`
// This is only done when the daemon is started up and `/daemon/root` is not
// already on a shared mountpoint.
if !shouldUnmountRoot(daemon.root, info) {
return nil
}
logrus.WithField("mountpoint", daemon.root).Debug("unmounting daemon root")
return mount.Unmount(daemon.root)
}
func getCleanPatterns(id string) (regexps []*regexp.Regexp) {
@ -91,3 +112,16 @@ func getCleanPatterns(id string) (regexps []*regexp.Regexp) {
func getRealPath(path string) (string, error) {
return fileutils.ReadSymlinkedDirectory(path)
}
func shouldUnmountRoot(root string, info *mount.Info) bool {
if info == nil {
return false
}
if info.Mountpoint != root {
return false
}
if !strings.HasSuffix(root, info.Root) {
return false
}
return hasMountinfoOption(info.Optional, sharedPropagationOption)
}

View File

@ -10,6 +10,7 @@ import (
"github.com/docker/docker/container"
"github.com/docker/docker/oci"
"github.com/docker/docker/pkg/idtools"
"github.com/docker/docker/pkg/mount"
"github.com/stretchr/testify/assert"
)
@ -164,3 +165,66 @@ func TestValidateContainerIsolationLinux(t *testing.T) {
_, err := d.verifyContainerSettings("linux", &containertypes.HostConfig{Isolation: containertypes.IsolationHyperV}, nil, false)
assert.EqualError(t, err, "invalid isolation 'hyperv' on linux")
}
func TestShouldUnmountRoot(t *testing.T) {
for _, test := range []struct {
desc string
root string
info *mount.Info
expect bool
}{
{
desc: "root is at /",
root: "/docker",
info: &mount.Info{Root: "/docker", Mountpoint: "/docker"},
expect: true,
},
{
desc: "not a mountpoint",
root: "/docker",
info: nil,
expect: false,
},
{
desc: "root is at in a submount from `/`",
root: "/foo/docker",
info: &mount.Info{Root: "/docker", Mountpoint: "/foo/docker"},
expect: true,
},
{
desc: "root is mounted in from a parent mount namespace same root dir", // dind is an example of this
root: "/docker",
info: &mount.Info{Root: "/docker/volumes/1234657/_data", Mountpoint: "/docker"},
expect: false,
},
{
desc: "root is mounted in from a parent mount namespace different root dir",
root: "/foo/bar",
info: &mount.Info{Root: "/docker/volumes/1234657/_data", Mountpoint: "/foo/bar"},
expect: false,
},
} {
t.Run(test.desc, func(t *testing.T) {
for _, options := range []struct {
desc string
Optional string
expect bool
}{
{desc: "shared", Optional: "shared:", expect: true},
{desc: "slave", Optional: "slave:", expect: false},
{desc: "private", Optional: "private:", expect: false},
} {
t.Run(options.desc, func(t *testing.T) {
expect := options.expect
if expect {
expect = test.expect
}
if test.info != nil {
test.info.Optional = options.Optional
}
assert.Equal(t, expect, shouldUnmountRoot(test.root, test.info))
})
}
})
}
}

View File

@ -29,6 +29,7 @@ import (
"github.com/docker/docker/pkg/containerfs"
"github.com/docker/docker/pkg/idtools"
"github.com/docker/docker/pkg/ioutils"
"github.com/docker/docker/pkg/mount"
"github.com/docker/docker/pkg/parsers"
"github.com/docker/docker/pkg/parsers/kernel"
"github.com/docker/docker/pkg/sysinfo"
@ -1228,6 +1229,12 @@ func setupDaemonRoot(config *config.Config, rootDir string, rootIDs idtools.IDPa
}
}
}
if err := ensureSharedOrSlave(config.Root); err != nil {
if err := mount.MakeShared(config.Root); err != nil {
logrus.WithError(err).WithField("dir", config.Root).Warn("Could not set daemon root propagation to shared, this is not generally critical but may cause some functionality to not work or fallback to less desirable behavior")
}
}
return nil
}

View File

@ -124,6 +124,7 @@ func (daemon *Daemon) cleanupContainer(container *container.Container, forceRemo
container.SetRemovalError(e)
return e
}
container.RWLayer = nil
}
if err := system.EnsureRemoveAll(container.Root); err != nil {

View File

@ -136,10 +136,6 @@ func Init(root string, options []string, uidMaps, gidMaps []idtools.IDMap) (grap
return nil, err
}
if err := mountpk.MakePrivate(root); err != nil {
return nil, err
}
// Populate the dir structure
for _, p := range paths {
if err := idtools.MkdirAllAndChown(path.Join(root, p), 0700, idtools.IDPair{UID: rootUID, GID: rootGID}); err != nil {
@ -610,7 +606,7 @@ func (a *Driver) Cleanup() error {
logrus.Debugf("aufs error unmounting %s: %s", m, err)
}
}
return mountpk.Unmount(a.root)
return mountpk.RecursiveUnmount(a.root)
}
func (a *Driver) aufsMount(ro []string, rw, target, mountLabel string) (err error) {

View File

@ -29,7 +29,6 @@ import (
"github.com/docker/docker/daemon/graphdriver"
"github.com/docker/docker/pkg/containerfs"
"github.com/docker/docker/pkg/idtools"
"github.com/docker/docker/pkg/mount"
"github.com/docker/docker/pkg/parsers"
"github.com/docker/docker/pkg/system"
"github.com/docker/go-units"
@ -77,10 +76,6 @@ func Init(home string, options []string, uidMaps, gidMaps []idtools.IDMap) (grap
return nil, err
}
if err := mount.MakePrivate(home); err != nil {
return nil, err
}
opt, userDiskQuota, err := parseOptions(options)
if err != nil {
return nil, err
@ -167,7 +162,7 @@ func (d *Driver) Cleanup() error {
return err
}
return mount.Unmount(d.home)
return nil
}
func free(p *C.char) {

View File

@ -42,10 +42,6 @@ func Init(home string, options []string, uidMaps, gidMaps []idtools.IDMap) (grap
return nil, err
}
if err := mount.MakePrivate(home); err != nil {
return nil, err
}
d := &Driver{
DeviceSet: deviceSet,
home: home,
@ -127,7 +123,7 @@ func (d *Driver) GetMetadata(id string) (map[string]string, error) {
func (d *Driver) Cleanup() error {
err := d.DeviceSet.Shutdown(d.home)
if err2 := mount.Unmount(d.home); err == nil {
if err2 := mount.RecursiveUnmount(d.home); err == nil {
err = err2
}

View File

@ -164,10 +164,6 @@ func Init(home string, options []string, uidMaps, gidMaps []idtools.IDMap) (grap
return nil, err
}
if err := mount.MakePrivate(home); err != nil {
return nil, err
}
d := &Driver{
home: home,
uidMaps: uidMaps,
@ -248,7 +244,7 @@ func (d *Driver) GetMetadata(id string) (map[string]string, error) {
// is being shutdown. For now, we just have to unmount the bind mounted
// we had created.
func (d *Driver) Cleanup() error {
return mount.Unmount(d.home)
return mount.RecursiveUnmount(d.home)
}
// CreateReadWrite creates a layer that is writable for use as a container

View File

@ -200,10 +200,6 @@ func Init(home string, options []string, uidMaps, gidMaps []idtools.IDMap) (grap
return nil, err
}
if err := mount.MakePrivate(home); err != nil {
return nil, err
}
d := &Driver{
home: home,
uidMaps: uidMaps,
@ -334,7 +330,7 @@ func (d *Driver) GetMetadata(id string) (map[string]string, error) {
// is being shutdown. For now, we just have to unmount the bind mounted
// we had created.
func (d *Driver) Cleanup() error {
return mount.Unmount(d.home)
return mount.RecursiveUnmount(d.home)
}
// CreateReadWrite creates a layer that is writable for use as a container

View File

@ -108,9 +108,6 @@ func Init(base string, opt []string, uidMaps, gidMaps []idtools.IDMap) (graphdri
return nil, fmt.Errorf("Failed to create '%s': %v", base, err)
}
if err := mount.MakePrivate(base); err != nil {
return nil, err
}
d := &Driver{
dataset: rootDataset,
options: options,
@ -181,7 +178,8 @@ func (d *Driver) String() string {
return "zfs"
}
// Cleanup is used to implement graphdriver.ProtoDriver. There is no cleanup required for this driver.
// Cleanup is called on daemon shutdown, it is a no-op for ZFS.
// TODO(@cpuguy83): Walk layer tree and check mounts?
func (d *Driver) Cleanup() error {
return nil
}

View File

@ -1,6 +1,7 @@
package daemon
import (
"errors"
"fmt"
"time"
@ -183,14 +184,24 @@ func (daemon *Daemon) getInspectData(container *container.Container) (*types.Con
contJSONBase.GraphDriver.Name = container.Driver
if container.RWLayer == nil {
if container.Dead {
return contJSONBase, nil
}
return nil, systemError{errors.New("RWLayer of container " + container.ID + " is unexpectedly nil")}
}
graphDriverData, err := container.RWLayer.Metadata()
// If container is marked as Dead, the container's graphdriver metadata
// could have been removed, it will cause error if we try to get the metadata,
// we can ignore the error if the container is dead.
if err != nil && !container.Dead {
return nil, systemError{err}
if err != nil {
if !container.Dead {
return nil, systemError{err}
}
} else {
contJSONBase.GraphDriver.Data = graphDriverData
}
contJSONBase.GraphDriver.Data = graphDriverData
return contJSONBase, nil
}

View File

@ -0,0 +1,33 @@
package daemon // import "github.com/docker/docker/daemon"
import (
"testing"
containertypes "github.com/docker/docker/api/types/container"
"github.com/docker/docker/container"
"github.com/docker/docker/daemon/config"
"github.com/docker/docker/daemon/exec"
"github.com/stretchr/testify/assert"
)
func TestGetInspectData(t *testing.T) {
c := &container.Container{
ID: "inspect-me",
HostConfig: &containertypes.HostConfig{},
State: container.NewState(),
ExecCommands: exec.NewStore(),
}
d := &Daemon{
linkIndex: newLinkIndex(),
configStore: &config.Config{},
}
_, err := d.getInspectData(c)
assert.Error(t, err)
c.Dead = true
_, err = d.getInspectData(c)
assert.NoError(t, err)
}

View File

@ -215,6 +215,7 @@ func New(info logger.Info) (logger.Logger, error) {
transport := &http.Transport{
TLSClientConfig: tlsConfig,
Proxy: http.ProxyFromEnvironment,
}
client := &http.Client{
Transport: transport,

View File

@ -4,12 +4,14 @@ import (
"compress/gzip"
"context"
"fmt"
"net/http"
"os"
"runtime"
"testing"
"time"
"github.com/docker/docker/daemon/logger"
"github.com/gotestyourself/gotestyourself/env"
"github.com/stretchr/testify/require"
)
@ -82,6 +84,36 @@ func TestNewMissedToken(t *testing.T) {
}
}
func TestNewWithProxy(t *testing.T) {
proxy := "http://proxy.testing:8888"
reset := env.Patch(t, "HTTP_PROXY", proxy)
defer reset()
// must not be localhost
splunkURL := "http://example.com:12345"
logger, err := New(logger.Info{
Config: map[string]string{
splunkURLKey: splunkURL,
splunkTokenKey: "token",
splunkVerifyConnectionKey: "false",
},
ContainerID: "containeriid",
})
require.NoError(t, err)
splunkLogger := logger.(*splunkLoggerInline)
proxyFunc := splunkLogger.transport.Proxy
require.NotNil(t, proxyFunc)
req, err := http.NewRequest("GET", splunkURL, nil)
require.NoError(t, err)
proxyURL, err := proxyFunc(req)
require.NoError(t, err)
require.NotNil(t, proxyURL)
require.Equal(t, proxy, proxyURL.String())
}
// Test default settings
func TestDefault(t *testing.T) {
hec := NewHTTPEventCollectorMock(t)

View File

@ -24,6 +24,7 @@ import (
"github.com/opencontainers/runc/libcontainer/devices"
"github.com/opencontainers/runc/libcontainer/user"
specs "github.com/opencontainers/runtime-spec/specs-go"
"github.com/pkg/errors"
"github.com/sirupsen/logrus"
"golang.org/x/sys/unix"
)
@ -419,52 +420,46 @@ func getSourceMount(source string) (string, string, error) {
return "", "", fmt.Errorf("Could not find source mount of %s", source)
}
const (
sharedPropagationOption = "shared:"
slavePropagationOption = "master:"
)
// hasMountinfoOption checks if any of the passed any of the given option values
// are set in the passed in option string.
func hasMountinfoOption(opts string, vals ...string) bool {
for _, opt := range strings.Split(opts, " ") {
for _, val := range vals {
if strings.HasPrefix(opt, val) {
return true
}
}
}
return false
}
// Ensure mount point on which path is mounted, is shared.
func ensureShared(path string) error {
sharedMount := false
sourceMount, optionalOpts, err := getSourceMount(path)
if err != nil {
return err
}
// Make sure source mount point is shared.
optsSplit := strings.Split(optionalOpts, " ")
for _, opt := range optsSplit {
if strings.HasPrefix(opt, "shared:") {
sharedMount = true
break
}
}
if !sharedMount {
return fmt.Errorf("path %s is mounted on %s but it is not a shared mount", path, sourceMount)
if !hasMountinfoOption(optionalOpts, sharedPropagationOption) {
return errors.Errorf("path %s is mounted on %s but it is not a shared mount", path, sourceMount)
}
return nil
}
// Ensure mount point on which path is mounted, is either shared or slave.
func ensureSharedOrSlave(path string) error {
sharedMount := false
slaveMount := false
sourceMount, optionalOpts, err := getSourceMount(path)
if err != nil {
return err
}
// Make sure source mount point is shared.
optsSplit := strings.Split(optionalOpts, " ")
for _, opt := range optsSplit {
if strings.HasPrefix(opt, "shared:") {
sharedMount = true
break
} else if strings.HasPrefix(opt, "master:") {
slaveMount = true
break
}
}
if !sharedMount && !slaveMount {
return fmt.Errorf("path %s is mounted on %s but it is not a shared or slave mount", path, sourceMount)
if !hasMountinfoOption(optionalOpts, sharedPropagationOption, slavePropagationOption) {
return errors.Errorf("path %s is mounted on %s but it is not a shared or slave mount", path, sourceMount)
}
return nil
}
@ -604,7 +599,8 @@ func setMounts(daemon *Daemon, s *specs.Spec, c *container.Container, mounts []c
//
// For private volumes any root propagation value should work.
pFlag := mountPropagationMap[m.Propagation]
if pFlag == mount.SHARED || pFlag == mount.RSHARED {
switch pFlag {
case mount.SHARED, mount.RSHARED:
if err := ensureShared(m.Source); err != nil {
return err
}
@ -612,13 +608,34 @@ func setMounts(daemon *Daemon, s *specs.Spec, c *container.Container, mounts []c
if rootpg != mount.SHARED && rootpg != mount.RSHARED {
s.Linux.RootfsPropagation = mountPropagationReverseMap[mount.SHARED]
}
} else if pFlag == mount.SLAVE || pFlag == mount.RSLAVE {
case mount.SLAVE, mount.RSLAVE:
var fallback bool
if err := ensureSharedOrSlave(m.Source); err != nil {
return err
// For backwards compatability purposes, treat mounts from the daemon root
// as special since we automatically add rslave propagation to these mounts
// when the user did not set anything, so we should fallback to the old
// behavior which is to use private propagation which is normally the
// default.
if !strings.HasPrefix(m.Source, daemon.root) && !strings.HasPrefix(daemon.root, m.Source) {
return err
}
cm, ok := c.MountPoints[m.Destination]
if !ok {
return err
}
if cm.Spec.BindOptions != nil && cm.Spec.BindOptions.Propagation != "" {
// This means the user explicitly set a propagation, do not fallback in that case.
return err
}
fallback = true
logrus.WithField("container", c.ID).WithField("source", m.Source).Warn("Falling back to default propagation for bind source in daemon root")
}
rootpg := mountPropagationMap[s.Linux.RootfsPropagation]
if rootpg != mount.SHARED && rootpg != mount.RSHARED && rootpg != mount.SLAVE && rootpg != mount.RSLAVE {
s.Linux.RootfsPropagation = mountPropagationReverseMap[mount.RSLAVE]
if !fallback {
rootpg := mountPropagationMap[s.Linux.RootfsPropagation]
if rootpg != mount.SHARED && rootpg != mount.RSHARED && rootpg != mount.SLAVE && rootpg != mount.RSLAVE {
s.Linux.RootfsPropagation = mountPropagationReverseMap[mount.RSLAVE]
}
}
}

View File

@ -1,6 +1,7 @@
package daemon
import (
"errors"
"fmt"
"io/ioutil"
"path/filepath"
@ -145,6 +146,9 @@ func (daemon *Daemon) createSpec(c *container.Container) (*specs.Spec, error) {
// Reverse order, expecting parent most first
s.Windows.LayerFolders = append([]string{layerPath}, s.Windows.LayerFolders...)
}
if c.RWLayer == nil {
return nil, errors.New("RWLayer of container " + c.ID + " is unexpectedly nil")
}
m, err := c.RWLayer.Metadata()
if err != nil {
return nil, fmt.Errorf("failed to get layer metadata - %s", err)

View File

@ -10,6 +10,7 @@ import (
"github.com/docker/docker/api/types"
containertypes "github.com/docker/docker/api/types/container"
"github.com/docker/docker/api/types/mount"
mounttypes "github.com/docker/docker/api/types/mount"
"github.com/docker/docker/container"
"github.com/docker/docker/volume"
@ -145,6 +146,13 @@ func (daemon *Daemon) registerMountPoints(container *container.Container, hostCo
if err != nil {
return err
}
needsSlavePropagation, err := daemon.validateBindDaemonRoot(bind.Spec)
if err != nil {
return err
}
if needsSlavePropagation {
bind.Propagation = mount.PropagationRSlave
}
// #10618
_, tmpfsExists := hostConfig.Tmpfs[bind.Destination]
@ -177,6 +185,13 @@ func (daemon *Daemon) registerMountPoints(container *container.Container, hostCo
if err != nil {
return validationError{err}
}
needsSlavePropagation, err := daemon.validateBindDaemonRoot(mp.Spec)
if err != nil {
return err
}
if needsSlavePropagation {
mp.Propagation = mount.PropagationRSlave
}
if binds[mp.Destination] {
return duplicateMountPointError(cfg.Target)

View File

@ -0,0 +1,35 @@
package daemon
import (
"strings"
"github.com/docker/docker/api/types/mount"
"github.com/pkg/errors"
)
// validateBindDaemonRoot ensures that if a given mountpoint's source is within
// the daemon root path, that the propagation is setup to prevent a container
// from holding private refereneces to a mount within the daemon root, which
// can cause issues when the daemon attempts to remove the mountpoint.
func (daemon *Daemon) validateBindDaemonRoot(m mount.Mount) (bool, error) {
if m.Type != mount.TypeBind {
return false, nil
}
// check if the source is within the daemon root, or if the daemon root is within the source
if !strings.HasPrefix(m.Source, daemon.root) && !strings.HasPrefix(daemon.root, m.Source) {
return false, nil
}
if m.BindOptions == nil {
return true, nil
}
switch m.BindOptions.Propagation {
case mount.PropagationRSlave, mount.PropagationRShared, "":
return m.BindOptions.Propagation == "", nil
default:
}
return false, validationError{errors.Errorf(`invalid mount config: must use either propagation mode "rslave" or "rshared" when mount source is within the daemon root, daemon root: %q, bind mount source: %q, propagation: %q`, daemon.root, m.Source, m.BindOptions.Propagation)}
}

View File

@ -0,0 +1,56 @@
package daemon
import (
"path/filepath"
"testing"
"github.com/docker/docker/api/types/mount"
)
func TestBindDaemonRoot(t *testing.T) {
t.Parallel()
d := &Daemon{root: "/a/b/c/daemon"}
for _, test := range []struct {
desc string
opts *mount.BindOptions
needsProp bool
err bool
}{
{desc: "nil propagation settings", opts: nil, needsProp: true, err: false},
{desc: "empty propagation settings", opts: &mount.BindOptions{}, needsProp: true, err: false},
{desc: "private propagation", opts: &mount.BindOptions{Propagation: mount.PropagationPrivate}, err: true},
{desc: "rprivate propagation", opts: &mount.BindOptions{Propagation: mount.PropagationRPrivate}, err: true},
{desc: "slave propagation", opts: &mount.BindOptions{Propagation: mount.PropagationSlave}, err: true},
{desc: "rslave propagation", opts: &mount.BindOptions{Propagation: mount.PropagationRSlave}, err: false, needsProp: false},
{desc: "shared propagation", opts: &mount.BindOptions{Propagation: mount.PropagationShared}, err: true},
{desc: "rshared propagation", opts: &mount.BindOptions{Propagation: mount.PropagationRSlave}, err: false, needsProp: false},
} {
t.Run(test.desc, func(t *testing.T) {
test := test
for desc, source := range map[string]string{
"source is root": d.root,
"source is subpath": filepath.Join(d.root, "a", "b"),
"source is parent": filepath.Dir(d.root),
"source is /": "/",
} {
t.Run(desc, func(t *testing.T) {
mount := mount.Mount{
Type: mount.TypeBind,
Source: source,
BindOptions: test.opts,
}
needsProp, err := d.validateBindDaemonRoot(mount)
if (err != nil) != test.err {
t.Fatalf("expected err=%v, got: %v", test.err, err)
}
if test.err {
return
}
if test.needsProp != needsProp {
t.Fatalf("expected needsProp=%v, got: %v", test.needsProp, needsProp)
}
})
}
})
}
}

View File

@ -5,6 +5,7 @@ package daemon
import (
"sort"
"github.com/docker/docker/api/types/mount"
"github.com/docker/docker/container"
"github.com/docker/docker/pkg/idtools"
"github.com/docker/docker/volume"
@ -46,3 +47,7 @@ func (daemon *Daemon) setupMounts(c *container.Container) ([]container.Mount, er
func setBindModeIfNull(bind *volume.MountPoint) {
return
}
func (daemon *Daemon) validateBindDaemonRoot(m mount.Mount) (bool, error) {
return false, nil
}

View File

@ -0,0 +1,139 @@
package container
import (
"context"
"fmt"
"path/filepath"
"testing"
"github.com/docker/docker/api/types"
"github.com/docker/docker/api/types/container"
"github.com/docker/docker/api/types/mount"
"github.com/docker/docker/integration/util/request"
)
func TestMountDaemonRoot(t *testing.T) {
t.Parallel()
client := request.NewAPIClient(t)
ctx := context.Background()
info, err := client.Info(ctx)
if err != nil {
t.Fatal(err)
}
for _, test := range []struct {
desc string
propagation mount.Propagation
expected mount.Propagation
}{
{
desc: "default",
propagation: "",
expected: mount.PropagationRSlave,
},
{
desc: "private",
propagation: mount.PropagationPrivate,
},
{
desc: "rprivate",
propagation: mount.PropagationRPrivate,
},
{
desc: "slave",
propagation: mount.PropagationSlave,
},
{
desc: "rslave",
propagation: mount.PropagationRSlave,
expected: mount.PropagationRSlave,
},
{
desc: "shared",
propagation: mount.PropagationShared,
},
{
desc: "rshared",
propagation: mount.PropagationRShared,
expected: mount.PropagationRShared,
},
} {
t.Run(test.desc, func(t *testing.T) {
test := test
t.Parallel()
propagationSpec := fmt.Sprintf(":%s", test.propagation)
if test.propagation == "" {
propagationSpec = ""
}
bindSpecRoot := info.DockerRootDir + ":" + "/foo" + propagationSpec
bindSpecSub := filepath.Join(info.DockerRootDir, "containers") + ":/foo" + propagationSpec
for name, hc := range map[string]*container.HostConfig{
"bind root": {Binds: []string{bindSpecRoot}},
"bind subpath": {Binds: []string{bindSpecSub}},
"mount root": {
Mounts: []mount.Mount{
{
Type: mount.TypeBind,
Source: info.DockerRootDir,
Target: "/foo",
BindOptions: &mount.BindOptions{Propagation: test.propagation},
},
},
},
"mount subpath": {
Mounts: []mount.Mount{
{
Type: mount.TypeBind,
Source: filepath.Join(info.DockerRootDir, "containers"),
Target: "/foo",
BindOptions: &mount.BindOptions{Propagation: test.propagation},
},
},
},
} {
t.Run(name, func(t *testing.T) {
hc := hc
t.Parallel()
c, err := client.ContainerCreate(ctx, &container.Config{
Image: "busybox",
Cmd: []string{"true"},
}, hc, nil, "")
if err != nil {
if test.expected != "" {
t.Fatal(err)
}
// expected an error, so this is ok and should not continue
return
}
if test.expected == "" {
t.Fatal("expected create to fail")
}
defer func() {
if err := client.ContainerRemove(ctx, c.ID, types.ContainerRemoveOptions{Force: true}); err != nil {
panic(err)
}
}()
inspect, err := client.ContainerInspect(ctx, c.ID)
if err != nil {
t.Fatal(err)
}
if len(inspect.Mounts) != 1 {
t.Fatalf("unexpected number of mounts: %+v", inspect.Mounts)
}
m := inspect.Mounts[0]
if m.Propagation != test.expected {
t.Fatalf("got unexpected propagation mode, expected %q, got: %v", test.expected, m.Propagation)
}
})
}
})
}
}

View File

@ -20,7 +20,8 @@ github.com/docker/go-connections 3ede32e2033de7505e6500d6c868c2b9ed9f169d
golang.org/x/text f72d8390a633d5dfb0cc84043294db9f6c935756
github.com/stretchr/testify 4d4bfba8f1d1027c4fdbe371823030df51419987
github.com/pmezard/go-difflib v1.0.0
github.com/gotestyourself/gotestyourself v1.1.0
github.com/gotestyourself/gotestyourself 511344eed30e4384f010579a593dfb442033a692
github.com/google/go-cmp v0.1.0
github.com/RackSec/srslog 456df3a81436d29ba874f3590eeeee25d666f8a5
github.com/imdario/mergo 0.2.1
@ -30,7 +31,7 @@ github.com/moby/buildkit aaff9d591ef128560018433fe61beb802e149de8
github.com/tonistiigi/fsutil dea3a0da73aee887fc02142d995be764106ac5e2
#get libnetwork packages
github.com/docker/libnetwork 7dd202d11b2078c063fefc614c9fb9e4615da0e1
github.com/docker/libnetwork 0ef1f8c94fa85acacf88b9be75ae07de91d0b2a5
github.com/docker/go-events 9461782956ad83b30282bf90e31fa6a70c255ba9
github.com/armon/go-radix e39d623f12e8e41c7b5529e9a9dd67a1e2261f80
github.com/armon/go-metrics eb0af217e5e9747e41dd5303755356b62d28e3ec

View File

@ -13,7 +13,6 @@ import (
"strings"
"sync"
"syscall"
"time"
"github.com/docker/docker/pkg/reexec"
"github.com/docker/libnetwork/datastore"
@ -693,6 +692,12 @@ func (n *network) initSandbox(restore bool) error {
n.driver.initSandboxPeerDB(n.id)
}
// If we are in swarm mode, we don't need anymore the watchMiss routine.
// This will save 1 thread and 1 netlink socket per network
if !n.driver.isSerfAlive() {
return nil
}
var nlSock *nl.NetlinkSocket
sbox.InvokeFunc(func() {
nlSock, err = nl.Subscribe(syscall.NETLINK_ROUTE, syscall.RTNLGRP_NEIGH)
@ -706,7 +711,7 @@ func (n *network) initSandbox(restore bool) error {
n.setNetlinkSocket(nlSock)
if err == nil {
go n.watchMiss(nlSock)
go n.watchMiss(nlSock, key)
} else {
logrus.Errorf("failed to subscribe to neighbor group netlink messages for overlay network %s in sbox %s: %v",
n.id, sbox.Key(), err)
@ -715,8 +720,23 @@ func (n *network) initSandbox(restore bool) error {
return nil
}
func (n *network) watchMiss(nlSock *nl.NetlinkSocket) {
t := time.Now()
func (n *network) watchMiss(nlSock *nl.NetlinkSocket, nsPath string) {
// With the new version of the netlink library the deserialize function makes
// requests about the interface of the netlink message. This can succeed only
// if this go routine is in the target namespace. For this reason following we
// lock the thread on that namespace
runtime.LockOSThread()
defer runtime.UnlockOSThread()
newNs, err := netns.GetFromPath(nsPath)
if err != nil {
logrus.WithError(err).Errorf("failed to get the namespace %s", nsPath)
return
}
defer newNs.Close()
if err = netns.Set(newNs); err != nil {
logrus.WithError(err).Errorf("failed to enter the namespace %s", nsPath)
return
}
for {
msgs, err := nlSock.Receive()
if err != nil {
@ -772,30 +792,13 @@ func (n *network) watchMiss(nlSock *nl.NetlinkSocket) {
continue
}
if n.driver.isSerfAlive() {
logrus.Debugf("miss notification: dest IP %v, dest MAC %v", ip, mac)
mac, IPmask, vtep, err := n.driver.resolvePeer(n.id, ip)
if err != nil {
logrus.Errorf("could not resolve peer %q: %v", ip, err)
continue
}
n.driver.peerAdd(n.id, "dummy", ip, IPmask, mac, vtep, l2Miss, l3Miss, false)
} else if l3Miss && time.Since(t) > time.Second {
// All the local peers will trigger a miss notification but this one is expected and the local container will reply
// autonomously to the ARP request
// In case the gc_thresh3 values is low kernel might reject new entries during peerAdd. This will trigger the following
// extra logs that will inform of the possible issue.
// Entries created would not be deleted see documentation http://man7.org/linux/man-pages/man7/arp.7.html:
// Entries which are marked as permanent are never deleted by the garbage-collector.
// The time limit here is to guarantee that the dbSearch is not
// done too frequently causing a stall of the peerDB operations.
pKey, pEntry, err := n.driver.peerDbSearch(n.id, ip)
if err == nil && !pEntry.isLocal {
t = time.Now()
logrus.Warnf("miss notification for peer:%+v l3Miss:%t l2Miss:%t, if the problem persist check the gc_thresh on the host pKey:%+v pEntry:%+v err:%v",
neigh, l3Miss, l2Miss, *pKey, *pEntry, err)
}
logrus.Debugf("miss notification: dest IP %v, dest MAC %v", ip, mac)
mac, IPmask, vtep, err := n.driver.resolvePeer(n.id, ip)
if err != nil {
logrus.Errorf("could not resolve peer %q: %v", ip, err)
continue
}
n.driver.peerAdd(n.id, "dummy", ip, IPmask, mac, vtep, l2Miss, l3Miss, false)
}
}
}

View File

@ -220,9 +220,11 @@ func NewSandbox(key string, osCreate, isRestore bool) (Sandbox, error) {
if err != nil {
logrus.Warnf("Failed to set the timeout on the sandbox netlink handle sockets: %v", err)
}
// In live-restore mode, IPV6 entries are getting cleaned up due to below code
// We should retain IPV6 configrations in live-restore mode when Docker Daemon
// comes back. It should work as it is on other cases
// As starting point, disable IPv6 on all interfaces
if !n.isDefault {
if !isRestore && !n.isDefault {
err = setIPv6(n.path, "all", false)
if err != nil {
logrus.Warnf("Failed to disable IPv6 on all interfaces on network namespace %q: %v", n.path, err)

View File

@ -0,0 +1,27 @@
Copyright (c) 2017 The Go Authors. All rights reserved.
Redistribution and use in source and binary forms, with or without
modification, are permitted provided that the following conditions are
met:
* Redistributions of source code must retain the above copyright
notice, this list of conditions and the following disclaimer.
* Redistributions in binary form must reproduce the above
copyright notice, this list of conditions and the following disclaimer
in the documentation and/or other materials provided with the
distribution.
* Neither the name of Google Inc. nor the names of its
contributors may be used to endorse or promote products derived from
this software without specific prior written permission.
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS
"AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT
LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR
A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT
OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL,
SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT
LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE,
DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY
THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT
(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

View File

@ -0,0 +1,44 @@
# Package for equality of Go values
[![GoDoc](https://godoc.org/github.com/google/go-cmp/cmp?status.svg)][godoc]
[![Build Status](https://travis-ci.org/google/go-cmp.svg?branch=master)][travis]
This package is intended to be a more powerful and safer alternative to
`reflect.DeepEqual` for comparing whether two values are semantically equal.
The primary features of `cmp` are:
* When the default behavior of equality does not suit the needs of the test,
custom equality functions can override the equality operation.
For example, an equality function may report floats as equal so long as they
are within some tolerance of each other.
* Types that have an `Equal` method may use that method to determine equality.
This allows package authors to determine the equality operation for the types
that they define.
* If no custom equality functions are used and no `Equal` method is defined,
equality is determined by recursively comparing the primitive kinds on both
values, much like `reflect.DeepEqual`. Unlike `reflect.DeepEqual`, unexported
fields are not compared by default; they result in panics unless suppressed
by using an `Ignore` option (see `cmpopts.IgnoreUnexported`) or explictly
compared using the `AllowUnexported` option.
See the [GoDoc documentation][godoc] for more information.
This is not an official Google product.
[godoc]: https://godoc.org/github.com/google/go-cmp/cmp
[travis]: https://travis-ci.org/google/go-cmp
## Install
```
go get -u github.com/google/go-cmp/cmp
```
## License
BSD - See [LICENSE][license] file
[license]: https://github.com/google/go-cmp/blob/master/LICENSE

View File

@ -0,0 +1,529 @@
// Copyright 2017, The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE.md file.
// Package cmp determines equality of values.
//
// This package is intended to be a more powerful and safer alternative to
// reflect.DeepEqual for comparing whether two values are semantically equal.
//
// The primary features of cmp are:
//
// • When the default behavior of equality does not suit the needs of the test,
// custom equality functions can override the equality operation.
// For example, an equality function may report floats as equal so long as they
// are within some tolerance of each other.
//
// • Types that have an Equal method may use that method to determine equality.
// This allows package authors to determine the equality operation for the types
// that they define.
//
// • If no custom equality functions are used and no Equal method is defined,
// equality is determined by recursively comparing the primitive kinds on both
// values, much like reflect.DeepEqual. Unlike reflect.DeepEqual, unexported
// fields are not compared by default; they result in panics unless suppressed
// by using an Ignore option (see cmpopts.IgnoreUnexported) or explictly compared
// using the AllowUnexported option.
package cmp
import (
"fmt"
"reflect"
"github.com/google/go-cmp/cmp/internal/diff"
"github.com/google/go-cmp/cmp/internal/function"
"github.com/google/go-cmp/cmp/internal/value"
)
// BUG: Maps with keys containing NaN values cannot be properly compared due to
// the reflection package's inability to retrieve such entries. Equal will panic
// anytime it comes across a NaN key, but this behavior may change.
//
// See https://golang.org/issue/11104 for more details.
var nothing = reflect.Value{}
// Equal reports whether x and y are equal by recursively applying the
// following rules in the given order to x and y and all of their sub-values:
//
// • If two values are not of the same type, then they are never equal
// and the overall result is false.
//
// • Let S be the set of all Ignore, Transformer, and Comparer options that
// remain after applying all path filters, value filters, and type filters.
// If at least one Ignore exists in S, then the comparison is ignored.
// If the number of Transformer and Comparer options in S is greater than one,
// then Equal panics because it is ambiguous which option to use.
// If S contains a single Transformer, then use that to transform the current
// values and recursively call Equal on the output values.
// If S contains a single Comparer, then use that to compare the current values.
// Otherwise, evaluation proceeds to the next rule.
//
// • If the values have an Equal method of the form "(T) Equal(T) bool" or
// "(T) Equal(I) bool" where T is assignable to I, then use the result of
// x.Equal(y). Otherwise, no such method exists and evaluation proceeds to
// the next rule.
//
// • Lastly, try to compare x and y based on their basic kinds.
// Simple kinds like booleans, integers, floats, complex numbers, strings, and
// channels are compared using the equivalent of the == operator in Go.
// Functions are only equal if they are both nil, otherwise they are unequal.
// Pointers are equal if the underlying values they point to are also equal.
// Interfaces are equal if their underlying concrete values are also equal.
//
// Structs are equal if all of their fields are equal. If a struct contains
// unexported fields, Equal panics unless the AllowUnexported option is used or
// an Ignore option (e.g., cmpopts.IgnoreUnexported) ignores that field.
//
// Arrays, slices, and maps are equal if they are both nil or both non-nil
// with the same length and the elements at each index or key are equal.
// Note that a non-nil empty slice and a nil slice are not equal.
// To equate empty slices and maps, consider using cmpopts.EquateEmpty.
// Map keys are equal according to the == operator.
// To use custom comparisons for map keys, consider using cmpopts.SortMaps.
func Equal(x, y interface{}, opts ...Option) bool {
s := newState(opts)
s.compareAny(reflect.ValueOf(x), reflect.ValueOf(y))
return s.result.Equal()
}
// Diff returns a human-readable report of the differences between two values.
// It returns an empty string if and only if Equal returns true for the same
// input values and options. The output string will use the "-" symbol to
// indicate elements removed from x, and the "+" symbol to indicate elements
// added to y.
//
// Do not depend on this output being stable.
func Diff(x, y interface{}, opts ...Option) string {
r := new(defaultReporter)
opts = Options{Options(opts), r}
eq := Equal(x, y, opts...)
d := r.String()
if (d == "") != eq {
panic("inconsistent difference and equality results")
}
return d
}
type state struct {
// These fields represent the "comparison state".
// Calling statelessCompare must not result in observable changes to these.
result diff.Result // The current result of comparison
curPath Path // The current path in the value tree
reporter reporter // Optional reporter used for difference formatting
// dynChecker triggers pseudo-random checks for option correctness.
// It is safe for statelessCompare to mutate this value.
dynChecker dynChecker
// These fields, once set by processOption, will not change.
exporters map[reflect.Type]bool // Set of structs with unexported field visibility
opts Options // List of all fundamental and filter options
}
func newState(opts []Option) *state {
s := new(state)
for _, opt := range opts {
s.processOption(opt)
}
return s
}
func (s *state) processOption(opt Option) {
switch opt := opt.(type) {
case nil:
case Options:
for _, o := range opt {
s.processOption(o)
}
case coreOption:
type filtered interface {
isFiltered() bool
}
if fopt, ok := opt.(filtered); ok && !fopt.isFiltered() {
panic(fmt.Sprintf("cannot use an unfiltered option: %v", opt))
}
s.opts = append(s.opts, opt)
case visibleStructs:
if s.exporters == nil {
s.exporters = make(map[reflect.Type]bool)
}
for t := range opt {
s.exporters[t] = true
}
case reporter:
if s.reporter != nil {
panic("difference reporter already registered")
}
s.reporter = opt
default:
panic(fmt.Sprintf("unknown option %T", opt))
}
}
// statelessCompare compares two values and returns the result.
// This function is stateless in that it does not alter the current result,
// or output to any registered reporters.
func (s *state) statelessCompare(vx, vy reflect.Value) diff.Result {
// We do not save and restore the curPath because all of the compareX
// methods should properly push and pop from the path.
// It is an implementation bug if the contents of curPath differs from
// when calling this function to when returning from it.
oldResult, oldReporter := s.result, s.reporter
s.result = diff.Result{} // Reset result
s.reporter = nil // Remove reporter to avoid spurious printouts
s.compareAny(vx, vy)
res := s.result
s.result, s.reporter = oldResult, oldReporter
return res
}
func (s *state) compareAny(vx, vy reflect.Value) {
// TODO: Support cyclic data structures.
// Rule 0: Differing types are never equal.
if !vx.IsValid() || !vy.IsValid() {
s.report(vx.IsValid() == vy.IsValid(), vx, vy)
return
}
if vx.Type() != vy.Type() {
s.report(false, vx, vy) // Possible for path to be empty
return
}
t := vx.Type()
if len(s.curPath) == 0 {
s.curPath.push(&pathStep{typ: t})
defer s.curPath.pop()
}
vx, vy = s.tryExporting(vx, vy)
// Rule 1: Check whether an option applies on this node in the value tree.
if s.tryOptions(vx, vy, t) {
return
}
// Rule 2: Check whether the type has a valid Equal method.
if s.tryMethod(vx, vy, t) {
return
}
// Rule 3: Recursively descend into each value's underlying kind.
switch t.Kind() {
case reflect.Bool:
s.report(vx.Bool() == vy.Bool(), vx, vy)
return
case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
s.report(vx.Int() == vy.Int(), vx, vy)
return
case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, reflect.Uintptr:
s.report(vx.Uint() == vy.Uint(), vx, vy)
return
case reflect.Float32, reflect.Float64:
s.report(vx.Float() == vy.Float(), vx, vy)
return
case reflect.Complex64, reflect.Complex128:
s.report(vx.Complex() == vy.Complex(), vx, vy)
return
case reflect.String:
s.report(vx.String() == vy.String(), vx, vy)
return
case reflect.Chan, reflect.UnsafePointer:
s.report(vx.Pointer() == vy.Pointer(), vx, vy)
return
case reflect.Func:
s.report(vx.IsNil() && vy.IsNil(), vx, vy)
return
case reflect.Ptr:
if vx.IsNil() || vy.IsNil() {
s.report(vx.IsNil() && vy.IsNil(), vx, vy)
return
}
s.curPath.push(&indirect{pathStep{t.Elem()}})
defer s.curPath.pop()
s.compareAny(vx.Elem(), vy.Elem())
return
case reflect.Interface:
if vx.IsNil() || vy.IsNil() {
s.report(vx.IsNil() && vy.IsNil(), vx, vy)
return
}
if vx.Elem().Type() != vy.Elem().Type() {
s.report(false, vx.Elem(), vy.Elem())
return
}
s.curPath.push(&typeAssertion{pathStep{vx.Elem().Type()}})
defer s.curPath.pop()
s.compareAny(vx.Elem(), vy.Elem())
return
case reflect.Slice:
if vx.IsNil() || vy.IsNil() {
s.report(vx.IsNil() && vy.IsNil(), vx, vy)
return
}
fallthrough
case reflect.Array:
s.compareArray(vx, vy, t)
return
case reflect.Map:
s.compareMap(vx, vy, t)
return
case reflect.Struct:
s.compareStruct(vx, vy, t)
return
default:
panic(fmt.Sprintf("%v kind not handled", t.Kind()))
}
}
func (s *state) tryExporting(vx, vy reflect.Value) (reflect.Value, reflect.Value) {
if sf, ok := s.curPath[len(s.curPath)-1].(*structField); ok && sf.unexported {
if sf.force {
// Use unsafe pointer arithmetic to get read-write access to an
// unexported field in the struct.
vx = unsafeRetrieveField(sf.pvx, sf.field)
vy = unsafeRetrieveField(sf.pvy, sf.field)
} else {
// We are not allowed to export the value, so invalidate them
// so that tryOptions can panic later if not explicitly ignored.
vx = nothing
vy = nothing
}
}
return vx, vy
}
func (s *state) tryOptions(vx, vy reflect.Value, t reflect.Type) bool {
// If there were no FilterValues, we will not detect invalid inputs,
// so manually check for them and append invalid if necessary.
// We still evaluate the options since an ignore can override invalid.
opts := s.opts
if !vx.IsValid() || !vy.IsValid() {
opts = Options{opts, invalid{}}
}
// Evaluate all filters and apply the remaining options.
if opt := opts.filter(s, vx, vy, t); opt != nil {
return opt.apply(s, vx, vy)
}
return false
}
func (s *state) tryMethod(vx, vy reflect.Value, t reflect.Type) bool {
// Check if this type even has an Equal method.
m, ok := t.MethodByName("Equal")
if !ok || !function.IsType(m.Type, function.EqualAssignable) {
return false
}
eq := s.callTTBFunc(m.Func, vx, vy)
s.report(eq, vx, vy)
return true
}
func (s *state) callTRFunc(f, v reflect.Value) reflect.Value {
if !s.dynChecker.Next() {
return f.Call([]reflect.Value{v})[0]
}
// Run the function twice and ensure that we get the same results back.
// We run in goroutines so that the race detector (if enabled) can detect
// unsafe mutations to the input.
c := make(chan reflect.Value)
go detectRaces(c, f, v)
want := f.Call([]reflect.Value{v})[0]
if got := <-c; !s.statelessCompare(got, want).Equal() {
// To avoid false-positives with non-reflexive equality operations,
// we sanity check whether a value is equal to itself.
if !s.statelessCompare(want, want).Equal() {
return want
}
fn := getFuncName(f.Pointer())
panic(fmt.Sprintf("non-deterministic function detected: %s", fn))
}
return want
}
func (s *state) callTTBFunc(f, x, y reflect.Value) bool {
if !s.dynChecker.Next() {
return f.Call([]reflect.Value{x, y})[0].Bool()
}
// Swapping the input arguments is sufficient to check that
// f is symmetric and deterministic.
// We run in goroutines so that the race detector (if enabled) can detect
// unsafe mutations to the input.
c := make(chan reflect.Value)
go detectRaces(c, f, y, x)
want := f.Call([]reflect.Value{x, y})[0].Bool()
if got := <-c; !got.IsValid() || got.Bool() != want {
fn := getFuncName(f.Pointer())
panic(fmt.Sprintf("non-deterministic or non-symmetric function detected: %s", fn))
}
return want
}
func detectRaces(c chan<- reflect.Value, f reflect.Value, vs ...reflect.Value) {
var ret reflect.Value
defer func() {
recover() // Ignore panics, let the other call to f panic instead
c <- ret
}()
ret = f.Call(vs)[0]
}
func (s *state) compareArray(vx, vy reflect.Value, t reflect.Type) {
step := &sliceIndex{pathStep{t.Elem()}, 0, 0}
s.curPath.push(step)
// Compute an edit-script for slices vx and vy.
eq, es := diff.Difference(vx.Len(), vy.Len(), func(ix, iy int) diff.Result {
step.xkey, step.ykey = ix, iy
return s.statelessCompare(vx.Index(ix), vy.Index(iy))
})
// Equal or no edit-script, so report entire slices as is.
if eq || es == nil {
s.curPath.pop() // Pop first since we are reporting the whole slice
s.report(eq, vx, vy)
return
}
// Replay the edit-script.
var ix, iy int
for _, e := range es {
switch e {
case diff.UniqueX:
step.xkey, step.ykey = ix, -1
s.report(false, vx.Index(ix), nothing)
ix++
case diff.UniqueY:
step.xkey, step.ykey = -1, iy
s.report(false, nothing, vy.Index(iy))
iy++
default:
step.xkey, step.ykey = ix, iy
if e == diff.Identity {
s.report(true, vx.Index(ix), vy.Index(iy))
} else {
s.compareAny(vx.Index(ix), vy.Index(iy))
}
ix++
iy++
}
}
s.curPath.pop()
return
}
func (s *state) compareMap(vx, vy reflect.Value, t reflect.Type) {
if vx.IsNil() || vy.IsNil() {
s.report(vx.IsNil() && vy.IsNil(), vx, vy)
return
}
// We combine and sort the two map keys so that we can perform the
// comparisons in a deterministic order.
step := &mapIndex{pathStep: pathStep{t.Elem()}}
s.curPath.push(step)
defer s.curPath.pop()
for _, k := range value.SortKeys(append(vx.MapKeys(), vy.MapKeys()...)) {
step.key = k
vvx := vx.MapIndex(k)
vvy := vy.MapIndex(k)
switch {
case vvx.IsValid() && vvy.IsValid():
s.compareAny(vvx, vvy)
case vvx.IsValid() && !vvy.IsValid():
s.report(false, vvx, nothing)
case !vvx.IsValid() && vvy.IsValid():
s.report(false, nothing, vvy)
default:
// It is possible for both vvx and vvy to be invalid if the
// key contained a NaN value in it. There is no way in
// reflection to be able to retrieve these values.
// See https://golang.org/issue/11104
panic(fmt.Sprintf("%#v has map key with NaNs", s.curPath))
}
}
}
func (s *state) compareStruct(vx, vy reflect.Value, t reflect.Type) {
var vax, vay reflect.Value // Addressable versions of vx and vy
step := &structField{}
s.curPath.push(step)
defer s.curPath.pop()
for i := 0; i < t.NumField(); i++ {
vvx := vx.Field(i)
vvy := vy.Field(i)
step.typ = t.Field(i).Type
step.name = t.Field(i).Name
step.idx = i
step.unexported = !isExported(step.name)
if step.unexported {
// Defer checking of unexported fields until later to give an
// Ignore a chance to ignore the field.
if !vax.IsValid() || !vay.IsValid() {
// For unsafeRetrieveField to work, the parent struct must
// be addressable. Create a new copy of the values if
// necessary to make them addressable.
vax = makeAddressable(vx)
vay = makeAddressable(vy)
}
step.force = s.exporters[t]
step.pvx = vax
step.pvy = vay
step.field = t.Field(i)
}
s.compareAny(vvx, vvy)
}
}
// report records the result of a single comparison.
// It also calls Report if any reporter is registered.
func (s *state) report(eq bool, vx, vy reflect.Value) {
if eq {
s.result.NSame++
} else {
s.result.NDiff++
}
if s.reporter != nil {
s.reporter.Report(vx, vy, eq, s.curPath)
}
}
// dynChecker tracks the state needed to periodically perform checks that
// user provided functions are symmetric and deterministic.
// The zero value is safe for immediate use.
type dynChecker struct{ curr, next int }
// Next increments the state and reports whether a check should be performed.
//
// Checks occur every Nth function call, where N is a triangular number:
// 0 1 3 6 10 15 21 28 36 45 55 66 78 91 105 120 136 153 171 190 ...
// See https://en.wikipedia.org/wiki/Triangular_number
//
// This sequence ensures that the cost of checks drops significantly as
// the number of functions calls grows larger.
func (dc *dynChecker) Next() bool {
ok := dc.curr == dc.next
if ok {
dc.curr = 0
dc.next++
}
dc.curr++
return ok
}
// makeAddressable returns a value that is always addressable.
// It returns the input verbatim if it is already addressable,
// otherwise it creates a new value and returns an addressable copy.
func makeAddressable(v reflect.Value) reflect.Value {
if v.CanAddr() {
return v
}
vc := reflect.New(v.Type()).Elem()
vc.Set(v)
return vc
}

View File

@ -0,0 +1,17 @@
// Copyright 2017, The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE.md file.
// +build !debug
package diff
var debug debugger
type debugger struct{}
func (debugger) Begin(_, _ int, f EqualFunc, _, _ *EditScript) EqualFunc {
return f
}
func (debugger) Update() {}
func (debugger) Finish() {}

View File

@ -0,0 +1,122 @@
// Copyright 2017, The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE.md file.
// +build debug
package diff
import (
"fmt"
"strings"
"sync"
"time"
)
// The algorithm can be seen running in real-time by enabling debugging:
// go test -tags=debug -v
//
// Example output:
// === RUN TestDifference/#34
// ┌───────────────────────────────┐
// │ \ · · · · · · · · · · · · · · │
// │ · # · · · · · · · · · · · · · │
// │ · \ · · · · · · · · · · · · · │
// │ · · \ · · · · · · · · · · · · │
// │ · · · X # · · · · · · · · · · │
// │ · · · # \ · · · · · · · · · · │
// │ · · · · · # # · · · · · · · · │
// │ · · · · · # \ · · · · · · · · │
// │ · · · · · · · \ · · · · · · · │
// │ · · · · · · · · \ · · · · · · │
// │ · · · · · · · · · \ · · · · · │
// │ · · · · · · · · · · \ · · # · │
// │ · · · · · · · · · · · \ # # · │
// │ · · · · · · · · · · · # # # · │
// │ · · · · · · · · · · # # # # · │
// │ · · · · · · · · · # # # # # · │
// │ · · · · · · · · · · · · · · \ │
// └───────────────────────────────┘
// [.Y..M.XY......YXYXY.|]
//
// The grid represents the edit-graph where the horizontal axis represents
// list X and the vertical axis represents list Y. The start of the two lists
// is the top-left, while the ends are the bottom-right. The '·' represents
// an unexplored node in the graph. The '\' indicates that the two symbols
// from list X and Y are equal. The 'X' indicates that two symbols are similar
// (but not exactly equal) to each other. The '#' indicates that the two symbols
// are different (and not similar). The algorithm traverses this graph trying to
// make the paths starting in the top-left and the bottom-right connect.
//
// The series of '.', 'X', 'Y', and 'M' characters at the bottom represents
// the currently established path from the forward and reverse searches,
// seperated by a '|' character.
const (
updateDelay = 100 * time.Millisecond
finishDelay = 500 * time.Millisecond
ansiTerminal = true // ANSI escape codes used to move terminal cursor
)
var debug debugger
type debugger struct {
sync.Mutex
p1, p2 EditScript
fwdPath, revPath *EditScript
grid []byte
lines int
}
func (dbg *debugger) Begin(nx, ny int, f EqualFunc, p1, p2 *EditScript) EqualFunc {
dbg.Lock()
dbg.fwdPath, dbg.revPath = p1, p2
top := "┌─" + strings.Repeat("──", nx) + "┐\n"
row := "│ " + strings.Repeat("· ", nx) + "│\n"
btm := "└─" + strings.Repeat("──", nx) + "┘\n"
dbg.grid = []byte(top + strings.Repeat(row, ny) + btm)
dbg.lines = strings.Count(dbg.String(), "\n")
fmt.Print(dbg)
// Wrap the EqualFunc so that we can intercept each result.
return func(ix, iy int) (r Result) {
cell := dbg.grid[len(top)+iy*len(row):][len("│ ")+len("· ")*ix:][:len("·")]
for i := range cell {
cell[i] = 0 // Zero out the multiple bytes of UTF-8 middle-dot
}
switch r = f(ix, iy); {
case r.Equal():
cell[0] = '\\'
case r.Similar():
cell[0] = 'X'
default:
cell[0] = '#'
}
return
}
}
func (dbg *debugger) Update() {
dbg.print(updateDelay)
}
func (dbg *debugger) Finish() {
dbg.print(finishDelay)
dbg.Unlock()
}
func (dbg *debugger) String() string {
dbg.p1, dbg.p2 = *dbg.fwdPath, dbg.p2[:0]
for i := len(*dbg.revPath) - 1; i >= 0; i-- {
dbg.p2 = append(dbg.p2, (*dbg.revPath)[i])
}
return fmt.Sprintf("%s[%v|%v]\n\n", dbg.grid, dbg.p1, dbg.p2)
}
func (dbg *debugger) print(d time.Duration) {
if ansiTerminal {
fmt.Printf("\x1b[%dA", dbg.lines) // Reset terminal cursor
}
fmt.Print(dbg)
time.Sleep(d)
}

View File

@ -0,0 +1,373 @@
// Copyright 2017, The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE.md file.
// Package diff implements an algorithm for producing edit-scripts.
// The edit-script is a sequence of operations needed to transform one list
// of symbols into another (or vice-versa). The edits allowed are insertions,
// deletions, and modifications. The summation of all edits is called the
// Levenshtein distance as this problem is well-known in computer science.
//
// This package prioritizes performance over accuracy. That is, the run time
// is more important than obtaining a minimal Levenshtein distance.
package diff
// EditType represents a single operation within an edit-script.
type EditType uint8
const (
// Identity indicates that a symbol pair is identical in both list X and Y.
Identity EditType = iota
// UniqueX indicates that a symbol only exists in X and not Y.
UniqueX
// UniqueY indicates that a symbol only exists in Y and not X.
UniqueY
// Modified indicates that a symbol pair is a modification of each other.
Modified
)
// EditScript represents the series of differences between two lists.
type EditScript []EditType
// String returns a human-readable string representing the edit-script where
// Identity, UniqueX, UniqueY, and Modified are represented by the
// '.', 'X', 'Y', and 'M' characters, respectively.
func (es EditScript) String() string {
b := make([]byte, len(es))
for i, e := range es {
switch e {
case Identity:
b[i] = '.'
case UniqueX:
b[i] = 'X'
case UniqueY:
b[i] = 'Y'
case Modified:
b[i] = 'M'
default:
panic("invalid edit-type")
}
}
return string(b)
}
// stats returns a histogram of the number of each type of edit operation.
func (es EditScript) stats() (s struct{ NI, NX, NY, NM int }) {
for _, e := range es {
switch e {
case Identity:
s.NI++
case UniqueX:
s.NX++
case UniqueY:
s.NY++
case Modified:
s.NM++
default:
panic("invalid edit-type")
}
}
return
}
// Dist is the Levenshtein distance and is guaranteed to be 0 if and only if
// lists X and Y are equal.
func (es EditScript) Dist() int { return len(es) - es.stats().NI }
// LenX is the length of the X list.
func (es EditScript) LenX() int { return len(es) - es.stats().NY }
// LenY is the length of the Y list.
func (es EditScript) LenY() int { return len(es) - es.stats().NX }
// EqualFunc reports whether the symbols at indexes ix and iy are equal.
// When called by Difference, the index is guaranteed to be within nx and ny.
type EqualFunc func(ix int, iy int) Result
// Result is the result of comparison.
// NSame is the number of sub-elements that are equal.
// NDiff is the number of sub-elements that are not equal.
type Result struct{ NSame, NDiff int }
// Equal indicates whether the symbols are equal. Two symbols are equal
// if and only if NDiff == 0. If Equal, then they are also Similar.
func (r Result) Equal() bool { return r.NDiff == 0 }
// Similar indicates whether two symbols are similar and may be represented
// by using the Modified type. As a special case, we consider binary comparisons
// (i.e., those that return Result{1, 0} or Result{0, 1}) to be similar.
//
// The exact ratio of NSame to NDiff to determine similarity may change.
func (r Result) Similar() bool {
// Use NSame+1 to offset NSame so that binary comparisons are similar.
return r.NSame+1 >= r.NDiff
}
// Difference reports whether two lists of lengths nx and ny are equal
// given the definition of equality provided as f.
//
// This function may return a edit-script, which is a sequence of operations
// needed to convert one list into the other. If non-nil, the following
// invariants for the edit-script are maintained:
// • eq == (es.Dist()==0)
// • nx == es.LenX()
// • ny == es.LenY()
//
// This algorithm is not guaranteed to be an optimal solution (i.e., one that
// produces an edit-script with a minimal Levenshtein distance). This algorithm
// favors performance over optimality. The exact output is not guaranteed to
// be stable and may change over time.
func Difference(nx, ny int, f EqualFunc) (eq bool, es EditScript) {
es = searchGraph(nx, ny, f)
st := es.stats()
eq = len(es) == st.NI
if !eq && st.NI < (nx+ny)/4 {
return eq, nil // Edit-script more distracting than helpful
}
return eq, es
}
func searchGraph(nx, ny int, f EqualFunc) EditScript {
// This algorithm is based on traversing what is known as an "edit-graph".
// See Figure 1 from "An O(ND) Difference Algorithm and Its Variations"
// by Eugene W. Myers. Since D can be as large as N itself, this is
// effectively O(N^2). Unlike the algorithm from that paper, we are not
// interested in the optimal path, but at least some "decent" path.
//
// For example, let X and Y be lists of symbols:
// X = [A B C A B B A]
// Y = [C B A B A C]
//
// The edit-graph can be drawn as the following:
// A B C A B B A
// ┌─────────────┐
// C │_|_|\|_|_|_|_│ 0
// B │_|\|_|_|\|\|_│ 1
// A │\|_|_|\|_|_|\│ 2
// B │_|\|_|_|\|\|_│ 3
// A │\|_|_|\|_|_|\│ 4
// C │ | |\| | | | │ 5
// └─────────────┘ 6
// 0 1 2 3 4 5 6 7
//
// List X is written along the horizontal axis, while list Y is written
// along the vertical axis. At any point on this grid, if the symbol in
// list X matches the corresponding symbol in list Y, then a '\' is drawn.
// The goal of any minimal edit-script algorithm is to find a path from the
// top-left corner to the bottom-right corner, while traveling through the
// fewest horizontal or vertical edges.
// A horizontal edge is equivalent to inserting a symbol from list X.
// A vertical edge is equivalent to inserting a symbol from list Y.
// A diagonal edge is equivalent to a matching symbol between both X and Y.
// Invariants:
// • 0 ≤ fwdPath.X ≤ (fwdFrontier.X, revFrontier.X) ≤ revPath.X ≤ nx
// • 0 ≤ fwdPath.Y ≤ (fwdFrontier.Y, revFrontier.Y) ≤ revPath.Y ≤ ny
//
// In general:
// • fwdFrontier.X < revFrontier.X
// • fwdFrontier.Y < revFrontier.Y
// Unless, it is time for the algorithm to terminate.
fwdPath := path{+1, point{0, 0}, make(EditScript, 0, (nx+ny)/2)}
revPath := path{-1, point{nx, ny}, make(EditScript, 0)}
fwdFrontier := fwdPath.point // Forward search frontier
revFrontier := revPath.point // Reverse search frontier
// Search budget bounds the cost of searching for better paths.
// The longest sequence of non-matching symbols that can be tolerated is
// approximately the square-root of the search budget.
searchBudget := 4 * (nx + ny) // O(n)
// The algorithm below is a greedy, meet-in-the-middle algorithm for
// computing sub-optimal edit-scripts between two lists.
//
// The algorithm is approximately as follows:
// • Searching for differences switches back-and-forth between
// a search that starts at the beginning (the top-left corner), and
// a search that starts at the end (the bottom-right corner). The goal of
// the search is connect with the search from the opposite corner.
// • As we search, we build a path in a greedy manner, where the first
// match seen is added to the path (this is sub-optimal, but provides a
// decent result in practice). When matches are found, we try the next pair
// of symbols in the lists and follow all matches as far as possible.
// • When searching for matches, we search along a diagonal going through
// through the "frontier" point. If no matches are found, we advance the
// frontier towards the opposite corner.
// • This algorithm terminates when either the X coordinates or the
// Y coordinates of the forward and reverse frontier points ever intersect.
//
// This algorithm is correct even if searching only in the forward direction
// or in the reverse direction. We do both because it is commonly observed
// that two lists commonly differ because elements were added to the front
// or end of the other list.
//
// Running the tests with the "debug" build tag prints a visualization of
// the algorithm running in real-time. This is educational for understanding
// how the algorithm works. See debug_enable.go.
f = debug.Begin(nx, ny, f, &fwdPath.es, &revPath.es)
for {
// Forward search from the beginning.
if fwdFrontier.X >= revFrontier.X || fwdFrontier.Y >= revFrontier.Y || searchBudget == 0 {
break
}
for stop1, stop2, i := false, false, 0; !(stop1 && stop2) && searchBudget > 0; i++ {
// Search in a diagonal pattern for a match.
z := zigzag(i)
p := point{fwdFrontier.X + z, fwdFrontier.Y - z}
switch {
case p.X >= revPath.X || p.Y < fwdPath.Y:
stop1 = true // Hit top-right corner
case p.Y >= revPath.Y || p.X < fwdPath.X:
stop2 = true // Hit bottom-left corner
case f(p.X, p.Y).Equal():
// Match found, so connect the path to this point.
fwdPath.connect(p, f)
fwdPath.append(Identity)
// Follow sequence of matches as far as possible.
for fwdPath.X < revPath.X && fwdPath.Y < revPath.Y {
if !f(fwdPath.X, fwdPath.Y).Equal() {
break
}
fwdPath.append(Identity)
}
fwdFrontier = fwdPath.point
stop1, stop2 = true, true
default:
searchBudget-- // Match not found
}
debug.Update()
}
// Advance the frontier towards reverse point.
if revPath.X-fwdFrontier.X >= revPath.Y-fwdFrontier.Y {
fwdFrontier.X++
} else {
fwdFrontier.Y++
}
// Reverse search from the end.
if fwdFrontier.X >= revFrontier.X || fwdFrontier.Y >= revFrontier.Y || searchBudget == 0 {
break
}
for stop1, stop2, i := false, false, 0; !(stop1 && stop2) && searchBudget > 0; i++ {
// Search in a diagonal pattern for a match.
z := zigzag(i)
p := point{revFrontier.X - z, revFrontier.Y + z}
switch {
case fwdPath.X >= p.X || revPath.Y < p.Y:
stop1 = true // Hit bottom-left corner
case fwdPath.Y >= p.Y || revPath.X < p.X:
stop2 = true // Hit top-right corner
case f(p.X-1, p.Y-1).Equal():
// Match found, so connect the path to this point.
revPath.connect(p, f)
revPath.append(Identity)
// Follow sequence of matches as far as possible.
for fwdPath.X < revPath.X && fwdPath.Y < revPath.Y {
if !f(revPath.X-1, revPath.Y-1).Equal() {
break
}
revPath.append(Identity)
}
revFrontier = revPath.point
stop1, stop2 = true, true
default:
searchBudget-- // Match not found
}
debug.Update()
}
// Advance the frontier towards forward point.
if revFrontier.X-fwdPath.X >= revFrontier.Y-fwdPath.Y {
revFrontier.X--
} else {
revFrontier.Y--
}
}
// Join the forward and reverse paths and then append the reverse path.
fwdPath.connect(revPath.point, f)
for i := len(revPath.es) - 1; i >= 0; i-- {
t := revPath.es[i]
revPath.es = revPath.es[:i]
fwdPath.append(t)
}
debug.Finish()
return fwdPath.es
}
type path struct {
dir int // +1 if forward, -1 if reverse
point // Leading point of the EditScript path
es EditScript
}
// connect appends any necessary Identity, Modified, UniqueX, or UniqueY types
// to the edit-script to connect p.point to dst.
func (p *path) connect(dst point, f EqualFunc) {
if p.dir > 0 {
// Connect in forward direction.
for dst.X > p.X && dst.Y > p.Y {
switch r := f(p.X, p.Y); {
case r.Equal():
p.append(Identity)
case r.Similar():
p.append(Modified)
case dst.X-p.X >= dst.Y-p.Y:
p.append(UniqueX)
default:
p.append(UniqueY)
}
}
for dst.X > p.X {
p.append(UniqueX)
}
for dst.Y > p.Y {
p.append(UniqueY)
}
} else {
// Connect in reverse direction.
for p.X > dst.X && p.Y > dst.Y {
switch r := f(p.X-1, p.Y-1); {
case r.Equal():
p.append(Identity)
case r.Similar():
p.append(Modified)
case p.Y-dst.Y >= p.X-dst.X:
p.append(UniqueY)
default:
p.append(UniqueX)
}
}
for p.X > dst.X {
p.append(UniqueX)
}
for p.Y > dst.Y {
p.append(UniqueY)
}
}
}
func (p *path) append(t EditType) {
p.es = append(p.es, t)
switch t {
case Identity, Modified:
p.add(p.dir, p.dir)
case UniqueX:
p.add(p.dir, 0)
case UniqueY:
p.add(0, p.dir)
}
debug.Update()
}
type point struct{ X, Y int }
func (p *point) add(dx, dy int) { p.X += dx; p.Y += dy }
// zigzag maps a consecutive sequence of integers to a zig-zag sequence.
// [0 1 2 3 4 5 ...] => [0 -1 +1 -2 +2 ...]
func zigzag(x int) int {
if x&1 != 0 {
x = ^x
}
return x >> 1
}

View File

@ -0,0 +1,49 @@
// Copyright 2017, The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE.md file.
// Package function identifies function types.
package function
import "reflect"
type funcType int
const (
_ funcType = iota
ttbFunc // func(T, T) bool
tibFunc // func(T, I) bool
trFunc // func(T) R
Equal = ttbFunc // func(T, T) bool
EqualAssignable = tibFunc // func(T, I) bool; encapsulates func(T, T) bool
Transformer = trFunc // func(T) R
ValueFilter = ttbFunc // func(T, T) bool
Less = ttbFunc // func(T, T) bool
)
var boolType = reflect.TypeOf(true)
// IsType reports whether the reflect.Type is of the specified function type.
func IsType(t reflect.Type, ft funcType) bool {
if t == nil || t.Kind() != reflect.Func || t.IsVariadic() {
return false
}
ni, no := t.NumIn(), t.NumOut()
switch ft {
case ttbFunc: // func(T, T) bool
if ni == 2 && no == 1 && t.In(0) == t.In(1) && t.Out(0) == boolType {
return true
}
case tibFunc: // func(T, I) bool
if ni == 2 && no == 1 && t.In(0).AssignableTo(t.In(1)) && t.Out(0) == boolType {
return true
}
case trFunc: // func(T) R
if ni == 1 && no == 1 {
return true
}
}
return false
}

View File

@ -0,0 +1,259 @@
// Copyright 2017, The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE.md file.
// Package value provides functionality for reflect.Value types.
package value
import (
"fmt"
"reflect"
"strings"
"unicode"
"unicode/utf8"
)
// formatFakePointers controls whether to substitute pointer addresses with nil.
// This is used for deterministic testing.
var formatFakePointers = false
var stringerIface = reflect.TypeOf((*fmt.Stringer)(nil)).Elem()
// Format formats the value v as a string.
//
// This is similar to fmt.Sprintf("%+v", v) except this:
// * Prints the type unless it can be elided
// * Avoids printing struct fields that are zero
// * Prints a nil-slice as being nil, not empty
// * Prints map entries in deterministic order
func Format(v reflect.Value, useStringer bool) string {
return formatAny(v, formatConfig{useStringer, true, true, !formatFakePointers}, nil)
}
type formatConfig struct {
useStringer bool // Should the String method be used if available?
printType bool // Should we print the type before the value?
followPointers bool // Should we recursively follow pointers?
realPointers bool // Should we print the real address of pointers?
}
func formatAny(v reflect.Value, conf formatConfig, visited map[uintptr]bool) string {
// TODO: Should this be a multi-line printout in certain situations?
if !v.IsValid() {
return "<non-existent>"
}
if conf.useStringer && v.Type().Implements(stringerIface) && v.CanInterface() {
if (v.Kind() == reflect.Ptr || v.Kind() == reflect.Interface) && v.IsNil() {
return "<nil>"
}
return fmt.Sprintf("%q", v.Interface().(fmt.Stringer).String())
}
switch v.Kind() {
case reflect.Bool:
return formatPrimitive(v.Type(), v.Bool(), conf)
case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
return formatPrimitive(v.Type(), v.Int(), conf)
case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, reflect.Uintptr:
if v.Type().PkgPath() == "" || v.Kind() == reflect.Uintptr {
// Unnamed uints are usually bytes or words, so use hexadecimal.
return formatPrimitive(v.Type(), formatHex(v.Uint()), conf)
}
return formatPrimitive(v.Type(), v.Uint(), conf)
case reflect.Float32, reflect.Float64:
return formatPrimitive(v.Type(), v.Float(), conf)
case reflect.Complex64, reflect.Complex128:
return formatPrimitive(v.Type(), v.Complex(), conf)
case reflect.String:
return formatPrimitive(v.Type(), fmt.Sprintf("%q", v), conf)
case reflect.UnsafePointer, reflect.Chan, reflect.Func:
return formatPointer(v, conf)
case reflect.Ptr:
if v.IsNil() {
if conf.printType {
return fmt.Sprintf("(%v)(nil)", v.Type())
}
return "<nil>"
}
if visited[v.Pointer()] || !conf.followPointers {
return formatPointer(v, conf)
}
visited = insertPointer(visited, v.Pointer())
return "&" + formatAny(v.Elem(), conf, visited)
case reflect.Interface:
if v.IsNil() {
if conf.printType {
return fmt.Sprintf("%v(nil)", v.Type())
}
return "<nil>"
}
return formatAny(v.Elem(), conf, visited)
case reflect.Slice:
if v.IsNil() {
if conf.printType {
return fmt.Sprintf("%v(nil)", v.Type())
}
return "<nil>"
}
if visited[v.Pointer()] {
return formatPointer(v, conf)
}
visited = insertPointer(visited, v.Pointer())
fallthrough
case reflect.Array:
var ss []string
subConf := conf
subConf.printType = v.Type().Elem().Kind() == reflect.Interface
for i := 0; i < v.Len(); i++ {
s := formatAny(v.Index(i), subConf, visited)
ss = append(ss, s)
}
s := fmt.Sprintf("{%s}", strings.Join(ss, ", "))
if conf.printType {
return v.Type().String() + s
}
return s
case reflect.Map:
if v.IsNil() {
if conf.printType {
return fmt.Sprintf("%v(nil)", v.Type())
}
return "<nil>"
}
if visited[v.Pointer()] {
return formatPointer(v, conf)
}
visited = insertPointer(visited, v.Pointer())
var ss []string
subConf := conf
subConf.printType = v.Type().Elem().Kind() == reflect.Interface
for _, k := range SortKeys(v.MapKeys()) {
sk := formatAny(k, formatConfig{realPointers: conf.realPointers}, visited)
sv := formatAny(v.MapIndex(k), subConf, visited)
ss = append(ss, fmt.Sprintf("%s: %s", sk, sv))
}
s := fmt.Sprintf("{%s}", strings.Join(ss, ", "))
if conf.printType {
return v.Type().String() + s
}
return s
case reflect.Struct:
var ss []string
subConf := conf
subConf.printType = true
for i := 0; i < v.NumField(); i++ {
vv := v.Field(i)
if isZero(vv) {
continue // Elide zero value fields
}
name := v.Type().Field(i).Name
subConf.useStringer = conf.useStringer && isExported(name)
s := formatAny(vv, subConf, visited)
ss = append(ss, fmt.Sprintf("%s: %s", name, s))
}
s := fmt.Sprintf("{%s}", strings.Join(ss, ", "))
if conf.printType {
return v.Type().String() + s
}
return s
default:
panic(fmt.Sprintf("%v kind not handled", v.Kind()))
}
}
func formatPrimitive(t reflect.Type, v interface{}, conf formatConfig) string {
if conf.printType && t.PkgPath() != "" {
return fmt.Sprintf("%v(%v)", t, v)
}
return fmt.Sprintf("%v", v)
}
func formatPointer(v reflect.Value, conf formatConfig) string {
p := v.Pointer()
if !conf.realPointers {
p = 0 // For deterministic printing purposes
}
s := formatHex(uint64(p))
if conf.printType {
return fmt.Sprintf("(%v)(%s)", v.Type(), s)
}
return s
}
func formatHex(u uint64) string {
var f string
switch {
case u <= 0xff:
f = "0x%02x"
case u <= 0xffff:
f = "0x%04x"
case u <= 0xffffff:
f = "0x%06x"
case u <= 0xffffffff:
f = "0x%08x"
case u <= 0xffffffffff:
f = "0x%010x"
case u <= 0xffffffffffff:
f = "0x%012x"
case u <= 0xffffffffffffff:
f = "0x%014x"
case u <= 0xffffffffffffffff:
f = "0x%016x"
}
return fmt.Sprintf(f, u)
}
// insertPointer insert p into m, allocating m if necessary.
func insertPointer(m map[uintptr]bool, p uintptr) map[uintptr]bool {
if m == nil {
m = make(map[uintptr]bool)
}
m[p] = true
return m
}
// isZero reports whether v is the zero value.
// This does not rely on Interface and so can be used on unexported fields.
func isZero(v reflect.Value) bool {
switch v.Kind() {
case reflect.Bool:
return v.Bool() == false
case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
return v.Int() == 0
case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, reflect.Uintptr:
return v.Uint() == 0
case reflect.Float32, reflect.Float64:
return v.Float() == 0
case reflect.Complex64, reflect.Complex128:
return v.Complex() == 0
case reflect.String:
return v.String() == ""
case reflect.UnsafePointer:
return v.Pointer() == 0
case reflect.Chan, reflect.Func, reflect.Interface, reflect.Ptr, reflect.Map, reflect.Slice:
return v.IsNil()
case reflect.Array:
for i := 0; i < v.Len(); i++ {
if !isZero(v.Index(i)) {
return false
}
}
return true
case reflect.Struct:
for i := 0; i < v.NumField(); i++ {
if !isZero(v.Field(i)) {
return false
}
}
return true
}
return false
}
// isExported reports whether the identifier is exported.
func isExported(id string) bool {
r, _ := utf8.DecodeRuneInString(id)
return unicode.IsUpper(r)
}

View File

@ -0,0 +1,111 @@
// Copyright 2017, The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE.md file.
package value
import (
"fmt"
"math"
"reflect"
"sort"
)
// SortKeys sorts a list of map keys, deduplicating keys if necessary.
// The type of each value must be comparable.
func SortKeys(vs []reflect.Value) []reflect.Value {
if len(vs) == 0 {
return vs
}
// Sort the map keys.
sort.Sort(valueSorter(vs))
// Deduplicate keys (fails for NaNs).
vs2 := vs[:1]
for _, v := range vs[1:] {
if v.Interface() != vs2[len(vs2)-1].Interface() {
vs2 = append(vs2, v)
}
}
return vs2
}
// TODO: Use sort.Slice once Google AppEngine is on Go1.8 or above.
type valueSorter []reflect.Value
func (vs valueSorter) Len() int { return len(vs) }
func (vs valueSorter) Less(i, j int) bool { return isLess(vs[i], vs[j]) }
func (vs valueSorter) Swap(i, j int) { vs[i], vs[j] = vs[j], vs[i] }
// isLess is a generic function for sorting arbitrary map keys.
// The inputs must be of the same type and must be comparable.
func isLess(x, y reflect.Value) bool {
switch x.Type().Kind() {
case reflect.Bool:
return !x.Bool() && y.Bool()
case reflect.Int, reflect.Int8, reflect.Int16, reflect.Int32, reflect.Int64:
return x.Int() < y.Int()
case reflect.Uint, reflect.Uint8, reflect.Uint16, reflect.Uint32, reflect.Uint64, reflect.Uintptr:
return x.Uint() < y.Uint()
case reflect.Float32, reflect.Float64:
fx, fy := x.Float(), y.Float()
return fx < fy || math.IsNaN(fx) && !math.IsNaN(fy)
case reflect.Complex64, reflect.Complex128:
cx, cy := x.Complex(), y.Complex()
rx, ix, ry, iy := real(cx), imag(cx), real(cy), imag(cy)
if rx == ry || (math.IsNaN(rx) && math.IsNaN(ry)) {
return ix < iy || math.IsNaN(ix) && !math.IsNaN(iy)
}
return rx < ry || math.IsNaN(rx) && !math.IsNaN(ry)
case reflect.Ptr, reflect.UnsafePointer, reflect.Chan:
return x.Pointer() < y.Pointer()
case reflect.String:
return x.String() < y.String()
case reflect.Array:
for i := 0; i < x.Len(); i++ {
if isLess(x.Index(i), y.Index(i)) {
return true
}
if isLess(y.Index(i), x.Index(i)) {
return false
}
}
return false
case reflect.Struct:
for i := 0; i < x.NumField(); i++ {
if isLess(x.Field(i), y.Field(i)) {
return true
}
if isLess(y.Field(i), x.Field(i)) {
return false
}
}
return false
case reflect.Interface:
vx, vy := x.Elem(), y.Elem()
if !vx.IsValid() || !vy.IsValid() {
return !vx.IsValid() && vy.IsValid()
}
tx, ty := vx.Type(), vy.Type()
if tx == ty {
return isLess(x.Elem(), y.Elem())
}
if tx.Kind() != ty.Kind() {
return vx.Kind() < vy.Kind()
}
if tx.String() != ty.String() {
return tx.String() < ty.String()
}
if tx.PkgPath() != ty.PkgPath() {
return tx.PkgPath() < ty.PkgPath()
}
// This can happen in rare situations, so we fallback to just comparing
// the unique pointer for a reflect.Type. This guarantees deterministic
// ordering within a program, but it is obviously not stable.
return reflect.ValueOf(vx.Type()).Pointer() < reflect.ValueOf(vy.Type()).Pointer()
default:
// Must be Func, Map, or Slice; which are not comparable.
panic(fmt.Sprintf("%T is not comparable", x.Type()))
}
}

View File

@ -0,0 +1,446 @@
// Copyright 2017, The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE.md file.
package cmp
import (
"fmt"
"reflect"
"runtime"
"strings"
"github.com/google/go-cmp/cmp/internal/function"
)
// Option configures for specific behavior of Equal and Diff. In particular,
// the fundamental Option functions (Ignore, Transformer, and Comparer),
// configure how equality is determined.
//
// The fundamental options may be composed with filters (FilterPath and
// FilterValues) to control the scope over which they are applied.
//
// The cmp/cmpopts package provides helper functions for creating options that
// may be used with Equal and Diff.
type Option interface {
// filter applies all filters and returns the option that remains.
// Each option may only read s.curPath and call s.callTTBFunc.
//
// An Options is returned only if multiple comparers or transformers
// can apply simultaneously and will only contain values of those types
// or sub-Options containing values of those types.
filter(s *state, vx, vy reflect.Value, t reflect.Type) applicableOption
}
// applicableOption represents the following types:
// Fundamental: ignore | invalid | *comparer | *transformer
// Grouping: Options
type applicableOption interface {
Option
// apply executes the option and reports whether the option was applied.
// Each option may mutate s.
apply(s *state, vx, vy reflect.Value) bool
}
// coreOption represents the following types:
// Fundamental: ignore | invalid | *comparer | *transformer
// Filters: *pathFilter | *valuesFilter
type coreOption interface {
Option
isCore()
}
type core struct{}
func (core) isCore() {}
// Options is a list of Option values that also satisfies the Option interface.
// Helper comparison packages may return an Options value when packing multiple
// Option values into a single Option. When this package processes an Options,
// it will be implicitly expanded into a flat list.
//
// Applying a filter on an Options is equivalent to applying that same filter
// on all individual options held within.
type Options []Option
func (opts Options) filter(s *state, vx, vy reflect.Value, t reflect.Type) (out applicableOption) {
for _, opt := range opts {
switch opt := opt.filter(s, vx, vy, t); opt.(type) {
case ignore:
return ignore{} // Only ignore can short-circuit evaluation
case invalid:
out = invalid{} // Takes precedence over comparer or transformer
case *comparer, *transformer, Options:
switch out.(type) {
case nil:
out = opt
case invalid:
// Keep invalid
case *comparer, *transformer, Options:
out = Options{out, opt} // Conflicting comparers or transformers
}
}
}
return out
}
func (opts Options) apply(s *state, _, _ reflect.Value) bool {
const warning = "ambiguous set of applicable options"
const help = "consider using filters to ensure at most one Comparer or Transformer may apply"
var ss []string
for _, opt := range flattenOptions(nil, opts) {
ss = append(ss, fmt.Sprint(opt))
}
set := strings.Join(ss, "\n\t")
panic(fmt.Sprintf("%s at %#v:\n\t%s\n%s", warning, s.curPath, set, help))
}
func (opts Options) String() string {
var ss []string
for _, opt := range opts {
ss = append(ss, fmt.Sprint(opt))
}
return fmt.Sprintf("Options{%s}", strings.Join(ss, ", "))
}
// FilterPath returns a new Option where opt is only evaluated if filter f
// returns true for the current Path in the value tree.
//
// The option passed in may be an Ignore, Transformer, Comparer, Options, or
// a previously filtered Option.
func FilterPath(f func(Path) bool, opt Option) Option {
if f == nil {
panic("invalid path filter function")
}
if opt := normalizeOption(opt); opt != nil {
return &pathFilter{fnc: f, opt: opt}
}
return nil
}
type pathFilter struct {
core
fnc func(Path) bool
opt Option
}
func (f pathFilter) filter(s *state, vx, vy reflect.Value, t reflect.Type) applicableOption {
if f.fnc(s.curPath) {
return f.opt.filter(s, vx, vy, t)
}
return nil
}
func (f pathFilter) String() string {
fn := getFuncName(reflect.ValueOf(f.fnc).Pointer())
return fmt.Sprintf("FilterPath(%s, %v)", fn, f.opt)
}
// FilterValues returns a new Option where opt is only evaluated if filter f,
// which is a function of the form "func(T, T) bool", returns true for the
// current pair of values being compared. If the type of the values is not
// assignable to T, then this filter implicitly returns false.
//
// The filter function must be
// symmetric (i.e., agnostic to the order of the inputs) and
// deterministic (i.e., produces the same result when given the same inputs).
// If T is an interface, it is possible that f is called with two values with
// different concrete types that both implement T.
//
// The option passed in may be an Ignore, Transformer, Comparer, Options, or
// a previously filtered Option.
func FilterValues(f interface{}, opt Option) Option {
v := reflect.ValueOf(f)
if !function.IsType(v.Type(), function.ValueFilter) || v.IsNil() {
panic(fmt.Sprintf("invalid values filter function: %T", f))
}
if opt := normalizeOption(opt); opt != nil {
vf := &valuesFilter{fnc: v, opt: opt}
if ti := v.Type().In(0); ti.Kind() != reflect.Interface || ti.NumMethod() > 0 {
vf.typ = ti
}
return vf
}
return nil
}
type valuesFilter struct {
core
typ reflect.Type // T
fnc reflect.Value // func(T, T) bool
opt Option
}
func (f valuesFilter) filter(s *state, vx, vy reflect.Value, t reflect.Type) applicableOption {
if !vx.IsValid() || !vy.IsValid() {
return invalid{}
}
if (f.typ == nil || t.AssignableTo(f.typ)) && s.callTTBFunc(f.fnc, vx, vy) {
return f.opt.filter(s, vx, vy, t)
}
return nil
}
func (f valuesFilter) String() string {
fn := getFuncName(f.fnc.Pointer())
return fmt.Sprintf("FilterValues(%s, %v)", fn, f.opt)
}
// Ignore is an Option that causes all comparisons to be ignored.
// This value is intended to be combined with FilterPath or FilterValues.
// It is an error to pass an unfiltered Ignore option to Equal.
func Ignore() Option { return ignore{} }
type ignore struct{ core }
func (ignore) isFiltered() bool { return false }
func (ignore) filter(_ *state, _, _ reflect.Value, _ reflect.Type) applicableOption { return ignore{} }
func (ignore) apply(_ *state, _, _ reflect.Value) bool { return true }
func (ignore) String() string { return "Ignore()" }
// invalid is a sentinel Option type to indicate that some options could not
// be evaluated due to unexported fields.
type invalid struct{ core }
func (invalid) filter(_ *state, _, _ reflect.Value, _ reflect.Type) applicableOption { return invalid{} }
func (invalid) apply(s *state, _, _ reflect.Value) bool {
const help = "consider using AllowUnexported or cmpopts.IgnoreUnexported"
panic(fmt.Sprintf("cannot handle unexported field: %#v\n%s", s.curPath, help))
}
// Transformer returns an Option that applies a transformation function that
// converts values of a certain type into that of another.
//
// The transformer f must be a function "func(T) R" that converts values of
// type T to those of type R and is implicitly filtered to input values
// assignable to T. The transformer must not mutate T in any way.
// If T and R are the same type, an additional filter must be applied to
// act as the base case to prevent an infinite recursion applying the same
// transform to itself (see the SortedSlice example).
//
// The name is a user provided label that is used as the Transform.Name in the
// transformation PathStep. If empty, an arbitrary name is used.
func Transformer(name string, f interface{}) Option {
v := reflect.ValueOf(f)
if !function.IsType(v.Type(), function.Transformer) || v.IsNil() {
panic(fmt.Sprintf("invalid transformer function: %T", f))
}
if name == "" {
name = "λ" // Lambda-symbol as place-holder for anonymous transformer
}
if !isValid(name) {
panic(fmt.Sprintf("invalid name: %q", name))
}
tr := &transformer{name: name, fnc: reflect.ValueOf(f)}
if ti := v.Type().In(0); ti.Kind() != reflect.Interface || ti.NumMethod() > 0 {
tr.typ = ti
}
return tr
}
type transformer struct {
core
name string
typ reflect.Type // T
fnc reflect.Value // func(T) R
}
func (tr *transformer) isFiltered() bool { return tr.typ != nil }
func (tr *transformer) filter(_ *state, _, _ reflect.Value, t reflect.Type) applicableOption {
if tr.typ == nil || t.AssignableTo(tr.typ) {
return tr
}
return nil
}
func (tr *transformer) apply(s *state, vx, vy reflect.Value) bool {
// Update path before calling the Transformer so that dynamic checks
// will use the updated path.
s.curPath.push(&transform{pathStep{tr.fnc.Type().Out(0)}, tr})
defer s.curPath.pop()
vx = s.callTRFunc(tr.fnc, vx)
vy = s.callTRFunc(tr.fnc, vy)
s.compareAny(vx, vy)
return true
}
func (tr transformer) String() string {
return fmt.Sprintf("Transformer(%s, %s)", tr.name, getFuncName(tr.fnc.Pointer()))
}
// Comparer returns an Option that determines whether two values are equal
// to each other.
//
// The comparer f must be a function "func(T, T) bool" and is implicitly
// filtered to input values assignable to T. If T is an interface, it is
// possible that f is called with two values of different concrete types that
// both implement T.
//
// The equality function must be:
// • Symmetric: equal(x, y) == equal(y, x)
// • Deterministic: equal(x, y) == equal(x, y)
// • Pure: equal(x, y) does not modify x or y
func Comparer(f interface{}) Option {
v := reflect.ValueOf(f)
if !function.IsType(v.Type(), function.Equal) || v.IsNil() {
panic(fmt.Sprintf("invalid comparer function: %T", f))
}
cm := &comparer{fnc: v}
if ti := v.Type().In(0); ti.Kind() != reflect.Interface || ti.NumMethod() > 0 {
cm.typ = ti
}
return cm
}
type comparer struct {
core
typ reflect.Type // T
fnc reflect.Value // func(T, T) bool
}
func (cm *comparer) isFiltered() bool { return cm.typ != nil }
func (cm *comparer) filter(_ *state, _, _ reflect.Value, t reflect.Type) applicableOption {
if cm.typ == nil || t.AssignableTo(cm.typ) {
return cm
}
return nil
}
func (cm *comparer) apply(s *state, vx, vy reflect.Value) bool {
eq := s.callTTBFunc(cm.fnc, vx, vy)
s.report(eq, vx, vy)
return true
}
func (cm comparer) String() string {
return fmt.Sprintf("Comparer(%s)", getFuncName(cm.fnc.Pointer()))
}
// AllowUnexported returns an Option that forcibly allows operations on
// unexported fields in certain structs, which are specified by passing in a
// value of each struct type.
//
// Users of this option must understand that comparing on unexported fields
// from external packages is not safe since changes in the internal
// implementation of some external package may cause the result of Equal
// to unexpectedly change. However, it may be valid to use this option on types
// defined in an internal package where the semantic meaning of an unexported
// field is in the control of the user.
//
// For some cases, a custom Comparer should be used instead that defines
// equality as a function of the public API of a type rather than the underlying
// unexported implementation.
//
// For example, the reflect.Type documentation defines equality to be determined
// by the == operator on the interface (essentially performing a shallow pointer
// comparison) and most attempts to compare *regexp.Regexp types are interested
// in only checking that the regular expression strings are equal.
// Both of these are accomplished using Comparers:
//
// Comparer(func(x, y reflect.Type) bool { return x == y })
// Comparer(func(x, y *regexp.Regexp) bool { return x.String() == y.String() })
//
// In other cases, the cmpopts.IgnoreUnexported option can be used to ignore
// all unexported fields on specified struct types.
func AllowUnexported(types ...interface{}) Option {
if !supportAllowUnexported {
panic("AllowUnexported is not supported on App Engine Classic or GopherJS")
}
m := make(map[reflect.Type]bool)
for _, typ := range types {
t := reflect.TypeOf(typ)
if t.Kind() != reflect.Struct {
panic(fmt.Sprintf("invalid struct type: %T", typ))
}
m[t] = true
}
return visibleStructs(m)
}
type visibleStructs map[reflect.Type]bool
func (visibleStructs) filter(_ *state, _, _ reflect.Value, _ reflect.Type) applicableOption {
panic("not implemented")
}
// reporter is an Option that configures how differences are reported.
type reporter interface {
// TODO: Not exported yet.
//
// Perhaps add PushStep and PopStep and change Report to only accept
// a PathStep instead of the full-path? Adding a PushStep and PopStep makes
// it clear that we are traversing the value tree in a depth-first-search
// manner, which has an effect on how values are printed.
Option
// Report is called for every comparison made and will be provided with
// the two values being compared, the equality result, and the
// current path in the value tree. It is possible for x or y to be an
// invalid reflect.Value if one of the values is non-existent;
// which is possible with maps and slices.
Report(x, y reflect.Value, eq bool, p Path)
}
// normalizeOption normalizes the input options such that all Options groups
// are flattened and groups with a single element are reduced to that element.
// Only coreOptions and Options containing coreOptions are allowed.
func normalizeOption(src Option) Option {
switch opts := flattenOptions(nil, Options{src}); len(opts) {
case 0:
return nil
case 1:
return opts[0]
default:
return opts
}
}
// flattenOptions copies all options in src to dst as a flat list.
// Only coreOptions and Options containing coreOptions are allowed.
func flattenOptions(dst, src Options) Options {
for _, opt := range src {
switch opt := opt.(type) {
case nil:
continue
case Options:
dst = flattenOptions(dst, opt)
case coreOption:
dst = append(dst, opt)
default:
panic(fmt.Sprintf("invalid option type: %T", opt))
}
}
return dst
}
// getFuncName returns a short function name from the pointer.
// The string parsing logic works up until Go1.9.
func getFuncName(p uintptr) string {
fnc := runtime.FuncForPC(p)
if fnc == nil {
return "<unknown>"
}
name := fnc.Name() // E.g., "long/path/name/mypkg.(mytype).(long/path/name/mypkg.myfunc)-fm"
if strings.HasSuffix(name, ")-fm") || strings.HasSuffix(name, ")·fm") {
// Strip the package name from method name.
name = strings.TrimSuffix(name, ")-fm")
name = strings.TrimSuffix(name, ")·fm")
if i := strings.LastIndexByte(name, '('); i >= 0 {
methodName := name[i+1:] // E.g., "long/path/name/mypkg.myfunc"
if j := strings.LastIndexByte(methodName, '.'); j >= 0 {
methodName = methodName[j+1:] // E.g., "myfunc"
}
name = name[:i] + methodName // E.g., "long/path/name/mypkg.(mytype)." + "myfunc"
}
}
if i := strings.LastIndexByte(name, '/'); i >= 0 {
// Strip the package name.
name = name[i+1:] // E.g., "mypkg.(mytype).myfunc"
}
return name
}

View File

@ -0,0 +1,293 @@
// Copyright 2017, The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE.md file.
package cmp
import (
"fmt"
"reflect"
"strings"
"unicode"
"unicode/utf8"
)
type (
// Path is a list of PathSteps describing the sequence of operations to get
// from some root type to the current position in the value tree.
// The first Path element is always an operation-less PathStep that exists
// simply to identify the initial type.
//
// When traversing structs with embedded structs, the embedded struct will
// always be accessed as a field before traversing the fields of the
// embedded struct themselves. That is, an exported field from the
// embedded struct will never be accessed directly from the parent struct.
Path []PathStep
// PathStep is a union-type for specific operations to traverse
// a value's tree structure. Users of this package never need to implement
// these types as values of this type will be returned by this package.
PathStep interface {
String() string
Type() reflect.Type // Resulting type after performing the path step
isPathStep()
}
// SliceIndex is an index operation on a slice or array at some index Key.
SliceIndex interface {
PathStep
Key() int // May return -1 if in a split state
// SplitKeys returns the indexes for indexing into slices in the
// x and y values, respectively. These indexes may differ due to the
// insertion or removal of an element in one of the slices, causing
// all of the indexes to be shifted. If an index is -1, then that
// indicates that the element does not exist in the associated slice.
//
// Key is guaranteed to return -1 if and only if the indexes returned
// by SplitKeys are not the same. SplitKeys will never return -1 for
// both indexes.
SplitKeys() (x int, y int)
isSliceIndex()
}
// MapIndex is an index operation on a map at some index Key.
MapIndex interface {
PathStep
Key() reflect.Value
isMapIndex()
}
// TypeAssertion represents a type assertion on an interface.
TypeAssertion interface {
PathStep
isTypeAssertion()
}
// StructField represents a struct field access on a field called Name.
StructField interface {
PathStep
Name() string
Index() int
isStructField()
}
// Indirect represents pointer indirection on the parent type.
Indirect interface {
PathStep
isIndirect()
}
// Transform is a transformation from the parent type to the current type.
Transform interface {
PathStep
Name() string
Func() reflect.Value
isTransform()
}
)
func (pa *Path) push(s PathStep) {
*pa = append(*pa, s)
}
func (pa *Path) pop() {
*pa = (*pa)[:len(*pa)-1]
}
// Last returns the last PathStep in the Path.
// If the path is empty, this returns a non-nil PathStep that reports a nil Type.
func (pa Path) Last() PathStep {
if len(pa) > 0 {
return pa[len(pa)-1]
}
return pathStep{}
}
// String returns the simplified path to a node.
// The simplified path only contains struct field accesses.
//
// For example:
// MyMap.MySlices.MyField
func (pa Path) String() string {
var ss []string
for _, s := range pa {
if _, ok := s.(*structField); ok {
ss = append(ss, s.String())
}
}
return strings.TrimPrefix(strings.Join(ss, ""), ".")
}
// GoString returns the path to a specific node using Go syntax.
//
// For example:
// (*root.MyMap["key"].(*mypkg.MyStruct).MySlices)[2][3].MyField
func (pa Path) GoString() string {
var ssPre, ssPost []string
var numIndirect int
for i, s := range pa {
var nextStep PathStep
if i+1 < len(pa) {
nextStep = pa[i+1]
}
switch s := s.(type) {
case *indirect:
numIndirect++
pPre, pPost := "(", ")"
switch nextStep.(type) {
case *indirect:
continue // Next step is indirection, so let them batch up
case *structField:
numIndirect-- // Automatic indirection on struct fields
case nil:
pPre, pPost = "", "" // Last step; no need for parenthesis
}
if numIndirect > 0 {
ssPre = append(ssPre, pPre+strings.Repeat("*", numIndirect))
ssPost = append(ssPost, pPost)
}
numIndirect = 0
continue
case *transform:
ssPre = append(ssPre, s.trans.name+"(")
ssPost = append(ssPost, ")")
continue
case *typeAssertion:
// Elide type assertions immediately following a transform to
// prevent overly verbose path printouts.
// Some transforms return interface{} because of Go's lack of
// generics, but typically take in and return the exact same
// concrete type. Other times, the transform creates an anonymous
// struct, which will be very verbose to print.
if _, ok := nextStep.(*transform); ok {
continue
}
}
ssPost = append(ssPost, s.String())
}
for i, j := 0, len(ssPre)-1; i < j; i, j = i+1, j-1 {
ssPre[i], ssPre[j] = ssPre[j], ssPre[i]
}
return strings.Join(ssPre, "") + strings.Join(ssPost, "")
}
type (
pathStep struct {
typ reflect.Type
}
sliceIndex struct {
pathStep
xkey, ykey int
}
mapIndex struct {
pathStep
key reflect.Value
}
typeAssertion struct {
pathStep
}
structField struct {
pathStep
name string
idx int
// These fields are used for forcibly accessing an unexported field.
// pvx, pvy, and field are only valid if unexported is true.
unexported bool
force bool // Forcibly allow visibility
pvx, pvy reflect.Value // Parent values
field reflect.StructField // Field information
}
indirect struct {
pathStep
}
transform struct {
pathStep
trans *transformer
}
)
func (ps pathStep) Type() reflect.Type { return ps.typ }
func (ps pathStep) String() string {
if ps.typ == nil {
return "<nil>"
}
s := ps.typ.String()
if s == "" || strings.ContainsAny(s, "{}\n") {
return "root" // Type too simple or complex to print
}
return fmt.Sprintf("{%s}", s)
}
func (si sliceIndex) String() string {
switch {
case si.xkey == si.ykey:
return fmt.Sprintf("[%d]", si.xkey)
case si.ykey == -1:
// [5->?] means "I don't know where X[5] went"
return fmt.Sprintf("[%d->?]", si.xkey)
case si.xkey == -1:
// [?->3] means "I don't know where Y[3] came from"
return fmt.Sprintf("[?->%d]", si.ykey)
default:
// [5->3] means "X[5] moved to Y[3]"
return fmt.Sprintf("[%d->%d]", si.xkey, si.ykey)
}
}
func (mi mapIndex) String() string { return fmt.Sprintf("[%#v]", mi.key) }
func (ta typeAssertion) String() string { return fmt.Sprintf(".(%v)", ta.typ) }
func (sf structField) String() string { return fmt.Sprintf(".%s", sf.name) }
func (in indirect) String() string { return "*" }
func (tf transform) String() string { return fmt.Sprintf("%s()", tf.trans.name) }
func (si sliceIndex) Key() int {
if si.xkey != si.ykey {
return -1
}
return si.xkey
}
func (si sliceIndex) SplitKeys() (x, y int) { return si.xkey, si.ykey }
func (mi mapIndex) Key() reflect.Value { return mi.key }
func (sf structField) Name() string { return sf.name }
func (sf structField) Index() int { return sf.idx }
func (tf transform) Name() string { return tf.trans.name }
func (tf transform) Func() reflect.Value { return tf.trans.fnc }
func (pathStep) isPathStep() {}
func (sliceIndex) isSliceIndex() {}
func (mapIndex) isMapIndex() {}
func (typeAssertion) isTypeAssertion() {}
func (structField) isStructField() {}
func (indirect) isIndirect() {}
func (transform) isTransform() {}
var (
_ SliceIndex = sliceIndex{}
_ MapIndex = mapIndex{}
_ TypeAssertion = typeAssertion{}
_ StructField = structField{}
_ Indirect = indirect{}
_ Transform = transform{}
_ PathStep = sliceIndex{}
_ PathStep = mapIndex{}
_ PathStep = typeAssertion{}
_ PathStep = structField{}
_ PathStep = indirect{}
_ PathStep = transform{}
)
// isExported reports whether the identifier is exported.
func isExported(id string) bool {
r, _ := utf8.DecodeRuneInString(id)
return unicode.IsUpper(r)
}
// isValid reports whether the identifier is valid.
// Empty and underscore-only strings are not valid.
func isValid(id string) bool {
ok := id != "" && id != "_"
for j, c := range id {
ok = ok && (j > 0 || !unicode.IsDigit(c))
ok = ok && (c == '_' || unicode.IsLetter(c) || unicode.IsDigit(c))
}
return ok
}

View File

@ -0,0 +1,53 @@
// Copyright 2017, The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE.md file.
package cmp
import (
"fmt"
"reflect"
"strings"
"github.com/google/go-cmp/cmp/internal/value"
)
type defaultReporter struct {
Option
diffs []string // List of differences, possibly truncated
ndiffs int // Total number of differences
nbytes int // Number of bytes in diffs
nlines int // Number of lines in diffs
}
var _ reporter = (*defaultReporter)(nil)
func (r *defaultReporter) Report(x, y reflect.Value, eq bool, p Path) {
if eq {
return // Ignore equal results
}
const maxBytes = 4096
const maxLines = 256
r.ndiffs++
if r.nbytes < maxBytes && r.nlines < maxLines {
sx := value.Format(x, true)
sy := value.Format(y, true)
if sx == sy {
// Stringer is not helpful, so rely on more exact formatting.
sx = value.Format(x, false)
sy = value.Format(y, false)
}
s := fmt.Sprintf("%#v:\n\t-: %s\n\t+: %s\n", p, sx, sy)
r.diffs = append(r.diffs, s)
r.nbytes += len(s)
r.nlines += strings.Count(s, "\n")
}
}
func (r *defaultReporter) String() string {
s := strings.Join(r.diffs, "")
if r.ndiffs == len(r.diffs) {
return s
}
return fmt.Sprintf("%s... %d more differences ...", s, len(r.diffs)-r.ndiffs)
}

View File

@ -0,0 +1,15 @@
// Copyright 2017, The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE.md file.
// +build appengine js
package cmp
import "reflect"
const supportAllowUnexported = false
func unsafeRetrieveField(reflect.Value, reflect.StructField) reflect.Value {
panic("unsafeRetrieveField is not implemented")
}

View File

@ -0,0 +1,23 @@
// Copyright 2017, The Go Authors. All rights reserved.
// Use of this source code is governed by a BSD-style
// license that can be found in the LICENSE.md file.
// +build !appengine,!js
package cmp
import (
"reflect"
"unsafe"
)
const supportAllowUnexported = true
// unsafeRetrieveField uses unsafe to forcibly retrieve any field from a struct
// such that the value has read-write permissions.
//
// The parent struct, v, must be addressable, while f must be a StructField
// describing the field to retrieve.
func unsafeRetrieveField(v reflect.Value, f reflect.StructField) reflect.Value {
return reflect.NewAt(f.Type, unsafe.Pointer(v.UnsafeAddr()+f.Offset)).Elem()
}

View File

@ -10,24 +10,23 @@ patterns.
## Packages
* [assert](http://godoc.org/github.com/gotestyourself/gotestyourself/assert) -
compare values and fail the test when the comparison fails
* [env](http://godoc.org/github.com/gotestyourself/gotestyourself/env) -
test code that uses environment variables
* [fs](http://godoc.org/github.com/gotestyourself/gotestyourself/fs) -
create test files and directories
* [golden](http://godoc.org/github.com/gotestyourself/gotestyourself/golden) -
compare large multi-line strings
* [testsum](http://godoc.org/github.com/gotestyourself/gotestyourself/testsum) -
a program to summarize `go test` output and test failures
* [icmd](http://godoc.org/github.com/gotestyourself/gotestyourself/icmd) -
execute binaries and test the output
* [poll](http://godoc.org/github.com/gotestyourself/gotestyourself/poll) -
test asynchronous code by polling until a desired state is reached
* [skip](http://godoc.org/github.com/gotestyourself/gotestyourself/skip) -
skip tests based on conditions
* [testsum](http://godoc.org/github.com/gotestyourself/gotestyourself/testsum) -
a program to summarize `go test` output and test failures
## Related
* [testify/assert](https://godoc.org/github.com/stretchr/testify/assert) and
[testify/require](https://godoc.org/github.com/stretchr/testify/require) -
assertion libraries with common assertions
* [golang/mock](https://github.com/golang/mock) - generate mocks for interfaces
* [testify/suite](https://godoc.org/github.com/stretchr/testify/suite) -
group test into suites to share common setup/teardown logic
* [maxbrunsfeld/counterfeiter](https://github.com/maxbrunsfeld/counterfeiter) - generate fakes for interfaces

View File

@ -0,0 +1,186 @@
/*Package assert provides assertions and checks for comparing expected values to
actual values. When an assertion or check fails a helpful error message is
printed.
Assert and Check
Assert() and Check() both accept a Comparison, and fail the test when the
comparison fails. The one difference is that Assert() will end the test execution
immediately (using t.FailNow()) whereas Check() will fail the test (using t.Fail()),
return the value of the comparison, then proceed with the rest of the test case.
Example Usage
The example below shows assert used with some common types.
import (
"testing"
"github.com/gotestyourself/gotestyourself/assert"
is "github.com/gotestyourself/gotestyourself/assert/cmp"
)
func TestEverything(t *testing.T) {
// booleans
assert.Assert(t, isOk)
assert.Assert(t, !missing)
// primitives
assert.Equal(t, count, 1)
assert.Equal(t, msg, "the message")
assert.Assert(t, total != 10) // NotEqual
// errors
assert.NilError(t, closer.Close())
assert.Assert(t, is.Error(err, "the exact error message"))
assert.Assert(t, is.ErrorContains(err, "includes this"))
// complex types
assert.Assert(t, is.Len(items, 3))
assert.Assert(t, len(sequence) != 0) // NotEmpty
assert.Assert(t, is.Contains(mapping, "key"))
assert.Assert(t, is.Compare(result, myStruct{Name: "title"}))
// pointers and interface
assert.Assert(t, is.Nil(ref))
assert.Assert(t, ref != nil) // NotNil
}
Comparisons
https://godoc.org/github.com/gotestyourself/gotestyourself/assert/cmp provides
many common comparisons. Additional comparisons can be written to compare
values in other ways.
Below is an example of a custom comparison using a regex pattern:
func RegexP(value string, pattern string) func() (bool, string) {
return func() (bool, string) {
re := regexp.MustCompile(pattern)
msg := fmt.Sprintf("%q did not match pattern %q", value, pattern)
return re.MatchString(value), msg
}
}
*/
package assert
import (
"fmt"
"github.com/gotestyourself/gotestyourself/assert/cmp"
"github.com/gotestyourself/gotestyourself/internal/format"
"github.com/gotestyourself/gotestyourself/internal/source"
)
// BoolOrComparison can be a bool, or Comparison. Other types will panic.
type BoolOrComparison interface{}
// Comparison is a function which compares values and returns true if the actual
// value matches the expected value. If the values do not match it returns a message
// with details about why it failed.
//
// https://godoc.org/github.com/gotestyourself/gotestyourself/assert/cmp
// provides many general purpose Comparisons.
type Comparison func() (success bool, message string)
// TestingT is the subset of testing.T used by the assert package.
type TestingT interface {
FailNow()
Fail()
Log(args ...interface{})
}
type helperT interface {
Helper()
}
// stackIndex = Assert()/Check(), assert()
const stackIndex = 2
const comparisonArgPos = 1
const failureMessage = "assertion failed: "
func assert(
t TestingT,
failer func(),
comparison BoolOrComparison,
msgAndArgs ...interface{},
) bool {
if ht, ok := t.(helperT); ok {
ht.Helper()
}
switch check := comparison.(type) {
case bool:
if check {
return true
}
source, err := source.GetCondition(stackIndex, comparisonArgPos)
if err != nil {
t.Log(err.Error())
}
msg := " is false"
t.Log(format.WithCustomMessage(failureMessage+source+msg, msgAndArgs...))
failer()
return false
case Comparison:
return runCompareFunc(failer, t, check, msgAndArgs...)
case func() (success bool, message string):
return runCompareFunc(failer, t, check, msgAndArgs...)
default:
panic(fmt.Sprintf("comparison arg must be bool or Comparison, not %T", comparison))
}
}
func runCompareFunc(failer func(), t TestingT, f Comparison, msgAndArgs ...interface{}) bool {
if ht, ok := t.(helperT); ok {
ht.Helper()
}
if success, message := f(); !success {
t.Log(format.WithCustomMessage(failureMessage+message, msgAndArgs...))
failer()
return false
}
return true
}
// Assert performs a comparison, marks the test as having failed if the comparison
// returns false, and stops execution immediately.
func Assert(t TestingT, comparison BoolOrComparison, msgAndArgs ...interface{}) {
if ht, ok := t.(helperT); ok {
ht.Helper()
}
assert(t, t.FailNow, comparison, msgAndArgs...)
}
// Check performs a comparison and marks the test as having failed if the comparison
// returns false. Returns the result of the comparison.
func Check(t TestingT, comparison BoolOrComparison, msgAndArgs ...interface{}) bool {
if ht, ok := t.(helperT); ok {
ht.Helper()
}
return assert(t, t.Fail, comparison, msgAndArgs...)
}
// NilError fails the test immediately if the last arg is a non-nil error.
// This is equivalent to Assert(t, cmp.NilError(err)).
func NilError(t TestingT, err error, msgAndArgs ...interface{}) {
if ht, ok := t.(helperT); ok {
ht.Helper()
}
assert(t, t.FailNow, cmp.NilError(err), msgAndArgs...)
}
// Equal uses the == operator to assert two values are equal and fails the test
// if they are not equal. This is equivalent to Assert(t, cmp.Equal(x, y)).
func Equal(t TestingT, x, y interface{}, msgAndArgs ...interface{}) {
if ht, ok := t.(helperT); ok {
ht.Helper()
}
assert(t, t.FailNow, cmp.Equal(x, y), msgAndArgs...)
}

View File

@ -0,0 +1,201 @@
/*Package cmp provides Comparisons for Assert and Check*/
package cmp
import (
"fmt"
"reflect"
"strings"
"github.com/google/go-cmp/cmp"
"github.com/pmezard/go-difflib/difflib"
)
// Compare two complex values using https://godoc.org/github.com/google/go-cmp/cmp
// and succeeds if the values are equal.
//
// The comparison can be customized using comparison Options.
func Compare(x, y interface{}, opts ...cmp.Option) func() (bool, string) {
return func() (bool, string) {
diff := cmp.Diff(x, y, opts...)
return diff == "", "\n" + diff
}
}
// Equal succeeds if x == y.
func Equal(x, y interface{}) func() (success bool, message string) {
return func() (bool, string) {
return x == y, fmt.Sprintf("%v (%T) != %v (%T)", x, x, y, y)
}
}
// Len succeeds if the sequence has the expected length.
func Len(seq interface{}, expected int) func() (bool, string) {
return func() (success bool, message string) {
defer func() {
if e := recover(); e != nil {
success = false
message = fmt.Sprintf("type %T does not have a length", seq)
}
}()
value := reflect.ValueOf(seq)
length := value.Len()
if length == expected {
return true, ""
}
msg := fmt.Sprintf("expected %s (length %d) to have length %d", seq, length, expected)
return false, msg
}
}
// NilError succeeds if the last argument is a nil error.
func NilError(arg interface{}, args ...interface{}) func() (bool, string) {
return func() (bool, string) {
msgFunc := func(value reflect.Value) string {
return fmt.Sprintf("error is not nil: %s", value.Interface().(error).Error())
}
if len(args) == 0 {
return isNil(arg, msgFunc)()
}
return isNil(args[len(args)-1], msgFunc)()
}
}
// Contains succeeds if item is in collection. Collection may be a string, map,
// slice, or array.
//
// If collection is a string, item must also be a string, and is compared using
// strings.Contains().
// If collection is a Map, contains will succeed if item is a key in the map.
// If collection is a slice or array, item is compared to each item in the
// sequence using reflect.DeepEqual().
func Contains(collection interface{}, item interface{}) func() (bool, string) {
return func() (bool, string) {
colValue := reflect.ValueOf(collection)
if !colValue.IsValid() {
return false, fmt.Sprintf("nil does not contain items")
}
msg := fmt.Sprintf("%v does not contain %v", collection, item)
itemValue := reflect.ValueOf(item)
switch colValue.Type().Kind() {
case reflect.String:
if itemValue.Type().Kind() != reflect.String {
return false, "string may only contain strings"
}
success := strings.Contains(colValue.String(), itemValue.String())
return success, fmt.Sprintf("string %q does not contain %q", collection, item)
case reflect.Map:
if itemValue.Type() != colValue.Type().Key() {
return false, fmt.Sprintf(
"%v can not contain a %v key", colValue.Type(), itemValue.Type())
}
index := colValue.MapIndex(itemValue)
return index.IsValid(), msg
case reflect.Slice, reflect.Array:
for i := 0; i < colValue.Len(); i++ {
if reflect.DeepEqual(colValue.Index(i).Interface(), item) {
return true, ""
}
}
return false, msg
default:
return false, fmt.Sprintf("type %T does not contain items", collection)
}
}
}
// Panics succeeds if f() panics.
func Panics(f func()) func() (bool, string) {
return func() (success bool, message string) {
defer func() {
if err := recover(); err != nil {
success = true
}
}()
f()
return false, "did not panic"
}
}
// EqualMultiLine succeeds if the two strings are equal. If they are not equal
// the failure message will be the difference between the two strings.
func EqualMultiLine(x, y string) func() (bool, string) {
return func() (bool, string) {
if x == y {
return true, ""
}
diff, err := difflib.GetUnifiedDiffString(difflib.UnifiedDiff{
A: difflib.SplitLines(x),
B: difflib.SplitLines(y),
FromFile: "left",
ToFile: "right",
Context: 3,
})
if err != nil {
return false, fmt.Sprintf("failed to produce diff: %s", err)
}
return false, "\n" + diff
}
}
// Error succeeds if err is a non-nil error, and the error message equals the
// expected message.
func Error(err error, message string) func() (bool, string) {
return func() (bool, string) {
switch {
case err == nil:
return false, "expected an error, got nil"
case err.Error() != message:
return false, fmt.Sprintf(
"expected error message %q, got %q", message, err.Error())
}
return true, ""
}
}
// ErrorContains succeeds if err is a non-nil error, and the error message contains
// the expected substring.
func ErrorContains(err error, substring string) func() (bool, string) {
return func() (bool, string) {
switch {
case err == nil:
return false, "expected an error, got nil"
case !strings.Contains(err.Error(), substring):
return false, fmt.Sprintf(
"expected error message to contain %q, got %q", substring, err.Error())
}
return true, ""
}
}
// Nil succeeds if obj is a nil interface, pointer, or function.
//
// Use NilError() for comparing errors. Use Len(obj, 0) for comparing slices,
// maps, and channels.
func Nil(obj interface{}) func() (bool, string) {
msgFunc := func(value reflect.Value) string {
return fmt.Sprintf("%v (type %s) is not nil", reflect.Indirect(value), value.Type())
}
return isNil(obj, msgFunc)
}
func isNil(obj interface{}, msgFunc func(reflect.Value) string) func() (bool, string) {
return func() (bool, string) {
if obj == nil {
return true, ""
}
value := reflect.ValueOf(obj)
kind := value.Type().Kind()
if kind >= reflect.Chan && kind <= reflect.Slice {
if value.IsNil() {
return true, ""
}
return false, msgFunc(value)
}
return false, fmt.Sprintf("%v (type %s) can not be nil", value, value.Type())
}
}

View File

@ -0,0 +1,92 @@
/*Package env provides functions to test code that read environment variables
or the current working directory.
*/
package env
import (
"os"
"strings"
"github.com/gotestyourself/gotestyourself/assert"
)
type helperT interface {
Helper()
}
// Patch changes the value of an environment variable, and returns a
// function which will reset the the value of that variable back to the
// previous state.
func Patch(t assert.TestingT, key, value string) func() {
if ht, ok := t.(helperT); ok {
ht.Helper()
}
oldValue, ok := os.LookupEnv(key)
assert.NilError(t, os.Setenv(key, value))
return func() {
if ht, ok := t.(helperT); ok {
ht.Helper()
}
if !ok {
assert.NilError(t, os.Unsetenv(key))
return
}
assert.NilError(t, os.Setenv(key, oldValue))
}
}
// PatchAll sets the environment to env, and returns a function which will
// reset the environment back to the previous state.
func PatchAll(t assert.TestingT, env map[string]string) func() {
if ht, ok := t.(helperT); ok {
ht.Helper()
}
oldEnv := os.Environ()
os.Clearenv()
for key, value := range env {
assert.NilError(t, os.Setenv(key, value))
}
return func() {
if ht, ok := t.(helperT); ok {
ht.Helper()
}
os.Clearenv()
for key, oldVal := range ToMap(oldEnv) {
assert.NilError(t, os.Setenv(key, oldVal))
}
}
}
// ToMap takes a list of strings in the format returned by os.Environ() and
// returns a mapping of keys to values.
func ToMap(env []string) map[string]string {
result := map[string]string{}
for _, raw := range env {
parts := strings.SplitN(raw, "=", 2)
switch len(parts) {
case 1:
result[raw] = ""
case 2:
result[parts[0]] = parts[1]
}
}
return result
}
// ChangeWorkingDir to the directory, and return a function which restores the
// previous working directory.
func ChangeWorkingDir(t assert.TestingT, dir string) func() {
if ht, ok := t.(helperT); ok {
ht.Helper()
}
cwd, err := os.Getwd()
assert.NilError(t, err)
assert.NilError(t, os.Chdir(dir))
return func() {
if ht, ok := t.(helperT); ok {
ht.Helper()
}
assert.NilError(t, os.Chdir(cwd))
}
}

View File

@ -8,29 +8,42 @@ import (
"os"
"path/filepath"
"github.com/stretchr/testify/require"
"github.com/gotestyourself/gotestyourself/assert"
)
// Path objects return their filesystem path. Both File and Dir implement Path.
type Path interface {
Path() string
Remove()
}
var (
_ Path = &Dir{}
_ Path = &File{}
)
// File is a temporary file on the filesystem
type File struct {
path string
}
type helperT interface {
Helper()
}
// NewFile creates a new file in a temporary directory using prefix as part of
// the filename. The PathOps are applied to the before returning the File.
func NewFile(t require.TestingT, prefix string, ops ...PathOp) *File {
func NewFile(t assert.TestingT, prefix string, ops ...PathOp) *File {
if ht, ok := t.(helperT); ok {
ht.Helper()
}
tempfile, err := ioutil.TempFile("", prefix+"-")
require.NoError(t, err)
assert.NilError(t, err)
file := &File{path: tempfile.Name()}
require.NoError(t, tempfile.Close())
assert.NilError(t, tempfile.Close())
for _, op := range ops {
require.NoError(t, op(file))
assert.NilError(t, op(file))
}
return file
}
@ -53,13 +66,16 @@ type Dir struct {
// NewDir returns a new temporary directory using prefix as part of the directory
// name. The PathOps are applied before returning the Dir.
func NewDir(t require.TestingT, prefix string, ops ...PathOp) *Dir {
func NewDir(t assert.TestingT, prefix string, ops ...PathOp) *Dir {
if ht, ok := t.(helperT); ok {
ht.Helper()
}
path, err := ioutil.TempDir("", prefix+"-")
require.NoError(t, err)
assert.NilError(t, err)
dir := &Dir{path: path}
for _, op := range ops {
require.NoError(t, op(dir))
assert.NilError(t, op(dir))
}
return dir
}

View File

@ -31,14 +31,17 @@ func AsUser(uid, gid int) PathOp {
}
// WithFile creates a file in the directory at path with content
func WithFile(filename, content string) PathOp {
func WithFile(filename, content string, ops ...PathOp) PathOp {
return func(path Path) error {
return createFile(path.Path(), filename, content)
fullpath := filepath.Join(path.Path(), filepath.FromSlash(filename))
if err := createFile(fullpath, content); err != nil {
return err
}
return applyPathOps(&File{path: fullpath}, ops)
}
}
func createFile(dir, filename, content string) error {
fullpath := filepath.Join(dir, filepath.FromSlash(filename))
func createFile(fullpath string, content string) error {
return ioutil.WriteFile(fullpath, []byte(content), 0644)
}
@ -46,7 +49,8 @@ func createFile(dir, filename, content string) error {
func WithFiles(files map[string]string) PathOp {
return func(path Path) error {
for filename, content := range files {
if err := createFile(path.Path(), filename, content); err != nil {
fullpath := filepath.Join(path.Path(), filepath.FromSlash(filename))
if err := createFile(fullpath, content); err != nil {
return err
}
}
@ -61,6 +65,35 @@ func FromDir(source string) PathOp {
}
}
// WithDir creates a subdirectory in the directory at path. Additional PathOp
// can be used to modify the subdirectory
func WithDir(name string, ops ...PathOp) PathOp {
return func(path Path) error {
fullpath := filepath.Join(path.Path(), filepath.FromSlash(name))
err := os.MkdirAll(fullpath, 0755)
if err != nil {
return err
}
return applyPathOps(&Dir{path: fullpath}, ops)
}
}
func applyPathOps(path Path, ops []PathOp) error {
for _, op := range ops {
if err := op(path); err != nil {
return err
}
}
return nil
}
// WithMode sets the file mode on the directory or file at path
func WithMode(mode os.FileMode) PathOp {
return func(path Path) error {
return os.Chmod(path.Path(), mode)
}
}
func copyDirectory(source, dest string) error {
entries, err := ioutil.ReadDir(source)
if err != nil {

View File

@ -7,8 +7,6 @@ import (
"fmt"
"io"
"os/exec"
"path/filepath"
"runtime"
"strings"
"sync"
"time"
@ -18,8 +16,12 @@ type testingT interface {
Fatalf(string, ...interface{})
}
type helperT interface {
Helper()
}
// None is a token to inform Result.Assert that the output should be empty
const None string = "[NOTHING]"
const None = "[NOTHING]"
type lockedBuffer struct {
m sync.RWMutex
@ -51,17 +53,16 @@ type Result struct {
// Assert compares the Result against the Expected struct, and fails the test if
// any of the expectations are not met.
// TODO: deprecate and replace with assert.CompareFunc
func (r *Result) Assert(t testingT, exp Expected) *Result {
if ht, ok := t.(helperT); ok {
ht.Helper()
}
err := r.Compare(exp)
if err == nil {
return r
}
_, file, line, ok := runtime.Caller(1)
if ok {
t.Fatalf("at %s:%d - %s\n", filepath.Base(file), line, err.Error())
} else {
t.Fatalf("(no file/line info) - %s", err.Error())
}
t.Fatalf(err.Error() + "\n")
return nil
}

View File

@ -0,0 +1,27 @@
package format
import "fmt"
// Message accepts a msgAndArgs varargs and formats it using fmt.Sprintf
func Message(msgAndArgs ...interface{}) string {
switch len(msgAndArgs) {
case 0:
return ""
case 1:
return fmt.Sprintf("%v", msgAndArgs[0])
default:
return fmt.Sprintf(msgAndArgs[0].(string), msgAndArgs[1:]...)
}
}
// WithCustomMessage accepts one or two messages and formats them appropriately
func WithCustomMessage(source string, msgAndArgs ...interface{}) string {
custom := Message(msgAndArgs...)
switch {
case custom == "":
return source
case source == "":
return custom
}
return fmt.Sprintf("%s: %s", source, custom)
}

View File

@ -0,0 +1,109 @@
package source
import (
"bytes"
"go/ast"
"go/format"
"go/parser"
"go/token"
"runtime"
"github.com/pkg/errors"
)
const baseStackIndex = 1
// GetCondition returns the condition string by reading it from the file
// identified in the callstack. In golang 1.9 the line number changed from
// being the line where the statement ended to the line where the statement began.
func GetCondition(stackIndex int, argPos int) (string, error) {
_, filename, lineNum, ok := runtime.Caller(baseStackIndex + stackIndex)
if !ok {
return "", errors.New("failed to get caller info")
}
node, err := getNodeAtLine(filename, lineNum)
if err != nil {
return "", err
}
return getArgSourceFromAST(node, argPos)
}
func getNodeAtLine(filename string, lineNum int) (ast.Node, error) {
fileset := token.NewFileSet()
astFile, err := parser.ParseFile(fileset, filename, nil, parser.AllErrors)
if err != nil {
return nil, errors.Wrapf(err, "failed to parse source file: %s", filename)
}
node := scanToLine(fileset, astFile, lineNum)
if node == nil {
return nil, errors.Wrapf(err,
"failed to find an expression on line %d in %s", lineNum, filename)
}
return node, nil
}
func scanToLine(fileset *token.FileSet, node ast.Node, lineNum int) ast.Node {
v := &scanToLineVisitor{lineNum: lineNum, fileset: fileset}
ast.Walk(v, node)
return v.matchedNode
}
type scanToLineVisitor struct {
lineNum int
matchedNode ast.Node
fileset *token.FileSet
}
func (v *scanToLineVisitor) Visit(node ast.Node) ast.Visitor {
if node == nil || v.matchedNode != nil {
return nil
}
var position token.Position
switch {
case runtime.Version() < "go1.9":
position = v.fileset.Position(node.End())
default:
position = v.fileset.Position(node.Pos())
}
if position.Line == v.lineNum {
v.matchedNode = node
return nil
}
return v
}
func getArgSourceFromAST(node ast.Node, argPos int) (string, error) {
visitor := &callExprVisitor{}
ast.Walk(visitor, node)
if visitor.expr == nil {
return "", errors.Errorf("unexpected ast")
}
buf := new(bytes.Buffer)
err := format.Node(buf, token.NewFileSet(), visitor.expr.Args[argPos])
return buf.String(), err
}
type callExprVisitor struct {
expr *ast.CallExpr
}
func (v *callExprVisitor) Visit(node ast.Node) ast.Visitor {
switch typed := node.(type) {
case nil:
return nil
case *ast.IfStmt:
ast.Walk(v, typed.Cond)
case *ast.CallExpr:
v.expr = typed
}
if v.expr != nil {
return nil
}
return v
}

View File

@ -19,6 +19,10 @@ type LogT interface {
Logf(format string, args ...interface{})
}
type helperT interface {
Helper()
}
// Settings are used to configure the behaviour of WaitOn
type Settings struct {
// Timeout is the maximum time to wait for the condition. Defaults to 10s
@ -101,6 +105,9 @@ func Error(err error) Result {
// check returns a done Result. To fail a test and exit polling with an error
// return a error result.
func WaitOn(t TestingT, check func(t LogT) Result, pollOps ...SettingOp) {
if ht, ok := t.(helperT); ok {
ht.Helper()
}
config := defaultConfig()
for _, pollOp := range pollOps {
pollOp(config)

View File

@ -3,19 +3,14 @@
package skip
import (
"bytes"
"fmt"
"go/ast"
"go/format"
"go/parser"
"go/token"
"io/ioutil"
"path"
"reflect"
"runtime"
"strings"
"github.com/pkg/errors"
"github.com/gotestyourself/gotestyourself/internal/format"
"github.com/gotestyourself/gotestyourself/internal/source"
)
type skipT interface {
@ -23,14 +18,29 @@ type skipT interface {
Log(args ...interface{})
}
type helperT interface {
Helper()
}
// BoolOrCheckFunc can be a bool or func() bool, other types will panic
type BoolOrCheckFunc interface{}
// If skips the test if the check function returns true. The skip message will
// contain the name of the check function. Extra message text can be passed as a
// format string with args
func If(t skipT, check func() bool, msgAndArgs ...interface{}) {
if check() {
t.Skip(formatWithCustomMessage(
getFunctionName(check),
formatMessage(msgAndArgs...)))
func If(t skipT, condition BoolOrCheckFunc, msgAndArgs ...interface{}) {
if ht, ok := t.(helperT); ok {
ht.Helper()
}
switch check := condition.(type) {
case bool:
ifCondition(t, check, msgAndArgs...)
case func() bool:
if check() {
t.Skip(format.WithCustomMessage(getFunctionName(check), msgAndArgs...))
}
default:
panic(fmt.Sprintf("invalid type for condition arg: %T", check))
}
}
@ -42,92 +52,30 @@ func getFunctionName(function func() bool) string {
// IfCondition skips the test if the condition is true. The skip message will
// contain the source of the expression passed as the condition. Extra message
// text can be passed as a format string with args.
//
// Deprecated: Use If() which now accepts bool arguments
func IfCondition(t skipT, condition bool, msgAndArgs ...interface{}) {
if ht, ok := t.(helperT); ok {
ht.Helper()
}
ifCondition(t, condition, msgAndArgs...)
}
func ifCondition(t skipT, condition bool, msgAndArgs ...interface{}) {
if ht, ok := t.(helperT); ok {
ht.Helper()
}
if !condition {
return
}
source, err := getConditionSource()
const (
stackIndex = 2
argPos = 1
)
source, err := source.GetCondition(stackIndex, argPos)
if err != nil {
t.Log(err.Error())
t.Skip(formatMessage(msgAndArgs...))
t.Skip(format.Message(msgAndArgs...))
}
t.Skip(formatWithCustomMessage(source, formatMessage(msgAndArgs...)))
}
func getConditionSource() (string, error) {
const callstackIndex = 3
lines, err := getSourceLine(callstackIndex)
if err != nil {
return "", err
}
for i := range lines {
source := strings.Join(lines[len(lines)-i-1:], "\n")
node, err := parser.ParseExpr(source)
if err == nil {
return getConditionArgFromAST(node)
}
}
return "", errors.Wrapf(err, "failed to parse source")
}
// maxContextLines is the maximum number of lines to scan for a complete
// skip.If() statement
const maxContextLines = 10
// getSourceLines returns the source line which called skip.If() along with a
// few preceding lines. To properly parse the AST a complete statement is
// required, and that statement may be split across multiple lines, so include
// up to maxContextLines.
func getSourceLine(stackIndex int) ([]string, error) {
_, filename, line, ok := runtime.Caller(stackIndex)
if !ok {
return nil, errors.New("failed to get caller info")
}
raw, err := ioutil.ReadFile(filename)
if err != nil {
return nil, errors.Wrapf(err, "failed to read source file: %s", filename)
}
lines := strings.Split(string(raw), "\n")
if len(lines) < line {
return nil, errors.Errorf("file %s does not have line %d", filename, line)
}
firstLine := line - maxContextLines
if firstLine < 0 {
firstLine = 0
}
return lines[firstLine:line], nil
}
func getConditionArgFromAST(node ast.Expr) (string, error) {
switch expr := node.(type) {
case *ast.CallExpr:
buf := new(bytes.Buffer)
err := format.Node(buf, token.NewFileSet(), expr.Args[1])
return buf.String(), err
}
return "", errors.New("unexpected ast")
}
func formatMessage(msgAndArgs ...interface{}) string {
switch len(msgAndArgs) {
case 0:
return ""
case 1:
return msgAndArgs[0].(string)
default:
return fmt.Sprintf(msgAndArgs[0].(string), msgAndArgs[1:]...)
}
}
func formatWithCustomMessage(source, custom string) string {
switch {
case custom == "":
return source
case source == "":
return custom
}
return fmt.Sprintf("%s: %s", source, custom)
t.Skip(format.WithCustomMessage(source, msgAndArgs...))
}

View File

@ -22,7 +22,7 @@ type lcowParser struct {
windowsParser
}
func (p *lcowParser) validateMountConfig(mnt *mount.Mount) error {
func (p *lcowParser) ValidateMountConfig(mnt *mount.Mount) error {
return p.validateMountConfigReg(mnt, rxLCOWDestination, lcowSpecificValidators)
}

View File

@ -40,7 +40,7 @@ func linuxValidateAbsolute(p string) error {
}
return fmt.Errorf("invalid mount path: '%s' mount path must be absolute", p)
}
func (p *linuxParser) validateMountConfig(mnt *mount.Mount) error {
func (p *linuxParser) ValidateMountConfig(mnt *mount.Mount) error {
// there was something looking like a bug in existing codebase:
// - validateMountConfig on linux was called with options skipping bind source existence when calling ParseMountRaw
// - but not when calling ParseMountSpec directly... nor when the unit test called it directly

View File

@ -26,8 +26,7 @@ type Parser interface {
IsBackwardCompatible(m *MountPoint) bool
HasResource(m *MountPoint, absPath string) bool
ValidateTmpfsMountDestination(dest string) error
validateMountConfig(mt *mount.Mount) error
ValidateMountConfig(mt *mount.Mount) error
}
// NewParser creates a parser for a given container OS, depending on the current host OS (linux on a windows host will resolve to an lcowParser)

View File

@ -31,13 +31,9 @@ func TestValidateMount(t *testing.T) {
{mount.Mount{Type: mount.TypeBind, Source: testDir, Target: testDestinationPath}, nil},
{mount.Mount{Type: "invalid", Target: testDestinationPath}, errors.New("mount type unknown")},
{mount.Mount{Type: mount.TypeBind, Source: testSourcePath, Target: testDestinationPath}, errBindNotExist},
}
if runtime.GOOS == "windows" {
cases = append(cases, struct {
input mount.Mount
expected error
}{mount.Mount{Type: mount.TypeBind, Source: testSourcePath, Target: testDestinationPath}, errBindNotExist}) // bind source existance is not checked on linux
}
lcowCases := []struct {
input mount.Mount
expected error
@ -54,7 +50,7 @@ func TestValidateMount(t *testing.T) {
}
parser := NewParser(runtime.GOOS)
for i, x := range cases {
err := parser.validateMountConfig(&x.input)
err := parser.ValidateMountConfig(&x.input)
if err == nil && x.expected == nil {
continue
}
@ -65,7 +61,7 @@ func TestValidateMount(t *testing.T) {
if runtime.GOOS == "windows" {
parser = &lcowParser{}
for i, x := range lcowCases {
err := parser.validateMountConfig(&x.input)
err := parser.ValidateMountConfig(&x.input)
if err == nil && x.expected == nil {
continue
}

View File

@ -189,7 +189,7 @@ func (p *windowsParser) ValidateVolumeName(name string) error {
}
return nil
}
func (p *windowsParser) validateMountConfig(mnt *mount.Mount) error {
func (p *windowsParser) ValidateMountConfig(mnt *mount.Mount) error {
return p.validateMountConfigReg(mnt, rxDestination, windowsSpecificValidators)
}