Docker daemon has a 16K buffer for log messages. If a message length
exceeds 16K, it should be split by the logger and merged at the
endpoint.
This change adds `PartialLogMetaData` struct for enhanced partial support
- LastPartial (bool) : indicates if this is the last of all partials.
- ID (string) : unique 32 bit ID. ID is same across all partials.
- Ordinal (int starts at 1) : indicates the position of msg in the series of partials.
Also, the timestamps across partials in the same.
Signed-off-by: Anusha Ragunathan <anusha.ragunathan@docker.com>
Upstream-commit: 0b4b0a7b5d5de8cb575b666312fceaa2cd58e658
Component: engine
It does not make any sense to vary this based on whether the
rootfs is read only. We removed all the other mount dependencies
on read-only eg see #35344.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
Upstream-commit: a729853bc712910574a7417f67764ec8c523928b
Component: engine
We have seen a panic when re-joining a node to a swarm cluster. The
cause of the issue is unknown, so we just need to add a test for nil
objects and log when we get the condition. Hopefully this can prevent
the crash and we can recover the config at a later time.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Upstream-commit: 454128c6e82cded211c1412e3eb350b1f7533ee2
Component: engine
commit 617c352e9225 "Don't create devices if in a user namespace"
introduced check, which meant to skip mknod operation when run
in user namespace, but instread skipped FIFO and socket files
copy.
Signed-off-by: Maxim Ivanov <ivanov.maxim@gmail.com>
Upstream-commit: 6f084f292932c464a30b56edb9edbe238bdcf0aa
Component: engine
Makes sure that if the user cancels a request that the daemon stops
trying to traverse a directory.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Upstream-commit: 9d46c4c138d7b3f7778c13fe84857712bd6c97a9
Component: engine
This call was added as part of commit a042e5a20 and at the time was
useful. sandbox.DisableService() basically calls
endpoint.deleteServiceInfoFromCluster() for every endpoint in the
sandbox. However, with the libnetwork change, endpoint.sbLeave()
invokes endpoint.deleteServiceInfoFromCluster(). The releaseNetwork()
call invokes sandbox.Delete() immediately after
sandbox.DisableService(). The sandbox.Delete() in turn ultimately
invokes endpoint.sbLeave() for every endpoint in the sandbox which thus
removes the endpoint's load balancing entry via
endpoint.deleteServiceInfoFromCluster(). So the call to
sandbox.DisableService() is now redundant.
It is noteworthy that, while redundant, the presence of the call would
not cause errors. It would just be sub-optimal. The DisableService()
call would cause libnetwork to down-weight the load balancing entries
while the call to sandbox.Delete() would cause it to remove the entries
immediately afterwards. Aside from the wasted computation, the extra
call would also propagate an extra state change in the networkDB gossip
messages. So, overall, it is much better to just avoid the extra
overhead.
Signed-off-by: Chris Telfer <ctelfer@docker.com>
Upstream-commit: c27417aa7de46daa415600b39fc8a9c411c8c493
Component: engine
If container will run as non root user, drop permitted, effective caps early
Upstream-commit: b67c1e078c7eeb20199dce301e95fa8999c98109
Component: engine
As soon as the initial executable in the container is executed as a non root user,
permitted and effective capabilities are dropped. Drop them earlier than this, so
that they are dropped before executing the file. The main effect of this is that
if `CAP_DAC_OVERRIDE` is set (the default) the user will not be able to execute
files they do not have permission to execute, which previously they could.
The old behaviour was somewhat surprising and the new one is definitely correct,
but it is not in any meaningful way exploitable, and I do not think it is
necessary to backport this fix. It is unlikely to have any negative effects as
almost all executables have world execute permission anyway.
Use the bounding set not the effective set as the canonical set of capabilities, as
effective will now vary.
Signed-off-by: Justin Cormack <justin.cormack@docker.com>
Upstream-commit: 15ff09395c001bcb0f284461abbc404a1d8bab4d
Component: engine
Commit fd0e24b7189374e0fe7c55b6d26ee916d3ee1655 changed
the stats collection loop to use a `sleep()` instead
of `time.Tick()` in the for-loop.
This change caused a regression in situations where
no stats are being collected, or an error is hit
in the loop (in which case the loop would `continue`,
and the `sleep()` is not hit).
This patch puts the sleep at the start of the loop
to guarantee it's always hit.
This will delay the sampling, which is similar to the
behavior before fd0e24b7189374e0fe7c55b6d26ee916d3ee1655.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Upstream-commit: 481b8e54b45955e40075f49a9af321afce439320
Component: engine
This PR adds support for compressibility of log file.
I added a new option conpression for the jsonfile log driver,
this option allows the user to specify compression algorithm to
compress the log files. By default, the log files will be
not compressed. At present, only support 'gzip'.
Signed-off-by: Yanqiang Miao <miao.yanqiang@zte.com.cn>
'docker logs' can read from compressed files
Signed-off-by: Yanqiang Miao <miao.yanqiang@zte.com.cn>
Add Metadata to the gzip header, optmize 'readlog'
Signed-off-by: Yanqiang Miao <miao.yanqiang@zte.com.cn>
Upstream-commit: f69f09f44ce9fedbc9d70f11980c1fc8d7f77cec
Component: engine
Commit 7a7357dae1bccc ("LCOW: Implemented support for docker cp + build")
changed `container.BaseFS` from being a string (that could be empty but
can't lead to nil pointer dereference) to containerfs.ContainerFS,
which could be be `nil` and so nil dereference is at least theoretically
possible, which leads to panic (i.e. engine crashes).
Such a panic can be avoided by carefully analysing the source code in all
the places that dereference a variable, to make the variable can't be nil.
Practically, this analisys are impossible as code is constantly
evolving.
Still, we need to avoid panics and crashes. A good way to do so is to
explicitly check that a variable is non-nil, returning an error
otherwise. Even in case such a check looks absolutely redundant,
further changes to the code might make it useful, and having an
extra check is not a big price to pay to avoid a panic.
This commit adds such checks for all the places where it is not obvious
that container.BaseFS is not nil (which in this case means we do not
call daemon.Mount() a few lines earlier).
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Upstream-commit: d6ea46cedaca0098c15843c5254a337d087f5cd6
Component: engine
In case ContainerExport() is called for an unmounted container, it leads
to a daemon panic as container.BaseFS, which is dereferenced here, is
nil.
To fix, do not rely on container.BaseFS; use the one returned from
rwlayer.Mount().
Fixes: 7a7357dae1bccc ("LCOW: Implemented support for docker cp + build")
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Upstream-commit: 81f6307eda44ab3a91de6e29304810a976161d74
Component: engine
In info, we only need the number of images, but `CountImages` was
getting the whole map of images and then grabbing the length from that.
This causes a lot of unnecessary CPU usage and memory allocations, which
increases with O(n) on the number of images.
Signed-off-by: Brian Goff <cpuguy83@gmail.com>
Upstream-commit: f6a7763b6f3256bed9a7352021745189d0ca8dc9
Component: engine
Ingress networks will no longer automatically remove their
load-balancing endpoint (and sandbox) automatically when the network is
otherwise upopulated. This is to prevent automatic removal of the
ingress networks when all the containers leave them. Therefore
explicit removal of an ingress network also requires explicit removal
of its load-balancing endpoint.
Signed-off-by: Chris Telfer <ctelfer@docker.com>
Upstream-commit: 3da4ebf355d3494d1403b2878a1ae6958b2724e9
Component: engine
It has been pointed out that if --read-only flag is given, /dev/shm
also becomes read-only in case of --ipc private.
This happens because in this case the mount comes from OCI spec
(since commit 7120976d74195), and is a regression caused by that
commit.
The meaning of --read-only flag is to only have a "main" container
filesystem read-only, not the auxiliary stuff (that includes /dev/shm,
other mounts and volumes, --tmpfs, /proc, /dev and so on).
So, let's make sure /dev/shm that comes from OCI spec is not made
read-only.
Fixes: 7120976d74195 ("Implement none, private, and shareable ipc modes")
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Upstream-commit: cad74056c09f6276b0f4a996a1511553177cd3d7
Component: engine
The test case checks that in case of IpcMode: private and
ReadonlyRootfs: true (as in "docker run --ipc private --read-only")
the resulting /dev/shm mount is NOT made read-only.
Signed-off-by: Kir Kolyshkin <kolyshkin@gmail.com>
Upstream-commit: 33dd562e3acff71ee18a2543d14fcbecf9bf0e62
Component: engine
To avoid noise in sampling CPU usage metrics, we now sample the system
usage closer to the actual response from the underlying runtime. Because
the response from the runtime may be delayed, this makes the sampling
more resilient in loaded conditions. In addition to this, we also
replace the tick with a sleep to avoid situations where ticks can backup
under loaded conditions.
The trade off here is slightly more load reading the system CPU usage
for each container. There may be an optimization required for large
amounts of containers but the cost is on the order of 15 ms per 1000
containers. If this becomes a problem, we can time slot the sampling,
but the complexity may not be worth it unless we can test further.
Unfortunately, there aren't really any good tests for this condition.
Triggering this behavior is highly system dependent. As a matter of
course, we should qualify the fix with the users that are affected.
Signed-off-by: Stephen J Day <stephen.day@docker.com>
Upstream-commit: fd0e24b7189374e0fe7c55b6d26ee916d3ee1655
Component: engine
With the inclusion of PR 30897, creating service for host network
fails in 18.02. Modified IsPreDefinedNetwork check and return
NetworkNameError instead of errdefs.Forbidden to address this issue
Signed-off-by: selansen <elango.siva@docker.com>
Upstream-commit: 7cf8b20762cc9491f52ff3f3d94c880378183696
Component: engine
