Antonio Murdaca
5965f0216f
It has observed defunct containerd processes accumulating over
time while dockerd was permanently failing to restart containerd.
Due to a bug in the runContainerdDaemon() function, dockerd does not clean up
its child process if containerd already exits very soon after the (re)start.
The reproducer and analysis below comes from docker 1.12.x but bug
still applies on latest master.
- from libcontainerd/remote_linux.go:
329 func (r *remote) runContainerdDaemon() error {
:
: // start the containerd child process
:
403 if err := cmd.Start(); err != nil {
404 return err
405 }
:
: // If containerd exits very soon after (re)start, it is
possible
: // that containerd is already in defunct state at the time
when
: // dockerd gets here. The setOOMScore() function tries to
write
: // to /proc/PID_OF_CONTAINERD/oom_score_adj. However, this
fails
: // with errno EINVAL because containerd is defunct. Please see
: // snippets of kernel source code and further explanation
below.
:
407 if err := setOOMScore(cmd.Process.Pid, r.oomScore); err != nil
{
408 utils.KillProcess(cmd.Process.Pid)
:
: // Due to the error from write() we return here. As
the
: // goroutine that would clean up the child has not
been
: // started yet, containerd remains in the defunct
state
: // and never gets reaped.
:
409 return err
410 }
:
417 go func() {
418 cmd.Wait()
419 close(r.daemonWaitCh)
420 }() // Reap our child when needed
:
423 }
This is the kernel function that gets invoked when dockerd tries to
write
to /proc/PID_OF_CONTAINERD/oom_score_adj.
- from fs/proc/base.c:
1197 static ssize_t oom_score_adj_write(struct file *file, ...
1198 size_t count, loff_t
*ppos)
1199 {
:
1223 task = get_proc_task(file_inode(file));
:
: // The defunct containerd process does not have a virtual
: // address space anymore, i.e. task->mm is NULL. Thus the
: // following code returns errno EINVAL to dockerd.
:
1230 if (!task->mm) {
1231 err = -EINVAL;
1232 goto err_task_lock;
1233 }
:
1253 err_task_lock:
:
1257 return err < 0 ? err : count;
1258 }
The purpose of the following program is to demonstrate the behavior of
the oom_score_adj_write() function in connection with a defunct process.
$ cat defunct_test.c
\#include <unistd.h>
main()
{
pid_t pid = fork();
if (pid == 0)
// child
_exit(0);
// parent
pause();
}
$ make defunct_test
cc defunct_test.c -o defunct_test
$ ./defunct_test &
[1] 3142
$ ps -f | grep defunct_test | grep -v grep
root 3142 2956 0 13:04 pts/0 00:00:00 ./defunct_test
root 3143 3142 0 13:04 pts/0 00:00:00 [defunct_test] <defunct>
$ echo "ps 3143" | crash -s
PID PPID CPU TASK ST %MEM VSZ RSS COMM
3143 3142 2 ffff880035def300 ZO 0.0 0 0
defunct_test
$ echo "px ((struct task_struct *)0xffff880035def300)->mm" | crash -s
$1 = (struct mm_struct *) 0x0
^^^ task->mm is NULL
$ cat /proc/3143/oom_score_adj
0
$ echo 0 > /proc/3143/oom_score_adj
-bash: echo: write error: Invalid argument"
---
This patch fixes the above issue by making sure we start the reaper
goroutine as soon as possible.
Signed-off-by: Antonio Murdaca <runcom@redhat.com>
(cherry picked from commit 27087eacbf96e6ef9d48a6d3dc89c7c1cff155b4)
Signed-off-by: Eli Uriegas <eli.uriegas@docker.com>
Signed-off-by: Eli Uriegas <eli.uriegas@docker.com>
Docker CE
This repository hosts open source components of Docker Community Edition
(CE) products. The master branch serves to unify the upstream components
on a regular basis. Long-lived release branches host the code that goes
into a product version for the lifetime of the product.
This repository is solely maintained by Docker, Inc.
Unifying upstream sources
The master branch is a combination of components adapted from
different upstream git repos into a unified directory structure using the
moby-components
tool.
You can view the upstream git repos in the components.conf file. Each component is isolated into its own directory under the components directory.
The tool will import each component git history within the appropriate path.
For example, this shows a commit
is imported into the component engine from
moby/moby@a27b4b8
into the components/engine directory.
commit 5c70746915d4589a692cbe50a43cf619ed0b7152
Author: Andrea Luzzardi <aluzzardi@gmail.com>
Date: Sat Jan 19 00:13:39 2013
Initial commit
Upstream-commit: a27b4b8cb8e838d03a99b6d2b30f76bdaf2f9e5d
Component: engine
components/engine/container.go | 203 ++++++++++++++++++++++++++++...
components/engine/container_test.go | 186 ++++++++++++++++++++++++++++...
components/engine/docker.go | 112 ++++++++++++++++++++++++++++...
components/engine/docker_test.go | 175 ++++++++++++++++++++++++++++...
components/engine/filesystem.go | 52 ++++++++++++++++++++++++++++...
components/engine/filesystem_test.go | 35 +++++++++++++++++++++++++++
components/engine/lxc_template.go | 94 ++++++++++++++++++++++++++++...
components/engine/state.go | 48 ++++++++++++++++++++++++++++...
components/engine/utils.go | 115 ++++++++++++++++++++++++++++...
components/engine/utils_test.go | 126 ++++++++++++++++++++++++++++...
10 files changed, 1146 insertions(+)
Updates to master branch
Main development of new features should be directed towards the upstream
git repos. The master branch of this repo will periodically pull in new
changes from upstream to provide a point for integration.
Branching for release
When a release is started for Docker CE, a new branch will be created
from master. Branch names will be YY.MM to represent the time-based
release version of the product, e.g. 17.06.
Adding fixes to release branch
Note: every commit of a fix should affect files only within one component directory.
Fix available upstream
A PR cherry-picking the necessary commits should be created against the release branch. If the the cherry-pick cannot be applied cleanly, the logic of the fix should be ported manually.
No fix yet
First create the PR with the fix for the release branch. Once the fix has been merged, be sure to port the fix to the respective upstream git repo.
Release tags
There will be a git tag for each release candidate (RC) and general availablilty (GA) release. The tag will only point to commits on release branches.