Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Docker reports incorrect exit code for containers if the docker service was restarted #41516

Closed
yujuhong opened this issue Feb 15, 2017 · 14 comments
Closed

Docker reports incorrect exit code for containers if the docker service was restarted #41516

yujuhong opened this issue Feb 15, 2017 · 14 comments
Labels
area/docker lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. sig/node Categorizes an issue or PR as relevant to SIG Node.

Comments

@yujuhong
Copy link
Contributor

To reproduce this, start several containers and restart docker with sudo systemctl restart docker or sudo service docker restart, depending on the OS distro.
After docker restarts properly, run docker ps -a and see containers were terminated with exit code 0.

$ sudo docker ps -a
CONTAINER ID        IMAGE               COMMAND                  CREATED              STATUS                      PORTS               NAMES
5039dd309f12        busybox             "/bin/sh -c 'sleep 10"   About a minute ago   Exited (0) 22 seconds ago                       compassionate_mcclintock
83322e7eedcb        busybox             "/bin/sh -c 'sleep 10"   About a minute ago   Exited (0) 22 seconds ago                       distracted_kirch
fd96e1c425df        busybox             "/bin/sh -c 'sleep 10"   About a minute ago   Exited (0) 22 seconds ago                       elegant_goldstine
a255470685c0        busybox             "/bin/sh -c 'sleep 10"   About a minute ago   Exited (0) 22 seconds ago                       serene_bhaskara
c726fe02b915        busybox             "/bin/sh -c 'sleep 10"   About a minute ago   Exited (0) 22 seconds ago                       hungry_boyd
fabf1984c604        busybox             "/bin/sh -c 'sleep 10"   About a minute ago   Exited (0) 22 seconds ago                       romantic_poitras

I tested this on three docker versions: 1.10.3, 1.11.2, and 1.12.6. Only docker 1.10.3 correctly reported the 137 exit code.
This issue was claimed to be fixed in moby/moby#26143
@yujuhong yujuhong added sig/node Categorizes an issue or PR as relevant to SIG Node. area/docker labels Feb 15, 2017
@yujuhong
Copy link
Contributor Author

/cc @kubernetes/sig-node-bugs

@yujuhong
Copy link
Contributor Author

Reported the issue to docker: moby/moby#31262

@resouer
Copy link
Contributor

resouer commented Feb 23, 2017 •
edited
Loading

[Update] I tested on docker 1.12.1, 1.12.4, 1.12.5 and all saw the same problem, while this seems already fixed in docker 1.13.0 & 1.13.1:

docker ps -a
CONTAINER ID        IMAGE               COMMAND             CREATED             STATUS                       PORTS               NAMES
4c262cd2e3e4        busybox             "sleep 60000"       30 seconds ago      Exited (137) 6 seconds ago                       modest_bardeen
1b0654cdc36b        busybox             "sleep 60000"       32 seconds ago      Exited (137) 6 seconds ago                       gracious_brattain

Server:
 Version:      1.13.1

Of course it is still a issue for 1.12.x.

@yujuhong
Copy link
Contributor Author

@mrunalp, this is the issue I talked to you last time. Have you seen the same issue in openshift?

/cc @derekwaynecarr

@yujuhong yujuhong mentioned this issue Mar 2, 2017
Closed
@mrunalp
Copy link
Contributor

mrunalp commented Mar 2, 2017

@yujuhong I don't see this issue on Fedora 25 with our docker 1.12.6

@yujuhong
Copy link
Contributor Author

yujuhong commented Mar 2, 2017

@yujuhong I don't see this issue on Fedora 25 with our docker 1.12.6

Hmm..Where can I find the list of patches for your docker?

@mrunalp
Copy link
Contributor

mrunalp commented Mar 3, 2017

https://github.com/projectatomic/docker/tree/docker-1.12.6

@mrunalp
Copy link
Contributor

mrunalp commented Mar 3, 2017

We also have similar trees for containerd and runc under projectatomic.

@ulzha ulzha mentioned this issue Mar 9, 2017
Merged
ulzha added a commit to spotify/styx that referenced this issue Mar 9, 2017
@ulzha
Get container exit code reliably
09b6c77 
This is a kludge to work around a Docker [exit code bug](kubernetes/kubernetes#41516).

Now, when docker_termination_logging is set to "true" for the scheduled workflow at submission time, a message in the termination log will be expected for determination of the container exit status.
@ulzha ulzha mentioned this issue Mar 9, 2017
Merged
ulzha added a commit to spotify/styx that referenced this issue Mar 9, 2017
@ulzha
Get container exit code reliably
a812c50 
This is a kludge to work around a Docker [exit code bug](kubernetes/kubernetes#41516).

Now, when docker_termination_logging is set to "true" for the scheduled workflow at submission time, a message in the termination log will be expected for determination of the container exit status.
ulzha added a commit to spotify/styx that referenced this issue Mar 10, 2017
@ulzha
Get container exit code reliably
1c912c0 
This is a kludge to work around a Docker [exit code bug](kubernetes/kubernetes#41516).

Now, when docker_termination_logging is set to "true" for the scheduled workflow at submission time, a message in the termination log will be expected for determination of the container exit status.
ulzha added a commit to spotify/styx that referenced this issue Mar 10, 2017
@ulzha
Get container exit code reliably
541d055 
This is a kludge to work around a Docker [exit code bug](kubernetes/kubernetes#41516).

Now, when docker_termination_logging is set to "true" for the scheduled workflow at submission time, a message in the termination log will be expected for determination of the container exit status.
@Random-Liu
Copy link
Member

FYI, I could still reproduce this in docker 17.03.1-ce.

moby/moby#31262 (comment)

@Random-Liu
Copy link
Member

Random-Liu commented Apr 5, 2017 •
edited
Loading

Updated that moby/moby#31262 (comment) is a red herring. The issue is actually fixed in newer Docker.

@warmchang
Copy link
Contributor

👍

@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Mar 14, 2018
@fejta-bot
Copy link

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Apr 13, 2018
@fejta-bot
Copy link

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/docker lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. sig/node Categorizes an issue or PR as relevant to SIG Node.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
7 participants
@mrunalp @resouer @Random-Liu @yujuhong @warmchang @k8s-ci-robot @fejta-bot