Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Flake in node e2e: "Container Runtime Conformance Test container conformance blackbox test when running a container that terminates [It] it should be possible to delete [Conformance] " #24937

Closed
smarterclayton opened this issue Apr 28, 2016 · 5 comments
Closed

Flake in node e2e: "Container Runtime Conformance Test container conformance blackbox test when running a container that terminates [It] it should be possible to delete [Conformance] " #24937

smarterclayton opened this issue Apr 28, 2016 · 5 comments
Assignees
@liangchenye
Labels
kind/flake Categorizes issue or PR as related to a flaky test. sig/node Categorizes an issue or PR as relevant to SIG Node.

Comments

@smarterclayton
Copy link
Contributor 

Summarizing 1 Failure:

[Fail] Container Runtime Conformance Test container conformance blackbox test when running a container that terminates [It] it should be possible to delete [Conformance] 
/var/lib/jenkins/workspace/node-pull-build-e2e-test@2/go/src/k8s.io/kubernetes/test/e2e_node/runtime_conformance_test.go:76

Ran 22 of 22 Specs in 1205.295 seconds
@smarterclayton smarterclayton added the kind/flake Categorizes issue or PR as related to a flaky test. label Apr 28, 2016
@smarterclayton smarterclayton mentioned this issue Apr 28, 2016
Merged
@mikedanese mikedanese added the sig/node Categorizes an issue or PR as relevant to SIG Node. label Apr 28, 2016
@mikedanese
Copy link
Member

Again here:

http://kubekins.dls.corp.google.com/view/Critical%20Builds/job/kubelet-gce-e2e-ci/2233

07:46:01 • Failure [0.085 seconds]
07:46:01 Kubelet Container Manager
07:46:01 /var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/test/e2e_node/container_manager_test.go:107
07:46:01   oom score adjusting
07:46:01   /var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/test/e2e_node/container_manager_test.go:105
07:46:01     when scheduling a busybox command that always fails in a pod
07:46:01     /var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/test/e2e_node/container_manager_test.go:104
07:46:01       it should be possible to delete [It]
07:46:01       /var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/test/e2e_node/container_manager_test.go:91
07:46:01 
07:46:01       Error deleting Pod pods "bin-false" not found
07:46:01       Expected
07:46:01           <*errors.StatusError | 0xc82017dc80>: {
07:46:01               ErrStatus: {
07:46:01                   TypeMeta: {Kind: "Status", APIVersion: "v1"},
07:46:01                   ListMeta: {SelfLink: "", ResourceVersion: ""},
07:46:01                   Status: "Failure",
07:46:01                   Message: "pods \"bin-false\" not found",
07:46:01                   Reason: "NotFound",
07:46:01                   Details: {Name: "bin-false", Group: "", Kind: "pods", Causes: nil, RetryAfterSeconds: 0},
07:46:01                   Code: 404,
07:46:01               },
07:46:01           }
07:46:01       to be nil
07:46:01 
07:46:01       /var/lib/jenkins/workspace/kubelet-gce-e2e-ci/go/src/k8s.io/kubernetes/test/e2e_node/container_manager_test.go:90

cc @pwittrock

@Random-Liu
Copy link
Member

@liangchenye

@liangchenye
Copy link
Contributor

liangchenye commented Apr 29, 2016 •
edited
Loading

Not sure how does this error come out. The 'Delete' test case only does two things: create a 'bin-false' pod and delete it immediately after that. Since there is no error comes out in pod creation, it means before the 'Delete' test case, there is no 'bin-false' pod (neither never been created nor has already been deleted). But this issue shows, after the creation, it is removed by another test case ('Terminate Reason' test case).
So, I guess it should be something like this: in the 'Delete' test, it creates a pod at B point
and delete the pod at D point. I made a test, if it creates a pod at A point, there will be a creation error.

'Terminate Reason' test case: 
     create -- check-- delete --  waiting --- real deleted 
                                 |         |                 |
'Delete' test case:              A         B                  D

So is this a bug of k8s apiserver? The 'real deleted' status seems not synchronized.

Anyway I'll choose different name in different cases in the later development. (and different namespace in different test file)

@smarterclayton smarterclayton mentioned this issue Apr 29, 2016
Closed
@jsafrane
Copy link
Member

This looks similar to the first comment, different to @mikedanese's one: https://storage.cloud.google.com/kubernetes-jenkins/pr-logs/pull/24861/node-pull-build-e2e-test/3451/build-log.txt

 Failure [0.046 seconds]
Container Runtime Conformance Test
/var/lib/jenkins/workspace/node-pull-build-e2e-test@3/go/src/k8s.io/kubernetes/test/e2e_node/runtime_conformance_test.go:139
  container conformance blackbox test
  /var/lib/jenkins/workspace/node-pull-build-e2e-test@3/go/src/k8s.io/kubernetes/test/e2e_node/runtime_conformance_test.go:138
    when running a container that terminates
    /var/lib/jenkins/workspace/node-pull-build-e2e-test@3/go/src/k8s.io/kubernetes/test/e2e_node/runtime_conformance_test.go:91
      it should be possible to delete [Conformance] [It]
      /var/lib/jenkins/workspace/node-pull-build-e2e-test@3/go/src/k8s.io/kubernetes/test/e2e_node/runtime_conformance_test.go:81

      Expected error:
          <*errors.StatusError | 0xc8205d4000>: {
              ErrStatus: {
                  TypeMeta: {Kind: "Status", APIVersion: "v1"},
                  ListMeta: {SelfLink: "", ResourceVersion: ""},
                  Status: "Failure",
                  Message: "pods \"busybox\" not found",
                  Reason: "NotFound",
                  Details: {Name: "busybox", Group: "", Kind: "pods", Causes: nil, RetryAfterSeconds: 0},
                  Code: 404,
              },
          }
          pods "busybox" not found
      not to have occurred

@jsafrane jsafrane mentioned this issue Apr 29, 2016
Closed
@liangchenye
Copy link
Contributor

will be fixed by #24191

@freehan freehan mentioned this issue May 2, 2016
Merged
@Random-Liu Random-Liu mentioned this issue May 3, 2016
Merged
@liangchenye liangchenye mentioned this issue May 4, 2016
Merged
This was referenced May 4, 2016
Closed
Merged
k8s-github-robot pushed a commit that referenced this issue May 6, 2016
Merge pull request #25224 from Random-Liu/delete-pod-with-uid
4a00266 
Automatic merge from submit-queue

Delete pod with uid as precondition.

Addressed #25169 (comment).

Fix #25169 
Fix #24937

This PR change status manager to delete pods with uid as a precondition, so that kubelet won't delete pods with different uid but the same name and namespace accidentally.

/cc @yujuhong
k8s-github-robot pushed a commit that referenced this issue May 6, 2016
Merge pull request #25121 from liangchenye/flakepodnotfound
2ebd771 
Automatic merge from submit-queue

fix #24937:  flake pod not found

#24937
Split from #24191
Signed-off-by: liang chenye <liangchenye@huawei.com>
derekwaynecarr pushed a commit to derekwaynecarr/kubernetes that referenced this issue May 10, 2016
@liangchenye @derekwaynecarr
fix kubernetes#24937: flake pod not found
b754dd6 
Signed-off-by: liang chenye <liangchenye@huawei.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@liangchenye liangchenye

Labels
kind/flake Categorizes issue or PR as related to a flaky test. sig/node Categorizes an issue or PR as relevant to SIG Node.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@smarterclayton @jsafrane @mikedanese @Random-Liu @liangchenye