Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

hack/test-integration.sh is flaky #4500

Closed
j3ffml opened this issue Feb 17, 2015 · 4 comments
Closed

hack/test-integration.sh is flaky #4500

j3ffml opened this issue Feb 17, 2015 · 4 comments
Assignees
@dchen1107
Labels
area/build-release area/test area/test-infra priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.

Comments

@j3ffml
Copy link
Contributor

j3ffml commented Feb 17, 2015

A significant number of builds fail with the error

F0217 20:14:13.799188   16880 integration.go:299] FAILED: pods never started running timed out waiting for the condition

If you look at shippable history, this is the leading cause of flakes (build passes on 1.4, fails on 1.3 or vice versa).
@j3ffml j3ffml added this to the Test Issue Fix-it milestone Feb 17, 2015
@goltermann goltermann added the priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. label Feb 18, 2015
@dchen1107 dchen1107 self-assigned this Feb 18, 2015
@dchen1107
Copy link
Member

I am going to look into this one.

dchen1107 added a commit to dchen1107/kubernetes-1 that referenced this issue Feb 18, 2015
@dchen1107
Debugging message for debugging kubernetes#4500
34df171
yujuhong added a commit that referenced this issue Feb 18, 2015
@yujuhong
Merge pull request #4547 from dchen1107/podstatus
9e291fb 
Debugging message for debugging #4500
@yujuhong
Copy link
Contributor

I kept getting emails about shippable failures...

Below is the snippet of the log that includes @dchen1107's new log messages:

I0219 01:00:35.995225   17113 integration.go:242] Check whether pod nginx-controller-133ol.test exists on node ""
I0219 01:00:35.995299   17113 integration.go:244] Pod nginx-controller-133ol.test is not bound to a host yet
F0219 01:00:35.995316   17113 integration.go:301] FAILED: pods never started running timed out waiting for the condition

The host field of the pod was not assigned and the check failed after 30 seconds of polling. This means either the scheduler didn't schedule the pod or it couldn't schedule the pods due to, say, conflicts.

The pod in question was from api/examples/controller.json

  {
    "id": "nginx-controller",
    "apiVersion": "v1beta1",
    "kind": "ReplicationController",
    "desiredState": {
      "replicas": 2,
      "replicaSelector": {"name": "nginx"},
      "podTemplate": {
        "desiredState": {
           "manifest": {
             "version": "v1beta1",
             "id": "nginx-controller",
             "containers": [{
               "name": "nginx",
               "image": "dockerfile/nginx",
               "ports": [{"containerPort": 80, "hostPort": 8080}]
             }]
           }
         },
         "labels": {"name": "nginx"}
        }},
    "labels": {"name": "nginx"}
  }

I have a wild guess. Since usually only one runtime fails (1.3 or 1.4), could it have been port conflicts (hostPort: 8080) that failed the tests?

@yujuhong yujuhong mentioned this issue Feb 19, 2015
Closed
@roberthbailey roberthbailey mentioned this issue Apr 1, 2015
Closed
@roberthbailey
Copy link
Contributor

Is this still a problem? Or can we close this issue now?

@roberthbailey roberthbailey removed this from the Test Issue Fix-it milestone Apr 27, 2015
@yujuhong
Copy link
Contributor

Some bugs were discovered and fixed, but in general, timeouts are still a problem in integration tests.

We can close this one though because there are other issue (#6651) tracking problems like this. #6651 is currently blocked on a reflector benchmark to stress test the watch performance.

@krzyzacy krzyzacy mentioned this issue Sep 18, 2017
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@dchen1107 dchen1107

Labels
area/build-release area/test area/test-infra priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Remove hostPort in api/examples/controller.json to avoid port conflicts yujuhong/kubernetes
5 participants
@j3ffml @dchen1107 @roberthbailey @goltermann @yujuhong