Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

integration flake: PersistentVolumeController could not update claim "default/fake-pvc" #26256

Closed
jsafrane opened this issue May 25, 2016 · 8 comments
Closed

integration flake: PersistentVolumeController could not update claim "default/fake-pvc" #26256

jsafrane opened this issue May 25, 2016 · 8 comments
Assignees
@jsafrane
Labels
kind/flake Categorizes issue or PR as related to a flaky test. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/storage Categorizes an issue or PR as relevant to SIG Storage.
Milestone
next-candidate

Comments

@jsafrane
Copy link
Member

PersistentVolume integration tests fail sometimes:
https://console.cltoud.google.com/storage/browser/kubernetes-jenkins/pr-logs/pull/26105/kubernetes-pull-test-unit-integration/27563/

E0523 22:12:02.268570   29064 controller_base.go:259] PersistentVolumeController could not update claim "default/fake-pvc": persistentvolumeclaims "fake-pvc" not found
panic: test timed out after 10m0s

There are some interesting reports in the log like:

E0523 22:06:52.344753   28799 controller_base.go:259] PersistentVolumeController could not update claim "default/fake-pvc-59": Put http://127.0.0.1:41421/api/v1/namespaces/default/persistentvolumeclaims/fake-pvc-59/status: dial tcp 127.0.0.1:41421: getsockopt: connection refused

Thans @quinton-hoole for report.
@jsafrane jsafrane added sig/storage Categorizes an issue or PR as relevant to SIG Storage. kind/flake Categorizes issue or PR as related to a flaky test. labels May 25, 2016
@jsafrane jsafrane self-assigned this May 25, 2016
This was referenced May 25, 2016
Closed
Merged
@jsafrane
Copy link
Member Author

It turns out that volume integration tests have no logs at all, err was not checked and I don't really know what's going on there. #26262 adds error checks to help me find out what's wrong there. Still, I suspect there are some troubles with etcd, not related to the test itself.

@jsafrane jsafrane mentioned this issue May 26, 2016
Closed
@jsafrane
Copy link
Member Author

This flake affects also TestPersistentVolumeMultiPVs:

goroutine 13886 [chan receive, 7 minutes]:
k8s.io/kubernetes/test/integration.waitForPersistentVolumePhase(0x7fd74c3ba278, 0xc82c328630, 0x23fe688, 0x5)
    /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/integration/persistent_volumes_test.go:281 +0x6b
k8s.io/kubernetes/test/integration.TestPersistentVolumeMultiPVs(0xc8290fd680)
    /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/test/integration/persistent_volumes_test.go:184 +0x89f
testing.tRunner(0xc8290fd680, 0x3568318)
    /usr/local/go/src/testing/testing.go:473 +0x98
created by testing.RunTests
    /usr/local/go/src/testing/testing.go:582 +0x892

csrwng pushed a commit to csrwng/kubernetes that referenced this issue May 26, 2016
Merge pull request kubernetes#26262 from jsafrane/integration-error-c…
9aace46 
…hecks

Automatic merge from submit-queue

Add error checks to volume integration tests

We need to know what's going on when a test fails, like in kubernetes#26256
@jsafrane
Copy link
Member Author

Extended logging got merged just now, please post links to logs when it breaks again!

@saad-ali saad-ali mentioned this issue May 30, 2016
Merged
@jszczepkowski jszczepkowski mentioned this issue May 31, 2016
Merged
@andyzheng0831 andyzheng0831 mentioned this issue Jun 1, 2016
Merged
@yifan-gu yifan-gu mentioned this issue Jun 2, 2016
Merged
@matchstick matchstick added the priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. label Jun 9, 2016
@matchstick matchstick added this to the v1.3 milestone Jun 9, 2016
@matchstick
Copy link
Contributor

Marking as P1 as per @saad-ali @jsafrane

@jsafrane jsafrane mentioned this issue Jun 9, 2016
Merged
@jsafrane
Copy link
Member Author

jsafrane commented Jun 9, 2016

I have no real usable logs, but I think it could be fixed by #26629, it hasn't flaked since it was merged. And I have #26894 (in merge queue) to further stabilize the tests.

k8s-github-robot pushed a commit that referenced this issue Jun 13, 2016
Merge pull request #26894 from jsafrane/fix-integration-pv-flake
1ed5342 
Automatic merge from submit-queue

Fix integration pv flakes

There are two fixes in this PR:
- run tests in separarate functions and use objects with different names, otherwise events from the beginning of the function are caught later when we watch for events of a different PV/PVC
- don't set PV.Spec.ClaimRef.UID of pre-bound PVs. PVs with UID set are considered as bound and they are deleted/recycled when appropriate PVC does not exists yet.

Fixes #26730 and probably also ~~#26894~~ #26256
@matchstick
Copy link
Contributor

Not blocking this on 1.3.

@matchstick matchstick modified the milestones: next-candidate, v1.3 Jun 15, 2016
@saad-ali saad-ali mentioned this issue Jun 15, 2016
Merged
@jsafrane
Copy link
Member Author

It hasn't flaked for almost a month, can we close this?

@childsb
Copy link
Contributor

childsb commented Jul 13, 2016

Closing this issue since previous fixes made the flake go away.

@childsb childsb closed this as completed Jul 13, 2016
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@jsafrane jsafrane

Labels
kind/flake Categorizes issue or PR as related to a flaky test. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/storage Categorizes an issue or PR as relevant to SIG Storage.
Projects
None yet
Milestone
next-candidate
Development

No branches or pull requests
4 participants
@childsb @jsafrane @a-robinson @matchstick