Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

TestCreateWithNonExistentOwner {garbagecollector} #30228

Closed
k8s-github-robot opened this issue Aug 8, 2016 · 26 comments
Closed

TestCreateWithNonExistentOwner {garbagecollector} #30228

k8s-github-robot opened this issue Aug 8, 2016 · 26 comments
Assignees
@caesarxuchao
Labels
kind/flake Categorizes issue or PR as related to a flaky test. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery.

Comments

@k8s-github-robot
Copy link

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-test-go/15489/

Failed: TestCreateWithNonExistentOwner {garbagecollector}

garbage_collector_test.go:281: garbage collector queues drained
garbage_collector_test.go:283: expect pod test.pod.1 to be garbage collected
@k8s-github-robot k8s-github-robot added priority/backlog Higher priority than priority/awaiting-more-evidence. kind/flake Categorizes issue or PR as related to a flaky test. labels Aug 8, 2016
@spxtr spxtr assigned girishkalele and unassigned spxtr Aug 8, 2016
@bprashanth bprashanth mentioned this issue Aug 15, 2016
Merged
@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-test-go/15861/

Failed: TestCreateWithNonExistentOwner {garbagecollector}

garbage_collector_test.go:288: garbage collector queues drained
garbage_collector_test.go:290: expect pod test.pod.1 to be garbage collected

@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-test-go/15872/

Failed: TestCreateWithNonExistentOwner {garbagecollector}

garbage_collector_test.go:288: garbage collector queues drained
garbage_collector_test.go:290: expect pod test.pod.1 to be garbage collected

@k8s-github-robot k8s-github-robot added priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. and removed priority/backlog Higher priority than priority/awaiting-more-evidence. labels Aug 17, 2016
@jsafrane jsafrane mentioned this issue Aug 18, 2016
Merged
@sttts sttts mentioned this issue Aug 22, 2016
Merged
@girishkalele
Copy link

Seems like this should be looked at by the node team - not sure what this feature does.

@girishkalele
Copy link

@pwittrock I might have missed this if you mentioned it in today's community meeting but are we allowed to move the flakes to the correct team once triaged ? This looks like it should be the node team (pod garbage collection).

@pwittrock
Copy link
Member

@girishkalele Yes. Please have someone from the node team take ownership. Remove the label once the correct owner has ack'ed

@pwittrock
Copy link
Member

@dchen1107 Does this look like an issue the Node team should own?

@pwittrock
Copy link
Member

@girishkalele Please coordinate with @dchen1107 to find the correct owner, and then remove the label

@lavalamp lavalamp added the sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery. label Aug 26, 2016
@lavalamp
Copy link
Member

@caesarxuchao maybe you can take a look?

@caesarxuchao
Copy link
Member

Yeah, I'll take it. @lavalamp could you approve https://github.com/kubernetes/kubernetes/pull/30316/files? It will help debug this issue.

@lavalamp
Copy link
Member

Done!

On Fri, Aug 26, 2016 at 3:53 PM, Chao Xu notifications@github.com wrote:

Yeah, I'll take it. @lavalamp https://github.com/lavalamp could you
approve https://github.com/kubernetes/kubernetes/pull/30316/files? It
will help debug this issue.

—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
#30228 (comment),
or mute the thread
https://github.com/notifications/unsubscribe-auth/AAngluRaO-CEz9ojPyCw_TGDzu4mH7ylks5qj25-gaJpZM4JfEhu
.

@ncdc ncdc mentioned this issue Aug 31, 2016
Merged
@k8s-github-robot
Copy link
Author

[FLAKE-PING] @caesarxuchao

This flaky-test issue would love to have more attention.

@aveshagarwal aveshagarwal mentioned this issue Sep 1, 2016
Merged
@caesarxuchao
Copy link
Member

https://console.cloud.google.com/storage/browser/kubernetes-jenkins/pr-logs/pull/31797/kubernetes-pull-test-unit-integration/42778/artifacts/
error syncing item...Get http://127.0.0.1:39048/api/v1/namespaces/gc-non-existing-owner/pods/test.pod.1: EOF

@caesarxuchao
Copy link
Member

https://console.cloud.google.com/storage/browser/kubernetes-jenkins/pr-logs/pull/31730/kubernetes-pull-test-unit-integration/42648/artifacts/

This one doesn't have the syncing error.

@k8s-github-robot
Copy link
Author

[FLAKE-PING] @caesarxuchao

This flaky-test issue would love to have more attention.

1 similar comment
@k8s-github-robot
Copy link
Author

[FLAKE-PING] @caesarxuchao

This flaky-test issue would love to have more attention.

@caesarxuchao caesarxuchao added priority/backlog Higher priority than priority/awaiting-more-evidence. and removed priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. labels Sep 9, 2016
@caesarxuchao
Copy link
Member

It doesn't happen for 8 days, down grade the priority.

@wojtek-t
Copy link
Member

It happened here: #32471

@wojtek-t wojtek-t mentioned this issue Sep 12, 2016
Merged
@soltysh soltysh mentioned this issue Sep 12, 2016
Merged
@caesarxuchao
Copy link
Member

https://console.cloud.google.com/storage/browser/kubernetes-jenkins/pr-logs/pull/32471/kubernetes-pull-test-unit-integration/44336/

E0912 03:13:46.378979   29472 garbagecollector.go:584] Error syncing item &garbagecollector.node{identity:garbagecollector.objectReference{OwnerReference:metatypes.OwnerReference{APIVersion:"v1", Kind:"Pod", UID:"915d256f-78d1-11e6-9961-0242ac110003", Name:"test.pod.1", Controller:(*bool)(nil)}, Namespace:"gc-non-existing-owner"}, dependentsLock:sync.RWMutex{w:sync.Mutex{state:0, sema:0x0}, writerSem:0x0, readerSem:0x0, readerCount:0, readerWait:0}, dependents:map[*garbagecollector.node]struct {}{}, owners:[]metatypes.OwnerReference{metatypes.OwnerReference{APIVersion:"v1", Kind:"ReplicationController", UID:"doesn't matter", Name:"test.rc.1", Controller:(*bool)(nil)}}}: Get http://127.0.0.1:57018/api/v1/namespaces/gc-non-existing-owner/pods/test.pod.1: dial tcp 127.0.0.1:57018: getsockopt: connection refused
--- FAIL: TestCreateWithNonExistentOwner (10.14s)
    garbage_collector_test.go:293: garbage collector queues drained
    garbage_collector_test.go:295: expect pod test.pod.1 to be garbage collected

@caesarxuchao caesarxuchao mentioned this issue Sep 12, 2016
Merged
@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-test-go/16991/

Failed: TestCreateWithNonExistentOwner {garbagecollector}

garbage_collector_test.go:293: garbage collector queues drained
garbage_collector_test.go:295: expect pod test.pod.1 to be garbage collected

@k8s-github-robot k8s-github-robot added priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. and removed priority/backlog Higher priority than priority/awaiting-more-evidence. labels Sep 14, 2016
@sttts sttts mentioned this issue Sep 14, 2016
Merged
k8s-github-robot pushed a commit that referenced this issue Sep 15, 2016
Merge pull request #32532 from caesarxuchao/log-integration
c594d49 
Automatic merge from submit-queue

Set --alsologtostderr=true in integration test

Without the flag, no glog output are stored in the test results. The logs are useful for debugging flaky tests like #30228.

The change also reveals a lot of messages like `W0912 14:19:32.306719   25386 cacher.go:468] Terminating all watchers from cacher *api.LimitRange`, which doesn't seem right.
@wojtek-t wojtek-t mentioned this issue Sep 16, 2016
Merged
@k8s-github-robot
Copy link
Author

[FLAKE-PING] @caesarxuchao

This flaky-test issue would love to have more attention.

@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-test-go/17114/

Failed: TestCreateWithNonExistentOwner {garbagecollector}

garbage_collector_test.go:287: garbage collector queues drained
garbage_collector_test.go:289: expect pod test.pod.1 to be garbage collected

@k8s-github-robot
Copy link
Author

[FLAKE-PING] @caesarxuchao

This flaky-test issue would love to have more attention.

@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-test-go/17263/

Failed: TestCreateWithNonExistentOwner {garbagecollector}

garbage_collector_test.go:287: garbage collector queues drained
garbage_collector_test.go:289: expect pod test.pod.1 to be garbage collected

@caesarxuchao
Copy link
Member

Log showed GC tried to delete the pod:

I0921 15:49:10.813234    1743 garbagecollector.go:755] none of object [v1/Pod, namespace: gc-non-existing-owner, name: test.pod.1, uid: 964aa6af-804d-11e6-9485-0242ac110002]'s owners exist any more, will garbage collect it

So the problem is in the test, it should just poll for the deletion of the pod, rather than relying on the gc.QueuesDrained(). I'll send a fix.

@caesarxuchao caesarxuchao mentioned this issue Sep 22, 2016
Merged
@k8s-github-robot
Copy link
Author

https://k8s-gubernator.appspot.com/build/kubernetes-jenkins/logs/kubernetes-test-go/17355/

Failed: TestCreateWithNonExistentOwner {garbagecollector}

garbage_collector_test.go:288: garbage collector queues drained
garbage_collector_test.go:290: expect pod test.pod.1 to be garbage collected

@k8s-github-robot
Copy link
Author

[FLAKE-PING] @caesarxuchao

This flaky-test issue would love to have more attention.

k8s-github-robot pushed a commit that referenced this issue Sep 28, 2016
Merge pull request #33235 from caesarxuchao/fix-TestCreateWithNonExis…
33d29b5 
…tentOwner

Automatic merge from submit-queue

Fix TestCreateWithNonExistentOwner

Fix #30228
As #30228 (comment) described, the GC did delete the garbage, it's the test logic failed. 
The test used to rely on `gc.QueuesDrained()`, which could return before the GC finished processing. It seems to be the only possible reason of the test failure. Hence, this PR changed the test to poll for the deletion of garbage.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@caesarxuchao caesarxuchao

Labels
kind/flake Categorizes issue or PR as related to a flaky test. priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/api-machinery Categorizes an issue or PR as relevant to SIG API Machinery.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
8 participants
@lavalamp @caesarxuchao @spxtr @dchen1107 @pwittrock @wojtek-t @k8s-github-robot @girishkalele