Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

ci-kubernetes-test-go: TestConfigMapController data race #42847

Closed
yujuhong opened this issue Mar 9, 2017 · 12 comments
Closed

ci-kubernetes-test-go: TestConfigMapController data race #42847

yujuhong opened this issue Mar 9, 2017 · 12 comments
Assignees
@mwielgus @csbell
Labels
kind/flake Categorizes issue or PR as related to a flaky test. priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now.
Milestone
next-candidate

Comments

@yujuhong
Copy link
Contributor

yujuhong commented Mar 9, 2017

https://k8s-testgrid.appspot.com/sq-blocking#test-go

FAIL k8s.io/kubernetes/federation/pkg/federation-controller/configmap 3.522s

=== RUN   TestConfigMapController
==================
WARNING: DATA RACE
Read at 0x00c4203ed3f8 by goroutine 28:
  reflect.Value.IsNil()
      /usr/local/go/src/reflect/value.go:975 +0xd8
  reflect.deepValueEqual()
      /usr/local/go/src/reflect/deepequal.go:70 +0xe8b
  reflect.deepValueEqual()
      /usr/local/go/src/reflect/deepequal.go:97 +0xe09
  reflect.deepValueEqual()
      /usr/local/go/src/reflect/deepequal.go:97 +0xe09
  reflect.deepValueEqual()
      /usr/local/go/src/reflect/deepequal.go:94 +0xc9b
  reflect.DeepEqual()
      /usr/local/go/src/reflect/deepequal.go:185 +0x2bb
  k8s.io/kubernetes/federation/pkg/federation-controller/util.NewTriggerOnAllChanges.func3()
      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/util/handlers.go:42 +0xc7
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.ResourceEventHandlerFuncs.OnUpdate()
      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/controller.go:199 +0x6f
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*ResourceEventHandlerFuncs).OnUpdate()
      <autogenerated>:55 +0xb0
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.NewInformer.func1()
      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/controller.go:265 +0x3ab
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*DeltaFIFO).Pop()
      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/delta_fifo.go:451 +0x3ea
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*controller).processLoop()
      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/controller.go:147 +0x70
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*controller).(k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.processLoop)-fm()
      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/controller.go:121 +0x41
  k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil.func1()
      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:96 +0x6f
  k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.JitterUntil()
      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:97 +0xbd
  k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait.Until()
      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/apimachinery/pkg/util/wait/wait.go:52 +0x5a
  k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache.(*controller).Run()
      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/vendor/k8s.io/client-go/tools/cache/controller.go:121 +0x31f

Previous write at 0x00c4203ed3f8 by goroutine 41:
  k8s.io/kubernetes/federation/pkg/federation-controller/configmap.(*ConfigMapController).addFinalizerFunc()
      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/configmap/configmap_controller.go:232 +0x167
  k8s.io/kubernetes/federation/pkg/federation-controller/configmap.(*ConfigMapController).(k8s.io/kubernetes/federation/pkg/federation-controller/configmap.addFinalizerFunc)-fm()
      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/configmap/configmap_controller.go:179 +0x86
  k8s.io/kubernetes/federation/pkg/federation-controller/util/deletionhelper.(*DeletionHelper).EnsureFinalizers()
      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/util/deletionhelper/deletion_helper.go:101 +0x3a9
  k8s.io/kubernetes/federation/pkg/federation-controller/configmap.(*ConfigMapController).reconcileConfigMap()
      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/configmap/configmap_controller.go:342 +0x2ea
  k8s.io/kubernetes/federation/pkg/federation-controller/configmap.(*ConfigMapController).Run.func2()
      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/configmap/configmap_controller.go:249 +0xe2
  k8s.io/kubernetes/federation/pkg/federation-controller/util.(*DelayingDeliverer).StartWithHandler.func1()
      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/util/delaying_deliverer.go:176 +0x105

Goroutine 28 (running) created at:
  k8s.io/kubernetes/federation/pkg/federation-controller/configmap.(*ConfigMapController).Run()
      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/configmap/configmap_controller.go:241 +0x73
  k8s.io/kubernetes/federation/pkg/federation-controller/configmap.TestConfigMapController()
      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/configmap/configmap_controller_test.go:88 +0xeb0
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:610 +0xc9

Goroutine 41 (running) created at:
  k8s.io/kubernetes/federation/pkg/federation-controller/util.(*DelayingDeliverer).StartWithHandler()
      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/util/delaying_deliverer.go:181 +0x56
  k8s.io/kubernetes/federation/pkg/federation-controller/configmap.(*ConfigMapController).Run()
      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/configmap/configmap_controller.go:250 +0x155
  k8s.io/kubernetes/federation/pkg/federation-controller/configmap.TestConfigMapController()
      /go/src/k8s.io/kubernetes/_output/local/go/src/k8s.io/kubernetes/federation/pkg/federation-controller/configmap/configmap_controller_test.go:88 +0xeb0
  testing.tRunner()
      /usr/local/go/src/testing/testing.go:610 +0xc9
==================
--- PASS: TestConfigMapController (1.05s)
PASS
Found 1 data race(s)
FAIL	k8s.io/kubernetes/federation/pkg/federation-controller/configmap	3.522s

@kubernetes/sig-federation-bugs
@yujuhong yujuhong added kind/flake Categorizes issue or PR as related to a flaky test. sig/federation labels Mar 9, 2017
@yujuhong
Copy link
Contributor Author

yujuhong commented Mar 9, 2017

This is blocking the submit queue.
/cc @k8s-oncall

@dchen1107
Copy link
Member

@yujuhong Thanks for filing the issue. I am just about to file it.

@dchen1107 dchen1107 added the priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now. label Mar 9, 2017
@dchen1107 dchen1107 added this to the v1.6 milestone Mar 9, 2017
@dchen1107
Copy link
Member

@csbell I am assigning this to you for triaging. Thanks!

@dchen1107
Copy link
Member

cc/ @kubernetes/sig-federation-bugs

@csbell
Copy link
Contributor

csbell commented Mar 10, 2017

This is a flakey test. We have a few flakes that we are trying to root out. What's the procedure for flakes that block. Can you just re-run to unblock the submit queue?

@madhusudancs
Copy link
Contributor

@yujuhong @dchen1107 do you know how often this is happening? Should we temporarily disable this test to unblock the submit queue?

@perotinus
Copy link
Contributor

@madhusudancs It looks like this particular flake has happened once this week (at least as far as I could find): https://k8s-testgrid.appspot.com/sq-blocking#test-go

@perotinus
Copy link
Contributor

And, FWIW, the runs just before and after were green, so this looks like an uncommon flake.

@madhusudancs
Copy link
Contributor

@madhusudancs It looks like this particular flake has happened once this week (at least as far as I could find):

I only looked at the runs in last 24 hours, but yeah.

@yujuhong
Copy link
Contributor Author

Whether this currently blocks the submit queue or not, unit test flakes should be minimized if possible. I think triaging this is definitely a high priority, i.e., figuring out whether the race is caused by a bug in the actual code or just the unit test itself...

@madhusudancs
Copy link
Contributor

@yujuhong fair enough.

@csbell csbell modified the milestones: v1.6, next-candidate Mar 13, 2017
@perotinus perotinus mentioned this issue Mar 15, 2017
Closed
@mwielgus mwielgus mentioned this issue Mar 16, 2017
Merged
k8s-github-robot pushed a commit that referenced this issue Mar 18, 2017
Merge pull request #43229 from mwielgus/cm-fix
8752bfb 
Automatic merge from submit-queue

Fix federated config map unit tests

Fixes #41419 and #42847 and possibly other issues in this area.

cc: @nikhiljindal @csbell @perotinus
@k8s-github-robot
Copy link

This Issue hasn't been active in 97 days. Closing this Issue. Please reopen if you would like to work towards merging this change, if/when the Issue is ready for the next round of review.

cc @csbell @mwielgus @yujuhong

You can add 'keep-open' label to prevent this from happening again, or add a comment to keep it open another 90 days

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@mwielgus mwielgus

@csbell csbell

Labels
kind/flake Categorizes issue or PR as related to a flaky test. priority/critical-urgent Highest priority. Must be actively worked on as someone's top priority right now.
Projects
None yet
Milestone
next-candidate
Development

No branches or pull requests
7 participants
@madhusudancs @perotinus @dchen1107 @yujuhong @mwielgus @k8s-github-robot @csbell