Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

operator: tests should use their own k8s client #7638

Closed
wants to merge 1 commit into from

Conversation

joejulian
Copy link
Contributor

@joejulian joejulian commented Dec 6, 2022

The manager-provided client is designed to do the right thing for controllers by default (which is to read from caches, meaning that it's not strongly consistent), which means it probably does the wrong thing for tests (which almost certainly want strong consistency). - kubernetes-sigs/controller-runtime#343

This addresses an error seen in #7605, ErrCacheNotStarted.
Associated with #7395

Backports Required

  • none - not a bug fix
  • none - issue does not exist in previous branches
  • none - papercut/not impactful enough to backport
  • v22.3.x
  • v22.2.x
  • v22.1.x

UX Changes

Release Notes

  • none

@joejulian joejulian added kind/bug Something isn't working area/k8s area/tests labels Dec 6, 2022
@joejulian joejulian requested a review from a team as a code owner December 6, 2022 14:35
@joejulian joejulian self-assigned this Dec 6, 2022
RafalKorepta
RafalKorepta previously approved these changes Dec 6, 2022
@RafalKorepta
Copy link
Contributor

Ehh it still happens

https://buildkite.com/redpanda/redpanda/builds/19506#0184e7f6-1ed2-42ef-a3ce-e6fcc551cf5f/1036-1053

=== RUN   TestAPIs
--
  | Running Suite: Controller Suite
  | ===============================
  | Random Seed: 1670339144
  | Will run 39 of 39 specs
  |  
  | •••••••••••••••••••
  | ------------------------------
  | • Failure [31.464 seconds]
  | Redpanda cluster scale resource
  | /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-05a631c3c06f55e56-1/redpanda/redpanda/src/go/k8s/controllers/redpanda/cluster_controller_scale_test.go:31
  | When scaling down a cluster
  | /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-05a631c3c06f55e56-1/redpanda/redpanda/src/go/k8s/controllers/redpanda/cluster_controller_scale_test.go:94
  | Can decommission nodes that never started [It]
  | /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-05a631c3c06f55e56-1/redpanda/redpanda/src/go/k8s/controllers/redpanda/cluster_controller_scale_test.go:210
  |  
  | Timed out after 30.001s.
  | Expected
  | <*int32 \| 0xc0007a2acc>: 2
  | to be nil
  |  
  | /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-05a631c3c06f55e56-1/redpanda/redpanda/src/go/k8s/controllers/redpanda/cluster_controller_scale_test.go:236

@joejulian
Copy link
Contributor Author

@RafalKorepta different error. This is for ErrCacheNotStarted, thought this might make the other failure more consistent.

@RafalKorepta
Copy link
Contributor

••••••••••••••••••E1206 15:07:24.629045    5075 event.go:273] Unable to write event: '&v1.Event{TypeMeta:v1.TypeMeta{Kind:"", APIVersion:""}, ObjectMeta:v1.ObjectMeta{Name:"test-console.172e3cb13f6e8f5c", GenerateName:"", Namespace:"default", SelfLink:"", UID:"", ResourceVersion:"902", Generation:0, CreationTimestamp:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), DeletionTimestamp:<nil>, DeletionGracePeriodSeconds:(*int64)(nil), Labels:map[string]string(nil), Annotations:map[string]string(nil), OwnerReferences:[]v1.OwnerReference(nil), Finalizers:[]string(nil), ClusterName:"", ManagedFields:[]v1.ManagedFieldsEntry(nil)}, InvolvedObject:v1.ObjectReference{Kind:"Console", Namespace:"default", Name:"test-console", UID:"dddecbfc-d2e2-46f7-b1ee-6fc123817a29", APIVersion:"redpanda.vectorized.io/v1alpha1", ResourceVersion:"901", FieldPath:""}, Reason:"NoSubdomain", Message:"No Ingress created because no subdomain is found in Cluster ExternalListener", Source:v1.EventSource{Component:"Console", Host:""}, FirstTimestamp:time.Date(2022, time.December, 6, 15, 7, 23, 0, time.Local), LastTimestamp:time.Date(2022, time.December, 6, 15, 7, 24, 628029999, time.Local), Count:11, Type:"Warning", EventTime:time.Date(1, time.January, 1, 0, 0, 0, 0, time.UTC), Series:(*v1.EventSeries)(nil), Action:"", Related:(*v1.ObjectReference)(nil), ReportingController:"", ReportingInstance:""}': 'Patch "https://127.0.0.1:46481/api/v1/namespaces/default/events/test-console.172e3cb13f6e8f5c": dial tcp 127.0.0.1:46481: connect: connection refused'(may retry after sleeping)
--
  |  
  |  
  |  
  | Summarizing 2 Failures:
  |  
  | [Fail] Redpanda cluster scale resource When scaling down a cluster [It] Can decommission nodes that never started
  | /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-05a631c3c06f55e56-1/redpanda/redpanda/src/go/k8s/controllers/redpanda/cluster_controller_scale_test.go:236
  |  
  | [Fail] Redpanda cluster scale resource When scaling down a cluster [It] Can decommission nodes that were lazy to start
  | /var/lib/buildkite-agent/builds/buildkite-amd64-builders-i-05a631c3c06f55e56-1/redpanda/redpanda/src/go/k8s/controllers/redpanda/cluster_controller_scale_test.go:289
  |  
  | Ran 39 of 39 Specs in 100.965 seconds
  | FAIL! -- 37 Passed \| 2 Failed \| 0 Pending \| 0 Skipped

https://buildkite.com/redpanda/redpanda/builds/19506#0184e7f6-1ed2-42ef-a3ce-e6fcc551cf5f/1036-2445

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/k8s area/tests kind/bug Something isn't working
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants