kubeadm: Make etcd member removal idempotent #117724

dlipovetsky · 2023-05-02T16:11:20Z

What type of PR is this?

/kind bug

What this PR does / why we need it:

With this change, if the etcd member is not found, then it has already been removed, and kubeadm reset immediately completes the remove-etcd-member phase.

Previously, the phase would complete only once the exponential-backoff retry expired, up to 3 minutes duration. Below is what this looks like; note that kubeadm reset fails to delete the member ID 0, but eventually completes the remove-etcd-member phase, and then the remaining phases:

> kubeadm reset -f -v 5
I0425 07:42:54.260045   17919 reset.go:100] [reset] Loaded client set from kubeconfig file: /etc/kubernetes/admin.conf
[reset] Reading configuration from the cluster...
[reset] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
I0425 07:42:54.286968   17919 kubelet.go:74] attempting to download the KubeletConfiguration from ConfigMap "kubelet-config"
[preflight] Running pre-flight checks
I0425 07:42:54.303838   17919 reset.go:124] [reset] Detected and using CRI socket: unix:///run/containerd/containerd.sock
I0425 07:42:54.303875   17919 removeetcdmember.go:56] [reset] Checking for etcd config
I0425 07:42:54.303892   17919 local.go:93] [etcd] creating etcd client that connects to etcd pods
I0425 07:42:54.303905   17919 etcd.go:168] retrieving etcd endpoints from "kubeadm.kubernetes.io/etcd.advertise-client-urls" annotation in etcd Pods
I0425 07:42:54.308865   17919 etcd.go:104] etcd endpoints read from pods: https://10.0.130.65:2379,https://10.0.131.205:2379,https://10.0.133.32:2379
I0425 07:42:54.319603   17919 etcd.go:224] etcd endpoints read from etcd: https://10.0.131.205:2379,https://10.0.133.32:2379
I0425 07:42:54.319618   17919 etcd.go:122] update etcd endpoints: https://10.0.131.205:2379,https://10.0.133.32:2379
I0425 07:42:54.325820   17919 local.go:118] [etcd] get the member id from peer: https://10.0.130.65:2380
I0425 07:42:54.333868   17919 local.go:124] [etcd] removing etcd member: https://10.0.130.65:2380, id: 0
{"level":"warn","ts":"2023-04-25T07:42:54.342Z","logger":"etcd-client","caller":"v3/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0005b56c0/10.0.131.205:2379","attempt":0,"error":"rpc error: code = NotFound desc = etcdserver: member not found"}
...
I0425 07:46:18.670046   17919 etcd.go:327] Failed to remove etcd member: etcdserver: member not found
W0425 07:46:18.670088   17919 removeetcdmember.go:64] [reset] Failed to remove etcd member: etcdserver: member not found, please manually remove this etcd member using etcdctl
I0425 07:46:18.670104   17919 cleanupnode.go:61] [reset] Getting init system
[reset] Stopping the kubelet service

This change also fixes a semantic bug in etcd.GetMemberID. The function now returns an error if no member ID is found for the peer. Previously, the function returned 0 if no member was found, but 0 is not a valid member ID.

Which issue(s) this PR fixes:

Fixes #

Special notes for your reviewer:

This change does not change the existing behavior of the remove-etcd-member phase. It only shortens the amount of time that kubeadm reset spends in that phase when the etcd member ID is not found for the peer.

Does this PR introduce a user-facing change?

If `kubeadm reset` finds no etcd member ID for the peer it removes during the `remove-etcd-member` phase, it continues immediately to other phases, instead of retrying the phase for up to 3 minutes before continuing.

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

dlipovetsky · 2023-05-02T16:12:02Z

/sig cluster-lifecycle

dlipovetsky · 2023-05-02T16:34:47Z

/cc @neolit123

dlipovetsky · 2023-05-02T17:16:33Z

I mistakenly had a double import (one already existed under an alias). Because it's a small change, I'll just amend my commit, and force-push.

dlipovetsky · 2023-05-02T18:05:18Z

Pod terminated unexpectedly.

/retest pull-kubernetes-dependencies

k8s-ci-robot · 2023-05-02T18:05:21Z

@dlipovetsky: The /retest command does not accept any targets.
The following commands are available to trigger required jobs:

/test pull-cadvisor-e2e-kubernetes
/test pull-kubernetes-conformance-kind-ga-only-parallel
/test pull-kubernetes-coverage-unit
/test pull-kubernetes-dependencies
/test pull-kubernetes-dependencies-go-canary
/test pull-kubernetes-e2e-gce
/test pull-kubernetes-e2e-gce-100-performance
/test pull-kubernetes-e2e-gce-big-performance
/test pull-kubernetes-e2e-gce-canary
/test pull-kubernetes-e2e-gce-cos
/test pull-kubernetes-e2e-gce-cos-canary
/test pull-kubernetes-e2e-gce-cos-no-stage
/test pull-kubernetes-e2e-gce-network-proxy-http-connect
/test pull-kubernetes-e2e-gce-scale-performance-manual
/test pull-kubernetes-e2e-kind
/test pull-kubernetes-e2e-kind-ipv6
/test pull-kubernetes-integration
/test pull-kubernetes-integration-go-canary
/test pull-kubernetes-kubemark-e2e-gce-scale
/test pull-kubernetes-node-e2e-containerd
/test pull-kubernetes-typecheck
/test pull-kubernetes-unit
/test pull-kubernetes-unit-go-canary
/test pull-kubernetes-update
/test pull-kubernetes-verify
/test pull-kubernetes-verify-go-canary

The following commands are available to trigger optional jobs:

/test check-dependency-stats
/test pull-ci-kubernetes-unit-windows
/test pull-e2e-gce-cloud-provider-disabled
/test pull-kubernetes-conformance-image-test
/test pull-kubernetes-conformance-kind-ga-only
/test pull-kubernetes-conformance-kind-ipv6-parallel
/test pull-kubernetes-cos-cgroupv1-containerd-node-e2e
/test pull-kubernetes-cos-cgroupv1-containerd-node-e2e-features
/test pull-kubernetes-cos-cgroupv2-containerd-node-e2e
/test pull-kubernetes-cos-cgroupv2-containerd-node-e2e-eviction
/test pull-kubernetes-cos-cgroupv2-containerd-node-e2e-features
/test pull-kubernetes-cos-cgroupv2-containerd-node-e2e-serial
/test pull-kubernetes-cross
/test pull-kubernetes-e2e-autoscaling-hpa-cm
/test pull-kubernetes-e2e-autoscaling-hpa-cpu
/test pull-kubernetes-e2e-capz-azure-disk
/test pull-kubernetes-e2e-capz-azure-disk-vmss
/test pull-kubernetes-e2e-capz-azure-file
/test pull-kubernetes-e2e-capz-azure-file-vmss
/test pull-kubernetes-e2e-capz-conformance
/test pull-kubernetes-e2e-capz-windows
/test pull-kubernetes-e2e-capz-windows-alpha-features
/test pull-kubernetes-e2e-capz-windows-serial-slow-hpa
/test pull-kubernetes-e2e-containerd-gce
/test pull-kubernetes-e2e-gce-correctness
/test pull-kubernetes-e2e-gce-cos-alpha-features
/test pull-kubernetes-e2e-gce-cos-kubetest2
/test pull-kubernetes-e2e-gce-csi-serial
/test pull-kubernetes-e2e-gce-device-plugin-gpu
/test pull-kubernetes-e2e-gce-network-proxy-grpc
/test pull-kubernetes-e2e-gce-serial
/test pull-kubernetes-e2e-gce-storage-disruptive
/test pull-kubernetes-e2e-gce-storage-slow
/test pull-kubernetes-e2e-gce-storage-snapshot
/test pull-kubernetes-e2e-gci-gce-autoscaling
/test pull-kubernetes-e2e-gci-gce-ingress
/test pull-kubernetes-e2e-gci-gce-ipvs
/test pull-kubernetes-e2e-inplace-pod-resize-containerd-main-v2
/test pull-kubernetes-e2e-kind-canary
/test pull-kubernetes-e2e-kind-dual-canary
/test pull-kubernetes-e2e-kind-ipv6-canary
/test pull-kubernetes-e2e-kind-ipvs-dual-canary
/test pull-kubernetes-e2e-kind-kms
/test pull-kubernetes-e2e-kind-multizone
/test pull-kubernetes-e2e-kops-aws
/test pull-kubernetes-e2e-kubelet-credential-provider
/test pull-kubernetes-e2e-ubuntu-gce-network-policies
/test pull-kubernetes-kind-dra
/test pull-kubernetes-kind-json-logging
/test pull-kubernetes-kind-text-logging
/test pull-kubernetes-kubemark-e2e-gce-big
/test pull-kubernetes-local-e2e
/test pull-kubernetes-node-crio-cgrpv1-evented-pleg-e2e
/test pull-kubernetes-node-crio-cgrpv2-e2e
/test pull-kubernetes-node-crio-cgrpv2-e2e-kubetest2
/test pull-kubernetes-node-crio-e2e
/test pull-kubernetes-node-crio-e2e-kubetest2
/test pull-kubernetes-node-e2e-containerd-alpha-features
/test pull-kubernetes-node-e2e-containerd-features
/test pull-kubernetes-node-e2e-containerd-features-kubetest2
/test pull-kubernetes-node-e2e-containerd-kubetest2
/test pull-kubernetes-node-e2e-containerd-sidecar-containers
/test pull-kubernetes-node-e2e-containerd-standalone-mode
/test pull-kubernetes-node-e2e-containerd-standalone-mode-all-alpha
/test pull-kubernetes-node-kubelet-credential-provider
/test pull-kubernetes-node-kubelet-serial-containerd
/test pull-kubernetes-node-kubelet-serial-containerd-kubetest2
/test pull-kubernetes-node-kubelet-serial-cpu-manager
/test pull-kubernetes-node-kubelet-serial-cpu-manager-kubetest2
/test pull-kubernetes-node-kubelet-serial-crio-cgroupv1
/test pull-kubernetes-node-kubelet-serial-crio-cgroupv2
/test pull-kubernetes-node-kubelet-serial-hugepages
/test pull-kubernetes-node-kubelet-serial-memory-manager
/test pull-kubernetes-node-kubelet-serial-pod-disruption-conditions
/test pull-kubernetes-node-kubelet-serial-topology-manager
/test pull-kubernetes-node-kubelet-serial-topology-manager-kubetest2
/test pull-kubernetes-node-memoryqos-cgrpv2
/test pull-kubernetes-node-swap-fedora
/test pull-kubernetes-node-swap-fedora-serial
/test pull-kubernetes-node-swap-ubuntu-serial
/test pull-kubernetes-unit-experimental
/test pull-kubernetes-verify-strict-lint
/test pull-publishing-bot-validate

Use /test all to run the following jobs that were automatically triggered:

pull-kubernetes-conformance-kind-ga-only-parallel
pull-kubernetes-dependencies
pull-kubernetes-e2e-gce
pull-kubernetes-e2e-kind
pull-kubernetes-e2e-kind-ipv6
pull-kubernetes-integration
pull-kubernetes-node-e2e-containerd
pull-kubernetes-typecheck
pull-kubernetes-unit
pull-kubernetes-verify

In response to this:

Pod terminated unexpectedly.

/retest pull-kubernetes-dependencies

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

dlipovetsky · 2023-05-02T18:05:43Z

/test pull-kubernetes-dependencies

neolit123

/approve
/hold

thanks, @dlipovetsky the change seems ok, but needs more eyes because this is a sensitive area and we don't have sufficient tests for all cases.

/cc @SataQiu @pacoxu

neolit123 · 2023-05-03T08:21:34Z

cmd/kubeadm/app/util/etcd/etcd.go

@@ -283,7 +285,7 @@ func (c *Client) GetMemberID(peerURL string) (uint64, error) {
 			return member.GetID(), nil
 		}
 	}
-	return 0, nil
+	return 0, ErrNoMemberIDForPeerURL


this seems valid, but i wonder if it will break something. are we running GetMemberID() in more places where we expect no error? if yes, we should probably adapt all code around it.

Thanks for your review! This is a great question.

The only other call site is https://github.com/kubernetes/kubernetes/blob/65e306d7d85dcfcbb9e3f5e7373f5469cc4437c7/cmd/kubeadm/app/phases/etcd/local.go#L180.

I've mentally walked this code.

Previous to this change, if GetMemberID did not find a member ID for the peer, it returned 0. Afterward, MemberPromote logged the message [etcd] Member 0 already promoted. The function continued to run, possibly returning an error later from the call to WaitForClusterAvailable.

With this change, if GetMemberID does not find the member ID, it returns an error, and that error is immediately returned.

This looks valid.

Only the learner mode will use it.

kubernetes/cmd/kubeadm/app/phases/etcd/local.go

Lines 175 to 184 in 28247d5

if features.Enabled(cfg.FeatureGates, features.EtcdLearnerMode) {

learnerID, err := etcdClient.GetMemberID(etcdPeerAddress)

if err != nil {

return err

}

err = etcdClient.MemberPromote(learnerID)

if err != nil {

return err

}

}

The change is OK if the GetMemberID returns the error, it is not valid and should return err. Or the promote will fail bellow.

dlipovetsky · 2023-05-03T15:30:51Z

we don't have sufficient tests for all cases.

I noticed that we don't have tests that exercise the etcd client.

I think we could address this with an integration test, i.e. run 3 etcd processes. I think that should work in prow...

neolit123 · 2023-05-03T16:11:10Z

we don't have sufficient tests for all cases.

I noticed that we don't have tests that exercise the etcd client.

I think we could address this with an integration test, i.e. run 3 etcd processes. I think that should work in prow...

do you mean the etcd client in etcd? we only use kinder for testing kubeadm binary calls, with the exception of some CLI integration tests in k/k. basing tests on kinder is a req since we'd ideally want to keep all test jobs generated and in the same place. not sure how we can tell the etcd client in kubeadm to do any arbitrary action foo.

IMO ideally we should have some kubeadm etcd code unit tests with fake etcd server / client. not sure to what extent etcd code supports that.

dlipovetsky · 2023-05-03T16:49:04Z

IMO ideally we should have some kubeadm etcd code unit tests with fake etcd server / client. not sure to what extent etcd code supports that.

👍 A fake etcd server seems doable.

Update: I took a look. I don't think a fake etcd server is doable after all. I would like to use a fake etcd client, but currently we instantiate the real etcd client every time we use it, so there's no way to replace it with a fake client. To do that, I'd need to refactor a bit. I'll take a closer look.

pacoxu · 2023-05-04T10:47:18Z

cmd/kubeadm/app/phases/etcd/local.go

+		if errors.Is(etcdutil.ErrNoMemberIDForPeerURL, err) {
+			klog.V(5).Infof("[etcd] member was already removed, because no member id exists for peer %s", etcdPeerAddress)
+			return nil
+		}


The RemoveMember will handle ErrNoMemberIDForPeerURL later in the function.

Can we pass through so that the log of removing an etcd member will be printed and for file delete, not-found is a successful state? I don't have a strong opinion on this, but I believe it would make the log more readable for users.

Can we pass through so that the log of removing an etcd member will be printed and for file delete, not-found is a successful state?

Thanks for the review @pacoxu!

I'm sorry, I can't understand what you're asking here. Could you please rephrase?

This change is inside func RemoveStackedEtcdMemberFromCluster.

With the current change, the log (if verbose is less than 5) will show only [etcd] get the member id from peer: %s and return.
I prefer to change the log level of line 122 higher so that users can know why. @neolit123 @SataQiu WDYT?

Sorry. My last comment was wrong. If there is an error here, the remove member will use an empty ID which is odd.

Thanks for rephrasing!

I prefer to change the log level of line 122 higher so that users can know why.

I chose log level 5 for two reasons. First, because it's not an error, so I don't want to alert the user unless they've asked for increased verbosity. Second, because all log messages that indicate a problem (i.e. start with Failed) use level 5.

Of course I can reduce the log level, if you feel strongly. Let me know.

/lgtm
overall

dlipovetsky · 2023-05-04T15:47:14Z

@neolit123 I created a fake etcd client and wrote a unit test of GetMemberID: master...dlipovetsky:kubernetes:kubeadm-etcd-client-refactor.

I could add this to my PR, but I'm not sure I should. WDYT?

neolit123 · 2023-05-04T16:19:27Z

new PR for these tests seems better

SataQiu · 2023-05-06T10:45:31Z

/approve

k8s-ci-robot · 2023-05-06T10:45:38Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: dlipovetsky, neolit123, pacoxu, SataQiu

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~cmd/kubeadm/OWNERS~~ [SataQiu,neolit123]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

dlipovetsky · 2023-05-08T20:38:18Z

I squashed the fixup.

dlipovetsky · 2023-05-09T16:19:11Z

Looks like #117792 will merge first. That adds a unit test for GetMemberID. I'll need to rebase and update the unit test to reflect the new behavior introduced in this PR.

dlipovetsky · 2023-05-09T18:54:41Z

I rebased on master to get the unit test added in #117792.

Reviewers: The code you reviewed is in the first commit, and the rebase did not change it.

I added a fixup that updates the unit test.

I'll squash the fixup once I get an /lgtm.

dlipovetsky · 2023-05-09T20:04:06Z

Failure appears unrelated:

  STEP: watching for the Pod to be deleted @ 05/09/23 19:22:50.518
  May  9 19:22:50.520: INFO: observed event type MODIFIED
  May  9 19:22:51.725: INFO: observed event type MODIFIED
  May  9 19:23:48.313: INFO: observed event type MODIFIED
  May  9 19:23:48.847: INFO: observed event type MODIFIED
  May  9 19:23:50.518: INFO: failed to see DELETED event: timed out waiting for the condition

/test pull-kubernetes-e2e-kind-ipv6

k8s-ci-robot · 2023-05-10T04:45:57Z

LGTM label has been added.

Git tree hash: 313e4bfca9e561239fe827832eca3fde1adaaff2

If the etcd member is not found, then it has already been removed, and kubeadm reset should immediately complete the 'remove-etcd-member' phase. Previously, the phase would complete only once the exponential-backoff retry expired, up to 3 minutes duration. This commit also fixes a semantic error in etcd.GetMemberID. Previously, the function returned 0 if no member was found, but 0 is not a valid member ID.

dlipovetsky · 2023-05-10T16:14:11Z

I've squashed the fixup commit.

dlipovetsky · 2023-05-10T16:28:40Z

@neolit123 I think this has gotten reviews you asked for, and it includes new unit tests from #117792. Are you ok to cancel the /hold?

neolit123 · 2023-05-10T17:03:39Z

/hold cancel

humblec · 2023-05-12T05:09:00Z

@dlipovetsky can you please cherry pick this to 1.24 as well ?

…#117792-#117724-upstream-release-1.27 Automated cherry pick of #117792: kubeadm: Use internal etcd client through an interface #117724: kubeadm: Make etcd member removal idempotent

…#117792-#117724-upstream-release-1.26 Automated cherry pick of #117792: kubeadm: Use internal etcd client through an interface #117724: kubeadm: Make etcd member removal idempotent

…#117792-#117724-upstream-release-1.24 Automated cherry pick of #117792: kubeadm: Use internal etcd client through an interface #117724: kubeadm: Make etcd member removal idempotent

…#117792-#117724-upstream-release-1.25 Automated cherry pick of #117792: kubeadm: Use internal etcd client through an interface #117724: kubeadm: Make etcd member removal idempotent

k8s-ci-robot added sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle. area/kubeadm and removed do-not-merge/needs-sig Indicates an issue or PR lacks a `sig/foo` label and requires one. labels May 2, 2023

k8s-ci-robot requested review from pacoxu and RA489 May 2, 2023 16:12

k8s-ci-robot requested a review from neolit123 May 2, 2023 16:34

dlipovetsky force-pushed the kubeadm-remove-etcd-member-idempotent branch from eb85ba6 to 65e306d Compare May 2, 2023 17:16

neolit123 reviewed May 3, 2023

View reviewed changes

k8s-ci-robot requested a review from SataQiu May 3, 2023 08:25

k8s-ci-robot added do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. approved Indicates a PR has been approved by an approver from all required OWNERS files. labels May 3, 2023

pacoxu reviewed May 4, 2023

View reviewed changes

dlipovetsky force-pushed the kubeadm-remove-etcd-member-idempotent branch from 52c3f55 to dccd157 Compare May 8, 2023 20:38

dlipovetsky force-pushed the kubeadm-remove-etcd-member-idempotent branch from dccd157 to 20d93a1 Compare May 9, 2023 18:52

k8s-ci-robot assigned pacoxu May 10, 2023

k8s-ci-robot added the lgtm "Looks good to me", indicates that a PR is ready to be merged. label May 10, 2023

dlipovetsky force-pushed the kubeadm-remove-etcd-member-idempotent branch from 20d93a1 to 5fd5768 Compare May 10, 2023 16:14

k8s-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label May 10, 2023

k8s-ci-robot merged commit 19aaf0c into kubernetes:master May 10, 2023

k8s-ci-robot added this to the v1.28 milestone May 10, 2023

pacoxu mentioned this pull request Aug 4, 2023

kubeadm: fix nil pointer when etcd member is already removed #119753

Merged

1 task

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kubeadm: Make etcd member removal idempotent #117724

kubeadm: Make etcd member removal idempotent #117724

dlipovetsky commented May 2, 2023 •

edited

Loading

dlipovetsky commented May 2, 2023

dlipovetsky commented May 2, 2023

dlipovetsky commented May 2, 2023

dlipovetsky commented May 2, 2023

k8s-ci-robot commented May 2, 2023

dlipovetsky commented May 2, 2023

neolit123 left a comment

neolit123 May 3, 2023

dlipovetsky May 3, 2023

neolit123 May 3, 2023

pacoxu May 5, 2023

dlipovetsky commented May 3, 2023

neolit123 commented May 3, 2023

dlipovetsky commented May 3, 2023 •

edited

Loading

pacoxu May 4, 2023

dlipovetsky May 4, 2023

pacoxu May 5, 2023 •

edited

Loading

dlipovetsky May 5, 2023

pacoxu May 10, 2023

dlipovetsky commented May 4, 2023

neolit123 commented May 4, 2023

SataQiu commented May 6, 2023

k8s-ci-robot commented May 6, 2023

dlipovetsky commented May 8, 2023

dlipovetsky commented May 9, 2023

dlipovetsky commented May 9, 2023

dlipovetsky commented May 9, 2023

k8s-ci-robot commented May 10, 2023

dlipovetsky commented May 10, 2023

dlipovetsky commented May 10, 2023

neolit123 commented May 10, 2023

humblec commented May 12, 2023

	if features.Enabled(cfg.FeatureGates, features.EtcdLearnerMode) {
	learnerID, err := etcdClient.GetMemberID(etcdPeerAddress)
	if err != nil {
	return err
	}
	err = etcdClient.MemberPromote(learnerID)
	if err != nil {
	return err
	}
	}

kubeadm: Make etcd member removal idempotent #117724

kubeadm: Make etcd member removal idempotent #117724

Conversation

dlipovetsky commented May 2, 2023 • edited Loading

What type of PR is this?

What this PR does / why we need it:

Which issue(s) this PR fixes:

Special notes for your reviewer:

Does this PR introduce a user-facing change?

Additional documentation e.g., KEPs (Kubernetes Enhancement Proposals), usage docs, etc.:

dlipovetsky commented May 2, 2023

dlipovetsky commented May 2, 2023

dlipovetsky commented May 2, 2023

dlipovetsky commented May 2, 2023

k8s-ci-robot commented May 2, 2023

dlipovetsky commented May 2, 2023

neolit123 left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dlipovetsky commented May 3, 2023

neolit123 commented May 3, 2023

dlipovetsky commented May 3, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

pacoxu May 5, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

dlipovetsky commented May 4, 2023

neolit123 commented May 4, 2023

SataQiu commented May 6, 2023

k8s-ci-robot commented May 6, 2023

dlipovetsky commented May 8, 2023

dlipovetsky commented May 9, 2023

dlipovetsky commented May 9, 2023

dlipovetsky commented May 9, 2023

k8s-ci-robot commented May 10, 2023

dlipovetsky commented May 10, 2023

dlipovetsky commented May 10, 2023

neolit123 commented May 10, 2023

humblec commented May 12, 2023

dlipovetsky commented May 2, 2023 •

edited

Loading

dlipovetsky commented May 3, 2023 •

edited

Loading

pacoxu May 5, 2023 •

edited

Loading