@wojtek-t @gmarek @kubernetes/goog-control-plane for 1k nodes and calling GCE api excessively

I don't think we have metrics for it. I'll try to add this on Monday.

FWIW, some initial internal numbers show we did 1136411 compute.instances.get, 966858 compute.routes.insert, and 11360 compute.routes.list. I thought I had used a version of k8s that included @cjcullen 's #21172 and #21012 patches, but perhaps not. I'll repro using a build from 9am today.

@gmarek found a way to see method calls: go to console, API Manager, Enabled APIs, Google Compute Engine, Usage, and toggle the "Response Codes" dropdown to be "Methods".

... and why are there so many compute.instances.get calls? If it's the master, can it get by with a compute.instances.list? And if the kubelet making them, perhaps we can either throttle that rate down or change the logic to rely on the metadata server in the VM or the k8s master instead?

I think some of it may be NodeController that queries cloud provider every sync period, but it seems way to much for NodeController itself. Does Kubelet communicates with cloud provider in any way? @dchen1107

FYI in the middle of turning up a new 1k node cluster using today ~9am cut.

• https://console.developers.google.com/home/dashboard?project=mohr-instance-manager1 shows API rate increasing to 118 qps so far.
• https://console.developers.google.com/compute/instanceGroups/list?project=mohr-instance-manager1
  reports "API rate limit exceeded. Rate limit may take several minutes to update if Google Compute Engine has just been enabled, or if this is the first time you use Google Compute Engine." and doesn't display anything.
• https://console.developers.google.com/apis/api/compute_component/usage?project=mohr-instance-manager1 shows Client errors (4xx)

cc/ @roberthbailey

AFAICT NodeController only calls into the cloud provider from one place,
https://github.com/kubernetes/kubernetes/blob/master/pkg/controller/node/nodecontroller.go#L471
but this should happen only once every nc.nodeMonitorPeriod which defaults to 5 seconds. And it's just one call (every 5s), not one call per node.

I think I found a problem in route controller (see #21561). It would also explain the bazillion calls to compute.instances.get, since GCECloud.CreateRoute() calls GCECloud.getInstanceByName() which calls gce.service.Instances.Get().

I looked through the Kubelet code for calls to the cloud provider and didn't find anything worrisome.

Everything looks merged on #21561. Anything remaining there or here?

I'm doing one last test to verify now that #21561 merged, but I'll mark closed and re-open if any other issues come up.

Oops, for posterity, it was PR #22094 together with PR #22044 that seems to have fixed both issues.

@alex-mohr did you mean to reopen this with your last comment?

I think he did, but I think the general opinion (not yet verified AFAIK) was that the more recent problem he was having was because his master size was too small. So maybe he can verify that and then close if it is the case.

We should probably add an optional flag to kube-up that asks it to pick MASTER_SIZE for you based on NUM_NODES. At the very least, we should add some recommended master sizes (ideally for both GCP and AWS) to
https://github.com/kubernetes/kubernetes/blob/master/docs/admin/cluster-large.md

@davidopp PTAL #22261

Sorry re: last comment. There are still a bunch of rate limit exceededs when creating a cluster using n1-standard-32 master, but it seems to eventually come up. And the rate limit exceededs eventually stop. It's not great, but does eventually result in a functional cluster, so may be enough to unblock 1.2 and pick improvements to 1.2.x?

FWIW, I suspect the current triggers for rate limits are some of the following:

• GetInstanceByName on each item in the node or route list, where GetInstanceByName makes a (synchronous) call to gce.service.Instances.Get. I didn't read the logic closely enough to see if that's necessary. If it is, perhaps we could replace with a single gce.service.Instances.List before the loop?
• We spawn a bunch of operations (like creating routes) in go routines and each independently polls the op for completion. I think we'd benefit from some form of shared rate limiter on polling operations to limit that to something like 5 qps, or adding some form of backoff when getting that error. As is, client seems to speed up when it gets rate limit exceededs because GCE api rejects them faster than it processes admitted requests -- yay.

Have you tested since #22099 was merged? Prior to that PR every node was making calls to the GCE API on startup.

@roberthbailey Yes, it was sync from Monday evening, so did include #22099.

Bummer.

@lavalamp Perhaps you'd be interested in extending the charter of the (Google) CSI team to include the GCE cloudprovider bits? Seems like there's a lot of overlap in terms of approach between preventing k8s clients from swamping k8s master and preventing k8s from swamping GCE?

Did you see the shared-rate limiter we put in for AWS? It's not the bestest, but maybe we could use it for GCE also! Maybe with some improvements from getting more eyes on it!

https://github.com/kubernetes/kubernetes/blob/master/pkg/cloudprovider/providers/aws/retry_handler.go

I think it makes sense to keep all of the cloud provider stuff as part of control-plane team for now, and I agree with Justin we should look at these problems holistically since the same themes will probably crop up on every cloud provider.

Whether anyone (on any team) has time to make the improvements in time for 1.2 is another story....

agree w/ @davidopp. Happy to consult though, if needed/desired.

Moving to next-candidate, upgrading to P1.

/sig cluster-lifecycle
/area platform/gce
since we lack a sig-gcp at the moment

/assign
actually, I think this was fixed by the linked PR

/close