Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[AWS] Repeated DescribeInstances calls when KCM restarts #25610

Closed
guoshimin opened this issue May 14, 2016 · 2 comments
Closed

[AWS] Repeated DescribeInstances calls when KCM restarts #25610

guoshimin opened this issue May 14, 2016 · 2 comments
Assignees
@therc
Labels
priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.
Milestone
v1.3

Comments

@guoshimin
Copy link
Contributor

On AWS, whenever the set of nodes changes, or when controller-manager is restarted, controller-manager calls DescribeInstances once for every service to get detailed information about the nodes. This, together with other AWS API calls, causes the AWS API request limit to be hit when you have enough services.

The call chain is

nodeSyncLoop --->
updateLoadBalancerHosts --->
(for each service) lockedUpdateLoadBalancerHosts --->
s.balancer.UpdateLoadBalancer --->
s.getInstancesByNodeNames
@justinsb justinsb added this to the v1.4 milestone Jun 4, 2016
@justinsb
Copy link
Member

justinsb commented Jun 4, 2016

Going to try to fix this in 1.3, but I don't know if we'll make it...

@justinsb justinsb modified the milestones: v1.3, v1.4 Jun 4, 2016
@justinsb justinsb added the priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. label Jun 4, 2016
@justinsb justinsb changed the title [AWS] Spammy DescribeInstances calls [AWS] Repeated DescribeInstances calls when KCM restarts Jun 4, 2016
@therc
Copy link
Member

therc commented Jun 5, 2016

If you're not going to look at this by Monday, I can give it a shot.

@therc therc mentioned this issue Jun 6, 2016
Merged
k8s-github-robot pushed a commit that referenced this issue Jun 12, 2016
Merge pull request #26900 from Clarifai/aws-instance-cache
6d32eba 
Automatic merge from submit-queue

AWS: cache instances during service reload to avoid rate limiting on restart

Fixes #25610 by reducing redundant calls to DescribeInstances()
 
```release-note
* The AWS cloudprovider will cache results from DescribeInstances() if the set of nodes hasn't changed
```

Also move int/stringSlicesEqual from servicecontroller.go to pkg/util/slice
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@therc therc

Labels
priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release.
Projects
None yet
Milestone
v1.3
Development

No branches or pull requests
4 participants
@justinsb @j3ffml @guoshimin @therc