Skip to content
This repository has been archived by the owner on Sep 30, 2020. It is now read-only.

aws ELB scaling event deletes network interfaces and makes nodes NotReady #854

Closed
meonkeys opened this issue Aug 22, 2017 · 2 comments
Closed
Labels
triage/duplicate Indicates an issue is a duplicate of other open issue.

Comments

@meonkeys
Copy link

We have a kube-aws-provisioned k8s cluster in us-west-2. A couple times a month EC2 "scaling events" take down our cluster. In their own words...

Thank you for contacting AWS Support. I have looked into why ENIs eni-deadbee0 and eni-deadbee1 were deleted, and found they both belonged to the underlying hardware that runs Elastic Load Balancer "k8s-test-ElbAPISe-XXXXXXXXX-999999999.us-west-2.elb.amazonaws.com". These interfaces were deleted because the load balancer itself experienced a scaling event, which automatically scales the underlying hardware of the ELB up and down in response to changes in traffic over time. This is a normal process, and nothing to worry about.
http://docs.aws.amazon.com/elasticloadbalancing/latest/classic/introduction.html
From the ENI page in the EC2 console, you can actually see ENIs associated with your load balancers. You can usually identify them by their description, which should be the name of the ELB.

Please let me know if you have any other questions or concerns.

Best regards,

Nick S.
Amazon Web Services

The root cause looks like this in CloudTrail:

cloudtrail management console

kubernetes/kubernetes#48638 looks like the same issue/symptom but I'm creating a bug in this github repo in case (a) there's something we can do in kube-aws to deal with the issue or (b) other folks experiencing this issue will be rightly led to the upstream issue.

I hope this helps!

-Adam

@camilb
Copy link
Contributor

camilb commented Aug 22, 2017

I think is the same problem described in #598 and kubernetes/kubernetes#41916 was closed in favor of kubernetes/kubernetes#48638

@danielfm
Copy link
Contributor

Closing this as dup of #598. If you find anything that diverges from what is described in that issue, please feel free to reopen this.

@danielfm danielfm added the triage/duplicate Indicates an issue is a duplicate of other open issue. label Aug 28, 2017
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
triage/duplicate Indicates an issue is a duplicate of other open issue.
Projects
None yet
Development

No branches or pull requests

3 participants