Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

AWS: Proposed set of annotations for ELB "advanced features" #24978

Closed
justinsb opened this issue Apr 29, 2016 · 12 comments
Closed

AWS: Proposed set of annotations for ELB "advanced features" #24978

justinsb opened this issue Apr 29, 2016 · 12 comments
Labels
area/api Indicates an issue on api area. area/federation lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. priority/backlog Higher priority than priority/awaiting-more-evidence. sig/multicluster Categorizes an issue or PR as relevant to SIG Multicluster. sig/network Categorizes an issue or PR as relevant to SIG Network.

Comments

@justinsb
Copy link
Member

There are a few PRs in flight to use annotations to add extra features to the ELBs that k8s sets up (DNS, SSL, Proxy Protocol), and I've written up here what I think a consistent set of those annotations would be. I think the PRs that we have are pretty close to what we have here.

Proxy protocol support:

  • service.beta.kubernetes.io/aws-load-balancer-proxy-protocol=*

This is required to get the client IP, but the service must be aware of the extra header bytes that are sent. Likely our ingress controllers will turn this on and then set an HTTP X-Forwarded-For header, and also some advanced services might set it also to indicate they "speak proxy protocol".

cc @williamsandrew and cf #24569

SSL support at the ELB:

  • service.beta.kubernetes.io/aws-load-balancer-ssl-cert=arn:aws:acm:us-east-1:123456789012:certificate/12345678-1234-1234-1234-123456789012
  • service.beta.kubernetes.io/aws-load-balancer-ssl-ports=* (or e.g. https)

If the user wants the ELB to do SSL decryption, they should set these annotations. ssl-cert specifies the certificate to use; ssl-ports specifies on which ports we should enable SSL. ssl-ports can be optional and default to "*". We need not implement ssl-ports right away, because the primary use for SSL and non-SSL is likely http and https for webservers, and they will likely use ingress, which will likely use SNI instead. (But I do think some people will want direct-to-the-service http & https, so I bet we'll want it eventually..)

cc @therc and cf #23495

DNS support at the service level:

  • service.beta.kubernetes.io/hostname=myservice.myzone.com

If set, Kubernetes will configure a DNS alias for the ELB. Because this isn't specific to AWS, I don't think it gets an "aws-" prefix.

cc @chbatey and @quinton-hoole and cf #21397

Sharing / reusing an ELB

  • service.beta.kubernetes.io/load-balancer-secret=some-sort-of-secret-value

This allows an ELB to be (safely?) shared between two k8s clusters. If set, k8s will look for an ELB with a tag derived from the secret. If it finds one, it will use that ELB, otherwise it will create one (and tag it). In this way, two k8s clusters with services with the same tag will share an ELB. If the secret is kept reasonably secret then a user won't be able to "steal" someone else's ELB (this will be more realistic when we have RBAC etc).

cf #23483
@therc
Copy link
Member

therc commented Apr 29, 2016

@justinsb what would the annotation for deciding whether to proxy SSL (L4, terminate at the pod) vs HTTPS (L7, terminate at the ELB, add X-Forwarded-For header)?

@chbatey
Copy link
Contributor

chbatey commented Apr 29, 2016

cc @jsravn

@justinsb
Copy link
Member Author

@therc good question - I forgot about that axis! I do think HTTP vs TCP should be a separate annotation. Perhaps service.beta.kubernetes.io/aws-load-balancer-mode with values tcp or http? What do you think?

@mml mml added area/api Indicates an issue on api area. team/api labels May 2, 2016
@therc
Copy link
Member

therc commented May 2, 2016

I think aws-load-balancer-layer might be more descriptive than a vague 'mode'. But we need to support four values, since users might ask for communication between the ELB and the backend to be encrypted, too. So perhaps we could use aws-load-balancer-backend-protocol with values https, http, ssl and tcp. EC2 also lets you specify a public SSL key to authenticate backends, but that would require yet another annotation and should probably be a separate PR.

This was referenced May 2, 2016
Merged
Closed
@justinsb justinsb mentioned this issue May 10, 2016
Closed
k8s-github-robot pushed a commit that referenced this issue May 12, 2016
Merge pull request #23495 from Clarifai/elb
4ac4e0f 
Automatic merge from submit-queue

AWS: SSL support for ELB listeners through annotations

In the API, ports have only either TCP or UDP as their protocols, but ELB distinguishes HTTPS->HTTP[S]? from SSL->(SSL|TCP). 

Per #24978, this is implemented through two separate annotations:

`service.beta.kubernetes.io/aws-load-balancer-ssl-cert=arn:aws:acm:us-east-1:123456789012:certificate/12345678-1234-1234-1234-123456789012`
`service.beta.kubernetes.io/aws-load-balancer-backend-protocol=(https|http|ssl|tcp)`

Mixing plain-text and encrypted listeners will be in a separate PR, implementing #24978's `aws-load-balancer-ssl-ports=LIST`
@therc therc mentioned this issue May 25, 2016
Closed
@StaymanHou StaymanHou mentioned this issue May 26, 2016
Closed
@StaymanHou
Copy link

+1

@therc therc mentioned this issue Jun 6, 2016
Closed
@therc therc mentioned this issue Jun 7, 2016
Merged
therc pushed a commit to Clarifai/kubernetes that referenced this issue Jun 7, 2016
AWS: support mixed plaintext/encrypted ports in ELBs
4ff9e93 
Fixes kubernetes#26268

Implements the second SSL ELB annotation, per kubernetes#24978

service.beta.kubernetes.io/aws-load-balancer-ssl-ports=* (or e.g. https)

If not specified, all ports are secure (SSL or HTTPS).
@2opremio 2opremio mentioned this issue Jun 8, 2016
Merged
3 tasks
k8s-github-robot pushed a commit that referenced this issue Jun 10, 2016
Merge pull request #26976 from Clarifai/elb
a2703a3 
Automatic merge from submit-queue

AWS: support mixed plaintext/encrypted ports in ELBs via service.beta.kubernetes.io/aws-load-balancer-ssl-ports annotation

Fixes #26268

Implements the second SSL ELB annotation, per #24978

`service.beta.kubernetes.io/aws-load-balancer-ssl-ports=*` (comma-separated list of port numbers or e.g. `https`)

If not specified, all ports are secure (SSL or HTTPS).
@ghost ghost added area/cluster-federation sig/network Categorizes an issue or PR as relevant to SIG Network. labels Nov 18, 2016
@ghost
Copy link

ghost commented Nov 18, 2016

@kubernetes/sig-cluster-federation

@philips
Copy link
Contributor

philips commented Jan 19, 2017

Just an update on this that #24569 and #23495 are merged. @justinsb would you mind updating that in the description and maybe reordering the bug?

@TylerRick TylerRick mentioned this issue Jan 20, 2017
Closed
@technologik
Copy link

technologik commented Jan 31, 2017 •
edited
Loading

@justinsb - Just to clarify, has an annotation been accepted for creating L4 vs L7 load balancers as mentioned above? I haven't seen anything explicitly noting that ELBv2 / L7 ALBs have an annotation, and if so, what that annotation may be. Thanks.

@ghost
Copy link

ghost commented Sep 8, 2017

Cleaning up old issues. @justinsb, do we still need this one, or can it be closed?

@ghost ghost added priority/backlog Higher priority than priority/awaiting-more-evidence. and removed team/api (deprecated - do not use) labels Sep 8, 2017
@csbell csbell added sig/multicluster Categorizes an issue or PR as relevant to SIG Multicluster. area/federation and removed sig/federation labels Oct 10, 2017
@fejta-bot
Copy link

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

Prevent issues from auto-closing with an /lifecycle frozen comment.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or @fejta.
/lifecycle stale

@k8s-ci-robot k8s-ci-robot added the lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. label Jan 8, 2018
@fejta-bot
Copy link

Stale issues rot after 30d of inactivity.
Mark the issue as fresh with /remove-lifecycle rotten.
Rotten issues close after an additional 30d of inactivity.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle rotten
/remove-lifecycle stale

@k8s-ci-robot k8s-ci-robot added lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. and removed lifecycle/stale Denotes an issue or PR has remained open with no activity and has become stale. labels Feb 10, 2018
@fejta-bot
Copy link

Rotten issues close after 30d of inactivity.
Reopen the issue with /reopen.
Mark the issue as fresh with /remove-lifecycle rotten.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/close

openshift-publish-robot pushed a commit to openshift/kubernetes that referenced this issue May 14, 2020
@k8s-publishing-bot
Merge pull request kubernetes#24978 from rphillips/backports/91066
d4dcaa2 
Bug 1835042: UPSTREAM: 91066: tweak exec-probe log statement to be more concise

Origin-commit: 9b487585358aef05dfbbe042d15cb93f1f563577
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/api Indicates an issue on api area. area/federation lifecycle/rotten Denotes an issue or PR that has aged beyond stale and will be auto-closed. priority/backlog Higher priority than priority/awaiting-more-evidence. sig/multicluster Categorizes an issue or PR as relevant to SIG Multicluster. sig/network Categorizes an issue or PR as relevant to SIG Network.
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
10 participants
@mml @philips @justinsb @chbatey @StaymanHou @therc @k8s-ci-robot @csbell @technologik @fejta-bot