Allow for mixed UDP/TCP ports on LoadBalancer Services #64471

Miouge1 · 2018-05-29T19:43:37Z

What this PR does / why we need it:
Allows for LoadBalancer Services to have both TCP and UDP ports.

Some LoadBalancer implementations support mixed TCP and UDP services on a single Service of type=LoadBalancer, like metallb.

So far users are forced to workaround this by creating 2 services like so:

apiVersion: v1
kind: Service
metadata:
  name: example-tcp
  annotations:
    metallb.universe.tf/allow-shared-ip: mykey
spec:
  ports:
  - port: 1234
    targetPort: 1234
    protocol: TCP
  selector:
    app: example
  type: LoadBalancer
  loadBalancerIP: 192.168.1.240

---
apiVersion: v1
kind: Service
metadata:
  name: example-udp
  annotations:
    metallb.universe.tf/allow-shared-ip: mykey
spec:
  ports:
  - port: 1234
    targetPort: 1234
    protocol: UDP
  selector:
    app: example
  type: LoadBalancer
  loadBalancerIP: 192.168.1.240

Instead of a clearer service manifest:

apiVersion: v1
kind: Service
metadata:
  name: example-app
spec:
  ports:
  - port: 1234
    targetPort: 1234
    protocol: TCP
  - port: 1234
    targetPort: 1234
    protocol: UDP
  selector:
    app: example
  type: LoadBalancer
  loadBalancerIP: 192.168.1.240

Which issue(s) this PR fixes
Fixes #23880, #35752

Special notes for your reviewer:
This is kind of like if the Ingress manifests didn't allow for mixed HTTP and HTTPS Ingress because not all Ingress controller implementation supported it.

Release note:

Allow for mixed UDP/TCP ports on type=LoadBalancer Services

k8s-ci-robot · 2018-05-29T19:43:43Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: Miouge1
To fully approve this pull request, please assign additional approvers.
We suggest the following additional approver: smarterclayton

Assign the PR to them by writing /assign @smarterclayton in a comment when ready.

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

pkg/apis/core/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

krmayankk · 2018-05-30T00:47:08Z

/ok-to-test

krmayankk · 2018-05-30T00:49:37Z

We should also add some ut . Also implementation wise is there any technical blockers to doing this

k8s-ci-robot · 2018-05-30T01:03:20Z

@Miouge1: The following test failed, say /retest to rerun them all:

Test name	Commit	Details	Rerun command
pull-kubernetes-bazel-test	`14f041d`	link	`/test pull-kubernetes-bazel-test`

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

gyliu513 · 2018-05-31T09:11:59Z

I also have some question for this, the checking was added in #20394 , but I donot know why the checking is needed.

@thockin I saw you opened #20394 , any comments why do not support mix protocols for lb services?

thockin · 2018-05-31T15:16:51Z

It's not that "not all implementations support it", it's that almost all implementations (as far as I know) do NOT support it. I would like to be able to allow this, but it would be just another source of support issues when people try to bridge environments.

This could be an implementation-specific annotation, which could build support for an argument that it IS portable, but to the best of my knowledge it is not.

@kubernetes/sig-network-api-reviews Does anyone else have evidence that this works in more places than I know?

gyliu513 · 2018-05-31T15:56:15Z

+1 to support this with annotation

Miouge1 · 2018-05-31T20:17:30Z

Is the idea an annotation like: allow-mixed-protocol: true that would disable the validation?

gyliu513 · 2018-06-01T02:14:14Z

@Miouge1 Yes, I think so.

@thockin can you please help confirm so that we can proceed this with a new PR? Thanks.

danderson · 2018-06-01T04:50:35Z

Coming at this problem from the "I've maintained load-balancers" side of the fence, I find this surprising. My expectation going in was that the GCP integration is the aberration. If the LB implementation supports TCP and UDP, I don't see why they wouldn't allow mixing the two in one Service.

If the argument is that most LBs only support TCP... Then we already have a support problem, because I can already tell k8s to create a UDP-only balancer, and that will presumably fail. Allowing mixed protocols doesn't change that surface, the LB implementation is still free to log a sadness event on the service and fail to converge it, or add an admission check to eagerly reject things.

As a point of reference, I did things the annotation way in MetalLB, and in my corner of the universe, this constraint of k8s generates an ongoing trickle of people confused that they need to resort to declarative hacks (2 services and implementation-specific annotations) to express something that their mental model says should be trivial.

@thockin What would it take to satisfy you that the proposed lack-of-constraint makes sense for the majority of k8s users? If I know what data we'd need, I'd be happy to go collect it (and possibly be proved wrong, I'm aware that I'm coming from a place of assumptions here). But if we're going to move the status quo, I'd rather move it towards a more intuitively sensible API than to enshrine vendor-specific hacks.

thockin · 2018-06-05T05:10:02Z

Build the matrix:

GCP:
AWS:
Azure:
OpenStack:

Of those how many support mixed protocols?

Of the out-of-tree clouds (alibaba, huawei, IBM, DigitalOcean, maybe others), how many support mixed protocols?

I don't LIKE being overly minimalist. I want this to work, so show me it will work for more people than it will fail for.

calder · 2018-08-09T00:58:04Z

It seems silly to impose this very artificial limitation on all platforms (but particularly MetalLB) simply because some [cloud] platforms don't support it.

PeterGrace · 2018-10-02T21:01:53Z

As a MetalLB user, I'm stumbling on this issue. I understand @thockin 's point. I also think that allowing an override of this check via an annotation would be very helpful for those of us who are using 'balancing' methods that support both tcp and udp (e.g., MetalLB).

stepin · 2018-12-04T12:42:53Z

Maybe this check can be done on per LB provider basis. As I know, there is already interface for LBs. So, that interface should include method to show if LB can or can not do mixed protocols.

zimmertr · 2019-02-02T06:22:30Z

Wait, so the reason this isn't supported is because cloud providers aren't doing it? That's the ONLY reason? That is very dumb. Who cares what they're doing?

For those coming into this thread fresh and using MetalLB, I got around this by creating two separate services of type loadBalancer that specify the same LoadBalancerIP

For each service, it will be necessary to pass an annotation allowing said service to have the same IP as another service.

    annotations:
        metallb.universe.tf/allow-shared-ip: "true"

Queuecumber · 2019-06-12T01:09:54Z

Kinda surprised this is still a thing

kiall · 2019-06-12T18:59:05Z

Another pass at this with opt in to make it more acceptable! #75831

ptescher · 2019-06-24T20:28:18Z

AWS now supports this, https://aws.amazon.com/about-aws/whats-new/2019/06/network-load-balancer-now-supports-udp-protocol/

boeremak · 2019-09-14T10:32:15Z

Wait, so the reason this isn't supported is because cloud providers aren't doing it? That's the ONLY reason? That is very dumb. Who cares what they're doing?

For those coming into this thread fresh and using MetalLB, I got around this by creating two separate services of type loadBalancer that specify the same LoadBalancerIP

For each service, it will be necessary to pass an annotation allowing said service to have the same IP as another service.
    annotations:
        metallb.universe.tf/allow-shared-ip: "true"

This is exactly what I needed to get RPI cluster running. Thanks.

golpa · 2019-10-04T19:01:35Z

I'd like to add that this feature is really needed

In our use case we don't want to manually manage IPs so we let MetalLB assign IPs to services and we use CoreDNS to resolve the external IP. In that scenario, being forced to define 2 services means

we have to use different hostnames for the same service based on protocol being used
we are now using 2 IP addresses for the same service

piersdd · 2020-04-07T11:20:48Z

bump

kenperkins · 2020-05-03T22:34:44Z

OpenStack supports UDP loadbalancing as of kubernetes/cloud-provider-openstack#844 via kubernetes/cloud-provider-openstack@594b9a3

Nick-The-Uncharted · 2021-03-17T02:52:42Z

bump

Aricg · 2021-03-23T21:42:06Z

bump

danderson · 2021-03-23T21:48:17Z

The issue is closed, bumping will do nothing. If you have new data, open a new bug and please stop spamming everyone subscribed to this one. Thanks!

swiftslee · 2021-11-24T02:54:31Z

bump

aojea · 2021-11-24T08:46:38Z

The issue is closed, bumping will do nothing. If you have new data, open a new bug and please stop spamming everyone subscribed to this one. Thanks!

you can track the development of this feature in kubernetes/enhancements#1435 (comment)

Please, don't "bump" issues 😄

Allow for mixed UDP/TCP ports on LoadBalancer Services

14f041d

k8s-ci-robot added release-note Denotes a PR that will be considered when it comes time to generate release notes. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels May 29, 2018

k8s-ci-robot added needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels May 29, 2018

k8s-ci-robot requested review from mikedanese and thockin May 29, 2018 19:43

k8s-ci-robot removed the needs-ok-to-test Indicates a PR that requires an org member to verify it is safe to test. label May 30, 2018

islinwb mentioned this pull request May 31, 2018

cannot create an external load balancer with mix protocols #64545

Closed

thockin closed this May 31, 2018

k8s-ci-robot added sig/network Categorizes an issue or PR as relevant to SIG Network. kind/api-change Categorizes issue or PR as related to adding, removing, or otherwise changing an API labels May 31, 2018

feiskyer mentioned this pull request Aug 2, 2018

Add support of mixed procotols LoadBalacer service for Azure #66887

Closed

kvaps mentioned this pull request Dec 10, 2018

Support mixed protocols in service.type=loadbalancer #23880

Closed

zimmertr mentioned this pull request Feb 11, 2019

Not releasing IPs for reuse after deleting a service metallb/metallb#399

Closed

kohrapha mentioned this pull request Nov 19, 2020

Abstract AOC as a service aws-observability/aws-otel-test-framework#102

Merged

ndhanushkodi mentioned this pull request Jun 7, 2022

Peering consul server external ip hashicorp/consul-k8s#1258

Closed

2 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow for mixed UDP/TCP ports on LoadBalancer Services #64471

Allow for mixed UDP/TCP ports on LoadBalancer Services #64471

Miouge1 commented May 29, 2018

k8s-ci-robot commented May 29, 2018

krmayankk commented May 30, 2018

krmayankk commented May 30, 2018

k8s-ci-robot commented May 30, 2018 •

edited

Loading

gyliu513 commented May 31, 2018

thockin commented May 31, 2018

gyliu513 commented May 31, 2018

Miouge1 commented May 31, 2018

gyliu513 commented Jun 1, 2018

danderson commented Jun 1, 2018

thockin commented Jun 5, 2018

calder commented Aug 9, 2018 •

edited

Loading

PeterGrace commented Oct 2, 2018

stepin commented Dec 4, 2018

zimmertr commented Feb 2, 2019 •

edited

Loading

Queuecumber commented Jun 12, 2019

kiall commented Jun 12, 2019

ptescher commented Jun 24, 2019

boeremak commented Sep 14, 2019

golpa commented Oct 4, 2019

piersdd commented Apr 7, 2020

kenperkins commented May 3, 2020 •

edited

Loading

Nick-The-Uncharted commented Mar 17, 2021

Aricg commented Mar 23, 2021

danderson commented Mar 23, 2021

swiftslee commented Nov 24, 2021

aojea commented Nov 24, 2021

Allow for mixed UDP/TCP ports on LoadBalancer Services #64471

Allow for mixed UDP/TCP ports on LoadBalancer Services #64471

Conversation

Miouge1 commented May 29, 2018

k8s-ci-robot commented May 29, 2018

krmayankk commented May 30, 2018

krmayankk commented May 30, 2018

k8s-ci-robot commented May 30, 2018 • edited Loading

gyliu513 commented May 31, 2018

thockin commented May 31, 2018

gyliu513 commented May 31, 2018

Miouge1 commented May 31, 2018

gyliu513 commented Jun 1, 2018

danderson commented Jun 1, 2018

thockin commented Jun 5, 2018

calder commented Aug 9, 2018 • edited Loading

PeterGrace commented Oct 2, 2018

stepin commented Dec 4, 2018

zimmertr commented Feb 2, 2019 • edited Loading

Queuecumber commented Jun 12, 2019

kiall commented Jun 12, 2019

ptescher commented Jun 24, 2019

boeremak commented Sep 14, 2019

golpa commented Oct 4, 2019

piersdd commented Apr 7, 2020

kenperkins commented May 3, 2020 • edited Loading

Nick-The-Uncharted commented Mar 17, 2021

Aricg commented Mar 23, 2021

danderson commented Mar 23, 2021

swiftslee commented Nov 24, 2021

aojea commented Nov 24, 2021

k8s-ci-robot commented May 30, 2018 •

edited

Loading

calder commented Aug 9, 2018 •

edited

Loading

zimmertr commented Feb 2, 2019 •

edited

Loading

kenperkins commented May 3, 2020 •

edited

Loading