Services v2 (ip-per-service) #1402

thockin · 2014-09-22T23:25:28Z

DO NOT COMMIT. Needs test updates and a new e2e test. Some FIXMEs still. 95% complete.

Add support for IP-per-service behavior. This creates virtual IP addresses for each service, and uses iptables to route traffic "magically".

The first few commits are relatively independent, and could be extracted to different PRs, if you feel like making me suffer.

See #1107 for background.

thockin · 2014-09-29T20:49:38Z

Anyone who was looking at this - I think it is almost done. I need to do better unit and e2e testing of the final solution, and there are a couple of FIXMEs left (any salt experts want to help?).

I am happy to start entertaining review comments at this time.

thockin · 2014-09-29T23:41:23Z

@eyakubovich We should consider how this affects flannel

eyakubovich · 2014-09-30T04:28:07Z

At first look I think it should be OK. If flannel is executed with --ip-masq, it installs a masquerade rule for traffic going from a flannel network to the outside world (same stuff Docker does). This PR is all about DNAT while flannel is doing SNAT so they should not conflict.

bgrant0607 · 2014-10-02T03:48:31Z

docs/services.md

+
+We expect that using iptables for portals will work at small scale, but will
+not scale to large clusters with thousands of services.  We hope that, by that
+time, we will have moved to a model wherein each pod declares which services


This is one possible solution, but I don't think it's a foregone conclusion that we'll go that way.

If we intercepted DNS resolutions, could we lazily set up iptables rules at DNS resolution time?

We could make that optimization if we decided it was important to keep global portals, no pre-declarations, and scaling was actually a problem. IMO it's too clever by half, but it's worth keeping in our pocket, I suppose.

I agree that we shouldn't do the optimization now. My point was that requiring users to declare services used isn't the only solution, and I have concerns about the impact that would have on ease of use and use-case coverage.

Maybe we should sketch out what the user flow for both user-directed services and global services looks like and determine whether we want both or just one. Also have the discussion about how the ordering problems as implemented today for global services and pods affects actual user interaction.

----- Original Message -----

+
+
+## Shortcomings
+
+Part of the service specification is a createExternalLoadBalancer
flag,
+which tells the master to make an external load balancer that points to
the
+service. In order to do this today, the service proxy must answer on a
known
+(i.e. not random) port. In this case, the service port is promoted to the
+proxy port. This means that is is still possible for users to collide
with
+each others services or with other pods. We expect most services will
not
+set this flag, mitigating the exposure.
+
+We expect that using iptables for portals will work at small scale, but
will
+not scale to large clusters with thousands of services. We hope that, by
that
+time, we will have moved to a model wherein each pod declares which
services

I agree that we shouldn't do the optimization now. My point was that
requiring users to declare services used isn't the only solution, and I have
concerns about the impact that would have on ease of use and use-case
coverage.

Reply to this email directly or view it on GitHub:
https://github.com/GoogleCloudPlatform/kubernetes/pull/1402/files#r18324012

Can you explain a bit more what you want to see? I'm too close to it, I
think.

On Thu, Oct 2, 2014 at 7:10 AM, Clayton Coleman notifications@github.com
wrote:

In docs/services.md:

+
+
+## Shortcomings
+
+Part of the service specification is a createExternalLoadBalancer flag,
+which tells the master to make an external load balancer that points to the
+service. In order to do this today, the service proxy must answer on a known
+(i.e. not random) port. In this case, the service port is promoted to the
+proxy port. This means that is is still possible for users to collide with
+each others services or with other pods. We expect most services will not
+set this flag, mitigating the exposure.
+
+We expect that using iptables for portals will work at small scale, but will
+not scale to large clusters with thousands of services. We hope that, by that
+time, we will have moved to a model wherein each pod declares which services

Maybe we should sketch out what the user flow for both user-directed
services and global services looks like and determine whether we want both
or just one. Also have the discussion about how the ordering problems as
implemented today for global services and pods affects actual user
interaction.
... <#148d1324a4b6f164_>
----- Original Message -----

+ > + > +##
Shortcomings > + > +Part of the service specification is a
createExternalLoadBalancer > flag, > +which tells the master to make an
external load balancer that points to > the > +service. In order to do this
today, the service proxy must answer on a > known > +(i.e. not random)
port. In this case, the service port is promoted to the > +proxy port. This
means that is is still possible for users to collide > with > +each others
services or with other pods. We expect most services will > not > +set
this flag, mitigating the exposure. > + > +We expect that using iptables
for portals will work at small scale, but > will > +not scale to large
clusters with thousands of services. We hope that, by > that > +time, we
will have moved to a model wherein each pod declares which > services I
agree that we shouldn't do the optimization now. My point was that
requiring users to declare services used isn't the only solution, and I
have concerns about the impact that would have on ease of use and use-case
coverage. --- Reply to this email directly or view it on GitHub:
https://github.com/GoogleCloudPlatform/kubernetes/pull/1402/files#r18324012

Reply to this email directly or view it on GitHub
https://github.com/GoogleCloudPlatform/kubernetes/pull/1402/files#r18341089
.

thockin · 2014-10-03T20:26:03Z

I know this is an enormous PR - I can break it up into smaller PRs, though the commits are pretty clean.

I'd really appreciate someone looking at at least the first few commits, though I think it is functionally complete modulo some improved testing.

@smarterclayton do you have some iptables comprehension? Any other volunteers?

bgrant0607 · 2014-10-03T20:33:35Z

@thockin I can take a look.

thockin · 2014-10-03T20:34:54Z

I welcome all comers :) The first 4 or 5 commits are self-contained and
not too subtle

On Fri, Oct 3, 2014 at 1:33 PM, bgrant0607 notifications@github.com wrote:

@thockin https://github.com/thockin I can take a look.

Reply to this email directly or view it on GitHub
#1402 (comment)
.

bgrant0607 · 2014-10-03T20:40:12Z

DESIGN.md

@@ -96,9 +96,9 @@ There are 4 ways that a container manifest can be provided to the Kubelet:

 ### Kubernetes Proxy

-Each node also runs a simple network proxy.  This reflects `services` as defined in the Kubernetes API on each node and can do simple TCP stream forwarding or round robin TCP forwarding across a set of backends.
+Each node also runs a simple network proxy.  This reflects `services` (see [here](https://github.com/GoogleCloudPlatform/kubernetes/blob/master/docs/serices.md) for more details) as defined in the Kubernetes API on each node and can do simple TCP and UDP stream forwarding (round robin) across a set of backends.


FYI, I have found that relative references work better than absolute ones -- docs/....md. Also, you have a typo: serices

Fixed in next push

filbranden · 2014-10-15T16:54:31Z

hack/e2e-suite/services.sh

+#   $3: service port
+function wait_for_service_down() {
+  for i in $(seq 1 20); do
+    $(ssh-to-node "${test_node}" "


Same as above...

back at you :)

thockin · 2014-10-15T22:56:00Z

e2e test passes on GCE every time. It still fails sometimes on vagrant - failing to connect in places that just shouldn't fail. I give up and am punting it to the Vagrant folks. @derekwaynecarr @pweil- - your ball.

derekwaynecarr · 2014-10-16T01:25:34Z

I am fine merging with the current behavior and we can separately address the consistency of the results.

There has been some talk about looking into a libvirt based VM to improve performance over VirtualBox for Linux based devs or kernel devs in general.

On Oct 15, 2014, at 6:56 PM, Tim Hockin notifications@github.com wrote:

e2e test passes on GCE every time. It still fails sometimes on vagrant - failing to connect in places that just shouldn't fail. I give up and am punting it to the Vagrant folks. @derekwaynecarr @pweil- - your ball.

—
Reply to this email directly or view it on GitHub.

thockin · 2014-10-16T02:26:37Z

As far as I can tell this PR is done.

pweil- · 2014-10-16T13:17:47Z

I agree with @derekwaynecarr. We can work on the specific env issues separately. LGTM

thockin · 2014-10-16T17:01:42Z

Bombs away!

Services v2 (ip-per-service)

In preparation for kubernetes#1402.

In preparation for kubernetes#1402. (cherry picked from commit fa24fac)

…-openshift-master Bug OCPBUGS-2991: Disable expansion in SC, if driver does not support it

thockin force-pushed the services_v2 branch 13 times, most recently from f23d4f6 to fa5e46d Compare September 29, 2014 20:45

thockin force-pushed the services_v2 branch from fa5e46d to b784e91 Compare September 29, 2014 23:18

thockin force-pushed the services_v2 branch 2 times, most recently from bcb4538 to 0d8b76d Compare September 30, 2014 04:21

thockin force-pushed the services_v2 branch from 0d8b76d to f825c1d Compare October 1, 2014 03:41

bgrant0607 reviewed Oct 2, 2014
View reviewed changes

thockin force-pushed the services_v2 branch from f825c1d to 8e5281c Compare October 2, 2014 16:50

thockin changed the title ~~WIP: Services v2 (ip-per-service)~~ Services v2 (ip-per-service) Oct 3, 2014

thockin force-pushed the services_v2 branch 2 times, most recently from 77d42a5 to 690dba9 Compare October 3, 2014 20:12

bgrant0607 reviewed Oct 3, 2014
View reviewed changes

filbranden reviewed Oct 15, 2014
View reviewed changes

thockin force-pushed the services_v2 branch from f704efd to d30fbf7 Compare October 15, 2014 22:05

thockin force-pushed the services_v2 branch 2 times, most recently from 41aef32 to 9b5162c Compare October 16, 2014 02:55

thockin added 6 commits October 16, 2014 08:36

Core support for ip-per-service

e907011

Add and update docs.

d258eca

Add e2e test

1c2f04b

Rackspace support

8f82d42

Add local-up support

99bca68

Add vagrant support

5c4bd55

thockin force-pushed the services_v2 branch from 9b5162c to 5c4bd55 Compare October 16, 2014 15:37

doublerr mentioned this pull request Oct 16, 2014

Rackspace - Switch to CoreOS for standard cluster #1832

Merged

thockin added a commit that referenced this pull request Oct 16, 2014

Merge pull request #1402 from thockin/services_v2

3436775

Services v2 (ip-per-service)

thockin merged commit 3436775 into kubernetes:master Oct 16, 2014

thockin mentioned this pull request Oct 16, 2014

DESIGN: Services v2 #1107

Closed

thockin deleted the services_v2 branch October 17, 2014 06:17

This was referenced Oct 20, 2014

Add an integration test for services #741

Closed

Service admission control should consider protocol #1276

Closed

jdef mentioned this pull request Oct 27, 2014

Networking TBD. d2iq-archive/kubernetes-mesos#5

Open

pietern pushed a commit to pietern/kubernetes that referenced this pull request Oct 29, 2014

Add 'portal_net' parameter to pillar

fa24fac

In preparation for kubernetes#1402.

jbeda pushed a commit to jbeda/kubernetes that referenced this pull request Oct 30, 2014

Add 'portal_net' parameter to pillar

f36f91f

In preparation for kubernetes#1402. (cherry picked from commit fa24fac)

spothanis pushed a commit to spothanis/kubernetes that referenced this pull request Mar 24, 2015

Add 'portal_net' parameter to pillar

048e75b

In preparation for kubernetes#1402. (cherry picked from commit fa24fac)

gnufied pushed a commit to gnufied/kubernetes that referenced this pull request Nov 14, 2022

Merge pull request kubernetes#1402 from gnufied/fix-vsphere-csi-tests…

e555d7c

…-openshift-master Bug OCPBUGS-2991: Disable expansion in SC, if driver does not support it

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Services v2 (ip-per-service) #1402

Services v2 (ip-per-service) #1402

thockin commented Sep 22, 2014

thockin commented Sep 29, 2014

thockin commented Sep 29, 2014

eyakubovich commented Sep 30, 2014

bgrant0607 Oct 2, 2014

bgrant0607 Oct 2, 2014

thockin Oct 2, 2014

bgrant0607 Oct 2, 2014

smarterclayton Oct 2, 2014

thockin Oct 2, 2014

thockin commented Oct 3, 2014

bgrant0607 commented Oct 3, 2014

thockin commented Oct 3, 2014

bgrant0607 Oct 3, 2014

thockin Oct 3, 2014

filbranden Oct 15, 2014

thockin Oct 16, 2014

thockin commented Oct 15, 2014

derekwaynecarr commented Oct 16, 2014

thockin commented Oct 16, 2014

pweil- commented Oct 16, 2014

thockin commented Oct 16, 2014

Services v2 (ip-per-service) #1402

Services v2 (ip-per-service) #1402

Conversation

thockin commented Sep 22, 2014

thockin commented Sep 29, 2014

thockin commented Sep 29, 2014

eyakubovich commented Sep 30, 2014

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thockin commented Oct 3, 2014

bgrant0607 commented Oct 3, 2014

thockin commented Oct 3, 2014

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

thockin commented Oct 15, 2014

derekwaynecarr commented Oct 16, 2014

thockin commented Oct 16, 2014

pweil- commented Oct 16, 2014

thockin commented Oct 16, 2014