Is the optimization (no DNS or IP) worth the complexity of a new noun?

On Mon, Oct 6, 2014 at 5:26 PM, bgrant0607 notifications@github.com wrote:

Related to #1437
#1437, though
this isn't the only solution to it.

A number of forms of discovery/naming/addressing are covered by other
issues:

• IP-per-service (DESIGN: Services v2 #1107
  DESIGN: Services v2 #1107, DESIGN: External IPs #1161
  DESIGN: External IPs #1161)
• DNS for services (Proposal: Internal Dynamic DNS service #1261
  Proposal: Internal Dynamic DNS service #1261, DNS #146
  DNS #146)
• Cardinal services for fixed-replicated (i.e., not dynamically
  scaled) services and singletons (PetSet (was nominal services) #260
  PetSet (was nominal services) #260)
• Master-elected services (Support master election #1542
  Support master election #1542)
• Sharded services (Reference implementation for sharded system #1064
  Reference implementation for sharded system #1064)

The remaining hole is individually addressable dynamically scaled services
and worker pools.

The user could tie use PreStart/PostStart and PreStop hooks to
self-register with a discovery system, such as etcd, consul, or eureka, as
mentioned in #386
#386 and
elsewhere.

A more integrated solution could be to support Headless services. The
endpoints_controller already creates endpoint records in etcd for services,
which are also made available via the API. Headless services would do the
same, just without creating an IP and/or DNS name for the service. This
would create an API that clients could use to watch the set of endpoints
associated with the headless service, and which would take advantage of
whatever UI, readiness probes, etc. we add for services more generally, and
which would be available for Kubernetes applications without the need to
run a separate discovery service.

The tradeoff for a developer would be whether to couple to the Kubernetes
API or to a particular discovery system. This API would not preclude the
self-registration approach, however, and adapters for other discovery
systems could be built upon this API, as well.

Improvements to the existing endpoints API and implementation (e.g.
efficient watching of joins/leaves) are orthogonal issues.

Reply to this email directly or view it on GitHub
#1607.

@thockin Not necessarily. This could be as little as blessing the endpoint API for client usage and documenting the pattern, or simply an option on services, or perhaps there will be some other way to provide this capability when we extend services to handle other cases, such as external IPs. I split #260 into individual use cases, of which this the last one, I believe.

These processes in worker pools: do none of them serve on any host:port, or do some serve application monitoring data such as /varz?

If some serve, how does application monitoring collection discover the host:ports it should collect from, if there are no IPs or DNS names?

@erictune The pod IP:port list would be published via the Endpoints API.

(For others, /varz is like Go's expvar /debug/vars.)

I think this could be as simple as accepting "none" for PortalIP.

I'd be fine with PortalIP=none. We shouldn't allocate an IP, create links vars, nor publish DNS for such services.

Just search for all uses of PortalIP (including kube2sky):
https://github.com/GoogleCloudPlatform/kubernetes/search?l=go&q=PortalIP&utf8=%E2%9C%93

Nit: Should be None, to obey our constant convention.

Hrm - but that means portalIP has a different type depending on what object you've selected.

I think this needs to be a separate field, distinct from portalIP, that does all of the things you describe.

Why wouldn't we publish DNS for headless services? I'd actually want that behavior in any case where it's a cluster.

@smarterclayton PortalIP is type string. :-)

I'm not opposed to a separate field. I took inspiration from your exploitation of a nil selector. :-)

I think of nominal services #260 as the solution for clusters with DNS.

You really want DNS dynamically updated every time a pod is added or removed?

On Mar 16, 2015, at 6:52 PM, Brian Grant notifications@github.com wrote:

@smarterclayton PortalIP is type string. :-)

I guess, although we wanted it to be typed more strongly I thought.
I'm not opposed to a separate field. I took inspiration from your exploitation of a nil selector. :-)

Guilty as charged. Maybe it's not so bad, that was just a gut reaction at first blush. I will look at our client code and see whether I still have a strong reaction (in terms of client complexity). Our deferred ip resolution (controller based allocation) will mean clients will need to wait for that value to be filled out anyway, so it may not be as bad as I thought. Would there be other magic variables in the future? An anycast/multicast address?
I think of nominal services #260 as the solution for clusters with DNS.

Sorry, I got nominal and headless confused. Agree, although some overlap is expected.
You really want DNS dynamically updated every time a pod is added or removed?

In OpenShift we went a step further and make DNS dynamic - membership is done at query time, not manipulated at state time. So there's no extra cost other than size of DNS response. It's really just exposing endpoints directly via DNS.
—
Reply to this email directly or view it on GitHub.

@bgrant0607 @smarterclayton Refer my early stage PR on this change --> #5556

#5556 is in. We're expecting a followup update to services.md.

We could also discuss whether some form of direct DNS publication for pod IPs makes sense.

Here's how OpenShift is doing headless service DNS: openshift/origin#1510

Should headless services be able to be assigned an external load balancer? I don't know that the Kube proxy properly creates an anonymous port for them. If headless means "don't give me a portalIP because I can't use it", the answer is no. If headless means "don't expose me into other containers or give me a portal ip" the answer might still be yes. As a consumer I might still stick a headless service behind a load balancer.

I think headless means "don't give me a portalIP because I can't use it"

On Wed, Apr 8, 2015 at 6:17 PM, Clayton Coleman notifications@github.com
wrote:

Should headless services be able to be assigned an external load balancer?
I don't know that the Kube proxy properly creates an anonymous port for
them. If headless means "don't give me a portalIP because I can't use it",
the answer is no. If headless means "don't expose me into other containers
or give me a portal ip" the answer might still be yes. As a consumer I
might still stick a headless service behind a load balancer.

—
Reply to this email directly or view it on GitHub
#1607 (comment)
.

Or I prefer to be addressed by my endpoints directly? Either way, I think services with portals are a subset of all portals, and it seems like all that changes with a portalIP is that your default DNS name resolves differently.

On Apr 25, 2015, at 11:32 PM, Tim Hockin notifications@github.com wrote:

I think headless means "don't give me a portalIP because I can't use it"

On Wed, Apr 8, 2015 at 6:17 PM, Clayton Coleman notifications@github.com
wrote:

Should headless services be able to be assigned an external load balancer?
I don't know that the Kube proxy properly creates an anonymous port for
them. If headless means "don't give me a portalIP because I can't use it",
the answer is no. If headless means "don't expose me into other containers
or give me a portal ip" the answer might still be yes. As a consumer I
might still stick a headless service behind a load balancer.

—
Reply to this email directly or view it on GitHub
#1607 (comment)
.

—
Reply to this email directly or view it on GitHub.

I think this is done