Copy/pasting comments from #2738.

@jbeda:

Not sure what issue this belongs to, but I'm thinking that we restructure the Service object. I talked with @brendandburns and I think he is on board.

Here are some ideas:

• we should split out "desired state" from "current state". (This is done in v1beta3 with spec/status)
• Reuse the PodPorts that are already on the pod
• Have a set of portals that expose the service both inside the cluster and out. Each portal can choose the ports to expose -- either by name or protocol and port number.
• Have different types of portals similar to volumes
• Reflect data about the IPs/ports picked in the status

kind: service
spec:
  selector:
    tier: frontend
  portals:
    - internalPortal: {}  # This will expose every port internally on a "portal IP"
    - name: admin # Name is optional but probably useful
      externalPortal: 
        pool: corp  # Indicate that the IP/ports for this should be pulled from a pool
    - name: www
      externalPortal:
        IP: 1.2.3.4 # We are naming the IP specifically, no pool needed. 
        ports:
          - portal: 80
            pod: 'http' # We are using the named port in the pod spec
          - portal: 443
            pod: 'https'
    - name: gce_www
      gceLoadBalancer:
        ports: [ 'http', 'https' ] # Since GCE L3 doesn't do port mapping, don't allow that to be specified

Finally, when the user does a GET on the resource, they'll see:

kind: service
spec:
  # [snip]
status:
  portals:
    - IP: 10.260.0.1
      internalPortal: {}
      ports:
        - name: 'http'
          portalPort: 80
          podPort: 80
          protocol: TCP
        - name: 'https'
          portalPort: 443
          podPort: 443
          protocol: TCP
        - name: 'admin_http'
          portalPort: 8080
          podPort: 8080
          protocol: TCP
    - name: admin
      IP: 192.168.1.100  # RFC 1918 range, but perhaps HA inside a corp
      externalPortal: {}
      ports:
        - name: 'http'
          portalPort: 6190  # Note the port mapping here
          podPort: 80
          protocol: TCP
        - name: 'https'
          portalPort: 6191
          podPort: 443
          protocol: TCP
        - name: 'admin_http'
          portalPort: 6192
          podPort: 8080
          protocol: TCP
    - name: www
      externalPortal: {}
      IP: 1.2.3.4
        - name: 'http'
          portalPort: 80
          podPort: 80
          protocol: TCP
        - name: 'https'
          portalPort: 443
          podPort: 443
          protocol: TCP
    - name: gce_www
      gceLoadBalancer:
        resourceLink: https://googleapis.com/compute/v1/...
      IP: 2.3.4.5
        - name: 'http'
          portalPort: 80
          podPort: 80
          protocol: TCP
        - name: 'https'
          portalPort: 443
          podPort: 443
          protocol: TCP      

Does this parse for folks? I'm happy to move this off to a new issue if there is a better place to discuss this.

@lavalamp:

I like explicitly listing the portals, that's a big improvement.

I think there's some more work to be done defining the contents of portals, though. Having portals inside ports inside portals is confusing.

@jbeda:

I don't think we have portals inside of ports. Basically it is like this:

Each service exports a set of ports. These ports are defined based on the pods that the service points too. (Open question on what to do if the pods aren't uniform. Perhaps this won't work. But we give ports on pods names, so we should try and use them if we can.)

Each service has a list of portals In status, a portal has an IP

Each portal is of a type Similar to volume, this is done with a "subtype". This can include extra info like the name of the LB that was created on your behalf for GCE.

Each portal has a port map -- what ports are exposed on that portal and how they map to the pods.

What can we remove to help simplify this?

@lavalamp:

Specifically, I was referring to this sequence in your example:

  portals:
    - name: www
      externalPortal:
        ports:
          - portal: 80
            pod: 'http' # We are using the named port in the pod spec

Is the idea behind the portal/port at the end the port that is exposed and the port that it is routed to? Maybe this is clearer:

        ports:
          - expose: 80
            routeTo: 'http' # We are using the named port in the pod spec

@jbeda:

Ah -- Gotcha. there, the last portal could be called portalPort. This is essentially a mapping from the port that the portal exposes to the port to use on the pod. expose sounds good but I'm not a fan of route in this context. We can bike shed on names after we get structure down.

@stp-ip:

Just to get a sense of the possibilities here.
We have internal services, with an internal IP.
We have exposed services, which use the proxy/portal to map the internal IP to an external IP.
We have internal services consuming external services. Would these be mapped via the portals too?
So instead of internal -> portal -> external, one would use external -> portal -> internal service.
This would make the overall structure more streamlined in my opinion.

Additionally with the portal and it's mapping of IP:Port to IP:Port, the next possibility could be routing.
Not just a mapping of the external IP to a service, but a mapping of example.com/service1 to service1 and example.com/service2 to service 2 for example. This would enable the definition of simplistic routing at the edge of the kubernetes cluster, perhaps even with the support from LBs.

In the last comment, I think @stp-ip was asking for what is essentially OpenShift's Route type:

type Route struct {
    TypeMeta   `json:",inline" yaml:",inline"`
    ObjectMeta `json:"metadata,omitempty" yaml:"metadata,omitempty"`
    // Required: Alias/DNS that points to the service
    // Can be host or host:port
    // host and port are combined to follow the net/url URL struct
    Host string `json:"host" yaml:"host"`
    // Optional: Path that the router watches for, to route traffic for to the service
    Path string `json:"path,omitempty" yaml:"path,omitempty"`
    // the name of the service that this route points to
    ServiceName string `json:"serviceName" yaml:"serviceName"`
}

Much like our service proxy watches endpoints, an HTTP reverse proxy, such as HAProxy, could watch routes.

First of all, bikeshedding about the name. I think "service" is too overloaded and brings too much baggage. Also, if we decompose this into multiple objects, they'll need more specific names. For at least the "frontend" portion, I propose Forwarder, akin to GCE's forwarding rules.

In terms of decomposing this, I think it divides nicely between the "frontend" parts and the "backend" target/endpoint specification, which could be inline but a separate object would make sense, too, in which case we could decouple production of the Endpoints list and consumption of it.

type EndpointsSpec struct {
    Selector map[string]string `json:"selector"`
    PodPorts []string `json:"podPorts,omitempty"` // port names
    // Explicit hosts or protocol://host:port, where hosts can be DNS names and/or IP addresses
    Endpoints []string `json:"endpoints,omitempty"`
}

UPDATE 01/14/2015: The manual Endpoints list above doesn't work well for multiple ports on the same IP address. It probably just needs to be a list of hosts. The PodPorts could be generalized to work for manual targets. They'd have to be protocol+port pairs.

We could distinguish internal vs. external portals by pool (i.e., have an internal pool).

I think GCELoadBalancer isn't something we want as a first-class API type. Ideally, we'd have a generic CloudBalancer that called into the cloudprovider API underneath, perhaps with plugins for cloud-specific parameters. Otherwise, the whole thing should just be a plugin.

I could also imagine a host pool, for contacting daemons on the same host, and support for mapping link-local addresses to per-node agents.

We should support an array of IP addresses rather than just a single address. This would permit DNS load balancing, such as in the case that the addresses don't map to Kubernetes backends.

I'd also like to support a one-to-one option on the portals, to indicate a nominal service (#260). In that case, no load-balancing details, such as SessionAffinity, would be specified. Perhaps those could be alternative MappingOptions.

I assume the portal names would be used to produce distinct DNS names for each portal. We may not support externally visible DNS yet, but I could imagine that we would in the future.

In the pod port specifications, it would be useful to indicate whether http/https were supported, in addition to TCP vs. UDP. Or perhaps we could accept HTTP and HTTPS in addition to just TCP.

We should add the ReadinessProbe spec to pods, also, which the Endpoints Controller could use. Since I've proposed we also target host:port, we might also need readiness probe spec in EndpointsSpec.

Another thought on provider-specific behavior: Ideally, the user could enumerate all provider-specific options, and the system would select out the appropriate ones based on which provider the cluster was actually running on. That would facilitate config reuse across multiple providers.

I thought of one difference between internal and external portals: For internal portals, we may want to add a whitelist of namespaces that could access the service and/or a selector to indicate which pods within the namespace should be permitted access. Note that the latter can't provide real protection until/unless we enforce application of labels by label namespace.

It would also be useful for a service provider to publish fairly arbitrary metadata to be consumed by clients, such as client connection parameters, shard maps, keys to look up resource usage data, etc. Not exactly labels or annotations, but would be a map of string to string (cf. DNS TXT key-value data). Maybe "serviceParameters" or "clientConfiguration". If we wanted it to be customizable per pod, it would need to be pulled from the pods, or even from the containers themselves, perhaps similar to readiness. Our internal RPC, load balancing, sharding, and other communication libraries/components depend more and more on this type of thing.

Starting with the easy part: the "backend". I think we have consensus that this is at least a logically separable part of the service. Certainly it's independently useful (#1607). My previous stab at this was here: #2585 (comment)

The only thing I'd add is the key-value payload described here: #2585 (comment)

An alternative name could be TargetController, TargetWatcher, .. See #3024 for general name ideas. But, if we change it, we should change "Endpoints" also, since the main job of this is to generate the Endpoints lists.

In the future, we could add configurable policies about what to filter from the endpoints list, such as whether to use readiness (#620) or not.

Like in the proposal to split out the pod template from ReplicationController (#170), we could support both an inline version of this in the other part of Service and a reference to a separate object.

Will work on the other part(s) of Service now.

/cc @thockin @jbeda

An observation: If we went with an independent EndpointsController object, the different portals shown in @jbeda's example #2585 (comment) could be all separate objects referring to the same EndpointsController object.

First stab at fully granular version (specs only):

//L7: host:port/path to portal mapping [we may not support L7, but here for completeness]
type RouteSpec struct {
    // Required: Alias/DNS that points to the service
    // Can be host or host:port
    // host and port are combined to follow the net/url URL struct
    Host string `json:"host"`
    // Optional: Path that the router watches for, to route traffic for to the service
    Path string `json:"path,omitempty"`
    // the reference to the portal that this route points to
    PortalRef  *ObjectReference `json:"templateRef,omitempty"`
}
//DNS to IPs mapping
type DNSSpec struct {
    Hostname string `json:"hostname"`
    // the reference to the portal that the name points to
    PortalRef  *ObjectReference `json:"templateRef,omitempty"`
}
// Port mapping (not a standalone object)
type PortalPortSpec struct {
    ExposedPort int `json:"exposedPort"`
    // Port names or protocol:port, where protocol could be UDP, TCP, HTTP, HTTPS
    TargetPort string `json:"targetPort"`
}
//IP and port remapper
type PortalSpec struct {
    // Optional list of port translations
    Ports []PortalPortSpec `json:"ports,omitempty"`
    // Optional port range (port or port-port) to forward, without translation
    PortRange string `json:"portRange,omitempty"`
    // Exposed addresses (AddressAllocation)
    ExposedAddresses *ObjectReference `json:"exposedAddresses,omitempty"`
    // Target addresses (Endpoints)
    TargetAddresses *ObjectReference `json:"targetAddresses,omitempty"`
    // If nil, it's expected that exposed addresses will be mapped 1-to-1 to target addresses
    LoadBalancer *LoadBalancerSpec `json:"loadBalancer,omitempty"`
}
//L3 LB (not a standalone object)
type LoadBalancerSpec struct {
    SessionAffinity AffinityType `json:"sessionAffinity,omitempty"`
//We should support plugins, however we specify that, for cloud-provider-specific arguments
}
//Address allocation
type AddressAllocationSpec struct {
    // Number of addresses to allocate
    Count int `json:"count"`
    // Pool from which to allocate. Kubernetes would provide some pre-defined pools, such as
    // "internal" and "linklocal"
    AddressPoolRef *ObjectReference `json:"addressPoolRef,omitempty"`
}
//Address pool: could be external or internal addresses
type ManualAddressPoolSpec struct {
    Addresses []string `json:"addresses"`
}
//Addresses allocated on demand by calling cloud provider
type CloudAddressPoolSpec struct {
//This should be a plugin, however we specify that, with cloud-provider-specific arguments
}
//"Backend" Endpoints generator
type EndpointsSpec struct {
    // Port names or protocol:port, where protocol could be UDP, TCP, HTTP, HTTPS
    Ports []string `json:"ports,omitempty"`
    // Selector to identify target pods
    Selector map[string]string `json:"selector,omitempty"`
    // Manually specified hosts
    Hosts []string `json:"endpoints,omitempty"`
    // Information for enlightened clients; copied into Endpoints
    Info map[string]string `json:"info,omitempty"`
}

Will explore alternatives next.

Low-hanging consolidation/simplifications:

• DNSSpec could just go away in favor of populating DNS automatically for Portals.
• AddressAllocationSpec could get pulled into PortalSpec

This would look like:

type PortalSpec struct {
    // Optional list of port translations
    Ports []PortalPortSpec `json:"ports,omitempty"`
    // Optional port range (port or port-port) to forward, without translation
    PortRange string `json:"portRange,omitempty"`
    // Number of addresses to allocate
    ExposedAddressCount int `json:"exposedAddressCount"`
    // Pool from which to allocate. Kubernetes would provide some pre-defined pools, such as
    // "internal" and "linklocal"
    ExposedAddressPool *ObjectReference `json:"exposedAddressPool,omitempty"`
    // Target addresses (Endpoints)
    TargetAddresses *ObjectReference `json:"targetAddresses,omitempty"`
    // If nil, it's expected that exposed addresses will be mapped 1-to-1 to target addresses
    LoadBalancer *LoadBalancerSpec `json:"loadBalancer,omitempty"`
}

What is the current status of this issue?

In progress.

On Thu, Feb 19, 2015 at 10:25 PM, Yuki Yugui Sonoda <
notifications@github.com> wrote:

What is the current status of this issue?

Reply to this email directly or view it on GitHub
#2585 (comment)
.

To revisit the previously posted slides about how the final endpoints struct is factored...

As I code it up, I find this factoring somewhat awkward. It's not a HUGE deal (it's just code) but what I realized is that the primary key I care about is the (service, port) tuple, not the (service, ip), which is what we have produced.

In effect I have to pivot all the data into a different struct to use it. A snippet:

        type hostPortPair struct {
                host string
                port int
        }

        // Update endpoints for services.
        for i := range allEndpoints {
                svcEndpoints := &allEndpoints[i]

                // We need to build a map of portname -> all ip:ports for that portname.
                portsToEndpoints := map[string][]hostPortPair{}

                // Explode the Endpoints.Endpoints[*].Ports[*] into the aforementioned map.
                for j := range svcEndpoints.Endpoints {
                        ep := &svcEndpoints.Endpoints[j]
                        for k := range ep.Ports {
                                epp := &ep.Ports[k]
                                portsToEndpoints[epp.Name] = append(portsToEndpoints[epp.Name], hostPortPair{ep.IP, epp.Port})
                                // Ignore the protocol field for now.
                        }
                }

                for portname := range portsToEndpoints {
                    // Finally I can process the (service, port) -> endpoints data.

I'm not sure this alone justifies revisting the proposed structure in #4370 but I wanted to put it out there - as the first consumer of my own work, I am not happy with it :)

I think there's no escaping that some use cases will want all ports for each IP, other use cases will want all endpoints for a given port name, and other use cases will expect the same port for all IPs.

Given that we expect the same ports for all IPs, would representing the ports and IPs separately be easier?

I don't think this is horrible, it's just a bit tedious. It's only a few
LoC, but "annoying". What is the case where someone cares about all the
different ports for a given Pod IP?

On Sun, Feb 22, 2015 at 10:23 PM, Brian Grant notifications@github.com
wrote:

I think there's no escaping that some use cases will want all ports for
each IP, other use cases will want all endpoints for a given port name, and
other use cases will expect the same port for all IPs.

Given that we expect the same ports for all IPs, would representing the
ports and IPs separately be easier?

Reply to this email directly or view it on GitHub
#2585 (comment)
.

Was discussed in person. Use cases:

• Peers of distributed applications, which keep track of peer IPs and use static ports
• Caching systems watching all endpoints, which otherwise need to synchronize updates to N lists
• Monitoring systems, which otherwise need to figure out which endpoints lists to monitor and which ones not to monitor
• Clients that need to get meta-info (API reflection, authentication) from a secondary port on the same IP

On a different topic, in #4440 I proposed using Endpoints for nodes and pods (not services) to replace ResourceLocation, which /proxy, /redirect, /bastion, etc. could be built on top of. This would suggest that we should change the service endpoints paths, to allow more flavors of endpoints.

xref d2iq-archive/kubernetes-mesos#59

multiple ports is in.

Removed from 1.0 and reduced priority. Will summarize later.

@thockin and @bgrant0607 ...

Trying to figure out the current state on "multiple ports", and am confused about what was removed from 1.0. Can you add clarification?

Multi-port services were completed: #6182, #5939. That was the only feature discussed in this issue required for 1.0, and none of the other ideas were implemented.

This issue is largely obsolete. Service/LB changes continue to be discussed in other issues, such as #561.

Access external database from inside K8S

I have workers inside K8S that want to talk to an external database. I have created an external mapping to SQL server like so:

apiVersion: v1
kind: Service
metadata:
  name: database
spec:
  ports:
  - port: 1433
    targetPort: 1433
    protocol: TCP
---
# Because this service has no selector, the corresponding Endpoints
# object will not be created. You can manually map the service to
# your own specific endpoints:
kind: Endpoints
apiVersion: v1
metadata:
  name: database
subsets:
  - addresses:
      - ip: "23.99.34.75"
    ports:
      - port: 1433

DNS seems to work and "database" resolves to a 10 net address which I think points to the external address. BUT pods can't seem to successfully access/login.

Thoughts/suggestions?

@smarterclayton Yep! Thanks for sharing.

That's why I posted here - I also saw your other thread that was circling around this concept.