Host port is more or less a hack. For real services at scale, you're going to want to run through load balancers, etc. Even for small scales, you'll most likely want some frontend nginx type thing to distribute requests. It's unlikely that you want to send raw requests right to your minions.

We definitely need to make it easier to get data/requests into the system, like telling a DNS system where to find our services (in the works, I think?).

Short answer: no. HTTP is but one protocol that is supported.

Longer answer: we need a more formalized model about externalizing ports
which would allow you to do what you want without doing it how you
describe. We started this discussion, it has just been back-burnered for a
while, as we sort out 800 other things :)

I know that is unsatisfying, but please know that we have some ideas, and I
think it will turn out great. Discussion of some of this on #1161

On Tue, Oct 14, 2014 at 8:24 PM, Daniel Smith notifications@github.com
wrote:

Host port is more or less a hack. For real services at scale, you're going
to want to run through load balancers, etc. Even for small scales, you'll
most likely want some frontend nginx type thing to distribute requests.
It's unlikely that you want to send raw requests right to your minions.

We definitely need to make it easier to get data/requests into the system,
like telling a DNS system where to find our services (in the works, I
think?).

Reply to this email directly or view it on GitHub
#1799 (comment)
.

#1161 discussion is pretty good. In my opinion I think providing support for both option 1 and 2 would be needed.

For now we need to build something custom.
In its current form does Kubernetes have some kind of plugin system where custom built plugins can be used to extend it's existing functionality? and if so any documentation or existing plugins to look at?

No plugin system yet-- see #991.

One thing you can do is query for service endpoints, either directly, or by visiting /api/v1beta1/redirect/services/, which will send back an HTTP redirection header. (You could just extract the destination if you don't have an HTTP service, of course.)

Not sure if I am following, how is querying services going to work to allow for many pods to share same port on same public ip?
Are you saying that for each pod I add a service with some random unique host port ? and then use that service to retrieve the real port and ip ? But with that I would still need a proxy above that which would forward to based on hostname/domain to proper internal pod ip and port, right ?

many pods to share same port on same public ip?

The idea is that the service is the public IP; incoming requests are load-balanced across all pods in the service. Pods will never literally "share same port on same public IP".

Are you talking about doing some analog of running a bunch of different websites on a box, and having nginx (or apache) dispatch stuff based on the requested URL? If that's what you want to do, you would accomplish that by running nginx in one pod and routing all requests to it, with it dispatching appropriately; if it sees a request for foo.com, it forwards on to kubernetes' foo-service, which is served by N foo-pods. If we make our services visible via DNS you should even be able to write a short rule.

It's questionable that you should do it that way, though-- instead of configuring your traffic source to send to your nginx pod it might make sense to teach it about kubernetes services and just route stuff to the right place to begin with.

Basically, the gateway to your system needs to be able to distribute requests. We need to make that easy information to get, because we're not supplying the gateway.

Yes that's exactly what I was talking about, running bunch of different websites on same box. I totally understand that might not be standard use case for kubernetes.
Thats what I was proposing as one option in my intial use case, to have nginx proxy on each node, forwarding to internal pods based on the host name.

I understand that for large traffic cases this might not be optimal and you would want to have the forwarding and load balancing done outside of nodes on dedicated hardware, but there could be a lot of use cases for a lot of smaller traffic pod instances where it could be enough without more complex LB setups. Like in my case, each of the pod instances will receive very little traffic since it is only used for development and testing by developers.

Thats what I was proposing as one option in my intial use case, to have nginx proxy on each node, forwarding to internal pods based on the host name.

For small scales, you should be able to have an nginx thingy like this on a single machine (not necessarily a node). For large scales, you shouldn't do this. :) At any rate, I don't think you should do this setup on every node.

I agree about the large scales.

Either way I think a lot of kubernetes users are probably going to run into a problem where they want to host multiple pods occupying the same port, I think it would be useful it there was some kind of default solution provided by kubernetes. It would be no different then how current node/minion proxy works its just instead of routing and load balancing based on the port you would route based on the domain/hostname.
Proxy is already there on each node so it's capability would just need to be expanded.

I think what you're really asking for is HTTP routing. I think that's probably more complicated than we want to build into our proxy, but we would be happy to see proposals about standard ways to set it up.

Well yeah it's both an http router and proxy since you need to get the requests inside private network.

I was thinking about what it would take to introduce the new http router capability. Since I would not be sure that my changes would end up in the main kubernetes code base, I would want to architect it in certain way so that it minimizes the amount of work it would take me to keep it in-sync with the main kubernetes upstream repo.

This is all very preliminary and I haven't spent much time really digging into the code to see if it's all possible.

First thing that I started looking into is the Service definition part. So far I have thought of 3 options.

OPTION 1: Expand existing "Service" definition

Modify existing service definition
(https://godoc.org/github.com/GoogleCloudPlatform/kubernetes/pkg/api/v1beta1#Service) to support a new property named "Router:" It is a new entity/type containing additional routing information like host, path ..etc
Example of expanded definition would be like so:

{
  "id": "wordpress1",
  "kind": "Service",
  "apiVersion": "v1beta1",
  "port": 80,
  "containerPort": 80,
  "router": {
    "host": "subdomain.domain.com",
    "path": "/path/.*"
  }
  "selector": {
    "name": "wordpress1"
  }
}

This option is probably easier to implement but would probably be harder to keep in sync with upstream repo.

OPTION 2: Create new "Router" definition

This would be a new type which would define the route (host, path) and point to the existing Service definition.
It would keep the routing definition part separate from service (proxy) part and that way keeping future updates easier.

Example :

# route definition
{
    "id" : "wordpress-route1"
    "kind": "Route",
    "host": "subdomain.domain.com",
    "path": "/path/.*",
    "selector": {
        "name": "wordpress-service1"
    }
}
# service definiton 
{
  "id": "wordpress-service1",
  "kind": "Service",
  "apiVersion": "v1beta1",
  "port": 80,
  "containerPort": 80,
  "selector": {
    "name": "wordpress-pod1"
  }
}

This option would probably require more work but I think it would keep it cleaner and more flexible

OPTION 3: Plugin system

This would introduce a plugin property for most of the existing types. Whenever kubernetes is reading or processing one of the definitions and encounters a plugin property it would instantiate the plugin and then just hand over the data to the plugin to handle it. There could also be different events issued based on when in the flow plugin is called.
All of the instructions and definitions would be left up to the plugin creator. This type of plugin system could support other types of services and features and not only routing.

Example :

# plugin definition for routes
{
    "id" : "route-plugin-1"
    "kind" : "Plugin",
    "host": "subdomain.domain.com",
    "path": "/path/.*",
    "selector": {
        "name": "wordpress-service1"
    }
    "someOtherFiled": "someValue"
}

# plugin definition for pods
{
    "id" : "pod-plugin-1"
    "kind" : "Plugin",
    "customData": "someValue"
}

# service definiton 
{
  "id": "wordpress-service1",
  "kind": "Service",
  "apiVersion": "v1beta1",
  "port": 80,
  "containerPort": 80,
  "plugin": "route-plugin-1"
  "selector": {
    "name": "wordpress-pod1"
  }
}

# pod definiton 
{
  "id": "redis-master-2",
  "kind": "Pod",
  "apiVersion": "v1beta1",
  "desiredState": {
    "manifest": {
      "version": "v1beta1",
      "id": "wordpress-pod1",
      "containers": [{
        "name": "master",
        "image": "dockerfile/wordpress",
        "ports": [{
          "containerPort": 80,
          "hostPort": 80
        }]
      }]
    }
  },
  "plugin": "pod-plugin-1"
  "labels": {
    "name": "wordpress-pod1"
  }
}

With either option I could expand the existing proxy capability or build a new reverse proxy that would handle the routing. Building a new separate one would probably make it easy to sync with main repo but it could take longer. I was also looking at http://www.vulcanproxy.com/library.html and it looks like a good library for it.

IMO, this should be done by making a pod with nginx/apache in it and a forwarding configuration that can direct traffic to k8s services.

Ok so option (4 would be nginx or vulcand way:

1. Create a new k8s service with unique or random port since we can't have all services running on host port 80.
2. Service info gets stored in k8s etcd.
3. There is an nginx w/ confd or vulcand container which would run on each node/minion and watch on etcd for new services created.
4. Once it is picked up it would create a proxy route to k8s service and make sure that random port is only accessed by nginx container. Can't have users just trying different ports to figure out websites.

Only thing is you have two types of proxying going , one from nginx to k8s proxy and another from k8s proxy to the actual container. I don't know if this would impact responsiveness.

That's not quite right.

1. We now have IP-per-service, so port 80 for every web service makes sense.
2. Make a frontend pod, running nginx or whatever else you want to use to distribute traffic.
3. frontend pod should be scaled according to traffic, NOT running on every host.
4. load balance incoming requests to frontend pod. Tools for this are in the "under construction" stage. When running on GCE, we have the concept of external services but this needs some attention. How to get your traffic to your frontend is going to depend on your exact environment.
5. frontend pod distributes requests to other k8s services via some sort of substitution rule. For this to be easy, we need some way to give frontend pod info about the other k8s services running.

For IP-per-service how do I enable that? do I not set host "port" in service definition or is there a special property?

We're enabling IP-per-service as we speak. Head is borken right now, though, so I'd wait a bit before trying :)

K thanks.

Also on another note It would also be great if there was property in each one of the definitions which would allow to pass in arbitrary data that would be passed around along with pod or services thru k8s.
This way it can be read back by external system to perform additional functionality without relying on any external mapping or key-value storage.

it could be called "meta" or "data" like :

{
  "id": "wordpress-service1",
  "kind": "Service",
  "apiVersion": "v1beta1",
  "port": 80,
  "containerPort": 80,
  "meta": "whatever text or serialized data we want in here"
  "selector": {
    "name": "wordpress-pod1"
  }
}

We are adding Annotations in v1beta3, which will do what you want.

That's great :)

For those who might have similar needs as I do, I've created a docker container that acts as a reverse proxy for kubernetes pods, allowing multiple pods to share same public ip. It's using nginx and confd to to read the k8s etcd specs and configure the nginx routing to the k8s service ip.

All you have to do is install the Container and add annotations to your k8s service.

It's not a perfect solution and it's still in its early stages but it gives basic capability to host multiple pod ports on same node.
It requires kubernetes ip-per-service and annotation capabilities, so your kubernetes binaries needs to be compiled with that.
If you need it you can also use my own kubernetes binaries which I try to keep up in sync often with latest main repo:
https://github.com/darkgaro/kubernetes-binaries

To find out more on how to use this reverse proxy you can check out the main github repo page :
https://github.com/darkgaro/kubernetes-reverseproxy

And get the docker image at:
https://registry.hub.docker.com/u/darkgaro/kubernetes-reverseproxy/

To quickly run the proxy do:

docker run -d -e CONFD_ETCD_NODE=<ETCD-IP>:<ETCD-PORT> -t -p 80:80 darkgaro/kubernetes-reverseproxy

##ETCD-IP## and ##ETCD-PORT## is one where kubernetes is installed and using.

You will also need to add following annotations to your k8s service definitions

"annotations":{
    "kubernetesReverseproxyHost":"some.host.name",
    "kubernetesReverseproxyPort":"port number"
    }

This has lost the meaning of the original bug - if there is work to do here, please reopen and retitle this or open a new issue.

The solution nowadays seems to be the Ingress resource