Do not break CoreOS when deprecating kubernetes-ro #9075

erictune · 2015-06-01T18:40:06Z

CoreOS does not have an equivalent to "kube-addons.sh", and so there are not system secrets created when the cluster is created. So, when some of our CoreOS users run things like the Hazelcast example, or the elasticsearch example, they currently rely on kubernetes-ro to reach the apiserver. When kuberenetes-ro support is removed (#8155), these examples will still work on GCE, because they can use the kubeconfig in their associated secret. But, on CoreOS, it seems they will break.

Possible fix alternatives:

Change Remove ro service #8155 to not remove support for the --readonly_port feature, but default it to off for GCE.
Change all known examples to include instructions on how to generate a secret when setting up the example.
Extend CoreOS cluster setup to include the kube-addons.sh secret-creation functionality.
@lavalamp has a plan that uses a proxy in a sidecar.

Option 1 is nice because it allows us to defer fixing the problem until after 1.0 freeze.

erictune · 2015-06-01T18:40:15Z

@pires

erictune · 2015-06-01T18:49:39Z

My preferred solution is #9076.

erictune · 2015-06-01T18:50:55Z

Under this solution, we would update the docs for hazelcast, elasticsearch, etc, to show how to create a service account for the pods, and tweak the code in the clients to look for the token in the right place.

pires · 2015-06-01T18:57:07Z

@erictune I'm ok with current elasticsearch example approach where we create a service account. Thing is we need a token that's supposedly available in the output of kubectl config view, but in my/our case this is empty. So no token, no service account, no fun.

I'm open to any of the proposed solutions but favor #9076 as well, as it feels more secure than just leaving a read-only port accessible.

lavalamp · 2015-06-02T00:30:17Z

Pods get a default service account, so you don't have to do anything special to get that.

You do have to either start using the auth info from the service account (preferred solution), or if you have a legacy container that you can't modify, run kubectl in a side-car container and use its proxy feature to get a local proxy of apiserver that doesn't need auth for your legacy application. #8155 adds support for this, and I'll soon have an example of how easy it is.

pires · 2015-06-02T09:39:01Z

Pods get a default service account, so you don't have to do anything special to get that.

@brendandburns said the same exact thing, but this is what happens when I try to run hazelcast example with @brendandburns patch on 0.18.0.

2015-06-01 11:56:34.641  INFO 12 --- [           main] c.g.p.h.HazelcastDiscoveryController     : Asking k8s registry at https://kubernetes.default.cluster.local..
2015-06-01 11:56:34.688  WARN 12 --- [           main] c.g.p.h.HazelcastDiscoveryController     : Request to Kubernetes API failed

java.nio.file.NoSuchFileException: /var/run/secrets/kubernetes.io/serviceaccount/token
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
    at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
    at java.nio.file.Files.newByteChannel(Files.java:361)
    at java.nio.file.Files.newByteChannel(Files.java:407)
    at java.nio.file.Files.readAllBytes(Files.java:3152)
    at com.github.pires.hazelcast.HazelcastDiscoveryController.getServiceAccountToken(HazelcastDiscoveryController.java:73)

That secret does not exist. I am not using kube-up or salt or whatever mechanisms you rely on to automate these things. I would like to replicate this in my Vagrant + CoreOS + Kubernetes binaries solution, and for that I need instructions on how does this work.

liggitt · 2015-06-02T15:29:14Z

what does the following show?

kubectl get serviceaccounts --namespace=yournamespace
kubectl get secrets --namespace=yournamespace

Service accounts and token are provisioned using controllers running in the kube-controller-manager. You can start them by passing --service_account_private_key_file="<private rsa key file>" to that command.

pires · 2015-06-02T16:07:16Z

$ kubectl get namespaces
NAME      LABELS    STATUS
default   <none>    Active
$ kubectl get serviceaccounts --namespace=default
Error: no resource "serviceaccounts" has been defined
$ kubectl get secrets --namespace=default
NAME      DATA

Do I need to replicate the steps from https://github.com/GoogleCloudPlatform/kubernetes/blob/master/hack/local-up-cluster.sh#L149 down, namely:

Generate SSL key
Start kube-apiserver with
- --admission_control=NamespaceLifecycle,NamespaceAutoProvision,Li...
- --service_account_key_file=GENERATED_SSL_FILE
Start kube-controller-manager with
- --service_account_private_key_file=GENERATED_SSL_FILE

And it's done?

liggitt · 2015-06-02T16:38:58Z

yes, that would enable service account creation, token generation, and token automounting

pires · 2015-06-02T20:58:25Z

@liggitt you are the man. Going to make a PR for the new version of the hazelcast example.

It probably won't make sense to you, but it works.

2015-06-02 20:56:40.626  INFO 12 --- [           main] com.github.pires.hazelcast.Application   : Starting Application v0.3.1 on hazelcast-ir7ca with PID 12 (/bootstrapper.jar started by root in /)
2015-06-02 20:56:40.689  INFO 12 --- [           main] s.c.a.AnnotationConfigApplicationContext : Refreshing org.springframework.context.annotation.AnnotationConfigApplicationContext@5424f110: startup date [Tue Jun 02 20:56:40 GMT 2015]; root of context hierarchy
2015-06-02 20:56:41.965  INFO 12 --- [           main] o.s.j.e.a.AnnotationMBeanExporter        : Registering beans for JMX exposure on startup
2015-06-02 20:56:41.989  INFO 12 --- [           main] c.g.p.h.HazelcastDiscoveryController     : Asking k8s registry at https://kubernetes.default.cluster.local..
2015-06-02 20:56:53.215  INFO 12 --- [           main] c.g.p.h.HazelcastDiscoveryController     : Found 2 pods running Hazelcast.
2015-06-02 20:56:53.350  INFO 12 --- [           main] c.h.instance.DefaultAddressPicker        : [LOCAL] [someGroup] [3.4.2] Interfaces is disabled, trying to pick one address from TCP-IP config addresses: [10.244.78.3, 10.244.30.2]
2015-06-02 20:56:53.350  INFO 12 --- [           main] c.h.instance.DefaultAddressPicker        : [LOCAL] [someGroup] [3.4.2] Prefer IPv4 stack is true.
2015-06-02 20:56:53.369  INFO 12 --- [           main] c.h.instance.DefaultAddressPicker        : [LOCAL] [someGroup] [3.4.2] Picked Address[10.244.30.2]:5701, using socket ServerSocket[addr=/0:0:0:0:0:0:0:0,localport=5701], bind any local is true
2015-06-02 20:56:53.773  INFO 12 --- [           main] com.hazelcast.spi.OperationService       : [10.244.30.2]:5701 [someGroup] [3.4.2] Backpressure is disabled
2015-06-02 20:56:53.778  INFO 12 --- [           main] c.h.spi.impl.BasicOperationScheduler     : [10.244.30.2]:5701 [someGroup] [3.4.2] Starting with 2 generic operation threads and 2 partition operation threads.
2015-06-02 20:56:53.974  INFO 12 --- [           main] com.hazelcast.system                     : [10.244.30.2]:5701 [someGroup] [3.4.2] Hazelcast 3.4.2 (20150326 - f6349a4) starting at Address[10.244.30.2]:5701
2015-06-02 20:56:53.975  INFO 12 --- [           main] com.hazelcast.system                     : [10.244.30.2]:5701 [someGroup] [3.4.2] Copyright (C) 2008-2014 Hazelcast.com
2015-06-02 20:56:53.976  INFO 12 --- [           main] com.hazelcast.instance.Node              : [10.244.30.2]:5701 [someGroup] [3.4.2] Creating TcpIpJoiner
2015-06-02 20:56:53.977  INFO 12 --- [           main] com.hazelcast.core.LifecycleService      : [10.244.30.2]:5701 [someGroup] [3.4.2] Address[10.244.30.2]:5701 is STARTING
2015-06-02 20:56:54.102  INFO 12 --- [cached.thread-2] com.hazelcast.nio.tcp.SocketConnector    : [10.244.30.2]:5701 [someGroup] [3.4.2] Connecting to /10.244.78.3:5701, timeout: 0, bind-any: true
2015-06-02 20:56:54.169  INFO 12 --- [cached.thread-2] c.h.nio.tcp.TcpIpConnectionManager       : [10.244.30.2]:5701 [someGroup] [3.4.2] Established socket connection between /10.244.30.2:53817 and 10.244.78.3/10.244.78.3:5701
2015-06-02 20:57:01.133  INFO 12 --- [ration.thread-0] com.hazelcast.cluster.ClusterService     : [10.244.30.2]:5701 [someGroup] [3.4.2]

Members [2] {
    Member [10.244.78.3]:5701
    Member [10.244.30.2]:5701 this
}

2015-06-02 20:57:03.156  INFO 12 --- [           main] com.hazelcast.core.LifecycleService      : [10.244.30.2]:5701 [someGroup] [3.4.2] Address[10.244.30.2]:5701 is STARTED
2015-06-02 20:57:03.156  INFO 12 --- [           main] com.github.pires.hazelcast.Application   : Started Application in 22.92 seconds (JVM running for 23.765)

pires · 2015-06-02T20:59:39Z

I will also need to update the CoreOS docs.

thockin · 2015-06-02T23:17:59Z

@kelseyhightower @pires Please take a look at this ASAP. I don't really want to go to 1.0 with CoreOS support broken.

pires · 2015-06-03T09:08:17Z

@thockin I am updating it as we speak.

pires · 2015-06-03T10:52:00Z

@thockin @erictune @AntonioMeireles I have questions.

Moved questions below to a #9178

~~1. Care if I remove single-node instructions? I don't think this makes sense anymore, unless someone is going to use hyperkube.~~

~~2. master.yaml needs instructions to generate service-account private key file, so it will depend on openssl. Any objections?~~

~~3. We should add DNS integration as many examples depend on it. Objections?~~

~~4. I'm moving aws folder from getting-started-guides folder, to getting-started-guides/coreos, in order to match how we did azure.~~
~~Speaking of which @errordeveloper @squillace @chanezon @crossorigin do you agree with questions 1, 2 and 3? Also, can we sync effort to get it homogeneous on all different CoreOS approaches?~~

AntonioMeireles · 2015-06-03T10:53:31Z

no objections on my part.

thockin · 2015-06-17T16:19:29Z

Status on this?

pires · 2015-06-18T16:54:24Z

I believe I will have the PR ready later today.

pires · 2015-06-19T09:38:10Z

Due to increased workload, I have to postpone this to next week.

yifan-gu · 2015-06-19T22:18:37Z

cc @yifan-gu

thockin · 2015-06-24T20:30:54Z

Are there still actions items open on this?

pires · 2015-06-24T22:58:45Z

Yes, just need to implement a way in the master.yaml to generate SSL key to use and we're done.

liggitt · 2015-06-24T23:02:28Z

See also #10264 in progress to include the root CA. You'll likely want to provide a CA for inclusion as well

pires · 2015-06-24T23:11:19Z

I am not aware of this root CA and tbh I've had enough breaking changes in the last couple weeks. It's amazing to see Kubernetes getting there, but troublesome to keep up in the CoreOS world.

As for me, my kubernetes-vagrant-coreos-cluster works right now by just providing the SSL key file, generated with openssl genrsa -out kube-serviceaccount.key 2048 2>/dev/null, to my master VM and run:

kube-apiserver \
  --service_account_key_file=/tmp/kube-serviceaccount.key \
  --service_account_lookup=false \
  --admission_control=NamespaceLifecycle,NamespaceAutoProvision,LimitRanger,SecurityContextDeny,ServiceAccount,ResourceQuota \
  --...

kube-controller-manager \
  --service_account_private_key_file=/tmp/kube-serviceaccount.key \
  --...

If this works @liggitt @thockin I'm happy to push the PR tomorrow morning (GMT). Otherwise, and mostly due to the lack of energy to do it, I can't deliver anytime soon.

liggitt · 2015-06-24T23:22:45Z

There are basically three levels of security talking to the API:

http, read only. This is what is going away
https, authenticated via token, client don't verify server's certificate. This is what you get by providing the service account key
https with server cert verification. This is enabled by providing the root CA bundle for clients inside pods to use to verify the server's cert

Enabling service account tokens is still worthwhile even if the CA part doesn't happen yet

pires · 2015-06-24T23:37:01Z

@liggitt yes, service-accounts are mandatory. I will push the PR tomorrow. Thanks for clarifying.

thockin · 2015-07-01T01:56:30Z

Status on this? I want to get it off the books...

pires · 2015-07-01T10:08:31Z

@thockin for real? #10344

Added instrumentation and configuration for service account tokens. Fixes kubernetes#9075

thockin · 2015-07-01T23:42:23Z

Excellent! Is this issue resolved now?

pires · 2015-07-01T23:54:50Z

@thockin the PR was up for a week but it got lost in the pipe. 0.19.3 with service-account support is done for CoreOS (tested on Vagrant, AWS and GCE).

pires · 2015-07-01T23:55:37Z

I will wait for v1 to come out before updating again. I want to add root CA and whatever may be needed then.

erictune added priority/backlog Higher priority than priority/awaiting-more-evidence. team/cluster labels Jun 1, 2015

erictune added this to the v1.0-candidate milestone Jun 1, 2015

goltermann assigned goltermann and kelseyhightower and unassigned goltermann Jun 2, 2015

pires mentioned this issue Jun 3, 2015

Revamp CoreOS getting started guides #9178

Closed

goltermann modified the milestones: v1.0, v1.0-candidate Jun 10, 2015

pires mentioned this issue Jun 25, 2015

Updates CoreOS cloud-configs to use Kubernetes 0.19.3. #10344

Merged

pires added a commit to pires/kubernetes that referenced this issue Jul 1, 2015

Updates CoreOS cloud-configs to use Kubernetes 0.19.3.

db00532

Added instrumentation and configuration for service account tokens. Fixes kubernetes#9075

zmerlynn closed this as completed in #10344 Jul 1, 2015

gaga-github mentioned this issue Jan 29, 2016

kubernetes 1.0.3 CentOS7, Request to Kubernetes API failed pires/hazelcast-kubernetes#8

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Do not break CoreOS when deprecating kubernetes-ro #9075

Do not break CoreOS when deprecating kubernetes-ro #9075

erictune commented Jun 1, 2015

erictune commented Jun 1, 2015

erictune commented Jun 1, 2015

erictune commented Jun 1, 2015

pires commented Jun 1, 2015

lavalamp commented Jun 2, 2015

pires commented Jun 2, 2015

liggitt commented Jun 2, 2015

pires commented Jun 2, 2015

liggitt commented Jun 2, 2015

pires commented Jun 2, 2015

pires commented Jun 2, 2015

thockin commented Jun 2, 2015

pires commented Jun 3, 2015

pires commented Jun 3, 2015

AntonioMeireles commented Jun 3, 2015

thockin commented Jun 17, 2015

pires commented Jun 18, 2015

pires commented Jun 19, 2015

yifan-gu commented Jun 19, 2015

thockin commented Jun 24, 2015

pires commented Jun 24, 2015

liggitt commented Jun 24, 2015

pires commented Jun 24, 2015

liggitt commented Jun 24, 2015

pires commented Jun 24, 2015

thockin commented Jul 1, 2015

pires commented Jul 1, 2015

thockin commented Jul 1, 2015

pires commented Jul 1, 2015

pires commented Jul 1, 2015

Do not break CoreOS when deprecating kubernetes-ro #9075

Do not break CoreOS when deprecating kubernetes-ro #9075

Comments

erictune commented Jun 1, 2015

erictune commented Jun 1, 2015

erictune commented Jun 1, 2015

erictune commented Jun 1, 2015

pires commented Jun 1, 2015

lavalamp commented Jun 2, 2015

pires commented Jun 2, 2015

liggitt commented Jun 2, 2015

pires commented Jun 2, 2015

liggitt commented Jun 2, 2015

pires commented Jun 2, 2015

pires commented Jun 2, 2015

thockin commented Jun 2, 2015

pires commented Jun 3, 2015

pires commented Jun 3, 2015

AntonioMeireles commented Jun 3, 2015

thockin commented Jun 17, 2015

pires commented Jun 18, 2015

pires commented Jun 19, 2015

yifan-gu commented Jun 19, 2015

thockin commented Jun 24, 2015

pires commented Jun 24, 2015

liggitt commented Jun 24, 2015

pires commented Jun 24, 2015

liggitt commented Jun 24, 2015

pires commented Jun 24, 2015

thockin commented Jul 1, 2015

pires commented Jul 1, 2015

thockin commented Jul 1, 2015

pires commented Jul 1, 2015

pires commented Jul 1, 2015