Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Demo service account to replace kubernetes-ro use cases in demos #7444

Closed
erictune opened this issue Apr 28, 2015 · 15 comments
Closed

Demo service account to replace kubernetes-ro use cases in demos #7444

erictune opened this issue Apr 28, 2015 · 15 comments
Labels
priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle.
Milestone
v1.0

Comments

@erictune
Copy link
Member

Problem

  • Several of our examples use kubernetes-ro.
    • cassandra, hazelcast, and rethinkdb examples use it for getting endpoints
    • phabricator examples uses it for getting pods
    • other examples may use it, pending further testing
    • probably other config not in our repo probably uses it.
  • We have use cases where nodes/pods need to reach the master over the internet, rather than over a private network. In those cases, we want to disable the kubernetes-ro service. Otherwise, it leaves cluster information exposed to the world.

Options

  1. disable kubernetes-ro and let examples be broken on those types of clusters described above.
    • problem: breaks examples, and provides inconsistent cluster semantics for config authors to deal with.
  2. something on the node proxies the kubernetes-ro service from the pod over ssl-with-auth to the master.
    • problem: introduces a mechanism with apparently limited general utility
  3. change all the examples on the site to use the http://kubernetes service, using an credential which allows readonly access.
    • problem: We could change the ones on this site, but there may be others
    • problem: It is a non-trivial amount of work to get all the examples working, as they use a variety of languages and libraries, which may not support https with token-based auth as well as our pkg/client.

Recommendation: Last option. It is a uses an existing, general mechanism (secrets) and keeps all clusters working the same.

Proposal

  • Add a "readonly-user"
    • adds a service account token/name at cluster startup, in the same way other service accounts have been added.
    • adds a secret containing this token.
  • Set policy for this service account to allow reading (events, endpoints, pods, services, rcs)
  • Merge ServiceAccounts #7101 by @liggitt
  • For each example currently using kubernetes-ro:
    1. update each pod to set "pod.spec.serviceAccount" to "readonly-user".
      • with ServiceAccounts #7101, causes pod to automatically be filled in with a secret volume containing a token for the readonly-user account.
    2. update places that use http and kubernetes-ro service to use https and kubernetes service.
    3. in those same places, add a token to header of the request.
    4. in those same places, set https options to use insecure connection. (Or find a way to inject the master certs into the pod)

There are a number of details to sort out once there is general agreement on the general approach.
@erictune
Copy link
Member Author

@cjcullen @roberthbailey

@erictune erictune mentioned this issue Apr 28, 2015
Closed
@cjcullen
Copy link
Member

SGTM.

I think the eventual right answer for iv is to have a canonical way to mount the cluster CA cert in the pod.

@a-robinson a-robinson added priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. team/cluster labels Apr 28, 2015
@a-robinson a-robinson added this to the v1.0 milestone Apr 28, 2015
@roberthbailey
Copy link
Contributor

This looks right to me. It would be nice if mounting the cluster CA could be done in a way that way mostly transparent to the callers. What about embedding the cluster CA in the kubeconfig file (like we do for kubectl) when it's created (see https://github.com/GoogleCloudPlatform/kubernetes/blob/master/cluster/gce/configure-vm.sh#L277)?

@erictune erictune mentioned this issue May 1, 2015
Closed
@roberthbailey roberthbailey added the sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle. label May 6, 2015
@brendandburns brendandburns self-assigned this May 7, 2015
@lavalamp lavalamp mentioned this issue May 8, 2015
Merged
7 tasks
@davidopp davidopp mentioned this issue May 11, 2015
Closed
@brendandburns
Copy link
Contributor

So this issue mentions #7343, but I think it actually requires #7101 to be merged...

@liggitt
Copy link
Member

liggitt commented May 11, 2015

#7101 is in (final?) review, and comments are slowing down. I may have just worn people out, but I'm optimistic the first pass is getting close

@lavalamp
Copy link
Member

@brendanburns This overlaps a lot with #5921-- I'm working on the Add a "readonly-user" step, so chat at me before you start working on it.

This was referenced May 18, 2015
Merged
Merged
Merged
@pires
Copy link
Contributor

pires commented May 19, 2015

@brendanburns @erictune I merged Brendan's PR and will update the Docker image and the example accordingly. But I wanted to know if you want to move the Dockerfile to the examples/hazelcast and at the same time control the image version on your side instead of pires/hazelcast-k8s?

This was referenced May 26, 2015
Merged
Merged
Closed
Merged
Merged
@brendandburns
Copy link
Contributor

@pires we'd love to take over the Dockerfile for the example, it will def. make it easier to make changes in the future.

@pires
Copy link
Contributor

pires commented May 27, 2015

@brendanburns I'm just worried about the current example not working with any of the Kubernetes releases. 0.18.0 was tagged but removed so I can't test it today as well and update it.

Anyway, do you need anything from me to take over the Dockerfile?

@brendandburns
Copy link
Contributor

Yeah, we should wait until post 0.18.0 before rebuilding that image.

@pires
Copy link
Contributor

pires commented Jun 1, 2015

@brendandburns I'm still getting errors with 0.18.0.

2015-06-01 11:56:34.641  INFO 12 --- [           main] c.g.p.h.HazelcastDiscoveryController     : Asking k8s registry at https://kubernetes.default.cluster.local..
2015-06-01 11:56:34.688  WARN 12 --- [           main] c.g.p.h.HazelcastDiscoveryController     : Request to Kubernetes API failed

java.nio.file.NoSuchFileException: /var/run/secrets/kubernetes.io/serviceaccount/token
    at sun.nio.fs.UnixException.translateToIOException(UnixException.java:86)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:102)
    at sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:107)
    at sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:214)
    at java.nio.file.Files.newByteChannel(Files.java:361)
    at java.nio.file.Files.newByteChannel(Files.java:407)
    at java.nio.file.Files.readAllBytes(Files.java:3152)
    at com.github.pires.hazelcast.HazelcastDiscoveryController.getServiceAccountToken(HazelcastDiscoveryController.java:73)

@pires
Copy link
Contributor

pires commented Jun 1, 2015

It wasn't clear but it seems one needs to create the secret as per Elasticsearch example.

@pires pires mentioned this issue Jun 1, 2015
Closed
@pires
Copy link
Contributor

pires commented Jun 1, 2015

Well, I'm gonna need help with this because

$ kubectl config view
apiVersion: v1
clusters: []
contexts: []
current-context: ""
kind: Config
preferences: {}
users: []

/cc @AntonioMeireles

@pires
Copy link
Contributor

pires commented Jun 2, 2015

OK just fixed it both in my Vagrant + CoreOS thing and Hazelcast. Going to open a couple PRs, one for Hazelcast example and another for CoreOS documentation.

@pires pires mentioned this issue Jun 2, 2015
Merged
@lavalamp
Copy link
Member

lavalamp commented Jun 5, 2015

I think this is fixed; canonical examples of upgrading are in contrib/prometheus (workload agnostic proxy method) and in contrib/for-tests/network-tester (go client construction method).

@lavalamp lavalamp closed this as completed Jun 5, 2015
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
priority/important-soon Must be staffed and worked on either currently, or very soon, ideally in time for the next release. sig/cluster-lifecycle Categorizes an issue or PR as relevant to SIG Cluster Lifecycle.
Projects
None yet
Milestone
v1.0
Development

No branches or pull requests
8 participants
@lavalamp @liggitt @pires @brendandburns @a-robinson @cjcullen @roberthbailey @erictune