Add hyper container runtime #13079

feiskyer · 2015-08-24T06:33:42Z

Add hyper container runtime for kubernetes.

Hyper is a Hypervisor-agnostic Docker Engine that allows you to run Docker images on any hypervisor (KVM, Xen, etc.).

Technically speaking, Hyper = Hypervisor + Kernel + Docker Image

By containing applications within separate VM instances and kernel spaces, Hyper is able to offer an excellent Hardware-enforced Isolation, which is much needed in multi-tenant environments.

Hyper also promises Immutable Infrastructure by eliminating the middle layer of Guest OS, along with the hassle to configure and manage them.

See https://hyper.sh for more documentation.

Task list:

There will be another commit on kubelet logs, since hyper doesn't support hyper log now.

Note that host network, socat-based port-forwarding and privileged containers is not supported.

k8s-bot · 2015-08-24T06:35:05Z

Can one of the admins verify that this patch is reasonable to test? (reply "ok to test", or if you trust the user, reply "add to whitelist")

If this message is too spammy, please complain to ixdy.

thockin · 2015-08-24T16:42:05Z

Cool!

WRT this as a patch - I think the hardest part is going to be sure that we're happy with the interface between core kubelet and runtimes. Since it's been on my plate this week, things like the networking tie-ins to runtimes is less-than-obvious.

Now that we have potentially 3 implementations of runtime, we should pause and think about abstractions.

feiskyer · 2015-08-25T11:40:37Z

@thockin The interface between core kublet and runtimes is ok except networking. Although it works well for current kubernetes networking model, it is very difficult to do things like multi-tenant networking.

bgrant0607 · 2015-08-26T03:36:15Z

cc @kubernetes/goog-node

yujuhong · 2015-08-26T19:51:18Z

Now that we have potentially 3 implementations of runtime, we should pause and think about abstractions.

We have some pod-level abstraction in kubelet/container, but we'd like to improve them.

We probably want to discuss this PR in the node team meeting, so I am assigning it to @dchen1107 for now.

k8s-github-robot · 2015-08-27T23:07:03Z

Labelling this PR as size/XXL

feiskyer · 2015-08-29T10:35:19Z

@yujuhong @dchen1107

Have you discussed this PR in the node team meeting?

pmorie · 2015-08-29T17:40:50Z

pkg/kubelet/hyper/hyper.go

+	for name, volume := range volumeMap {
+		glog.V(4).Infof("Hyper: volume %s %s", name, volume.GetPath())
+		v := make(map[string]string)
+		v["name"] = name


Nit: might be good to make "name","source", etc constants to avoid magic strings all over.

@pmorie Thanks for suggestion, commits updated.

yujuhong · 2015-08-31T14:45:34Z

@feiskyer, @dchen1107 was out last week. We'll talk about this either today or tomorrow.

feiskyer · 2015-09-07T07:17:10Z

@thockin @dchen1107 @yujuhong What's the progress of node team meeting?

dchen1107 · 2015-09-09T21:56:42Z

Thanks for your pr, and really nice work by implementing today's Container Runtime interface. But I am hesitated to take this as is for now:

I am not very happy with today's Runtime API. You must notice it already you have to duplicate most of code for SyncPod for both rkt and docker support including all those Kubernetes defined policy: StartPolicy, PullPolicy, etc. Runtime API is in today's shape due to many reasons, and it is not final one yet. One reason is mismatching network model between docker and rkt. Both docker and rkt are evolving, which help us to get right level of abstraction between Kubernetes and runtime. But not today.
Can we make this as part of contrib (https://github.com/kubernetes/contrib) instead of Kubernetes core? There are many features missing here, and you already listed a bunch of them. But there should be more. How does cAdvisor integrate with this model?
We are working on adding a bunch of resource management related features: such as qos, overcommit. What are the impact with this new Runtime? cc @bgrant0607 @davidopp
Who should be the maintainer of this runtime? Can we have unittests for this? Can we have integration tests? Can we enable e2e for a new runtime. CoreOS rkt team is working with us closely to have all those tests I listed above. How about this one?

On another hand, I love this since I believe it could enable new usecases for the customer. Is it possible you give a demo to our community hangout every Friday 10:00am? If there are big demanding on this, we could re prioritize this?

cc/ @bgrant0607 @thockin @brendandburns

brendandburns · 2015-09-09T22:14:15Z

First off, let me say that it would be awesome to have support for hyper and I would love to see a demo of this at community meeting sometime.

I think the right way forward for this PR (and other PRs like this in the future) is to implement a client/server version of the container runtime, so that container runtimes can be run outside of the kubelet process and implementations can be built in the kubernetes/contrib tree rather than the core tree.

To that end, I have filed #13768.

@feiskyer if you are up for it, we would be happy to merge an implementation of such a client/server interface (if you can build it...)

And then you can use that as the basis for integrating hyper into kubernetes. (and it can also be the basis for additional experimental container runtimes)

Let me know what you think of this, I would be happy to help with design, reviews, etc.

Best!
--brendan

feiskyer · 2015-09-10T00:40:20Z

I am not very happy with today's Runtime API. You must notice it already you have to duplicate most of code for SyncPod for both rkt and docker support including all those Kubernetes defined policy: StartPolicy, PullPolicy, etc. Runtime API is in today's shape due to many reasons, and it is not final one yet. One reason is mismatching network model between docker and rkt. Both docker and rkt are evolving, which help us to get right level of abstraction between Kubernetes and runtime. But not today.

I think so, there should be an abstraction that most pod management codes could be reused accross all container runtimes.

Can we make this as part of contrib (https://github.com/kubernetes/contrib) instead of Kubernetes core? There are many features missing here, and you already listed a bunch of them. But there should be more. How does cAdvisor integrate with this model?

Currently, kubelet container runtime is strongly binding to Kubernetes core. I don't think it's easy to make a runtime as part of contrib. If you have any way, please tell me.

cAdvisor is not working with hyper, since hyper is vm-based. I think a new plugin/agent for Heapster is more appropriate.

We are working on adding a bunch of resource management related features: such as qos, overcommit. What are the impact with this new Runtime? cc @bgrant0607 @davidopp

Hyper is still on quick evolving, more features can be added to hyper if needed.

Who should be the maintainer of this runtime? Can we have unittests for this? Can we have integration tests? Can we enable e2e for a new runtime. CoreOS rkt team is working with us closely to have all those tests I listed above. How about this one?

Hyper team is the maintainer of this runtime. We could work togather on setting up all the tests needed.

On another hand, I love this since I believe it could enable new usecases for the customer. Is it possible you give a demo to our community hangout every Friday 10:00am? If there are big demanding on this, we could re prioritize this?

I could show some hyper demos on community hangout, please tell me how to join community hangout. By the way, is every Friday 10:00am US-Pacific time?

cc/ @bgrant0607 @thockin @brendandburns @dchen1107

feiskyer · 2015-09-10T00:46:40Z

@brendandburns It's a great idea of making a c/s version of the container runtime and I'm very glad to be up for it.

dchen1107 · 2015-09-11T07:23:00Z

Yes, community hangout is every Friday 10:00am US-Pacific time. I can put your demo to hangout schedule. How about Sept 18th?

feiskyer · 2015-09-11T07:30:52Z

@dchen1107 Time is ok, how can I join community hangout?

dchen1107 · 2015-09-11T18:37:31Z

@feiskyer I scheduled your demo on Sept 18th, you can access it at https://docs.google.com/document/d/1VQDIAB0OqiSjIHI8AWMvSdceWhnz56jNpZrLs6o7NJY/edit#

Can you access this link: https://plus.google.com/hangouts/_/google.com/k8s-community?authuser=0

feiskyer · 2015-09-15T00:49:11Z

@dchen1107 Thanks.

dchen1107 · 2015-09-17T21:30:03Z

@feiskyer I am assuming you cannot access hangout link I provided above without any issue, and the demo is tomorrow at 10:am PST.

Also could you please send me your personal email address to dawnchen@google.com? I want to invite you to our discussion on Container Runtime interface. Thanks!

feiskyer · 2015-09-17T23:36:08Z

@dchen1107 Thanks for tip, I can access hangout without any issue.

Hyper is a hypervisor-agnostic docker engine, see https://hyper.sh for more documentation.

feiskyer · 2015-12-23T04:59:49Z

I'm closing this because of #17048

googlebot added the cla: yes label Aug 24, 2015

feiskyer force-pushed the kubelet/hyper branch 4 times, most recently from 9fb80a1 to 38dbabb Compare August 24, 2015 07:39

feiskyer force-pushed the kubelet/hyper branch from 38dbabb to 91df0e6 Compare August 25, 2015 11:27

feiskyer force-pushed the kubelet/hyper branch 2 times, most recently from 9d9ac63 to 29d9564 Compare August 26, 2015 02:01

feiskyer changed the title ~~[WIP] Add hyper container runtime~~ Add hyper container runtime Aug 26, 2015

yujuhong added [deprecated--DO NOT USE]old-team/master sig/node Categorizes an issue or PR as relevant to SIG Node. labels Aug 26, 2015

yujuhong assigned dchen1107 Aug 26, 2015

k8s-github-robot added the size/XXL Denotes a PR that changes 1000+ lines, ignoring generated files. label Aug 27, 2015

pmorie reviewed Aug 29, 2015
View reviewed changes

feiskyer force-pushed the kubelet/hyper branch from 29d9564 to 06e0e72 Compare August 30, 2015 05:50

feiskyer force-pushed the kubelet/hyper branch from 06e0e72 to 8b62cb7 Compare September 6, 2015 08:28

feiskyer force-pushed the kubelet/hyper branch from 8b62cb7 to d09ecb8 Compare September 9, 2015 13:24

feiskyer mentioned this pull request Sep 10, 2015

Add a client/server implementation of the container runtime #13768

Closed

feiskyer force-pushed the kubelet/hyper branch 5 times, most recently from 65942fb to 5451a2e Compare September 21, 2015 01:27

feiskyer added 3 commits September 22, 2015 11:07

Add hyper container runtime

623009f

Hyper is a hypervisor-agnostic docker engine, see https://hyper.sh for more documentation.

Refactor some strings to constants

6843b0a

update hyper spec

c86e832

feiskyer force-pushed the kubelet/hyper branch from 5451a2e to 8e990d8 Compare September 22, 2015 03:10

Fix hyper listpods

ef13667

feiskyer force-pushed the kubelet/hyper branch from 8e990d8 to ef13667 Compare September 22, 2015 03:23

k8s-github-robot added the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Oct 30, 2015

k8s-github-robot removed the needs-rebase Indicates a PR cannot be merged because it has merge conflicts with HEAD. label Dec 22, 2015

feiskyer closed this Dec 23, 2015

v1k0d3n mentioned this pull request May 13, 2016

discussion / plan LXC integration #6862

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add hyper container runtime #13079

Add hyper container runtime #13079

feiskyer commented Aug 24, 2015

k8s-bot commented Aug 24, 2015

thockin commented Aug 24, 2015

feiskyer commented Aug 25, 2015

bgrant0607 commented Aug 26, 2015

yujuhong commented Aug 26, 2015

k8s-github-robot commented Aug 27, 2015

feiskyer commented Aug 29, 2015

pmorie Aug 29, 2015

feiskyer Aug 30, 2015

yujuhong commented Aug 31, 2015

feiskyer commented Sep 7, 2015

dchen1107 commented Sep 9, 2015

brendandburns commented Sep 9, 2015

feiskyer commented Sep 10, 2015

feiskyer commented Sep 10, 2015

dchen1107 commented Sep 11, 2015

feiskyer commented Sep 11, 2015

dchen1107 commented Sep 11, 2015

feiskyer commented Sep 15, 2015

dchen1107 commented Sep 17, 2015

feiskyer commented Sep 17, 2015

feiskyer commented Dec 23, 2015

Add hyper container runtime #13079

Add hyper container runtime #13079

Conversation

feiskyer commented Aug 24, 2015

k8s-bot commented Aug 24, 2015

thockin commented Aug 24, 2015

feiskyer commented Aug 25, 2015

bgrant0607 commented Aug 26, 2015

yujuhong commented Aug 26, 2015

k8s-github-robot commented Aug 27, 2015

feiskyer commented Aug 29, 2015

pmorie Aug 29, 2015

Choose a reason for hiding this comment

feiskyer Aug 30, 2015

Choose a reason for hiding this comment

yujuhong commented Aug 31, 2015

feiskyer commented Sep 7, 2015

dchen1107 commented Sep 9, 2015

brendandburns commented Sep 9, 2015

feiskyer commented Sep 10, 2015

feiskyer commented Sep 10, 2015

dchen1107 commented Sep 11, 2015

feiskyer commented Sep 11, 2015

dchen1107 commented Sep 11, 2015

feiskyer commented Sep 15, 2015

dchen1107 commented Sep 17, 2015

feiskyer commented Sep 17, 2015

feiskyer commented Dec 23, 2015