[Proposal] Make Kubernetes run on ARM devices #17981

luxas · 2015-11-30T20:58:21Z

Edited to reflect current status
Since Kubernetes is written in Go, it's possible to add support for ARM devices like Raspberry Pi.
I have done this, it's published here: https://github.com/luxas/kubernetes-on-arm
And it would be great to merge this to mainline k8s.

This work focuses on making it easier for other architectures too, like arm64 and ppc64le

TODO:

Status of multiarch images:
Get docker-multinode working:

gcr.io/google_containers/hyperkube, fixed for all Cross-build hyperkube and debian-iptables for ARM. Also add a flannel image #21617, Upgrade to golang 1.6 #22149
gcr.io/google_containers/etcd, fixed for all Make the etcd image cross-platform #19216, Upgrade to golang 1.6 #22149
gcr.io/google_containers/flannel, fixed for all Cross-build hyperkube and debian-iptables for ARM. Also add a flannel image #21617, Upgrade to golang 1.6 #22149
- Depends on gcr.io/google_containers/debian-iptables, fixed for all Cross-build hyperkube and debian-iptables for ARM. Also add a flannel image #21617
gcr.io/google_containers/pause, fixed for all Reimplement 'pause' in C - smaller footprint all around #23009

Get DNS working:

gcr.io/google_containers/kube2sky, fixed for all Make kube2sky and skydns docker images cross-platform #19376
gcr.io/google_containers/skydns, fixed for all Make kube2sky and skydns docker images cross-platform #19376
gcr.io/google_containers/exechealthz fixed for all Add cross-compiling support for exechealthz kubernetes-retired/contrib#646

Docker wrapped Kubernetes binaries:

gcr.io/google_containers/kube-proxy, fixed for all Cross-build hyperkube and debian-iptables for ARM. Also add a flannel image #21617, Upgrade to golang 1.6 #22149
gcr.io/google_containers/kube-apiserver, fixed for all Official ARM builds for every new release #19769, Upgrade to golang 1.6 #22149
gcr.io/google_containers/kube-controller-manager, fixed for all Official ARM builds for every new release #19769, Upgrade to golang 1.6 #22149
gcr.io/google_containers/kube-scheduler, fixed for all Official ARM builds for every new release #19769, Upgrade to golang 1.6 #22149
gcr.io/google_containers/kubelet, doesn't really exist yet: Add docker images for kubelet and kubelet-checker. #19177

I worked with the dashboard team, and it has had multiarch support from the first release.
I'm maybe gonna add multiarch support for official heapster too, now when it has reached v1.
On kubernetes-on-arm, I already have heapster, influxdb and grafana nicely running as a cluster addon.

Things to have in mind:

All binaries have to be compiled for ARM, and therefore a common naming structure is important

@brendandburns has already proposed this as binary-ARCH:VERSION, which is logical

Also, etcd and flannel should be built. etcd is already in cluster/images/etcd, maybe flannel could be included there also? Pushed to gcr.io/google_containers/flannel

One big question is how they should be compiled

It might be possible to compile ARM images on a amd64 machine.

https://github.com/dweinstein/dockerfile-qemu-arm-chroot

https://github.com/moul/docker-binfmt-register

runtime: qemu-arm fails to emulate go user programs golang/go#13024 go1.5.2 is probably going to support qemu
~~- Cross-compilation doesn't work AFAIK, because e.g. hyperkube needs to be built dynamically, and one can't build ARM docker images on amd64~~

~~Also all docker images must be compiled on ARM (except those FROM scratch)~~

Base docker images have to be chosen

One deployment method that would fit well for Raspberry Pis is k8s-in-docker

I'm using a k8s-in-docker-like setup combined with systemd services. I've also written a kube-config script that handles the whole process.

kubernetes cluster deployment & management using docker containers #13901

Bugs and patches

~~Sometimes, things fail (randomly) on ARM.~~

Example: Kubernetes just compiles on amd64 #17060

Often, it's easy to patch. ~~But there should be a specific place for ARM patches.~~

GOARM values

It's difference between armv6 and armv7 when compiling. We will have to set ENV GOARM=6 when we compile the images, so e.g. Raspberry Pi 1 is supported.
Outdated tasks:

Use qemu emulation for hyperkube and flannel cross-building Makefiles. Cross-build hyperkube and debian-iptables for ARM. Also add a flannel image #21617

Related: https://github.com/resin-io-projects/armv7hf-debian-qemu

I thought about making a proposal earlier, and now when #17629 was merged, it might be time

/cc
@dalanlan (k8s-in-docker interest)
@resouer (k8s-in-docker interest)
@guybrush (k8s-in-docker interest)
@davidopp (who was involved in #17629)
@roberthbailey (who was involved in #17629)
@zmerlynn (who was mentioned in #17629)
@DieterReuter (interested in Raspberry Pi things)

The text was updated successfully, but these errors were encountered:

luxas · 2015-12-05T20:24:52Z

I've digged into this a little bit more, and here are the results:

hyperkube must be compiled dynamically, I tested it and it exited if it was built statically:

Error: failed to create kubelet: cAdvisor is unsupported in this build
failed to create kubelet: cAdvisor is unsupported in this build

armbuild/debian, which is the image currently used for ARM, doesn't work on ARMv6
- the reason why armbuild fails is probably because the packages are built for ARMv7
- the other option would be resin/rpi-raspbian which is a very good debian image
Now I may run any ARM docker image on my amd64 machine. This is very interesting.
- ~~I'm using moul's binfmt-register. However, it should only be able to run and compile binaries with go 1.5.2 or newer.~~
- Edit: Even better method: https://resin.io/blog/building-arm-containers-on-any-x86-machine-even-dockerhub/
- If we use this method, we may compile the binaries dynamically for ARM on a amd64 machine.
- Some drawbacks though:
  - We can't use the normal kube-build:cross method as it is right now. It just cross-compiles statically. So just adding linux/arm to server platforms in hack/lib/golang.sh won't work. I'm working on this Fixed in Official ARM builds for every new release #19769
  - ~~Only the go base image must be built on a real ARM machine. That's because, using qemu we can't compile go1.4.3, which is required for go1.5.2. The chicken-or-egg problem.~~
- ~~So if we create a gcr.io/google_containers/golang-arm:v1.5.2 (maybe with some other prefix) and build that on a ARM machine, we could build all binaries on amd64~~
- Edit: It's possible to cross-compile dynamically with golang:1.5.2 as base image.

So here are some questions for you to answer

Should I start working on a WIP: Kubernetes on ARM PR? Yes, but splitting the work in separate PRs
Is it acceptable to include flannel in cluster/images and maybe build them there? The same method as skydns... Yes
Is using qemu for building ARM docker images OK?
Should ARMv6 be supported? That would be great. You've already set GOARM=5 in kube-build:cross Yes
Could we switch armbuild/debian for resin/rpi-raspbian or some other ARMv6 compatible image? Yes

At least I would like to discuss these things
Just say if you don't want this functionality at this time in Kubernetes
Personally, I don't think it's that good to promote usage of k8s on Pis on your blog when you have to hack things for it to work (on the other hand, it's very fun 😄)

ikehz · 2015-12-11T23:34:37Z

@brendandburns I know you were working on some Pi stuff; want to chime in here?

luxas · 2015-12-12T09:37:39Z

Yes #17629

I thought about making a proposal earlier, and now when #17629 was merged, it might be time

There are lots of folks who want this, and since 1st Oct, my project may help with this.
Or one have to hack it by hand...

@ihmccreery At least, should I start writing on the ARM docs?
E.g. about what's different with ARM and what needs to be customized.

ikehz · 2015-12-16T01:20:11Z

Assigning to @brendandburns to advise.

brendandburns · 2015-12-17T14:20:03Z

@luxas I have a working Raspberry PI cluster and have built a bunch of ARM binaries for various pieces. I'm happy to advise and review the PRs..

We can keep this open as a tracking issue. Please assign the PRs to me.

luxas · 2015-12-17T15:15:22Z

@brendandburns Are you OK with using resin/rpi-raspbian instead for armbuild/debian for ARMv6 compability?
And how about including flannel besides etcd in cluster/images?

luxas · 2015-12-26T15:38:25Z

@brendandburns Great news! I've been able to compile all kubernetes binaries on my amd64 laptop.
This makes ARM building a lot easier and faster, and the ARM binaries works on both ARMv6 and ARMv7 (aka RPi 1 and 2)

The code is here and it uses golang:1.5.2 as the base image. I could test if it also works with golang:1.4-cross, which you are using today.

hyperkube and flannel is built dynamically for ARM on amd64 thanks to some libraries described here

I also think it would be possible to build ARM docker images on amd64 (only images with COPY works today) using qemu
Here is a blog post about that: https://resin.io/blog/building-arm-containers-on-any-x86-machine-even-dockerhub/

WDYT?

ikehz · 2015-12-29T21:27:17Z

cc @david-mcmahon

luxas · 2016-04-05T15:27:52Z

Again, a major milestone was merged for cross-platform Kubernetes: #21617!
When the v1.3.0-alpha.2 release is out, one is able to run Kubernetes on ARM by following the docker guide.

@brendandburns

Automatic merge from submit-queue Make kube2sky and skydns docker images cross-platform ARM tracking issue: #17981 Continues on: #19216 Make it possible to create `kube2sky` and `skydns` docker images for ARM and other architectures too Build in a container, so `golang` isn't a dependency I've preserved the original default behaviour: - `skydns`: It just compiles with go on host - `kube2sky`: Build an image @brendandburns @dchen1107 @ArtfulCoder @thockin @fgrzadkowski

coryjamesfisher · 2016-04-30T20:16:22Z

@luxas Thanks so much for the work you and the others have put into this. I was smacking my head against the wall trying to get Kubernetes running on a rpi 3 last night using the guide... Then I saw your comment this morning so I changed the k8 version number to the latest alpha and everything just works :) Thank you thank you thank you :)

@davidopp

Automatic merge from submit-queue Automatically add node labels beta.kubernetes.io/{os,arch} Proposal: #17981 As discussed in #22623: > @davidopp: #9044 says cloud provider but can also cover platform stuff. Adds a label `beta.kubernetes.io/platform` to `kubelet` that informs about the os/arch it's running on. Makes it easy to specify `nodeSelectors` for different arches in multi-arch clusters. ```console $ kubectl get no --show-labels NAME STATUS AGE LABELS 127.0.0.1 Ready 1m beta.kubernetes.io/platform=linux-amd64,kubernetes.io/hostname=127.0.0.1 $ kubectl describe no Name: 127.0.0.1 Labels: beta.kubernetes.io/platform=linux-amd64,kubernetes.io/hostname=127.0.0.1 CreationTimestamp: Thu, 31 Mar 2016 20:39:15 +0300 ``` @davidopp @vishh @fgrzadkowski @thockin @wojtek-t @ixdy @bgrant0607 @dchen1107 @preillyme

@ixdy

Automatic merge from submit-queue Use pause image depending on the server's platform when testing Removed all pause image constant strings, now the pause image is chosen by arch. Part of the effort of making e2e arch-agnostic. The pause image name and version is also now only in two places, and it's documented to bump both Also removed "amd64" constants in the code. Such constants should be replaced by `runtime.GOARCH` or by looking up the server platform Fixes: #22876 and #15140 Makes it easier for: #25730 Related: #17981 This is for `v1.3` @ixdy @thockin @vishh @kubernetes/sig-testing @andyzheng0831 @Pensu

luxas · 2016-06-25T14:10:09Z

@david-mcmahon How is the release note writing process for v1.3 managed?
It would be really great to mention a few words about the new non-business-supported architectures in the v1.3 release notes, ideally both on the blog and in the changelog.
WDYT? Please respond when you're seeing this and if you want me to write a short summary, just ping me

roberthbailey · 2016-06-25T15:37:58Z

Release notes are handled in a few chunks. There are the auto-generated notes from github labels that are bundled up since the last release. And there are hand-curated release notes, which this falls under. Please write up what you'd like to see and send it to @david-mcmahon

luxas · 2016-06-29T21:14:09Z

@david-mcmahon Did you get my email message I sent some days ago?

david-mcmahon · 2016-06-29T23:07:48Z

@luxas Yes and I replied.

luxas · 2016-07-02T20:33:42Z

Yes, thanks!
Looking forward to see the full release notes for v1.3 😄

m1093782566 · 2016-08-20T07:49:00Z

/cc

luxas · 2016-10-16T19:27:24Z

For everyone interested, arm binaries has been released for kubernetes since v1.2.

For v1.2.x, use my project kubernetes-on-arm v0.7.0
For v1.3.x, use docker-multinode or kubernetes-on-arm v0.8.0
For v1.4.x, use kubeadm

You may also read https://github.com/kubernetes/kubernetes/blob/master/docs/proposals/multi-platform.md

I consider this work done since a while ago thanks to the linked PRs above.
Still, there are work to do mentioned in the proposal, but the original issue Make Kubernetes on ARM devices has worked "officially" since v1.2.x, which is more than a half year ago, so closing this issue

fabioy added sig/node Categorizes an issue or PR as relevant to SIG Node. team/cluster area/test-infra kind/feature Categorizes issue or PR as related to a new feature. area/release-eng Issues or PRs related to the Release Engineering subproject labels Dec 1, 2015

luxas mentioned this issue Dec 6, 2015

Kubernetes just compiles on amd64 #17060

Closed

ikehz added area/build-release and removed area/release-eng Issues or PRs related to the Release Engineering subproject labels Dec 11, 2015

luxas mentioned this issue Dec 14, 2015

Don't restrict kube2sky to build only on amd64 #18669

Merged

ikehz assigned brendandburns Dec 16, 2015

This was referenced Dec 18, 2015

Update docker-multinode instructions and version #18894

Merged

Provide an 'all-in-one' binary for server components (hyperkube) #5755

Closed

ikehz added priority/backlog Higher priority than priority/awaiting-more-evidence. and removed area/test-infra priority/backlog Higher priority than priority/awaiting-more-evidence. labels Dec 22, 2015

This was referenced Dec 23, 2015

RFC: Determine the scalability goal for v1.2 kubelet #16943

Closed

Update kube-build to go1.5.2 #19110

Closed

This was referenced Mar 30, 2016

Add Windows Containers Support #22623

Closed

Automatically add node labels beta.kubernetes.io/{os,arch} #23684

Merged

Support emulating arm64 go binaries with qemu golang/go#15050

Closed

luxas mentioned this issue Apr 6, 2016

Build Kubernetes, etcd and flannel for arm64 and ppc64le #23931

Merged

vishh mentioned this issue Apr 19, 2016

Document experimental support for other architectures kubernetes/website#407

Merged

luxas assigned luxas and unassigned brendandburns Apr 21, 2016

luxas added area/images-registry area/test-infra labels Apr 21, 2016

luxas mentioned this issue Apr 23, 2016

Installation issues with latest hypriot-os for rpi3 luxas/kubernetes-on-arm#95

Closed

luxas mentioned this issue May 2, 2016

Kubelet need to run w/ --containerized flag and mount / to /rootfs? luxas/kubernetes-on-arm#98

Closed

luxas closed this as completed Oct 16, 2016

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Proposal] Make Kubernetes run on ARM devices #17981

[Proposal] Make Kubernetes run on ARM devices #17981

luxas commented Nov 30, 2015 •

edited

Loading

luxas commented Dec 5, 2015

ikehz commented Dec 11, 2015

luxas commented Dec 12, 2015

ikehz commented Dec 16, 2015

brendandburns commented Dec 17, 2015

luxas commented Dec 17, 2015

luxas commented Dec 26, 2015

ikehz commented Dec 29, 2015

luxas commented Apr 5, 2016

coryjamesfisher commented Apr 30, 2016

luxas commented Jun 25, 2016

roberthbailey commented Jun 25, 2016

luxas commented Jun 29, 2016

david-mcmahon commented Jun 29, 2016

luxas commented Jul 2, 2016

m1093782566 commented Aug 20, 2016

luxas commented Oct 16, 2016

[Proposal] Make Kubernetes run on ARM devices #17981

[Proposal] Make Kubernetes run on ARM devices #17981

Comments

luxas commented Nov 30, 2015 • edited Loading

luxas commented Dec 5, 2015

ikehz commented Dec 11, 2015

luxas commented Dec 12, 2015

ikehz commented Dec 16, 2015

brendandburns commented Dec 17, 2015

luxas commented Dec 17, 2015

luxas commented Dec 26, 2015

ikehz commented Dec 29, 2015

luxas commented Apr 5, 2016

coryjamesfisher commented Apr 30, 2016

luxas commented Jun 25, 2016

roberthbailey commented Jun 25, 2016

luxas commented Jun 29, 2016

david-mcmahon commented Jun 29, 2016

luxas commented Jul 2, 2016

m1093782566 commented Aug 20, 2016

luxas commented Oct 16, 2016

luxas commented Nov 30, 2015 •

edited

Loading