Description
Edited to reflect current status
Since Kubernetes is written in Go, it's possible to add support for ARM devices like Raspberry Pi.
I have done this, it's published here: https://github.com/luxas/kubernetes-on-arm
And it would be great to merge this to mainline k8s.
This work focuses on making it easier for other architectures too, like arm64
and ppc64le
TODO:
- Upgrade to
go1.5
go1.6
: Upgrade to golang 1.6 #22149 - Compile ARM binaries on every CI build and for every release. Official ARM builds for every new release #19769!
- Compile ARM64 and ppc64le binaries on every CI build and build images for every release. Upgrade to golang 1.6 #22149
- Use the
gcr.io/google_containers/binary-arch:version
names for images. Discussions: Push server docker images during the release (attempt 2) #19061, Make kubelet use an arch-specific pause image depending on GOARCH #23059, Reimplement 'pause' in C - smaller footprint all around #23009 - Make
etcd
cross-platform: Make the etcd image cross-platform #19216 - Add a
gcr.io
flannel image that usesquay.io/coreos/flannel:version
foramd64
and cross-compile for other architectures. Cross-build hyperkube and debian-iptables for ARM. Also add a flannel image #21617 - Merge the PR that pushes docker images (including
hyperkube
) when releasing. Push server docker images during the release (attempt 2) #19061 - Edit the
hyperkube
Makefile to support ARM. Cross-build hyperkube and debian-iptables for ARM. Also add a flannel image #21617 - Make
skydns
,kube2sky
andexechealthz
cross-platform. Make kube2sky and skydns docker images cross-platform #19376, Add cross-compiling support for exechealthz kubernetes-retired/contrib#646 - Add a k8s-in-docker deployment (like kubernetes cluster deployment & management using docker containers #13901) that supports multiple architectures Initial add of docker-multinode kubernetes-retired/kube-deploy#12
- Update
docker-multinode
instructions: Update docker-multinode instructions and version #18894 - Update
docker
guide to support other architectures also. Document experimental support for other architectures website#407 - Write a multiarch proposal Add a multi-platform proposal #26863
- Build
dashboard
for ARM (gcr.io/google_containers/kubernetes-dashboard-arm
) - Make
pause
totally cross-platform (and written in C). Make kubelet use an arch-specific pause image depending on GOARCH #23059, Reimplement 'pause' in C - smaller footprint all around #23009 - Be able to detect server platform Include Go version, platform, and other build info in version string #19905
- Label nodes with os/arch Automatically add node labels beta.kubernetes.io/{os,arch} #23684
Status of multiarch images:
Get docker-multinode working:
- gcr.io/google_containers/hyperkube, fixed for all Cross-build hyperkube and debian-iptables for ARM. Also add a flannel image #21617, Upgrade to golang 1.6 #22149
- gcr.io/google_containers/etcd, fixed for all Make the etcd image cross-platform #19216, Upgrade to golang 1.6 #22149
- gcr.io/google_containers/flannel, fixed for all Cross-build hyperkube and debian-iptables for ARM. Also add a flannel image #21617, Upgrade to golang 1.6 #22149
- Depends on
gcr.io/google_containers/debian-iptables
, fixed for all Cross-build hyperkube and debian-iptables for ARM. Also add a flannel image #21617
- Depends on
- gcr.io/google_containers/pause, fixed for all Reimplement 'pause' in C - smaller footprint all around #23009
Get DNS working:
- gcr.io/google_containers/kube2sky, fixed for all Make kube2sky and skydns docker images cross-platform #19376
- gcr.io/google_containers/skydns, fixed for all Make kube2sky and skydns docker images cross-platform #19376
- gcr.io/google_containers/exechealthz fixed for all Add cross-compiling support for exechealthz kubernetes-retired/contrib#646
Docker wrapped Kubernetes binaries:
- gcr.io/google_containers/kube-proxy, fixed for all Cross-build hyperkube and debian-iptables for ARM. Also add a flannel image #21617, Upgrade to golang 1.6 #22149
- gcr.io/google_containers/kube-apiserver, fixed for all Official ARM builds for every new release #19769, Upgrade to golang 1.6 #22149
- gcr.io/google_containers/kube-controller-manager, fixed for all Official ARM builds for every new release #19769, Upgrade to golang 1.6 #22149
- gcr.io/google_containers/kube-scheduler, fixed for all Official ARM builds for every new release #19769, Upgrade to golang 1.6 #22149
- gcr.io/google_containers/kubelet, doesn't really exist yet: Add docker images for kubelet and kubelet-checker. #19177
I worked with the dashboard team, and it has had multiarch support from the first release.
I'm maybe gonna add multiarch support for official heapster too, now when it has reached v1.
On kubernetes-on-arm, I already have heapster, influxdb and grafana nicely running as a cluster addon.
Things to have in mind:
- All binaries have to be compiled for ARM, and therefore a common naming structure is important
- @brendandburns has already proposed this as
binary-ARCH:VERSION
, which is logical- Also,
etcd
andflannel
should be built.etcd
is already incluster/images/etcd
, maybeflannel
could be included there also? Pushed togcr.io/google_containers/flannel
- One big question is how they should be compiled
- It might be possible to compile ARM images on a
amd64
machine.
- https://github.com/dweinstein/dockerfile-qemu-arm-chroot
- https://github.com/moul/docker-binfmt-register
- runtime: qemu-arm fails to emulate go user programs golang/go#13024
go1.5.2
is probably going to supportqemu
- Cross-compilation doesn't work AFAIK, because e.g.hyperkube
needs to be built dynamically, and one can't build ARM docker images onamd64
Also all docker images must be compiled on ARM (except thoseFROM scratch
)- Base docker images have to be chosen
- One deployment method that would fit well for Raspberry Pis is
k8s-in-docker
- I'm using a
k8s-in-docker
-like setup combined withsystemd
services. I've also written akube-config
script that handles the whole process.- kubernetes cluster deployment & management using docker containers #13901
- Bugs and patches
Sometimes, things fail (randomly) on ARM.- Example: Kubernetes just compiles on amd64 #17060
- Often, it's easy to patch.
But there should be a specific place for ARM patches.- GOARM values
- It's difference between
armv6
andarmv7
when compiling. We will have to setENV GOARM=6
when we compile the images, so e.g.Raspberry Pi 1
is supported.
Outdated tasks:- Use
qemu
emulation forhyperkube
andflannel
cross-buildingMakefile
s. Cross-build hyperkube and debian-iptables for ARM. Also add a flannel image #21617
I thought about making a proposal earlier, and now when #17629 was merged, it might be time
/cc
@dalanlan (k8s-in-docker
interest)
@resouer (k8s-in-docker
interest)
@guybrush (k8s-in-docker
interest)
@davidopp (who was involved in #17629)
@roberthbailey (who was involved in #17629)
@zmerlynn (who was mentioned in #17629)
@DieterReuter (interested in Raspberry Pi things)