libvirt-coreos

libvirt-coreos is a cluster provider for kubernetes that starts local VMs and runs kubernetes on it.
Its goal is to provide a multi-machines environment to develop and test kubernetes.

The purpose is mostly the same as the vagrant provider but with a big focus on efficiency. Indeed, the vagrant cluster takes a long time to boot and consumes a huge amount of disk space.

The following array summarizes the differences between the vagrant and the libvirt-coreos kubernetes provider for a cluster with 1 master and 3 minions:

The first time the cluster (either vagrant or libvirt-coreos) is started, a base image (named “box” in vagrant language) for the VMs is downloaded. This image don’t need to be downloaded again for the subsequent boot.

The above figures were measured for the second or following boot. The time doesn’t take into account the download of the base image.

export NUM_MINIONS=3
export KUBERNETES_PROVIDER=…
cluster/kube-up.sh # May download the VM base image
cluster/kube-down.sh
# Note the disk and memory usage of the host.
time cluster/kube-up.sh
# Note the disk and memory usage of the host.
cluster/kube-down.sh

¹ The time to boot is the elapsed time measured by time cluster/kube-up.sh.

² The disk space is measured with df /home before and after cluster/kube-up.sh

³ The memory usage is measured with free before and after cluster/kube-up.sh

As the libvirt-coreos cluster provider is cheaper than the vagrant one, it allows to start bigger clusters.

How to it works

It uses libvirt and qemu-kvm as hypervisor (in order to benefit from disk copy-on-write and linux Kernel Samepage Merging feature).

As OS, it uses CoreOS because it has the advantages of:

• being light;
• coming with docker and etcd already functional;
• being configurable with a cloud-init file.

Reduced disk space

Vagrant copies the base image to create the disk for VMs whereas libvirt-coreos uses the Copy-on-Write feature of QEMU to create cheap VM drive.

Vagrant installs stuff inside each VM whereas in libvirt-coreos, all the VMs mount a drive from the host which contains the kubernetes binaries.

All in all, vagrant creates a 2.4 Gib file per VM whereas libvirt-coreos creates a 120 Mib file per VM.

Reduced memory footprint

libvirt-coreos uses qemu-kvm as hypervisor. qemu-kvm can use KSM to deduplicate memory pages.

Reduced boot time

Vagrant needs to:

1. start the VM;
2. Wait for sshd to be started;
3. Connect via sshd to the VM to provision it;
4. Vagrant finishes the setup of one VM before starting with the next VM.

libvirt-coreos uses cloud-config to let the VM provision themselves as soon as possible. libvirt-coreos’s kube-up starts all the VM and let them being provisioned in parallel.