This gist compiles all necessary information on how to setup a Kubernetes with both Linux and Windows Nodes, enabling both use of Windows and Linux docker containers, using ovn-kubernetes as the network plugin on bare metal servers.

Two machines are required to run this, one will be running Linux Ubuntu and will act as our Master in the cluster and the other machine will be running Windows which will be added to our cluster as a Work Node able to run Windows Containers.

This configuration is based on the available vagrant setup.

I would like to thank Alin Balutoiu and Alin Gabriel Serdean for their amazing help.

The following diagram represents the internal architecture details of the overlay mode.

• Hybrid Kubernetes with OVN
• Table of Contents
• Versions
  • Linux
  • Windows
• Hostnames
• Variables
• Master Setup
  • Master Initial Setup
    • Add external repos to install docker and OVS from packages
    • Docker installation
    • OVS installation and necessary dependencies
    • Golang installation
    • Setup CNI directory
    • Install OVN+K8S Integration
    • Download and install K8s
  • Kubernetes Setup
    • Install an etcd cluster
    • Start k8s daemons
    • Wait for kube-apiserver to start up, then launch both Controller and Scheduler
    • Create a kubeconfig file.
    • Start ovnkube
• Linux Node Setup
  • Linux Initial Setup
  • K8s Startup
    • Install CNI
    • Add kubeconfig
    • Start k8s daemons
    • Setup OVNKube
• Windows Node Setup
  • Windows Initial Setup
    • Installing both Hyper-V and Container features
    • Force install Docker CE edition
    • Download and Install OVS
    • Download and Install Kubernetes
  • OVS Configuration
    • Setup a transparent docker network
    • Configure VMSwitch and OVS
    • [Optiona] Fix DHCP
  • Startup
    • Add kubeconfig file
    • Launch Kubelet
    • Run OVNkube
• Launching Test Pod
  • Test Pod
• Debugging
• Sources and Further Reading

Compilation of all versions used and tested, feel free to implement this setup with more recent versions and leave feedback so that this gist can be updated.

To facilitate deployment, hostnames will be used on all machines and their respective scripts.

This example setup will only add one linux node and one windows node, however adding more for both OS should be exactly the same process, since new registered nodes will receive new ips from the available cluster subnet space.

sudo apt-get update
sudo apt-get install -y apt-transport-https ca-certificates
echo "deb https://packages.wand.net.nz $(lsb_release -sc) main" | sudo tee /etc/apt/sources.list.d/wand.list
sudo curl https://packages.wand.net.nz/keyring.gpg -o /etc/apt/trusted.gpg.d/wand.gpg
sudo apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
sudo su -c "echo \"deb https://apt.dockerproject.org/repo ubuntu-xenial main\" >> /etc/apt/sources.list.d/docker.list"
sudo apt-get update

sudo apt-get purge lxc-docker
sudo apt-get install -y linux-image-extra-$(uname -r) linux-image-extra-virtual
sudo apt-get install -y docker-engine
sudo service docker start

sudo apt-get build-dep dkms
sudo apt-get install python-six openssl python-pip -y
sudo -H pip install --upgrade pip

sudo apt-get install openvswitch-datapath-dkms=2.9.0-1 -y
sudo apt-get install openvswitch-switch=2.9.0-2 openvswitch-common=2.9.0-2 libopenvswitch=2.9.0-2 -y
sudo -H pip install ovs

sudo apt-get install ovn-central=2.9.0-2 ovn-common=2.9.0-2 ovn-host=2.9.0-2 -y

wget -nv https://dl.google.com/go/go1.9.2.linux-amd64.tar.gz
sudo tar -C /usr/local -xzf go1.9.2.linux-amd64.tar.gz
export PATH="/usr/local/go/bin:echo $PATH"
export GOPATH=$HOME/work

sudo mkdir -p /opt/cni/bin/

mkdir -p $HOME/work/src/github.com/openvswitch
pushd $HOME/work/src/github.com/openvswitch
git clone https://github.com/openvswitch/ovn-kubernetes
popd
pushd $HOME/work/src/github.com/openvswitch/ovn-kubernetes/go-controller
make 1>&2 2>/dev/null
sudo make install
popd

mkdir k8s
cd k8s
wget https://github.com/kubernetes/kubernetes/releases/download/v1.9.3/kubernetes.tar.gz
tar xvzf kubernetes.tar.gz
./kubernetes/cluster/get-kube-binaries.sh
mkdir server
cd server
tar xvzf ../kubernetes/server/kubernetes-server-linux-amd64.tar.gz

sudo docker run --net=host -v /var/etcd/data:/var/etcd/data -d \
        gcr.io/google_containers/etcd:3.0.17 /usr/local/bin/etcd \
        --listen-peer-urls http://127.0.0.1:2380 \
        --advertise-client-urls=http://127.0.0.1:4001 \
        --listen-client-urls=http://0.0.0.0:4001 \
        --data-dir=/var/etcd/data

sudo sh -c 'echo "PATH=$PATH:$HOME/k8s/server/kubernetes/server/bin" >> /etc/profile'
cd k8s/server/kubernetes/server/bin
sudo ./kube-apiserver --service-cluster-ip-range=$SERVICE_CLUSTER_RANGE/24 \
                            --address=0.0.0.0 \
                            --etcd-servers=http://127.0.0.1:4001 \
                            --advertise-address=$GW_IP \
                            --v=2 2>&1 0<&- &>/dev/null &

sudo ./kube-controller-manager --master=127.0.0.1:8080 --v=2 2>&1 0<&- &>/dev/null &
sudo ./kube-scheduler --master=127.0.0.1:8080 --v=2 2>&1 0<&- &>/dev/null &

apiVersion: v1
clusters:
- cluster:
    server: http://localhost:8080
  name: default-cluster
- cluster:
    server: http://localhost:8080
  name: local-server
- cluster:
    server: http://localhost:8080
  name: ubuntu
contexts:
- context:
    cluster: ubuntu
    user: ubuntu
  name: ubuntu
current-context: ubuntu
kind: Config
preferences: {}
users:
- name: ubuntu
  user:
    password: p1NVMZqhOOOqkWQq
    username: admin
KUBECONFIG

sudo ovnkube -k8s-kubeconfig $HOME/kubeconfig.yaml -net-controller -loglevel=4 \
 -k8s-apiserver="http://$OVERLAY_IP:8080" \
 -logfile="/var/log/openvswitch/ovnkube.log" \
 -init-master=$MASTER -cluster-subnet="$CLUSTER_SUBNET" \
 -service-cluster-ip-range="$SERVICE_CLUSTER_RANGE" \
 -nodeport \
 -nb-address="$PROTOCOL://$OVERLAY_IP:6631" \
 -sb-address="$PROTOCOL://$OVERLAY_IP:6632" 2>&1 &

Repeat Initial setup from Master.

pushd ~/
wget -nv https://github.com/containernetworking/cni/releases/download/v0.5.2/cni-amd64-v0.5.2.tgz
popd
sudo mkdir -p /opt/cni/bin
pushd /opt/cni/bin
sudo tar xvzf ~/cni-amd64-v0.5.2.tgz
popd

apiVersion: v1
clusters:
- cluster:
    server: http://imperial:8080
  name: default-cluster
- cluster:
    server: http://imperial:8080
  name: local-server
- cluster:
    server: http://imperial:8080
  name: ubuntu
contexts:
- context:
    cluster: ubuntu
    user: ubuntu
  name: ubuntu
current-context: ubuntu
kind: Config
preferences: {}
users:
- name: ubuntu
  user:
    password: p1NVMZqhOOOqkWQq
    username: admin
KUBECONFIG

pushd k8s/server/kubernetes/server/bin
nohup sudo ./kubelet --kubeconfig $HOME/kubeconfig.yaml \
                     --v=2 --address=0.0.0.0 \
                     --fail-swap-on=false \
                     --runtime-cgroups=/systemd/system.slice \
                     --kubelet-cgroups=/systemd/system.slice \
                     --enable-server=true --network-plugin=cni \
                     --cni-conf-dir=/etc/cni/net.d \
                     --cni-bin-dir="/opt/cni/bin/" 2>&1 0<&- &>/dev/null &
sleep 10
popd

nohup sudo ovnkube -k8s-kubeconfig $HOME/kubeconfig.yaml -loglevel=4 \
    -logfile="/var/log/openvswitch/ovnkube.log" \
    -k8s-apiserver="http://$MASTER:8080" \
    -init-node="$MINION_NAME"  \
    -nodeport \
    -nb-address="$PROTOCOL://$MASTER:6631" \
    -sb-address="$PROTOCOL://$MASTER:6632" \
    -init-gateways -gateway-interface="$LINUX_INTERFACE" -gateway-nexthop="$GW_IP" \
    -service-cluster-ip-range=$SERVICE_CLUSTER_RANGE/24 \
    -cluster-subnet=$CLUSTER_SUBNET 2>&1 &

Enable-WindowsOptionalFeature -Online -FeatureName containers –All
Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V –All

Install docker for windows. CE edition was choosen and force installed since it seems to function better, however the Windows Server recommended EE edition should work too.

Invoke-WebRequest https://download.docker.com/win/static/test/x86_64/docker-17.11.0-ce-rc3.zip -UseBasicParsing -OutFile docker.zip
Expand-Archive docker.zip -DestinationPath $Env:ProgramFiles
Remove-Item -Force docker.zip
$env:Path = $env:Path + "C:\Program Files\docker\;"
cd "C:\Program Files\docker\"
dockerd --register-service
Start-Service docker
setx PATH "$env:Path;C:\Program Files\docker\;" -m

Download OVS for Windows from Cloudbase 2.9.0_beta.

Since this version is beta, we will have to disable driver certification:

bcdedit /set testsigning yes

To run this command it may be required to have safe boot disabled.

Make sure to check OVN Host on installation.

You can find the client binaries here.

docker network create -d transparent --gateway $GATEWAY_IP --subnet $SUBNET `
    -o com.docker.network.windowsshim.interface="$WINDOWS_INTERFACE" external

Get-VMSwitch -SwitchType External | Disable-VMSwitchExtension "Cloudbase Open vSwitch Extension"

ovs-vsctl --if-exists --no-wait del-br br-ex

Get-VMSwitch -SwitchType External | Set-VMSwitch -AllowManagementOS $false

Get-VMSwitch -SwitchType External | Set-VMSwitch -AllowManagementOS $false

ovs-vsctl --no-wait --may-exist add-br br-ex

ovs-vsctl --no-wait add-port br-ex '$WINDOWS_INTERFACE'

Stop-Service ovs-vswitchd

Get-VMSwitch -SwitchType External | Enable-VMSwitchExtension "Cloudbase Open vSwitch Extension"

Start-Service ovs-vswitchd

sleep 2

Restart-Service ovs-vswitchd

Enable-NetAdapter br-ex

Remove-NetIPAddress -ifAlias br-ex -Confirm:$false

New-NetIPAddress -ifAlias br-ex -IPAddress $INTERFACE_IP -DefaultGateway $DEFAULT_GW -PrefixLength 24

Set-DnsClientServerAddress -InterfaceAlias br-ex -ServerAddresses ("1.1.1.1")

$GUID = (New-Guid).Guid

ovs-vsctl set Open_vSwitch . external_ids:system-id="$($GUID)"

The last script will change from dhcp to static ip, to change back to dhcp it will be necessary to give the our external interface macaddress to our new virtual interface and generate a new macaddress to give to our external interface, this will make sure we receive from dhcp the same IP to our virtual interface.

Set-NetAdapter -Name <interface_name> -MacAddress <new_interface_mac> -Confirm:$false
Set-NetAdapter -Name br-ex -MacAddress <interface_mac> -Confirm:$false

After running both commands we change the network settings to dhcp.

We will first need to register the node in the cluster. For this we will launch kubelet and wait for node registration to be checked, we can then stop the kubelet process and run ovnkube once, after that we can rerun the kubelet.

apiVersion: v1
clusters:
- cluster:
    server: http://imperial:8080
  name: default-cluster
- cluster:
    server: http://imperial:8080
  name: local-server
- cluster:
    server: http://imperial:8080
  name: ubuntu
contexts:
- context:
    cluster: ubuntu
    user: ubuntu
  name: ubuntu
current-context: ubuntu
kind: Config
preferences: {}
users:
- name: ubuntu
  user:
    password: p1NVMZqhOOOqkWQq
    username: admin
KUBECONFIG

.\kubelet.exe --hostname-override="desperados" `
        --pod-infra-container-image="cloudbase/pause" `
        --resolv-conf="" `
        --kubeconfig="C:\k\kubeconfig.yaml" `
        --network-plugin=cni `
        --cni-bin-dir="C:\k" `
        --cni-conf-dir="C:\etc\cni\net.d"

.\ovnkube.exe -k8s-kubeconfig kubeconfig.yaml -loglevel=4 `
    -k8s-apiserver="http://$MASTER:8080" `
    -init-node="desperados"  `
    -nodeport `
    -nb-address="$PROTOCOL://$MASTER:6631" `
    -sb-address="$PROTOCOL://$MASTER:6632" -k8s-token="test" `
    -service-cluster-ip-range="$SERVICE_CLUSTER_RANGE" `
    -cluster-subnet="$CLUSTER_SUBNET"

To test our hybrid cluster, we will deploy a simple IIS pod. Since these pods will only run on windows, we can force node selection with the following:

nodeSelector:
    beta.kubernetes.io/os: windows

apiVersion: v1
kind: Pod
metadata:
  name: nano-1709
  labels:
    name: webserver
spec:
  containers:
  - name: nano
    image: cloudbase/pause
    imagePullPolicy: IfNotPresent
  - name: nano2
    image: microsoft/iis:windowsservercore-1709
    imagePullPolicy: IfNotPresent
  nodeSelector:
    beta.kubernetes.io/os: windows

Check here for official information on debugging.

• ovn-kubernetes
• Open vSwitch
• CloudBase OvS for Windows
• ovn-kubernetes Vars