Skip to content

Instantly share code, notes, and snippets.

@luisg0nc

luisg0nc/hybrid_kubernetes_using_ovn.md

Last active November 8, 2024 17:31
Show Gist options
  • Save luisg0nc/5213d6c7ad548c61de9090c0931d8f74 to your computer and use it in GitHub Desktop.
Save luisg0nc/5213d6c7ad548c61de9090c0931d8f74 to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. luisg0nc revised this gist Nov 8, 2024. No changes.
  2. luisg0nc revised this gist May 3, 2018. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion hybrid_kubernetes_using_ovn.md
    View file
    Original file line number Diff line number Diff line change
    @@ -26,7 +26,7 @@ Table of Contents
    - [Table of Contents](#table-of-contents)
    - [Versions](#versions)
    - [Linux](#linux)
    - [Windows__](#windows)
    - [Windows](#windows)
    - [Hostnames](#hostnames)
    - [Variables](#variables)
    - [Master Setup](#master-setup)
  3. luisg0nc revised this gist May 2, 2018. 1 changed file with 13 additions and 11 deletions.
    24 changes: 13 additions & 11 deletions hybrid_kubernetes_using_ovn.md
    View file
    Original file line number Diff line number Diff line change
    @@ -26,7 +26,7 @@ Table of Contents
    - [Table of Contents](#table-of-contents)
    - [Versions](#versions)
    - [Linux](#linux)
    - [Windows](#windows)
    - [Windows__](#windows)
    - [Hostnames](#hostnames)
    - [Variables](#variables)
    - [Master Setup](#master-setup)
    @@ -107,6 +107,8 @@ To facilitate deployment, hostnames will be used on all machines and their respe
    | Node | Linux Ubuntu | SAGRES |
    | Node | Windows Server 1709 | DESPERADOS |

    This example setup will only add one linux node and one windows node, however adding more for both OS should be exactly the same process, since new registered nodes will receive new ips from the available cluster subnet space.

    Variables
    ---------

    @@ -118,7 +120,8 @@ Variables
    | CLUSTER_SUBNET | Cluster wide IP subnet to use (default "11.11.0.0/16") | 192.168.0.0/16 |
    | SERVICE_CLUSTER_RANGE | A CIDR notation IP range from which k8s assigns service cluster IPs. This should be the same as the one provided for kube-apiserver's. | 172.16.1.0/24 |
    | MASTER | The ip or hostname of clusters master | IMPERIAL |

    | LINUX_INTERFACE | Name of interface with access to the cluster network on Linux | enp0s9 |
    | WINDOWS_INTERFACE | Name of interface with access to the cluster network on Windows | "Ethernet" |

    Master Setup
    ------------
    @@ -319,7 +322,6 @@ KUBECONFIG
    #### Start k8s daemons
    ```
    pushd k8s/server/kubernetes/server/bin
    echo "Starting kubelet ..."
    nohup sudo ./kubelet --kubeconfig $HOME/kubeconfig.yaml \
    --v=2 --address=0.0.0.0 \
    --fail-swap-on=false \
    @@ -336,14 +338,14 @@ popd
    ```
    nohup sudo ovnkube -k8s-kubeconfig $HOME/kubeconfig.yaml -loglevel=4 \
    -logfile="/var/log/openvswitch/ovnkube.log" \
    -k8s-apiserver="http://$MASTER_OVERLAY_IP:8080" \
    -k8s-apiserver="http://$MASTER:8080" \
    -init-node="$MINION_NAME" \
    -nodeport \
    -nb-address="$PROTOCOL://$MASTER_OVERLAY_IP:6631" \
    -sb-address="$PROTOCOL://$MASTER_OVERLAY_IP:6632" \
    -init-gateways -gateway-interface=enp0s9 -gateway-nexthop="$GW_IP" \
    -service-cluster-ip-range=172.16.1.0/24 \
    -cluster-subnet="192.168.0.0/16" 2>&1 &
    -nb-address="$PROTOCOL://$MASTER:6631" \
    -sb-address="$PROTOCOL://$MASTER:6632" \
    -init-gateways -gateway-interface="$LINUX_INTERFACE" -gateway-nexthop="$GW_IP" \
    -service-cluster-ip-range=$SERVICE_CLUSTER_RANGE/24 \
    -cluster-subnet=$CLUSTER_SUBNET 2>&1 &
    ```

    Windows Node Setup
    @@ -396,7 +398,7 @@ You can find the client binaries [here](https://github.com/kubernetes/kubernetes

    ```
    docker network create -d transparent --gateway $GATEWAY_IP --subnet $SUBNET `
    -o com.docker.network.windowsshim.interface="$INTERFACE_ALIAS" external
    -o com.docker.network.windowsshim.interface="$WINDOWS_INTERFACE" external
    ```

    #### Configure VMSwitch and OVS
    @@ -412,7 +414,7 @@ Get-VMSwitch -SwitchType External | Set-VMSwitch -AllowManagementOS $false
    ovs-vsctl --no-wait --may-exist add-br br-ex
    ovs-vsctl --no-wait add-port br-ex '$INTERFACE_ALIAS'
    ovs-vsctl --no-wait add-port br-ex '$WINDOWS_INTERFACE'
    Stop-Service ovs-vswitchd
  4. luisg0nc revised this gist May 2, 2018. 1 changed file with 4 additions and 2 deletions.
    6 changes: 4 additions & 2 deletions hybrid_kubernetes_using_ovn.md
    View file
    Original file line number Diff line number Diff line change
    @@ -439,13 +439,15 @@ ovs-vsctl set Open_vSwitch . external_ids:system-id="$($GUID)"

    #### [Optiona] Fix DHCP

    The last script will change from dhcp to static ip, to change back to dhcp it will be necessary to give the our external interface macaddress to our new virtual interface and generate a new macaddress to give to our external interfacem, this will make sure we receive from dhcp the same IP to our virtual interface.
    The last script will change from dhcp to static ip, to change back to dhcp it will be necessary to give the our external interface macaddress to our new virtual interface and generate a new macaddress to give to our external interface, this will make sure we receive from dhcp the same IP to our virtual interface.

    ```
    Set-NetAdapter -Name <interface_name> -MacAddress <new_interface_mac> -Confirm:$false
    Set-NetAdapter -Name br-ex -MacAddress <interface_mac> -Confirm:$false
    ```

    After running both commands we change the network settings to dhcp.

    ### Startup

    We will first need to register the node in the cluster. For this we will launch kubelet and wait for node registration to be checked, we can then stop the kubelet process and run ovnkube once, after that we can rerun the kubelet.
    @@ -549,4 +551,4 @@ Sources and Further Reading
    * [ovn-kubernetes](https://github.com/openvswitch/ovn-kubernetes/)
    * [Open vSwitch](https://www.openvswitch.org/)
    * [CloudBase OvS for Windows](https://cloudbase.it/openvswitch/)
    * [ovn-kubeernetes Vars](https://github.com/openvswitch/ovn-kubernetes/blob/master/docs/ovnkube.1)
    * [ovn-kubernetes Vars](https://github.com/openvswitch/ovn-kubernetes/blob/master/docs/ovnkube.1)
  5. luisg0nc revised this gist May 2, 2018. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions hybrid_kubernetes_using_ovn.md
    View file
    Original file line number Diff line number Diff line change
    @@ -25,8 +25,8 @@ Table of Contents
    - [Hybrid Kubernetes with OVN](#hybrid-kubernetes-with-ovn)
    - [Table of Contents](#table-of-contents)
    - [Versions](#versions)
    - [__Linux__](#linux)
    - [__Windows__](#windows)
    - [Linux](#linux)
    - [Windows](#windows)
    - [Hostnames](#hostnames)
    - [Variables](#variables)
    - [Master Setup](#master-setup)
  6. luisg0nc revised this gist May 2, 2018. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions hybrid_kubernetes_using_ovn.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,4 +1,4 @@
    Hybrid Kubernetes and OVN
    Hybrid Kubernetes with OVN
    =========================

    This gist compiles all necessary information on how to setup a Kubernetes with both Linux and Windows Nodes,
    @@ -22,7 +22,7 @@ of the overlay mode.

    Table of Contents
    -----

    - [Hybrid Kubernetes with OVN](#hybrid-kubernetes-with-ovn)
    - [Table of Contents](#table-of-contents)
    - [Versions](#versions)
    - [__Linux__](#linux)
  7. luisg0nc revised this gist May 2, 2018. 1 changed file with 82 additions and 8 deletions.
    90 changes: 82 additions & 8 deletions hybrid_kubernetes_using_ovn.md
    View file
    Original file line number Diff line number Diff line change
    @@ -22,10 +22,11 @@ of the overlay mode.

    Table of Contents
    -----

    - [Table of Contents](#table-of-contents)
    - [Versions](#versions)
    - [Linux](#linux)
    - [Windows](#windows)
    - [__Linux__](#linux)
    - [__Windows__](#windows)
    - [Hostnames](#hostnames)
    - [Variables](#variables)
    - [Master Setup](#master-setup)
    @@ -44,8 +45,12 @@ Table of Contents
    - [Create a kubeconfig file.](#create-a-kubeconfig-file)
    - [Start ovnkube](#start-ovnkube)
    - [Linux Node Setup](#linux-node-setup)
    - [Initial Setup](#initial-setup)
    - [Kubernetes Setup](#kubernetes-setup)
    - [Linux Initial Setup](#linux-initial-setup)
    - [K8s Startup](#k8s-startup)
    - [Install CNI](#install-cni)
    - [Add kubeconfig](#add-kubeconfig)
    - [Start k8s daemons](#start-k8s-daemons)
    - [Setup OVNKube](#setup-ovnkube)
    - [Windows Node Setup](#windows-node-setup)
    - [Windows Initial Setup](#windows-initial-setup)
    - [Installing both Hyper-V and Container features](#installing-both-hyper-v-and-container-features)
    @@ -264,13 +269,82 @@ sudo ovnkube -k8s-kubeconfig $HOME/kubeconfig.yaml -net-controller -loglevel=4 \
    Linux Node Setup
    ----------------

    ### Initial Setup
    ### Linux Initial Setup

    `todo`
    Repeat Initial setup from Master.

    ### Kubernetes Setup
    ### K8s Startup

    #### Install CNI
    ```
    pushd ~/
    wget -nv https://github.com/containernetworking/cni/releases/download/v0.5.2/cni-amd64-v0.5.2.tgz
    popd
    sudo mkdir -p /opt/cni/bin
    pushd /opt/cni/bin
    sudo tar xvzf ~/cni-amd64-v0.5.2.tgz
    popd
    ```

    `todo`
    #### Add kubeconfig

    ```
    apiVersion: v1
    clusters:
    - cluster:
    server: http://imperial:8080
    name: default-cluster
    - cluster:
    server: http://imperial:8080
    name: local-server
    - cluster:
    server: http://imperial:8080
    name: ubuntu
    contexts:
    - context:
    cluster: ubuntu
    user: ubuntu
    name: ubuntu
    current-context: ubuntu
    kind: Config
    preferences: {}
    users:
    - name: ubuntu
    user:
    password: p1NVMZqhOOOqkWQq
    username: admin
    KUBECONFIG
    ```

    #### Start k8s daemons
    ```
    pushd k8s/server/kubernetes/server/bin
    echo "Starting kubelet ..."
    nohup sudo ./kubelet --kubeconfig $HOME/kubeconfig.yaml \
    --v=2 --address=0.0.0.0 \
    --fail-swap-on=false \
    --runtime-cgroups=/systemd/system.slice \
    --kubelet-cgroups=/systemd/system.slice \
    --enable-server=true --network-plugin=cni \
    --cni-conf-dir=/etc/cni/net.d \
    --cni-bin-dir="/opt/cni/bin/" 2>&1 0<&- &>/dev/null &
    sleep 10
    popd
    ```

    #### Setup OVNKube
    ```
    nohup sudo ovnkube -k8s-kubeconfig $HOME/kubeconfig.yaml -loglevel=4 \
    -logfile="/var/log/openvswitch/ovnkube.log" \
    -k8s-apiserver="http://$MASTER_OVERLAY_IP:8080" \
    -init-node="$MINION_NAME" \
    -nodeport \
    -nb-address="$PROTOCOL://$MASTER_OVERLAY_IP:6631" \
    -sb-address="$PROTOCOL://$MASTER_OVERLAY_IP:6632" \
    -init-gateways -gateway-interface=enp0s9 -gateway-nexthop="$GW_IP" \
    -service-cluster-ip-range=172.16.1.0/24 \
    -cluster-subnet="192.168.0.0/16" 2>&1 &
    ```

    Windows Node Setup
    ------------------
  8. luisg0nc revised this gist May 2, 2018. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions hybrid_kubernetes_using_ovn.md
    View file
    Original file line number Diff line number Diff line change
    @@ -24,8 +24,8 @@ Table of Contents
    -----
    - [Table of Contents](#table-of-contents)
    - [Versions](#versions)
    - [__Linux__](#linux)
    - [__Windows__](#windows)
    - [Linux](#linux)
    - [Windows](#windows)
    - [Hostnames](#hostnames)
    - [Variables](#variables)
    - [Master Setup](#master-setup)
  9. luisg0nc revised this gist May 2, 2018. 1 changed file with 77 additions and 31 deletions.
    108 changes: 77 additions & 31 deletions hybrid_kubernetes_using_ovn.md
    View file
    Original file line number Diff line number Diff line change
    @@ -20,6 +20,52 @@ of the overlay mode.

    ![alt text](https://i.imgur.com/i7sci9O.png "Overlay mode diagram")

    Table of Contents
    -----
    - [Table of Contents](#table-of-contents)
    - [Versions](#versions)
    - [__Linux__](#linux)
    - [__Windows__](#windows)
    - [Hostnames](#hostnames)
    - [Variables](#variables)
    - [Master Setup](#master-setup)
    - [Master Initial Setup](#master-initial-setup)
    - [Add external repos to install docker and OVS from packages](#add-external-repos-to-install-docker-and-ovs-from-packages)
    - [Docker installation](#docker-installation)
    - [OVS installation and necessary dependencies](#ovs-installation-and-necessary-dependencies)
    - [Golang installation](#golang-installation)
    - [Setup CNI directory](#setup-cni-directory)
    - [Install OVN+K8S Integration](#install-ovnk8s-integration)
    - [Download and install K8s](#download-and-install-k8s)
    - [Kubernetes Setup](#kubernetes-setup)
    - [Install an etcd cluster](#install-an-etcd-cluster)
    - [Start k8s daemons](#start-k8s-daemons)
    - [Wait for kube-apiserver to start up, then launch both Controller and Scheduler](#wait-for-kube-apiserver-to-start-up--then-launch-both-controller-and-scheduler)
    - [Create a kubeconfig file.](#create-a-kubeconfig-file)
    - [Start ovnkube](#start-ovnkube)
    - [Linux Node Setup](#linux-node-setup)
    - [Initial Setup](#initial-setup)
    - [Kubernetes Setup](#kubernetes-setup)
    - [Windows Node Setup](#windows-node-setup)
    - [Windows Initial Setup](#windows-initial-setup)
    - [Installing both Hyper-V and Container features](#installing-both-hyper-v-and-container-features)
    - [Force install Docker CE edition](#force-install-docker-ce-edition)
    - [Download and Install OVS](#download-and-install-ovs)
    - [Download and Install Kubernetes](#download-and-install-kubernetes)
    - [OVS Configuration](#ovs-configuration)
    - [Setup a transparent docker network](#setup-a-transparent-docker-network)
    - [Configure VMSwitch and OVS](#configure-vmswitch-and-ovs)
    - [[Optiona] Fix DHCP](#optiona-fix-dhcp)
    - [Startup](#startup)
    - [Add kubeconfig file](#add-kubeconfig-file)
    - [Launch Kubelet](#launch-kubelet)
    - [Run OVNkube](#run-ovnkube)
    - [Launching Test Pod](#launching-test-pod)
    - [Test Pod](#test-pod)
    - [Debugging](#debugging)
    - [Sources and Further Reading](#sources-and-further-reading)


    Versions
    --------

    @@ -72,9 +118,9 @@ Variables
    Master Setup
    ------------

    ### Initial Setup
    ### Master Initial Setup

    ##### Add external repos to install docker and OVS from packages
    #### Add external repos to install docker and OVS from packages
    ```
    sudo apt-get update
    sudo apt-get install -y apt-transport-https ca-certificates
    @@ -85,15 +131,15 @@ sudo su -c "echo \"deb https://apt.dockerproject.org/repo ubuntu-xenial main\" >
    sudo apt-get update
    ```

    ##### Docker installation
    #### Docker installation
    ```
    sudo apt-get purge lxc-docker
    sudo apt-get install -y linux-image-extra-$(uname -r) linux-image-extra-virtual
    sudo apt-get install -y docker-engine
    sudo service docker start
    ```

    ##### OVS installation and necessary dependencies
    #### OVS installation and necessary dependencies
    ```
    sudo apt-get build-dep dkms
    sudo apt-get install python-six openssl python-pip -y
    @@ -106,20 +152,20 @@ sudo -H pip install ovs
    sudo apt-get install ovn-central=2.9.0-2 ovn-common=2.9.0-2 ovn-host=2.9.0-2 -y
    ```

    ##### Golang installation
    #### Golang installation
    ```
    wget -nv https://dl.google.com/go/go1.9.2.linux-amd64.tar.gz
    sudo tar -C /usr/local -xzf go1.9.2.linux-amd64.tar.gz
    export PATH="/usr/local/go/bin:echo $PATH"
    export GOPATH=$HOME/work
    ```

    ##### Setup CNI directory
    #### Setup CNI directory
    ```
    sudo mkdir -p /opt/cni/bin/
    ```

    ##### Install OVN+K8S Integration
    #### Install OVN+K8S Integration
    ```
    mkdir -p $HOME/work/src/github.com/openvswitch
    pushd $HOME/work/src/github.com/openvswitch
    @@ -131,7 +177,7 @@ sudo make install
    popd
    ```

    ##### Download and install K8s
    #### Download and install K8s
    ```
    mkdir k8s
    cd k8s
    @@ -143,9 +189,9 @@ cd server
    tar xvzf ../kubernetes/server/kubernetes-server-linux-amd64.tar.gz
    ```

    #### Kubernetes Setup
    ### Kubernetes Setup

    ##### Install an etcd cluster
    #### Install an etcd cluster
    ```
    sudo docker run --net=host -v /var/etcd/data:/var/etcd/data -d \
    gcr.io/google_containers/etcd:3.0.17 /usr/local/bin/etcd \
    @@ -155,7 +201,7 @@ sudo docker run --net=host -v /var/etcd/data:/var/etcd/data -d \
    --data-dir=/var/etcd/data
    ```

    ##### Start k8s daemons
    #### Start k8s daemons
    ```
    sudo sh -c 'echo "PATH=$PATH:$HOME/k8s/server/kubernetes/server/bin" >> /etc/profile'
    cd k8s/server/kubernetes/server/bin
    @@ -166,13 +212,13 @@ sudo ./kube-apiserver --service-cluster-ip-range=$SERVICE_CLUSTER_RANGE/24 \
    --v=2 2>&1 0<&- &>/dev/null &
    ```

    ##### Wait for kube-apiserver to start up, then launch both Controller and Scheduler
    #### Wait for kube-apiserver to start up, then launch both Controller and Scheduler
    ```
    sudo ./kube-controller-manager --master=127.0.0.1:8080 --v=2 2>&1 0<&- &>/dev/null &
    sudo ./kube-scheduler --master=127.0.0.1:8080 --v=2 2>&1 0<&- &>/dev/null &
    ```

    ##### Create a kubeconfig file.
    #### Create a kubeconfig file.
    ```
    apiVersion: v1
    clusters:
    @@ -202,7 +248,7 @@ KUBECONFIG
    ```


    ##### Start ovnkube
    #### Start ovnkube
    ```
    sudo ovnkube -k8s-kubeconfig $HOME/kubeconfig.yaml -net-controller -loglevel=4 \
    -k8s-apiserver="http://$OVERLAY_IP:8080" \
    @@ -215,29 +261,29 @@ sudo ovnkube -k8s-kubeconfig $HOME/kubeconfig.yaml -net-controller -loglevel=4 \
    ```


    Minion Node Setup
    Linux Node Setup
    ----------------

    #### Initial Setup
    ### Initial Setup

    `todo`

    #### Kubernetes Setup
    ### Kubernetes Setup

    `todo`

    Windows Node Setup
    ------------------

    #### Initial Setup
    ### Windows Initial Setup

    ##### Installing both Hyper-V and Container features
    #### Installing both Hyper-V and Container features
    ```
    Enable-WindowsOptionalFeature -Online -FeatureName containers –All
    Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V –All
    ```

    ##### Force install Docker CE edition
    #### Force install Docker CE edition

    Install docker for windows. CE edition was choosen and force installed since it seems to function better, however the Windows Server recommended EE edition should work too.

    @@ -252,7 +298,7 @@ Start-Service docker
    setx PATH "$env:Path;C:\Program Files\docker\;" -m
    ```

    ##### Download and Install OVS
    #### Download and Install OVS

    Download OVS for Windows from Cloudbase [2.9.0_beta](https://docs.google.com/uc?export=download&id=1QnAtn8EhfbBFwn-1KPCkJ4EqCmEiT8g_).

    @@ -265,21 +311,21 @@ To run this command it may be required to have safe boot disabled.

    Make sure to check __OVN Host__ on installation.

    ##### Download and Install Kubernetes
    #### Download and Install Kubernetes

    You can find the client binaries [here](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.9.md#downloads-for-v193).


    #### OVS Configuration
    ### OVS Configuration

    ##### Setup a transparent docker network
    #### Setup a transparent docker network

    ```
    docker network create -d transparent --gateway $GATEWAY_IP --subnet $SUBNET `
    -o com.docker.network.windowsshim.interface="$INTERFACE_ALIAS" external
    ```

    ##### Configure VMSwitch and OVS
    #### Configure VMSwitch and OVS

    ```
    Get-VMSwitch -SwitchType External | Disable-VMSwitchExtension "Cloudbase Open vSwitch Extension"
    @@ -317,7 +363,7 @@ $GUID = (New-Guid).Guid
    ovs-vsctl set Open_vSwitch . external_ids:system-id="$($GUID)"
    ```

    ##### [Optiona] Fix DHCP
    #### [Optiona] Fix DHCP

    The last script will change from dhcp to static ip, to change back to dhcp it will be necessary to give the our external interface macaddress to our new virtual interface and generate a new macaddress to give to our external interfacem, this will make sure we receive from dhcp the same IP to our virtual interface.

    @@ -326,7 +372,7 @@ Set-NetAdapter -Name <interface_name> -MacAddress <new_interface_mac> -Confirm:$
    Set-NetAdapter -Name br-ex -MacAddress <interface_mac> -Confirm:$false
    ```

    #### Startup
    ### Startup

    We will first need to register the node in the cluster. For this we will launch kubelet and wait for node registration to be checked, we can then stop the kubelet process and run ovnkube once, after that we can rerun the kubelet.

    @@ -360,7 +406,7 @@ users:
    KUBECONFIG
    ```

    ##### Launch Kubelet
    #### Launch Kubelet

    ```
    .\kubelet.exe --hostname-override="desperados" `
    @@ -372,7 +418,7 @@ KUBECONFIG
    --cni-conf-dir="C:\etc\cni\net.d"
    ```

    ##### Run OVNkube
    #### Run OVNkube

    ```
    .\ovnkube.exe -k8s-kubeconfig kubeconfig.yaml -loglevel=4 `
    @@ -385,7 +431,7 @@ KUBECONFIG
    -cluster-subnet="$CLUSTER_SUBNET"
    ```

    Launching a Pod
    Launching Test Pod
    ---------------

    To test our hybrid cluster, we will deploy a simple IIS pod. Since these pods will only run on windows,
    @@ -398,7 +444,7 @@ nodeSelector:



    #### Test Pod
    ### Test Pod
    ```
    apiVersion: v1
    kind: Pod
  10. luisg0nc revised this gist Apr 24, 2018. 1 changed file with 13 additions and 9 deletions.
    22 changes: 13 additions & 9 deletions hybrid_kubernetes_using_ovn.md
    View file
    Original file line number Diff line number Diff line change
    @@ -20,12 +20,6 @@ of the overlay mode.

    ![alt text](https://i.imgur.com/i7sci9O.png "Overlay mode diagram")


    ### Read more:
    * [ovn-kubernetes](https://github.com/openvswitch/ovn-kubernetes/)
    * [Open vSwitch](https://www.openvswitch.org/)
    * [CloudBase OvS for Windows](https://cloudbase.it/openvswitch/)

    Versions
    --------

    @@ -70,8 +64,8 @@ Variables
    | GW_IP | `TODO` | Master IP |
    | OVERLAY_IP | `TODO` | `TODO` |
    | PROTOCOL | The protocol used, options are SSL and TCP, for sake of simplicity tcp will be used. Look [here](https://github.com/openvswitch/ovn-kubernetes/blob/master/vagrant/provisioning/setup-k8s-minion.sh#L67) for more details. | tcp |
    | CLUSTER_SUBNET | `TODO` | 192.168.0.0/16 |
    | SERVICE_CLUSTER_RANGE | `TODO` | 172.16.1.0/24 |
    | CLUSTER_SUBNET | Cluster wide IP subnet to use (default "11.11.0.0/16") | 192.168.0.0/16 |
    | SERVICE_CLUSTER_RANGE | A CIDR notation IP range from which k8s assigns service cluster IPs. This should be the same as the one provided for kube-apiserver's. | 172.16.1.0/24 |
    | MASTER | The ip or hostname of clusters master | IMPERIAL |


    @@ -314,7 +308,7 @@ Enable-NetAdapter br-ex
    Remove-NetIPAddress -ifAlias br-ex -Confirm:$false
    New-NetIPAddress -ifAlias br-ex -IPAddress <interface_ip> -DefaultGateway <default_gateway> -PrefixLength 24
    New-NetIPAddress -ifAlias br-ex -IPAddress $INTERFACE_IP -DefaultGateway $DEFAULT_GW -PrefixLength 24
    Set-DnsClientServerAddress -InterfaceAlias br-ex -ServerAddresses ("1.1.1.1")
    @@ -337,6 +331,7 @@ Set-NetAdapter -Name br-ex -MacAddress <interface_mac> -Confirm:$false
    We will first need to register the node in the cluster. For this we will launch kubelet and wait for node registration to be checked, we can then stop the kubelet process and run ovnkube once, after that we can rerun the kubelet.

    #### Add kubeconfig file

    ```
    apiVersion: v1
    clusters:
    @@ -366,6 +361,7 @@ KUBECONFIG
    ```

    ##### Launch Kubelet

    ```
    .\kubelet.exe --hostname-override="desperados" `
    --pod-infra-container-image="cloudbase/pause" `
    @@ -377,6 +373,7 @@ KUBECONFIG
    ```

    ##### Run OVNkube

    ```
    .\ovnkube.exe -k8s-kubeconfig kubeconfig.yaml -loglevel=4 `
    -k8s-apiserver="http://$MASTER:8080" `
    @@ -426,3 +423,10 @@ Debugging

    Check [here](https://github.com/openvswitch/ovn-kubernetes/blob/master/docs/debugging.md) for official information on debugging.

    Sources and Further Reading
    ---------------------------

    * [ovn-kubernetes](https://github.com/openvswitch/ovn-kubernetes/)
    * [Open vSwitch](https://www.openvswitch.org/)
    * [CloudBase OvS for Windows](https://cloudbase.it/openvswitch/)
    * [ovn-kubeernetes Vars](https://github.com/openvswitch/ovn-kubernetes/blob/master/docs/ovnkube.1)
  11. luisg0nc revised this gist Apr 24, 2018. 1 changed file with 30 additions and 17 deletions.
    47 changes: 30 additions & 17 deletions hybrid_kubernetes_using_ovn.md
    View file
    Original file line number Diff line number Diff line change
    @@ -3,7 +3,7 @@ Hybrid Kubernetes and OVN

    This gist compiles all necessary information on how to setup a Kubernetes with both Linux and Windows Nodes,
    enabling both use of Windows and Linux docker containers,
    using [ovn-kubernetes](https://github.com/openvswitch/ovn-kubernetes/) as the network plugin.
    using [ovn-kubernetes](https://github.com/openvswitch/ovn-kubernetes/) as the network plugin on bare metal servers.

    Two machines are required to run this, one will be running Linux Ubuntu and will act as our Master in the cluster and
    the other machine will be running Windows which will be added to our cluster as a Work Node able to run Windows Containers.
    @@ -15,16 +15,23 @@ their amazing help.

    ### Overlay mode architecture diagram:

    The following digaram represents the internal architecture details
    The following diagram represents the internal architecture details
    of the overlay mode.

    ![alt text](https://i.imgur.com/i7sci9O.png "Overlay mode diagram")


    ### Read more:
    * [ovn-kubernetes](https://github.com/openvswitch/ovn-kubernetes/)
    * [Open vSwitch](https://www.openvswitch.org/)
    * [CloudBase OvS for Windows](https://cloudbase.it/openvswitch/)

    Versions
    --------

    ### Linux
    Compilation of all versions used and tested, feel free to implement this setup with more recent versions and leave feedback so that this gist can be updated.

    ### __Linux__
    || Versions
    | ------------- |-------------:|
    | Ubuntu | 16.04.4 LTS |
    @@ -35,7 +42,7 @@ Versions
    | kubernetes | 1.9.3 |
    | docker | 17.0-ce-rc3 |

    ### Windows
    ### __Windows__
    || Versions
    | ------------- |-------------:|
    | Windows Server | 1709 |
    @@ -51,20 +58,21 @@ To facilitate deployment, hostnames will be used on all machines and their respe

    | K8s Role | OS | Hostname |
    | --------- | ------- |-------- |
    | Master | Linux Ubuntu | Imperial |
    | Node | Linux Ubuntu | Sagres |
    | Node | Windows Server 1709 | Desperados |
    | Master | Linux Ubuntu | IMPERIAL |
    | Node | Linux Ubuntu | SAGRES |
    | Node | Windows Server 1709 | DESPERADOS |

    Variables
    ---------

    | Variable | Description | Ex. Value |
    | --- | --- |------:|
    | GW_IP | `todo` | `todo` |
    | OVERLAY_IP | `todo` | `todo` |
    | GW_IP | `TODO` | Master IP |
    | OVERLAY_IP | `TODO` | `TODO` |
    | PROTOCOL | The protocol used, options are SSL and TCP, for sake of simplicity tcp will be used. Look [here](https://github.com/openvswitch/ovn-kubernetes/blob/master/vagrant/provisioning/setup-k8s-minion.sh#L67) for more details. | tcp |
    | CLUSTER_SUBNET | `todo` | 192.168.0.0/16 |
    | SERVICE_CLUSTER_RANGE | `todo` | 172.16.1.0/24 |
    | CLUSTER_SUBNET | `TODO` | 192.168.0.0/16 |
    | SERVICE_CLUSTER_RANGE | `TODO` | 172.16.1.0/24 |
    | MASTER | The ip or hostname of clusters master | IMPERIAL |


    Master Setup
    @@ -157,7 +165,7 @@ sudo docker run --net=host -v /var/etcd/data:/var/etcd/data -d \
    ```
    sudo sh -c 'echo "PATH=$PATH:$HOME/k8s/server/kubernetes/server/bin" >> /etc/profile'
    cd k8s/server/kubernetes/server/bin
    sudo ./kube-apiserver --service-cluster-ip-range=172.16.1.0/24 \
    sudo ./kube-apiserver --service-cluster-ip-range=$SERVICE_CLUSTER_RANGE/24 \
    --address=0.0.0.0 \
    --etcd-servers=http://127.0.0.1:4001 \
    --advertise-address=$GW_IP \
    @@ -236,6 +244,9 @@ Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V –All
    ```

    ##### Force install Docker CE edition

    Install docker for windows. CE edition was choosen and force installed since it seems to function better, however the Windows Server recommended EE edition should work too.

    ```
    Invoke-WebRequest https://download.docker.com/win/static/test/x86_64/docker-17.11.0-ce-rc3.zip -UseBasicParsing -OutFile docker.zip
    Expand-Archive docker.zip -DestinationPath $Env:ProgramFiles
    @@ -256,6 +267,8 @@ Since this version is beta, we will have to disable driver certification:
    bcdedit /set testsigning yes
    ```

    To run this command it may be required to have safe boot disabled.

    Make sure to check __OVN Host__ on installation.

    ##### Download and Install Kubernetes
    @@ -303,7 +316,7 @@ Remove-NetIPAddress -ifAlias br-ex -Confirm:$false
    New-NetIPAddress -ifAlias br-ex -IPAddress <interface_ip> -DefaultGateway <default_gateway> -PrefixLength 24
    Set-DnsClientServerAddress -InterfaceAlias br-ex -ServerAddresses ("8.8.8.8")
    Set-DnsClientServerAddress -InterfaceAlias br-ex -ServerAddresses ("1.1.1.1")
    $GUID = (New-Guid).Guid
    @@ -366,11 +379,11 @@ KUBECONFIG
    ##### Run OVNkube
    ```
    .\ovnkube.exe -k8s-kubeconfig kubeconfig.yaml -loglevel=4 `
    -k8s-apiserver="http://10.20.130.7:8080" `
    -k8s-apiserver="http://$MASTER:8080" `
    -init-node="desperados" `
    -nodeport `
    -nb-address="$PROTOCOL://10.20.130.7:6631" `
    -sb-address="$PROTOCOL://10.20.130.7:6632" -k8s-token="test" `
    -nb-address="$PROTOCOL://$MASTER:6631" `
    -sb-address="$PROTOCOL://$MASTER:6632" -k8s-token="test" `
    -service-cluster-ip-range="$SERVICE_CLUSTER_RANGE" `
    -cluster-subnet="$CLUSTER_SUBNET"
    ```
    @@ -411,5 +424,5 @@ spec:
    Debugging
    ---------

    Check [here](https://github.com/openvswitch/ovn-kubernetes/blob/master/docs/debugging.md) for information on debugging.
    Check [here](https://github.com/openvswitch/ovn-kubernetes/blob/master/docs/debugging.md) for official information on debugging.

  12. luisg0nc revised this gist Apr 20, 2018. 1 changed file with 2 additions and 2 deletions.
    4 changes: 2 additions & 2 deletions hybrid_kubernetes_using_ovn.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,8 +1,8 @@
    Hybrid Kubernetes and OVN
    =========================

    This file compiles all necessary information on how to setup a Kubernetes with both Linux and Windows Nodes,
    enabling both use of Windows and Linuxcontainers,
    This gist compiles all necessary information on how to setup a Kubernetes with both Linux and Windows Nodes,
    enabling both use of Windows and Linux docker containers,
    using [ovn-kubernetes](https://github.com/openvswitch/ovn-kubernetes/) as the network plugin.

    Two machines are required to run this, one will be running Linux Ubuntu and will act as our Master in the cluster and
  13. luisg0nc revised this gist Apr 20, 2018. 1 changed file with 9 additions and 5 deletions.
    14 changes: 9 additions & 5 deletions hybrid_kubernetes_using_ovn.md
    View file
    Original file line number Diff line number Diff line change
    @@ -58,11 +58,14 @@ To facilitate deployment, hostnames will be used on all machines and their respe
    Variables
    ---------

    | Variable | Description | Value |
    | --- | --- |---:|
    | Variable | Description | Ex. Value |
    | --- | --- |------:|
    | GW_IP | `todo` | `todo` |
    | OVERLAY_IP | `todo` | `todo` |
    | PROTOCOL | The protocol used, options are SSL and TCP, for sake of simplicity tcp will be used. Look [here](https://github.com/openvswitch/ovn-kubernetes/blob/master/vagrant/provisioning/setup-k8s-minion.sh#L67) for more details. | tcp |
    | CLUSTER_SUBNET | `todo` | 192.168.0.0/16 |
    | SERVICE_CLUSTER_RANGE | `todo` | 172.16.1.0/24 |


    Master Setup
    ------------
    @@ -203,7 +206,7 @@ sudo ovnkube -k8s-kubeconfig $HOME/kubeconfig.yaml -net-controller -loglevel=4 \
    -k8s-apiserver="http://$OVERLAY_IP:8080" \
    -logfile="/var/log/openvswitch/ovnkube.log" \
    -init-master=$MASTER -cluster-subnet="$CLUSTER_SUBNET" \
    -service-cluster-ip-range=172.16.1.0/24 \
    -service-cluster-ip-range="$SERVICE_CLUSTER_RANGE" \
    -nodeport \
    -nb-address="$PROTOCOL://$OVERLAY_IP:6631" \
    -sb-address="$PROTOCOL://$OVERLAY_IP:6632" 2>&1 &
    @@ -306,6 +309,7 @@ $GUID = (New-Guid).Guid
    ovs-vsctl set Open_vSwitch . external_ids:system-id="$($GUID)"
    ```

    ##### [Optiona] Fix DHCP

    The last script will change from dhcp to static ip, to change back to dhcp it will be necessary to give the our external interface macaddress to our new virtual interface and generate a new macaddress to give to our external interfacem, this will make sure we receive from dhcp the same IP to our virtual interface.
    @@ -367,8 +371,8 @@ KUBECONFIG
    -nodeport `
    -nb-address="$PROTOCOL://10.20.130.7:6631" `
    -sb-address="$PROTOCOL://10.20.130.7:6632" -k8s-token="test" `
    -service-cluster-ip-range="172.16.1.0/24" `
    -cluster-subnet="192.168.0.0/16"
    -service-cluster-ip-range="$SERVICE_CLUSTER_RANGE" `
    -cluster-subnet="$CLUSTER_SUBNET"
    ```

    Launching a Pod
  14. luisg0nc revised this gist Apr 20, 2018. 1 changed file with 42 additions and 4 deletions.
    46 changes: 42 additions & 4 deletions hybrid_kubernetes_using_ovn.md
    View file
    Original file line number Diff line number Diff line change
    @@ -13,6 +13,12 @@ This configuration is based on the available [vagrant](https://github.com/openvs
    I would like to thank [Alin Balutoiu](https://github.com/alinbalutoiu) and [Alin Gabriel Serdean](https://github.com/aserdean) for
    their amazing help.

    ### Overlay mode architecture diagram:

    The following digaram represents the internal architecture details
    of the overlay mode.

    ![alt text](https://i.imgur.com/i7sci9O.png "Overlay mode diagram")


    Versions
    @@ -43,8 +49,11 @@ Hostnames

    To facilitate deployment, hostnames will be used on all machines and their respective scripts.

    `complete this with table`

    | K8s Role | OS | Hostname |
    | --------- | ------- |-------- |
    | Master | Linux Ubuntu | Imperial |
    | Node | Linux Ubuntu | Sagres |
    | Node | Windows Server 1709 | Desperados |

    Variables
    ---------
    @@ -273,7 +282,7 @@ Get-VMSwitch -SwitchType External | Set-VMSwitch -AllowManagementOS $false
    ovs-vsctl --no-wait --may-exist add-br br-ex
    ovs-vsctl --no-wait add-port br-ex 'SWITCH'
    ovs-vsctl --no-wait add-port br-ex '$INTERFACE_ALIAS'
    Stop-Service ovs-vswitchd
    @@ -289,7 +298,7 @@ Enable-NetAdapter br-ex
    Remove-NetIPAddress -ifAlias br-ex -Confirm:$false
    New-NetIPAddress -ifAlias br-ex -IPAddress 10.20.130.19 -DefaultGateway 10.20.130.254 -PrefixLength 24
    New-NetIPAddress -ifAlias br-ex -IPAddress <interface_ip> -DefaultGateway <default_gateway> -PrefixLength 24
    Set-DnsClientServerAddress -InterfaceAlias br-ex -ServerAddresses ("8.8.8.8")
    @@ -310,6 +319,35 @@ Set-NetAdapter -Name br-ex -MacAddress <interface_mac> -Confirm:$false

    We will first need to register the node in the cluster. For this we will launch kubelet and wait for node registration to be checked, we can then stop the kubelet process and run ovnkube once, after that we can rerun the kubelet.

    #### Add kubeconfig file
    ```
    apiVersion: v1
    clusters:
    - cluster:
    server: http://imperial:8080
    name: default-cluster
    - cluster:
    server: http://imperial:8080
    name: local-server
    - cluster:
    server: http://imperial:8080
    name: ubuntu
    contexts:
    - context:
    cluster: ubuntu
    user: ubuntu
    name: ubuntu
    current-context: ubuntu
    kind: Config
    preferences: {}
    users:
    - name: ubuntu
    user:
    password: p1NVMZqhOOOqkWQq
    username: admin
    KUBECONFIG
    ```

    ##### Launch Kubelet
    ```
    .\kubelet.exe --hostname-override="desperados" `
  15. luisg0nc revised this gist Apr 20, 2018. 1 changed file with 34 additions and 2 deletions.
    36 changes: 34 additions & 2 deletions hybrid_kubernetes_using_ovn.md
    View file
    Original file line number Diff line number Diff line change
    @@ -41,7 +41,7 @@ Versions
    Hostnames
    ---------

    To facilitate deployment, hostnames will be used on all machines and respective scripts.
    To facilitate deployment, hostnames will be used on all machines and their respective scripts.

    `complete this with table`

    @@ -53,7 +53,7 @@ Variables
    | --- | --- |---:|
    | GW_IP | `todo` | `todo` |
    | OVERLAY_IP | `todo` | `todo` |
    | PROTOCOL | The protocol used by OVN, options are SSL and TCP, for sake of simplicity we will use tcp. Look [here](https://github.com/openvswitch/ovn-kubernetes/blob/master/vagrant/provisioning/setup-k8s-minion.sh#L67) for more details. | tcp |
    | PROTOCOL | The protocol used, options are SSL and TCP, for sake of simplicity tcp will be used. Look [here](https://github.com/openvswitch/ovn-kubernetes/blob/master/vagrant/provisioning/setup-k8s-minion.sh#L67) for more details. | tcp |

    Master Setup
    ------------
    @@ -333,6 +333,38 @@ We will first need to register the node in the cluster. For this we will launch
    -cluster-subnet="192.168.0.0/16"
    ```

    Launching a Pod
    ---------------

    To test our hybrid cluster, we will deploy a simple IIS pod. Since these pods will only run on windows,
    we can force node selection with the following:

    ```
    nodeSelector:
    beta.kubernetes.io/os: windows
    ```



    #### Test Pod
    ```
    apiVersion: v1
    kind: Pod
    metadata:
    name: nano-1709
    labels:
    name: webserver
    spec:
    containers:
    - name: nano
    image: cloudbase/pause
    imagePullPolicy: IfNotPresent
    - name: nano2
    image: microsoft/iis:windowsservercore-1709
    imagePullPolicy: IfNotPresent
    nodeSelector:
    beta.kubernetes.io/os: windows
    ```

    Debugging
    ---------
  16. luisg0nc revised this gist Apr 20, 2018. 1 changed file with 8 additions and 0 deletions.
    8 changes: 8 additions & 0 deletions hybrid_kubernetes_using_ovn.md
    View file
    Original file line number Diff line number Diff line change
    @@ -297,6 +297,14 @@ $GUID = (New-Guid).Guid
    ovs-vsctl set Open_vSwitch . external_ids:system-id="$($GUID)"
    ```
    ##### [Optiona] Fix DHCP

    The last script will change from dhcp to static ip, to change back to dhcp it will be necessary to give the our external interface macaddress to our new virtual interface and generate a new macaddress to give to our external interfacem, this will make sure we receive from dhcp the same IP to our virtual interface.

    ```
    Set-NetAdapter -Name <interface_name> -MacAddress <new_interface_mac> -Confirm:$false
    Set-NetAdapter -Name br-ex -MacAddress <interface_mac> -Confirm:$false
    ```

    #### Startup

  17. luisg0nc revised this gist Apr 20, 2018. 1 changed file with 100 additions and 8 deletions.
    108 changes: 100 additions & 8 deletions hybrid_kubernetes_using_ovn.md
    View file
    Original file line number Diff line number Diff line change
    @@ -41,16 +41,19 @@ Versions
    Hostnames
    ---------

    To facilitate deployment, hostnames will be used on all machines and respective scripts.

    `complete this with table`


    Variables
    ---------

    | Variable | Description | Value |
    | --- | --- |---:|
    | GW_IP | These | |
    | OVERLAY_IP | | |
    | PROTOCOL | tcp | |
    | GW_IP | `todo` | `todo` |
    | OVERLAY_IP | `todo` | `todo` |
    | PROTOCOL | The protocol used by OVN, options are SSL and TCP, for sake of simplicity we will use tcp. Look [here](https://github.com/openvswitch/ovn-kubernetes/blob/master/vagrant/provisioning/setup-k8s-minion.sh#L67) for more details. | tcp |

    Master Setup
    ------------
    @@ -203,8 +206,12 @@ Minion Node Setup

    #### Initial Setup

    `todo`

    #### Kubernetes Setup

    `todo`

    Windows Node Setup
    ------------------

    @@ -230,12 +237,97 @@ setx PATH "$env:Path;C:\Program Files\docker\;" -m

    ##### Download and Install OVS

    Link to download installer:
    [Cloudbase_OVS_Windows_2.9.0_BETA](https://docs.google.com/uc?export=download&id=1QnAtn8EhfbBFwn-1KPCkJ4EqCmEiT8g_)
    Download OVS for Windows from Cloudbase [2.9.0_beta](https://docs.google.com/uc?export=download&id=1QnAtn8EhfbBFwn-1KPCkJ4EqCmEiT8g_).

    Since this version is beta, we will have to disable driver certification:
    ```
    bcdedit /set testsigning yes
    ```

    Make sure to check __OVN Host__ on installation.

    ##### Download and Install Kubernetes

    You can find the client binaries [here](https://github.com/kubernetes/kubernetes/blob/master/CHANGELOG-1.9.md#downloads-for-v193).


    #### OVS Configuration

    ##### Setup a transparent docker network

    ```
    docker network create -d transparent --gateway $GATEWAY_IP --subnet $SUBNET `
    -o com.docker.network.windowsshim.interface="$INTERFACE_ALIAS" external
    ```

    ##### Configure VMSwitch and OVS

    ```
    Get-VMSwitch -SwitchType External | Disable-VMSwitchExtension "Cloudbase Open vSwitch Extension"
    ovs-vsctl --if-exists --no-wait del-br br-ex
    Get-VMSwitch -SwitchType External | Set-VMSwitch -AllowManagementOS $false
    Get-VMSwitch -SwitchType External | Set-VMSwitch -AllowManagementOS $false
    ovs-vsctl --no-wait --may-exist add-br br-ex
    Make sure to check __OVN Host__.
    ovs-vsctl --no-wait add-port br-ex 'SWITCH'
    Stop-Service ovs-vswitchd
    Get-VMSwitch -SwitchType External | Enable-VMSwitchExtension "Cloudbase Open vSwitch Extension"
    Start-Service ovs-vswitchd
    sleep 2
    Restart-Service ovs-vswitchd
    Enable-NetAdapter br-ex
    Remove-NetIPAddress -ifAlias br-ex -Confirm:$false
    New-NetIPAddress -ifAlias br-ex -IPAddress 10.20.130.19 -DefaultGateway 10.20.130.254 -PrefixLength 24
    Set-DnsClientServerAddress -InterfaceAlias br-ex -ServerAddresses ("8.8.8.8")
    $GUID = (New-Guid).Guid
    ovs-vsctl set Open_vSwitch . external_ids:system-id="$($GUID)"
    ```

    #### Startup

    We will first need to register the node in the cluster. For this we will launch kubelet and wait for node registration to be checked, we can then stop the kubelet process and run ovnkube once, after that we can rerun the kubelet.

    ##### Launch Kubelet
    ```
    .\kubelet.exe --hostname-override="desperados" `
    --pod-infra-container-image="cloudbase/pause" `
    --resolv-conf="" `
    --kubeconfig="C:\k\kubeconfig.yaml" `
    --network-plugin=cni `
    --cni-bin-dir="C:\k" `
    --cni-conf-dir="C:\etc\cni\net.d"
    ```

    ##### Run OVNkube
    ```
    .\ovnkube.exe -k8s-kubeconfig kubeconfig.yaml -loglevel=4 `
    -k8s-apiserver="http://10.20.130.7:8080" `
    -init-node="desperados" `
    -nodeport `
    -nb-address="$PROTOCOL://10.20.130.7:6631" `
    -sb-address="$PROTOCOL://10.20.130.7:6632" -k8s-token="test" `
    -service-cluster-ip-range="172.16.1.0/24" `
    -cluster-subnet="192.168.0.0/16"
    ```


    Debugging
    ---------

    #### OVS Setup
    Check [here](https://github.com/openvswitch/ovn-kubernetes/blob/master/docs/debugging.md) for information on debugging.

    #### Kubelet Setup
  18. luisg0nc revised this gist Apr 19, 2018. 1 changed file with 48 additions and 8 deletions.
    56 changes: 48 additions & 8 deletions hybrid_kubernetes_using_ovn.md
    View file
    Original file line number Diff line number Diff line change
    @@ -38,9 +38,22 @@ Versions
    | docker | 17.0-ce-rc3 |


    Hostnames
    ---------



    Variables
    ---------

    | Variable | Description | Value |
    | --- | --- |---:|
    | GW_IP | These | |
    | OVERLAY_IP | | |
    | PROTOCOL | tcp | |

    Master Setup
    --------
    ------------

    ### Initial Setup

    @@ -101,8 +114,6 @@ sudo make install
    popd
    ```

    #### Kubernetes Setup

    ##### Download and install K8s
    ```
    mkdir k8s
    @@ -115,6 +126,7 @@ cd server
    tar xvzf ../kubernetes/server/kubernetes-server-linux-amd64.tar.gz
    ```

    #### Kubernetes Setup

    ##### Install an etcd cluster
    ```
    @@ -186,16 +198,44 @@ sudo ovnkube -k8s-kubeconfig $HOME/kubeconfig.yaml -net-controller -loglevel=4 \
    ```


    ### Minion Node Setup
    Minion Node Setup
    ----------------

    #### Machine Configuration
    #### Initial Setup

    #### Kubernetes Setup

    ### Windows Node Setup
    Windows Node Setup
    ------------------

    #### Machine Configuration
    #### Initial Setup

    ##### Installing both Hyper-V and Container features
    ```
    Enable-WindowsOptionalFeature -Online -FeatureName containers –All
    Enable-WindowsOptionalFeature -Online -FeatureName Microsoft-Hyper-V –All
    ```

    ##### Force install Docker CE edition
    ```
    Invoke-WebRequest https://download.docker.com/win/static/test/x86_64/docker-17.11.0-ce-rc3.zip -UseBasicParsing -OutFile docker.zip
    Expand-Archive docker.zip -DestinationPath $Env:ProgramFiles
    Remove-Item -Force docker.zip
    $env:Path = $env:Path + "C:\Program Files\docker\;"
    cd "C:\Program Files\docker\"
    dockerd --register-service
    Start-Service docker
    setx PATH "$env:Path;C:\Program Files\docker\;" -m
    ```

    ##### Download and Install OVS

    Link to download installer:
    [Cloudbase_OVS_Windows_2.9.0_BETA](https://docs.google.com/uc?export=download&id=1QnAtn8EhfbBFwn-1KPCkJ4EqCmEiT8g_)

    Make sure to check __OVN Host__.

    #### Kubernetes Setup

    #### OVS Setup

    #### Kubelet Setup
  19. luisg0nc revised this gist Apr 19, 2018. 1 changed file with 1 addition and 0 deletions.
    1 change: 1 addition & 0 deletions hybrid_kubernetes_using_ovn.md
    View file
    Original file line number Diff line number Diff line change
    @@ -32,6 +32,7 @@ Versions
    ### Windows
    || Versions
    | ------------- |-------------:|
    | Windows Server | 1709 |
    | ovs-ovn-cloudbase | 2.9.0 (BETA) |
    | kubelet | 1.9.3 |
    | docker | 17.0-ce-rc3 |
  20. luisg0nc revised this gist Apr 19, 2018. 1 changed file with 83 additions and 0 deletions.
    83 changes: 83 additions & 0 deletions hybrid_kubernetes_using_ovn.md
    View file
    Original file line number Diff line number Diff line change
    @@ -102,6 +102,89 @@ popd

    #### Kubernetes Setup

    ##### Download and install K8s
    ```
    mkdir k8s
    cd k8s
    wget https://github.com/kubernetes/kubernetes/releases/download/v1.9.3/kubernetes.tar.gz
    tar xvzf kubernetes.tar.gz
    ./kubernetes/cluster/get-kube-binaries.sh
    mkdir server
    cd server
    tar xvzf ../kubernetes/server/kubernetes-server-linux-amd64.tar.gz
    ```


    ##### Install an etcd cluster
    ```
    sudo docker run --net=host -v /var/etcd/data:/var/etcd/data -d \
    gcr.io/google_containers/etcd:3.0.17 /usr/local/bin/etcd \
    --listen-peer-urls http://127.0.0.1:2380 \
    --advertise-client-urls=http://127.0.0.1:4001 \
    --listen-client-urls=http://0.0.0.0:4001 \
    --data-dir=/var/etcd/data
    ```

    ##### Start k8s daemons
    ```
    sudo sh -c 'echo "PATH=$PATH:$HOME/k8s/server/kubernetes/server/bin" >> /etc/profile'
    cd k8s/server/kubernetes/server/bin
    sudo ./kube-apiserver --service-cluster-ip-range=172.16.1.0/24 \
    --address=0.0.0.0 \
    --etcd-servers=http://127.0.0.1:4001 \
    --advertise-address=$GW_IP \
    --v=2 2>&1 0<&- &>/dev/null &
    ```

    ##### Wait for kube-apiserver to start up, then launch both Controller and Scheduler
    ```
    sudo ./kube-controller-manager --master=127.0.0.1:8080 --v=2 2>&1 0<&- &>/dev/null &
    sudo ./kube-scheduler --master=127.0.0.1:8080 --v=2 2>&1 0<&- &>/dev/null &
    ```

    ##### Create a kubeconfig file.
    ```
    apiVersion: v1
    clusters:
    - cluster:
    server: http://localhost:8080
    name: default-cluster
    - cluster:
    server: http://localhost:8080
    name: local-server
    - cluster:
    server: http://localhost:8080
    name: ubuntu
    contexts:
    - context:
    cluster: ubuntu
    user: ubuntu
    name: ubuntu
    current-context: ubuntu
    kind: Config
    preferences: {}
    users:
    - name: ubuntu
    user:
    password: p1NVMZqhOOOqkWQq
    username: admin
    KUBECONFIG
    ```


    ##### Start ovnkube
    ```
    sudo ovnkube -k8s-kubeconfig $HOME/kubeconfig.yaml -net-controller -loglevel=4 \
    -k8s-apiserver="http://$OVERLAY_IP:8080" \
    -logfile="/var/log/openvswitch/ovnkube.log" \
    -init-master=$MASTER -cluster-subnet="$CLUSTER_SUBNET" \
    -service-cluster-ip-range=172.16.1.0/24 \
    -nodeport \
    -nb-address="$PROTOCOL://$OVERLAY_IP:6631" \
    -sb-address="$PROTOCOL://$OVERLAY_IP:6632" 2>&1 &
    ```


    ### Minion Node Setup

    #### Machine Configuration
  21. luisg0nc revised this gist Apr 19, 2018. 1 changed file with 5 additions and 1 deletion.
    6 changes: 5 additions & 1 deletion hybrid_kubernetes_using_ovn.md
    View file
    Original file line number Diff line number Diff line change
    @@ -13,6 +13,8 @@ This configuration is based on the available [vagrant](https://github.com/openvs
    I would like to thank [Alin Balutoiu](https://github.com/alinbalutoiu) and [Alin Gabriel Serdean](https://github.com/aserdean) for
    their amazing help.



    Versions
    --------

    @@ -34,10 +36,12 @@ Versions
    | kubelet | 1.9.3 |
    | docker | 17.0-ce-rc3 |



    Master Setup
    --------

    #### Initial Setup
    ### Initial Setup

    ##### Add external repos to install docker and OVS from packages
    ```
  22. luisg0nc revised this gist Apr 19, 2018. 1 changed file with 2 additions and 1 deletion.
    3 changes: 2 additions & 1 deletion hybrid_kubernetes_using_ovn.md
    View file
    Original file line number Diff line number Diff line change
    @@ -34,7 +34,8 @@ Versions
    | kubelet | 1.9.3 |
    | docker | 17.0-ce-rc3 |

    ### Master Setup
    Master Setup
    --------

    #### Initial Setup

  23. luisg0nc revised this gist Apr 19, 2018. 1 changed file with 3 additions and 2 deletions.
    5 changes: 3 additions & 2 deletions hybrid_kubernetes_using_ovn.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,8 +1,9 @@
    Hybrid Kubernetes and OVN
    =========================

    This file contains information on how to setup a Kubernetes with both Linux and Windows Nodes, enabling both use of Windows and Linux
    containers, using [ovn-kubernetes](https://github.com/openvswitch/ovn-kubernetes/) as the network plugin.
    This file compiles all necessary information on how to setup a Kubernetes with both Linux and Windows Nodes,
    enabling both use of Windows and Linuxcontainers,
    using [ovn-kubernetes](https://github.com/openvswitch/ovn-kubernetes/) as the network plugin.

    Two machines are required to run this, one will be running Linux Ubuntu and will act as our Master in the cluster and
    the other machine will be running Windows which will be added to our cluster as a Work Node able to run Windows Containers.
  24. luisg0nc revised this gist Apr 19, 2018. 1 changed file with 63 additions and 4 deletions.
    67 changes: 63 additions & 4 deletions hybrid_kubernetes_using_ovn.md
    View file
    Original file line number Diff line number Diff line change
    @@ -2,13 +2,15 @@ Hybrid Kubernetes and OVN
    =========================

    This file contains information on how to setup a Kubernetes with both Linux and Windows Nodes, enabling both use of Windows and Linux
    containers, using [ovn-kubernetes](https://github.com/openvswitch/ovn-kubernetes/) as the network plugin.
    containers, using [ovn-kubernetes](https://github.com/openvswitch/ovn-kubernetes/) as the network plugin.

    Two machines are required to run this, one will be running Linux Ubuntu and will act as our Master in the cluster and
    the other machine will be running Windows which will be added to our cluster as a Work Node able to run Windows Containers.

    I would like to thank [Alin Balutoiu](https://github.com/alinbalutoiu) and [Alin Gabriel Serdean](https://github.com/aserdean) for
    helping me in this process.
    This configuration is based on the available [vagrant](https://github.com/openvswitch/ovn-kubernetes/tree/master/vagrant) setup.

    I would like to thank [Alin Balutoiu](https://github.com/alinbalutoiu) and [Alin Gabriel Serdean](https://github.com/aserdean) for
    their amazing help.

    Versions
    --------
    @@ -33,7 +35,64 @@ Versions

    ### Master Setup

    #### Machine Configuration
    #### Initial Setup

    ##### Add external repos to install docker and OVS from packages
    ```
    sudo apt-get update
    sudo apt-get install -y apt-transport-https ca-certificates
    echo "deb https://packages.wand.net.nz $(lsb_release -sc) main" | sudo tee /etc/apt/sources.list.d/wand.list
    sudo curl https://packages.wand.net.nz/keyring.gpg -o /etc/apt/trusted.gpg.d/wand.gpg
    sudo apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D
    sudo su -c "echo \"deb https://apt.dockerproject.org/repo ubuntu-xenial main\" >> /etc/apt/sources.list.d/docker.list"
    sudo apt-get update
    ```

    ##### Docker installation
    ```
    sudo apt-get purge lxc-docker
    sudo apt-get install -y linux-image-extra-$(uname -r) linux-image-extra-virtual
    sudo apt-get install -y docker-engine
    sudo service docker start
    ```

    ##### OVS installation and necessary dependencies
    ```
    sudo apt-get build-dep dkms
    sudo apt-get install python-six openssl python-pip -y
    sudo -H pip install --upgrade pip
    sudo apt-get install openvswitch-datapath-dkms=2.9.0-1 -y
    sudo apt-get install openvswitch-switch=2.9.0-2 openvswitch-common=2.9.0-2 libopenvswitch=2.9.0-2 -y
    sudo -H pip install ovs
    sudo apt-get install ovn-central=2.9.0-2 ovn-common=2.9.0-2 ovn-host=2.9.0-2 -y
    ```

    ##### Golang installation
    ```
    wget -nv https://dl.google.com/go/go1.9.2.linux-amd64.tar.gz
    sudo tar -C /usr/local -xzf go1.9.2.linux-amd64.tar.gz
    export PATH="/usr/local/go/bin:echo $PATH"
    export GOPATH=$HOME/work
    ```

    ##### Setup CNI directory
    ```
    sudo mkdir -p /opt/cni/bin/
    ```

    ##### Install OVN+K8S Integration
    ```
    mkdir -p $HOME/work/src/github.com/openvswitch
    pushd $HOME/work/src/github.com/openvswitch
    git clone https://github.com/openvswitch/ovn-kubernetes
    popd
    pushd $HOME/work/src/github.com/openvswitch/ovn-kubernetes/go-controller
    make 1>&2 2>/dev/null
    sudo make install
    popd
    ```

    #### Kubernetes Setup

  25. luisg0nc revised this gist Apr 19, 2018. 1 changed file with 38 additions and 6 deletions.
    44 changes: 38 additions & 6 deletions hybrid_kubernetes_using_ovn.md
    View file
    Original file line number Diff line number Diff line change
    @@ -7,14 +7,46 @@ containers, using [ovn-kubernetes](https://github.com/openvswitch/ovn-kubernetes
    Two machines are required to run this, one will be running Linux Ubuntu and will act as our Master in the cluster and
    the other machine will be running Windows which will be added to our cluster as a Work Node able to run Windows Containers.

    I would like to thank [Alin Balutoiu](https://github.com/alinbalutoiu) and [Alin Gabriel Serdean](https://github.com/aserdean) for
    helping me in this process.

    Versions
    --------

    ### Linux
    || Version
    || Versions
    | ------------- |-------------:|
    | Ubuntu | 16.04.4 LTS |
    | ovs-dkms | 2.9.0-1 |
    | ovs-common-switch | 2.9.0-2 |
    | ovn-central | 2.9.0-2 |
    | ovn-common-host |2.9.0-2 |
    | kubernetes | 1.9.3 |
    | docker | 17.0-ce-rc3 |

    ### Windows
    || Versions
    | ------------- |-------------:|
    | ovs-dkms-version | 2.9.0-1 |
    | ovs-common-switch-version | 2.9.0-2 |
    | ovn-central-version | 2.9.0-2 |
    | ovn-common-host-version |2.9.0-2 |
    | kubernetes-version | 1.9.3 |
    | ovs-ovn-cloudbase | 2.9.0 (BETA) |
    | kubelet | 1.9.3 |
    | docker | 17.0-ce-rc3 |

    ### Master Setup

    #### Machine Configuration

    #### Kubernetes Setup

    ### Minion Node Setup

    #### Machine Configuration

    #### Kubernetes Setup

    ### Windows Node Setup

    #### Machine Configuration

    #### Kubernetes Setup


  26. luisg0nc revised this gist Apr 12, 2018. 1 changed file with 17 additions and 1 deletion.
    18 changes: 17 additions & 1 deletion hybrid_kubernetes_using_ovn.md
    View file
    Original file line number Diff line number Diff line change
    @@ -1,4 +1,20 @@
    Hybrid Kubernetes and OVN
    =========================

    This file contains information on how to setup a Kubernetes with both Linux and Windows Nodes, enabling both use of Windows and Linux containers, using [ovn-kubernetes](https://github.com/openvswitch/ovn-kubernetes/) as the network plugin.
    This file contains information on how to setup a Kubernetes with both Linux and Windows Nodes, enabling both use of Windows and Linux
    containers, using [ovn-kubernetes](https://github.com/openvswitch/ovn-kubernetes/) as the network plugin.

    Two machines are required to run this, one will be running Linux Ubuntu and will act as our Master in the cluster and
    the other machine will be running Windows which will be added to our cluster as a Work Node able to run Windows Containers.

    Versions
    --------

    ### Linux
    || Version
    | ------------- |-------------:|
    | ovs-dkms-version | 2.9.0-1 |
    | ovs-common-switch-version | 2.9.0-2 |
    | ovn-central-version | 2.9.0-2 |
    | ovn-common-host-version |2.9.0-2 |
    | kubernetes-version | 1.9.3 |
  27. luisg0nc created this gist Apr 12, 2018.
    4 changes: 4 additions & 0 deletions hybrid_kubernetes_using_ovn.md
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,4 @@
    Hybrid Kubernetes and OVN
    =========================

    This file contains information on how to setup a Kubernetes with both Linux and Windows Nodes, enabling both use of Windows and Linux containers, using [ovn-kubernetes](https://github.com/openvswitch/ovn-kubernetes/) as the network plugin.