This example is based on the Titanic OpenVaccine competition (https://www.kaggle.com/c/stanford-covid-vaccine). The objective of this exercise is to develop models and design rules for RNA degradation.

This pipeline was tested using Kubeflow 1.4 and kfp 1.1.2 and x86-64 and ARM based system which includes all Intel and AMD based CPU's and M1/M2 series Macbooks.

• If you haven’t already, sign up (https://www.arrikto.com/kubeflow-as-a-service/)
• Deploy Kubeflow

• Bump memory to 2GB and vCPUs to 2

• (Kubeflow as a Service) Open up a terminal in the Notebook Server and git clone the kubeflow/examples repository

git clone https://github.com/kubeflow/examples

• If you haven’t already, sign up (https://hub.docker.com/) for DockerHub
• If you haven’t already, install Docker Desktop locally (https://www.docker.com/products/docker-desktop/) OR install the Docker command line utility (https://docs.docker.com/get-docker/) and enter sudo docker login command in your terminal and log into Docker with your your DockerHub username and password

• If you haven’t already done so, sign up (https://www.kaggle.com/) for Kaggle
• (On Kaggle) Generate an API token (https://www.kaggle.com/docs/api)
• (Kubeflow as a Service) Create a Kubernetes secret

kubectl create secret generic kaggle-secret --from-literal=KAGGLE_USERNAME=<username> --from-literal=KAGGLE_KEY=<api_token> 

• (Locally) If you don’t have it already, install Git (https://github.com/git-guides/install-git)

• (Locally) Git clone the kubeflow/examples repository

git clone https://github.com/kubeflow/examples

• (Kubeflow as a Service) Navigate to the openvaccine-kaggle-competition directory
• Create a resource.yaml file

resource.yaml:

apiVersion: "kubeflow.org/v1alpha1"
kind: PodDefault
metadata:
  name: kaggle-access
spec:
 selector:
  matchLabels:
    kaggle-secret: "true"
 desc: "kaggle-access"
 volumeMounts:
 - name: secret-volume
   mountPath: /secret/kaggle
 volumes:
 - name: secret-volume
   secret:
    secretName: kaggle-secret

• Apply created resource using: kubectl apply -f resource.yaml

• (Locally) Navigate to the openvaccine-kaggle-competition/pipeline-components/load-data directory
• Open up the load.py file
• Note the code in this file that will perform the actions required in the “load-data” pipeline step

• (Locally) Navigate to the openvaccine-kaggle-competition/pipeline-components/load-data directory
• Build the Docker image if locally you are using arm64 (Apple M1)

docker build --platform=linux/amd64 -t <docker_username>/<docker_imagename>:<tag>-amd64 . 

• OR build the Docker image if locally you are using amd64

docker build -t <docker_username>/<docker_imagename>:<tag> .

• (Locally) Navigate to the openvaccine-kaggle-competition/pipeline-components/load-data directory
• Push the Docker image if locally you are using arm64 (Apple M1)

docker push <docker_username>/<docker_imagename>:<tag>-amd64 

• OR build the Docker image if locally you are using amd64

docker push <docker_username>/<docker_imagename>:<tag>

• (Locally) Navigate to the openvaccine-kaggle-competition/pipeline-components/preprocess-data directory
• Open up the preprocess.py file
• Note the code in this file that will perform the actions required in the “preprocess” pipeline step

• (Locally) Navigate to the openvaccine-kaggle-competition/pipeline-components/preprocess-data directory
• Build the Docker image if locally you are using arm64 (Apple M1)

docker build --platform=linux/amd64 -t <docker_username>/<docker_imagename>:<tag>-amd64 . 

• OR build the Docker image if locally you are using amd64

docker build -t <docker_username>/<docker_imagename>:<tag> .

• (Locally) Navigate to the openvaccine-kaggle-competition/pipeline-components/preprocess-data directory
• Push the Docker image if locally you are using arm64 (Apple M1)

docker push <docker_username>/<docker_imagename>:<tag>-amd64 

• OR build the Docker image if locally you are using amd64

docker push <docker_username>/<docker_imagename>:<tag>

• (Locally) Navigate to the openvaccine-kaggle-competition/pipeline-components/model-training directory
• Open up the model.py file
• Note the code in this file that will perform the actions required in the “train” pipeline step

• (Locally) Navigate to the openvaccine-kaggle-competition/pipeline-components/model-training directory
• Build the Docker image if locally you are using arm64 (Apple M1)

docker build --platform=linux/amd64 -t <docker_username>/<docker_imagename>:<tag>-amd64 . 

• OR build the Docker image if locally you are using amd64

docker build -t <docker_username>/<docker_imagename>:<tag> .

• (Locally) Navigate to the openvaccine-kaggle-competition/pipeline-components/model-training directory
• Push the Docker image if locally you are using arm64 (Apple M1)

docker push <docker_username>/<docker_imagename>:<tag>-amd64 

• OR build the Docker image if locally you are using amd64

docker push <docker_username>/<docker_imagename>:<tag>

• (Locally) Navigate to the openvaccine-kaggle-competition/pipeline-components/model-evaluation directory
• Open up the eval.py file
• Note the code in this file that will perform the actions required in the “test” pipeline step

• (Locally) Navigate to the openvaccine-kaggle-competition/pipeline-components/model-evaluation directory
• Build the Docker image if locally you are using arm64 (Apple M1)

docker build --platform=linux/amd64 -t <docker_username>/<docker_imagename>:<tag>-amd64 . 

• OR build the Docker image if locally you are using amd64

docker build -t <docker_username>/<docker_imagename>:<tag> .

• (Locally) Navigate to the openvaccine-kaggle-competition/pipeline-components/model-evaluation directory
• Push the Docker image if locally you are using arm64 (Apple M1)

docker push <docker_username>/<docker_imagename>:<tag>-amd64 

• OR build the Docker image if locally you are using amd64

docker push <docker_username>/<docker_imagename>:<tag>

• (Kubeflow as a Service) Navigate to the openvaccine-kaggle-competition directory
• Update the openvaccine-kaggle-competiton-kfp.py with accurate Docker Image inputs

   return dsl.ContainerOp(
        name = 'load-data', 
        image = '<dockerhub username>/<image name>:<tag>',

—-----

def GetMsg(comp1):
    return dsl.ContainerOp(
        name = 'preprocess',
        image = '<dockerhub username>/<image name>:<tag>',

—-----

def Train(comp2, trial, epoch, batchsize, embeddim, hiddendim, dropout, spdropout, trainsequencelength):
    return dsl.ContainerOp(
        name = 'train',
        image = '<dockerhub username>/<image name>:<tag>',

—-----

def Eval(comp1, trial, epoch, batchsize, embeddim, hiddendim, dropout, spdropout, trainsequencelength):
    return dsl.ContainerOp(
        name = 'Evaluate',
  image = '<dockerhub username>/<image name>:<tag>',

• (Locally) Navigate to the openvaccine-kaggle-competition directory and delete the existing openvaccine-kaggle-competition-kfp.yaml file
• (Kubeflow as a Service) Navigate to the openvaccine-kaggle-competition directory

Build a python virtual environment :

Step a) Update pip

python3 -m pip install --upgrade pip

Step b) Install virtualenv

sudo pip3 install virtualenv

Step c) Check the installed version of venv

virtualenv --version

Step d) Name your virtual enviornment as kfp

virtualenv kfp

Step e) Activate your venv.

source kfp/bin/activate

After this virtual environment will get activated. Now in our activated venv we need to install following packages:

sudo apt-get update
sudo apt-get upgrade
sudo apt-get install -y git python3-pip

python3 -m pip install  kfp==1.1.2

After installing packages create the yaml file

Inside venv point your terminal to a path which contains our kfp file to build pipeline (openvaccine-kaggle-competition-kfp.py) and run these commands to generate a yaml file for the Pipeline:

python3 openvaccine-kaggle-competition-kfp.py

Download the openvaccine-kaggle-competition-kfp.yaml file that was created to your local openvaccine-kaggle-competition directory

• (Kubeflow as a Service) Within the Kubeflow Central Dashboard, navigate to the Experiments (KFP) > Create Experiment view
• Name the experiment and click Next
• Click on Experiments (KFP) to view the experiment you just created

• (Kubeflow as a Service) Within the Kubeflow Central Dashboard, navigate to the Pipelines > +Upload Pipeline view
• Name the pipeline
• Click on Upload a file
• Upload the local openvaccine-kaggle-competition-kfp.yaml file
• Click Create

• (Kubeflow as a Service) Click on Create Run in the view from the previous step
• Choose the experiment we created in Step 23
• Input your desired run parameters. For example:

TRIAL = 1
EPOCHS = 2
BATCH_SIZE = 64
EMBED_DIM = 100
HIDDEN_DIM = 128
DROPOUT = .2
SP_DROPOUT = .3
TRAIN_SEQUENCE_LENGTH = 107

• Click Start
• Click on the run name to view the runtime execution graph

While running the pipeline as mentioned above you may come across this error:

errorlog:

kaggle.rest.ApiException: (403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Content-Type': 'application/json', 'Date': 'Thu, 23 Jun 2022 11:31:18 GMT', 'Access-Control-Allow-Credentials': 'true', 'Set-Cookie': 'ka_sessionid=6817a347c75399a531148e19cad0aaeb; max-age=2626560; path=/, GCLB=CIGths3--ebbUg; path=/; HttpOnly', 'Transfer-Encoding': 'chunked', 'Vary': 
HTTP response body: b'{"code":403,"message":"You must accept this competition\\u0027s rules before you\\u0027ll be able to download files."}'

This error occours for two reasons:

• Your Kaggle account is not verified with your phone number.
• Rules for this specific competitions are not accepted.

Lets accept Rules of competition

Click on "I Understand and Accept". After this you will be prompted to verify your account using your phone number:

Add your phone number and Kaggle will send the code to your number, enter this code and verify your account. ( Note: pipeline wont run if your Kaggle account is not verified )

After the kaggle account is verified pipeline run is successful we will get the following: