KFP version: 1.7.0+ Kubernetes version: 1.17+ Orchestrate Spark Jobs using Kubeflow pipelines Install kubeflow pipelines standalone or full kubeflow for standalone kubeflow pipelines installation https://www.kubeflow.org/docs/components/pipelines/installation/ for full kubeflow installation https://www.kubeflow.org/docs/started/installing-kubeflow/ Install Spark Operator https://github.com/GoogleCloudPlatform/spark-on-k8s-operator#installation Create Spark Service Account and add permissions kubectl apply -f ./scripts/spark-rbac.yaml Run the notebok kubeflow-pipeline.ipynb Access Kubflow/KFP UI OR Upload pipeline Upload the spark_job_pipeline.yaml file Create Run Start Pipeline add service account spark-sa Wait till the execution is finished. check the print-message logs to view the result
