Open
Description
It's not clear what the state of Pachyderm-kubeflow integration is, and AFAICT there's no mention of Kubeflow integration in the pach docs
Questions to answer
- Can Pachyderm and Kubeflow be used together? It seems yes, there's an example of using KF with pach in KF's docs, thanks to @Nick-Harvey: Adding Pachyderm Example (squashed) kubeflow/examples#522 (building on an example started by @dwhitena )
- What do we consider the current best practice for integration?
- What's missing?
- Expand Kubeflow example to leverage TFjobs Expand KubeFlow example to leverage TFjobs #3210
- Pachyderm File System Source Type for KVC Pachyderm File System Source Type for KVC intel/vck#22
- I think "proper" kubeflow integration was why we wanted CSIs. What's the status of that work, and how does that relate to the rest of this?
- There still seems to be something missing from the example about "how to use pachyderm within kubeflow pipelines" Kubeflow pipelines with pachyderm. kubeflow/pipelines#1162 (I don't actually know enough about kubeflow to fully understand this question, but we should engage in that thread to figure out what's missing to address this user's use-case)
- how does this relate to the proposed Pachyderm Pipeline Improvements Design: Pachyderm Pipeline Improvements #3345 ?