One of the things that's really useful about Kubeflow is that the Kubeflow deployments will take care of setting up secured ingress behind which you can easily setup and deploy web applications like TensorBoard, Tektkon, a custom flask app, etc...

We should document this process.

The process basically consists of 3 steps

1. Create a K8s resource (Deployment, Statefulset, etc...) to deploy your service
2. Create a K8s service pointing at your service
3. Create a virtual service to add a reverse proxy route to your service.

The mnist E2E example does this.
https://github.com/kubeflow/examples/blob/master/mnist/mnist_gcp.ipynb

If you look at the section "Deploy the UI for the mnist app" it contains K8s resources for all 3 of the above.