Originally, this issue was tracking a 1.8.1 release, but now that 1.9.0 is so close, we will just include them in that.

We are planning to do an overall Kubeflow 1.8.1 patch release which will include a number of fixes to issues uncovered/missed in the 1.8.0 release.

Missing Stuff

There may have been other critical PRs / Issues raised since we created this issue.

We should review the recent activity kubeflow/kubeflow repo before finalizing this list:

• https://github.com/kubeflow/kubeflow/issues
• https://github.com/kubeflow/kubeflow/pulls

Other Tracking Issues

• Manifests (All Components): KF release 1.9 tracking manifests#2600
• Distributions (All Components): [Kubeflow 1.9] Distributions and Kubeflow 1.9 manifests#2611
• Roadmap (Notebooks WG): Notebooks WG roadmap for KF 1.9 manifests#2608

Changes for 1.9.0

We need to finish/fix these PRs and then cherry-pick them into the v1.9-branch before cutting the 1.9.0 tag.

Notebook Images

• jupyter notebook not connect only #7370
  • Fixed By: fix: jupyterlab runtime volume permissions #7407
• Notebook containers can time out starting with very slow storage:
  • Fixed By: Prevent S6 timeouts on container start by setting S6_CMD_WAIT_FOR_SER… #7421
  • Follow Up PR: fix: increase S6_CMD_WAIT_FOR_SERVICES_MAXTI… #7496
• Update Notebook Images:
  • Fixed By: feat: update notebook server images #7590

Notebooks/TensorBoards Controller

• Hard-coded VirtualService hosts:
  • Fixed By: feat: allow setting ISTIO_HOST in notebook/tensorboard controller for Istio VirtualServices #6902
• Cannot see the list of notebooks [500] An error occurred in the backend.  #6886
  • Fixed by: fix(notebooks): add CRD validation for empty image and container #7495

PVC Viewer Controller

• chore: update pvcviewer dependencies #7561

Notebook Spawner UI (Jupyter Web App)

• Trim whitespace on custom images:
  • Fixed By: Trim the custom Jupyter image string #6529
• Notebook doesn't created after upgrade to Kubeflow 1.8 #7456
  • Fixed By: fix: 🏷️ Ensure username is returned by get_username as a string #7457
• Create notebooks page and get [500] error #7441
  • Caused by: Handle edge case when node capacity is undefined gracefully. Closes #7280. #7281
  • Fixed by: fix(crud-web-apps/jupyter): dict variable reference for node.status.capacity #7485
• Creating or starting notebook throws 500 error - KeyError: 'message' #7454
  • Caused by: jwa: Improve how JWA exposes errors #6952 (see get_status_from_container_state())
  • Fixed by: fix(notebooks): correctly handle status.containerState messages #7585

PVC Viewer UI

• Show "waiting" status on PVC Viewer Pods:
  • Fixed By: fix: return waiting on no pvcviewer status #7350

Tensorboards Web App

• fix: remove incorrect space in TWA_APP_SECURE_COOKIES config #7274

Central Dashboard

• fix: update central dashboard to NodeJS 16 #7578
• feat: big update for manage contributors page #7582
• feat: update dashboard styles and links #7583
• feat(dashboard): allow using {ns} template in externalLinks #7138
• centraldashboard: don't display All namespaces in Manage Contributors screen (#7188) #7242
• fix: incorrect references DASHBOARD_LINKS_CONFIGMAP env-var #7264
• fix: email validation parity for profile creation #7597
• fix: kserve models link in sidebar of dashboard #7603

Future Changes for 1.9.1

Repo Cleanup and CI

• ci: read-only token-permissions for all workflows #7278

Notebook Images

• Add kubeflow-katib and kserve packages to default notebook images notebooks#122
• feat: add Intel Gaudi example notebook images #7635

Notebooks Controller

• tensorboard fails to connect if there is a notebook with the same name dashboard#50
  • Fixed By: fix: resolve service name conflict for notebooks and tensorboards #7468
  • WARNING: this needs a lot of testing and work to ensure it is safe to do in-place upgrades
• notebook-controller: Update dependency versions #7152
  • WARNING: this can be deferred to a future update as it will need to be rebased and retested.
• feat: always set a Pod status in Notebook CRD #7015
  • WARNING: needs to be reviewed
• fix: add lease RBAC to notebook controller role (for leader election) #6942
  • WARNING: needs to be reviewed
• feat: expose runningCount and totalCount in notebook controller metrics #6838
  • WARNING: needs to be reviewed
• feat: expose NotebookCullingCount and NotebookCullingTimestamp in prometheus metrics #6488
  • WARNING: needs to be reviewed
• fix: unnecessary notebook updates caused by compare issues #6474
  • WARNING: needs to be reviewed
• fix: add lease RBAC to notebook controller role (for leader election) #6942
  • WARNING: needs to be reviewed

Profile Controller

• profile controller fails to update profile owner dashboard#33
  • Fixed by: Fix (profile controller): Profile controller fails to update profile owner, fixes #6576 and #5449 #7276
  • Fixed By: feat: allow updating profile owners #7547
    • WARNING: this needs a bit of work, it is NOT ready to merge
• feat(profile-controller): propagate labels from CR to namespace #7481
  • THIS SHOULD NOT BE INCLUDED
  • WARNING: including this introduces issues for ArgoCD and possible security issues by allowing users to set arbitrary namespace labels by updating profile labels.
• fix: extend profile AuthorizationPolicy for KServe models #6627
  • WARNING: needs to be updated
• fix: stop unnecessary updates of profiles #6435
  • WARNING: needs to be reviewed
• feat: make profile controller error logs more descriptive #6231
  • WARNING: needs to be reviewed

Tensorboard Controller

• Stop using gcr.io/kubebuilder/kube-rbac-proxy in TensorBoards before March 18, 2025 dashboard#58
  • NEEDS A PR
• fix: add error handling to tensorboard controller #7067
  • WARNING: needs to be reviewed
• fix: use "required" in tensorboard node affinity, rather than "preferred" #7024
  • WARNING: needs to be reviewed
• feat: make TensorBoard image configurable in CRD #6783
  • WARNING: needs to be reviewed

Notebook Spawner UI

• chore: remove unused jupyter-notebook-role and jupyter-notebook-role-binding #7616
  • This looks mostly ok to me, as I cant see any usage of these roles/rolebinding.
• feat: add gpusCount to spawner_ui_config.yaml #7295
  • WARNING: in its current state, this is a breaking change, we need to find a way to make it backwards compatible.
• feat: add memoryVendors and arbitrary counts for notebook GPU resource limits #6657
  • WARNING: there are a bunch of PRs which do similar things, we should only merge one of them
• feat(notebooks): only display allocatable GPU configuration when creating notebook #7124
  • WARNING: needs to be reviewed
• feat(notebook): dynamically display the allocatable GPU type/num in notebook server #7134
  • WARNING: needs to be reviewed
• web-apps: Implement watch to avoid constant polling from backend to K8s #7080
  • WARNING: needs to be reviewed
• feat: support selector.matchExpression in PodDefaults #7551
  • WARNING: needs to be reviewed
• fix: notebooks ui should support PodDefaults with matchExpressions #6983
  • WARNING: needs to be reviewed
• fix: typo when setting server_type for jupyter notebook (no in app affect) #6666
  • this change does nothing, but is fine to fix a typo
• fix: correctly display notebook images from registries with port #6351
  • WARNING: needs to be reviewed

PVC Viewer UI / Controller

• volume app list pvcs facing 404 error after upgrade kubeflow from v1.7 to v1.8 #7414
  • Caused by: Add PVCViewer Frontend Integration #7179 (see this part)
  • Fixed By: (NEED A PR)
• pvcviewer-controller-manager creates pvcviewer pod on different node from notebook server pod dashboard#60:
  • Fixed By: (NEED A PR)

Central Dashboard

• fix: show correct BUILD_VERSION in centraldashboard #6978
  • THIS SHOULD NOT BE INCLUDED
  • WARNING: we can probably just remove the build version section of the UI instead
• fix: double URL encoding in central-dashboard links #6564
  • WARNING: needs to be reviewed
• feat(dashboard): include fonts as assets #6009
  • WARNING: we need to confirm we are allowed to include these fonts in Apache 2.0 licenced code
• fix: show error logs for YAML parsing in notebooks ui #7497
  • WARNING: needs to be reviewed

Admission Webhook (PodDefaults)

• jupyter-web-app's PodDefault configurations are keyed by their label selector's key, not their name notebooks#121
  • (Need PR)
• feat: allow same volume to be mounted multiple times in single PodDefault #6847
  • WARNING: needs to be reviewed

KFAM

• feat: use native RBAC subject to find RoleBindings in KFAM, rather than annotations #6839
  • WARNING: needs to be reviewed

Future Changes for 1.10.0

See #7459