Hi @xwt-ml, have you found any solution ? I saw on the kubeflow github that it should be fixed with kubeflow 1.9.0, but deployKF seem to not be able to upgrade kubeflow version.

I downgraded the kfp to use 1.8.22. Have not found a solution to work with kfp 2.0.

@xwt-ml @haiminh2001 can you please try restarting the Deployment/metadata-envoy-deployment in the kubeflow namespace?

I see from others that there is some kind of race condition on that pod which causes this error.

Run the following command to do the restart:

kubectl rollout restart deployment/metadata-envoy-deployment --namespace kubeflow

Assuming that restarting that Pod fixes it, I have made a PR to add a livenessProbe which will do it automatically:

• fix: add liveness probe for metadata-envoy-deployment #191

@xwt-ml @haiminh2001 can you please try restarting the Deployment/metadata-envoy-deployment in the kubeflow namespace?

I see from others that there is some kind of race condition on that pod which causes this error.

Run the following command to do the restart:

kubectl rollout restart deployment/metadata-envoy-deployment --namespace kubeflow

I also saw that issue yesterday from charmed kubeflow, I did restart the deployment and now it worked. But I am not 100% sure that restarting did the trick because I was applying other changes too. But adding probes can only benefit so I think you should definitely add the probes.

I restarted deployment/metadata-envoy-deployment and run a kfp2.0 pipeline in a new profile namespace. It works fine without the error.