Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Enable and document for Kubeflow 1.10 Kserve secure inferencing from inside and outside the cluster with tokens #2811

Open
7 tasks done
juliusvonkohout opened this issue Jul 22, 2024 · 16 comments
Open
7 tasks done

Enable and document for Kubeflow 1.10 Kserve secure inferencing from inside and outside the cluster with tokens #2811

juliusvonkohout opened this issue Jul 22, 2024 · 16 comments
Assignees
@kromanow94 @juliusvonkohout
Labels
good first issue Good for newcomers help wanted Extra attention is needed lifecycle/frozen

Comments

@juliusvonkohout
Copy link
Member

Validation Checklist

Version

master

Describe your issue

From @kromanow94

I did some investigation and I found out that this is because the VirtualServices created by kserve are configured by default to use cluster-local-gateway. istio-ingressgateway is configured with AuthorizationPolicy istio-ingressgateway-oauth2-proxy which enforces the traffic to go through oauth2-proxy. There is no such AuthorizationPolicy for cluster-local-gateway.

So, I see two options:

  1. Configure Istio auth for current setup with cluster-local-gateway
    1. Create cluster-local-gateway-oauth2-proxy AuthorizationPolicy to enforce authentication with oauth2-proxy: 
      apiVersion: security.istio.io/v1
kind: AuthorizationPolicy
metadata:
  name: cluster-local-gateway-oauth2-proxy
  namespace: istio-system
spec:
  action: CUSTOM
  provider:
    name: oauth2-proxy
  rules:
  - {}
  selector:
    matchLabels:
      app: cluster-local-gateway
    2. Depending on your setup, if the model is deployed in Kubeflow managed namespace (KF Profile, for example kubeflow-user-example-com), you also have to configure access to the sklearn-iris deployment: 
      apiVersion: security.istio.io/v1beta1
kind: AuthorizationPolicy
metadata:
  name: sklearn-iris-predictor-allow
  namespace: kubeflow-user-example-com
spec:
  selector:
    matchLabels:
      serving.knative.dev/service: sklearn-iris-predictor
  action: ALLOW
  rules:
  - {}
    3. Testing with curl: 
      $ curl -XPOST -v "http://sklearn-iris.kubeflow-user-example-com.svc.cluster.local/v1/models/sklearn-iris:predict" -H "Authorization: Bearer $(cat /run/secrets/kubernetes.io/serviceaccount/token)" -d '{"instances": [[6.8,  2.8,  4.8,  1.4], [6.0,  3.4,  4.5,  1.6]]}' -H "Content-Type: application/json"
Note: Unnecessary use of -X or --request, POST is already inferred.
* Host sklearn-iris.kubeflow-user-example-com.svc.cluster.local:80 was resolved.
* IPv6: (none)
* IPv4: 172.20.1.23
*   Trying 172.20.1.23:80...
* Connected to sklearn-iris.kubeflow-user-example-com.svc.cluster.local (172.20.1.23) port 80
> POST /v1/models/sklearn-iris:predict HTTP/1.1
> Host: sklearn-iris.kubeflow-user-example-com.svc.cluster.local
> User-Agent: curl/8.7.1
> Accept: */*
> Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6Ikh3ZUQ2enNYYnRZNUFZQk8xX1ZKc3ZCZGwwRmR3dTdwRURiQXpDN3c5MncifQ.eyJhdWQiOlsiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiLCJodHRwczovL2t1YmVybmV0ZXMuZGVmYXVsdC5zdmMiLCJodHRwczovL2t1YmVybmV0ZXMuZGVmYXVsdCJdLCJleHAiOjIwMjg1NDA4NzAsImlhdCI6MTcxMzE4MDg3MCwiaXNzIjoiaHR0cHM6Ly9rdWJlcm5ldGVzLmRlZmF1bHQuc3ZjLmNsdXN0ZXIubG9jYWwiLCJrdWJlcm5ldGVzLmlvIjp7Im5hbWVzcGFjZSI6Imt1YmVmbG93LXVzZXItZXhhbXBsZS1jb20iLCJwb2QiOnsibmFtZSI6ImN1cmwiLCJ1aWQiOiI3ZGI2ZjliNC0zZTliLTQ3ZDUtOWI4ZC0yZjhiMWVkNTVhZjkifSwic2VydmljZWFjY291bnQiOnsibmFtZSI6ImRlZmF1bHQtZWRpdG9yIiwidWlkIjoiODZhZmM3OGYtMTIzYS00MDMwLWI5YjQtZDllYWQ5YmE2NTc4In19LCJuYmYiOjE3MTMxODA4NzAsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlZmxvdy11c2VyLWV4YW1wbGUtY29tOmRlZmF1bHQtZWRpdG9yIn0.iY9WY7vqFQvxv3mzFYlnKQ3arG631movAfkIM1eWH_UdsQuWUupIz7wak81pOM23gBPpYxMT5HR1ZgVHYWG07Neh4e1ySUzhmPNNfydSIs-jUP1P8BjEPq3BdSQ9j_1pGggMDXFM4msFnEdAjlmpl23yDKOoJCj0RDV3fZIiA-mf7wLyiv_E38ah1ygZXYjrTCdzstCH02aZ7VCLc1dPETttE7nlF3YoaurwHJzZF6WHXmQlVdU2yMg0RT8uRDBUDI6WTq_guxjuEBEJrj166pXbp1MBvslMBUYXPV3StQ-AXnvQUyCBDoa5NOlJKOht3UOhGeS_-1A50ctjsl8xKw
> Content-Type: application/json
> Content-Length: 65
> 
* upload completely sent off: 65 bytes
< HTTP/1.1 200 OK
< content-length: 21
< content-type: application/json
< date: Mon, 15 Apr 2024 12:47:28 GMT
< server: envoy
< x-envoy-upstream-service-time: 9
< 
* Connection #0 to host sklearn-iris.kubeflow-user-example-com.svc.cluster.local left intact
{"predictions":[1,1]}
  2. Change the kserve config to use istio-ingressgateway instead of cluster-local-gateway. This touches kserve which I don't have a lot of experience with. I tried changing the inferenceservice-config ConfigMap to define the "localGatewayService": "istio-ingressgateway.istio-system.svc.cluster.local" and "localGateway": "kubeflow/kubeflow-gateway" but that didn't work for some reason, probably something is missing...

@juliusvonkohout do you think we should add this AuthorizationPolicy for cluster-local-gateway to the manifests?

Steps to reproduce the issue

See above

Put here any screenshots or videos (optional)

No response
This was referenced Jul 22, 2024
Closed
Closed
Closed
@juliusvonkohout
Copy link
Member Author

@kromanow94 yes we should tackle this for 1.10

@juliusvonkohout juliusvonkohout changed the title Document Kserve secure inferencing from inside and outside the cluster with tokens Enable and document for Kubeflow 1.10 Kserve secure inferencing from inside and outside the cluster with tokens Jul 22, 2024
@juliusvonkohout juliusvonkohout mentioned this issue Jul 22, 2024
Closed
@kromanow94
Copy link
Contributor

@juliusvonkohout , makes sense. From conceptual point of view, what do we expect and what are the assumptions?

I guess it's easy decision for Out Of Cluster - the Authorization header should contain a valid JWT.

What about from in-cluster? Do we expect that access to given kserve endpoint should be enabled without Authorization header when called from the same namespace? From other namespace we probably want to use a valid JWT either way so it's similar as from out of cluster, just the istio ingress gateway would be cluster-local.

@juliusvonkohout
Copy link
Member Author

"Do we expect that access to given kserve endpoint should be enabled without Authorization header when called from the same namespace?" I am not sure. Actually i like to enforce security everywhere. @kimwnasptd what do you think?

@kromanow94
Copy link
Contributor

Actually i like to enforce security everywhere

That would also ease and unify the approach from implementation perspective.

dpoulopoulos added a commit to dpoulopoulos/skin-cancer-detection that referenced this issue Jul 24, 2024
@dpoulopoulos
client: Add a client Notebook
a136885 
Add a Notebook that demonstrates how to invoke the KServe Inference
Service.

NOTE: Do not forget to create an Authorization Policy for the Inference
Service. See here: kubeflow/manifests#2811

Signed-off-by: Dimitris Poulopoulos <dimitris.a.poulopoulos@gmail.com>
dpoulopoulos added a commit to dpoulopoulos/skin-cancer-detection that referenced this issue Jul 25, 2024
@dpoulopoulos
client: Add a client Notebook
cbf7629 
Add a Notebook that demonstrates how to invoke the KServe Inference
Service.

NOTE: Do not forget to create an Authorization Policy for the Inference
Service. See here: kubeflow/manifests#2811

Signed-off-by: Dimitris Poulopoulos <dimitris.a.poulopoulos@gmail.com>
@padrian2s
Copy link

I bump into approval_prompt for v1.9, but yes the docs are missing

curl -XPOST -v "http://sklearn-iris.kubeflow-user-example-com.svc.cluster.local/v1/models/sklearn-iris:predict" -H "Authorization: Bearer $(cat /run/secrets/kubernetes.io/serviceaccount/token)" -d '{"instances": [[6.8, 2.8, 4.8, 1.4], [6.0, 3.4, 4.5, 1.6]]}' -H "Content-Type: application/json"
Note: Unnecessary use of -X or --request, POST is already inferred.

  • Trying 172.20.112.202:80...
  • Connected to sklearn-iris.kubeflow-user-example-com.svc.cluster.local (172.20.112.202) port 80 (#0)

POST /v1/models/sklearn-iris:predict HTTP/1.1
Host: sklearn-iris.kubeflow-user-example-com.svc.cluster.local
User-Agent: curl/7.88.1
Accept: /
Authorization: Bearer <>
Content-Type: application/json
Content-Length: 65

< HTTP/1.1 302 Found
< location: /dex/auth?approval_prompt=force&client_id=kubeflow-oidc-authservice&code_challenge=xzijon7LEPKr6gr3FsDcXOYcw9dSVgpN0aof3dyehSo&code_challenge_method=S256&redirect_uri=%2Foauth2%2Fcallback&response_type=code&scope=profile+email+groups+openid&state=YwJa9xq0i_wZuyC_RE8P90Fgb5eybkRasgrk6q5Nd0o%3A%2Fv1%2Fmodels%2Fsklearn-iris%3Apredict
< set-cookie: oauth2_proxy_kubeflow_csrf=avnayyw40z0m5rKMuBfHOjugZwMS94CxZTj04EJ_SVtoCAM5EixXNlzgRLj9Qf2V0Y3ARRgtBRnZGkqQQR7DVQaZuAUBENgnfPoxQD3wrWS6c3r5GwZaFhryWgssC39nnsN92984toftiQDWenP1NtqY6UQUMd4mt6gjQb3rWB62Rd3hwv6l1p36HAOHwFi-FhFBwPHgsgA7hcXiULacdx9ycmjkWoJZHFchLUGPleVpjkdTxQXfVgWDF7Gh5A==|1723062375|VMndKKL-bTECRokaEjTfvrEAPhfoZfGLsIJmf9BvhYI=; Path=/; Expires=Wed, 07 Aug 2024 20:41:15 GMT; HttpOnly
< date: Wed, 07 Aug 2024 20:26:15 GMT
< server: envoy
< x-envoy-upstream-service-time: 1
< content-length: 0
<

  • Connection #0 to host sklearn-iris.kubeflow-user-example-com.svc.cluster.local left intact

@juliusvonkohout
Copy link
Member Author

"@juliusvonkohout do you think we should add this AuthorizationPolicy for cluster-local-gateway to the manifests?" Yes, we have to move one. This is too complex for many users to understand and we need to provide it out of the box.

@juliusvonkohout
Copy link
Member Author

So @kromanow94 please create a PR if you have time.

@juliusvonkohout
Copy link
Member Author

There seems to be a lot of wrong/outdated stuff in kserve/kserve#3260 and we should probably fix it there as well.

@juliusvonkohout juliusvonkohout mentioned this issue Aug 15, 2024
Closed
@jaffe-fly
Copy link

by my test in kubeflow 1.9 cluster, https://github.com/kserve/kserve/tree/master/docs/samples/istio-dex,this is cant run .
_auth_session["authservice_session"] is None.my _auth_session is

{'endpoint_url': 'http://xxxxxxx:18080/', 'redirect_url': 'http://xxxxxxx:18080/dex/auth/local/login?back=&state=sl6amu5zckyntv35z65solp4t', 'dex_login_url': 'http://xxxxxxx:18080/dex/auth/local/login?back=&state=sl6amu5zckyntv35z65solp4t', 'is_secured': True, 'session_cookie': 'oauth2_proxy_kubeflow_csrf=-GOrJQeg-v5_Dc_3HD12Td80HAuE9ld2eT3Y7dT2lyw1b2wUKEZaYJVjhqaBF7UW_elSQ8i08WsjXTGO2kys6VTPJWcHQ2uI_qMFqFp7Wn2_SIpjihvsfa-JX8VC-8Qag25f0jkbi-LMSu0IlRjVYoQZ2VDkriRa3bCq8j3zVOboKjcxVGujyrggNU1X4r-PyYVOujmdv7oxhw1wcS2HbYiQ84GZxPbb0S9FeFI8u2gkSvT5U_HQrpfl2MkoBg==|1723645690|pZzKszJaoId5Vb5vQN0-a99zY29WdQ2TfHtHdsIga-U=', 'authservice_session': None}

this issue will solve the question?

@juliusvonkohout
Copy link
Member Author

by my test in kubeflow 1.9 cluster, https://github.com/kserve/kserve/tree/master/docs/samples/istio-dex,this is cant run . _auth_session["authservice_session"] is None.my _auth_session is

{'endpoint_url': 'http://xxxxxxx:18080/', 'redirect_url': 'http://xxxxxxx:18080/dex/auth/local/login?back=&state=sl6amu5zckyntv35z65solp4t', 'dex_login_url': 'http://xxxxxxx:18080/dex/auth/local/login?back=&state=sl6amu5zckyntv35z65solp4t', 'is_secured': True, 'session_cookie': 'oauth2_proxy_kubeflow_csrf=-GOrJQeg-v5_Dc_3HD12Td80HAuE9ld2eT3Y7dT2lyw1b2wUKEZaYJVjhqaBF7UW_elSQ8i08WsjXTGO2kys6VTPJWcHQ2uI_qMFqFp7Wn2_SIpjihvsfa-JX8VC-8Qag25f0jkbi-LMSu0IlRjVYoQZ2VDkriRa3bCq8j3zVOboKjcxVGujyrggNU1X4r-PyYVOujmdv7oxhw1wcS2HbYiQ84GZxPbb0S9FeFI8u2gkSvT5U_HQrpfl2MkoBg==|1723645690|pZzKszJaoId5Vb5vQN0-a99zY29WdQ2TfHtHdsIga-U=', 'authservice_session': None}

this issue will solve the question?

Hello, as said before, the kserve documentation is wrong there. Check the first post here on how to get it done with oauth2-proxy.

@padrian2s
Copy link

KServe is right but for internal K8s JWT tokens not with OAuth-proxy tokens that are generated outside the cluster.

@juliusvonkohout
Copy link
Member Author

juliusvonkohout commented Aug 16, 2024 •
edited
Loading

No, oauth2-proxy is explicitly there to use serviceaccountokens. Whether from inside or outside does not matter.

@github-actions GitHub Actions
Copy link

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@juliusvonkohout
Copy link
Member Author

/lifecycle frozen

@juliusvonkohout juliusvonkohout added help wanted Extra attention is needed good first issue Good for newcomers labels Oct 21, 2024
@juliusvonkohout
Copy link
Member Author

See also
kubeflow/dashboard#13 and
kubeflow/kubeflow#6627

@juliusvonkohout juliusvonkohout mentioned this issue Dec 18, 2024
Open
@juliusvonkohout
Copy link
Member Author

Implementation started in #2936

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@kromanow94 kromanow94

@juliusvonkohout juliusvonkohout

Labels
good first issue Good for newcomers help wanted Extra attention is needed lifecycle/frozen
Projects
Needs Triage
Status: To Do
Milestone
No milestone
Development

No branches or pull requests
4 participants
@padrian2s @kromanow94 @juliusvonkohout @jaffe-fly