-
Notifications
You must be signed in to change notification settings - Fork 2.4k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Jupyter UI that manages Notebook CRs #2357
Jupyter UI that manages Notebook CRs #2357
Conversation
Hi @kimwnasptd. Thanks for your PR. I'm waiting for a kubeflow member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
/ok-to-test |
Traveling in Europe but might be able to look tomorrow |
Related to testing |
/assign @jlewi |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewed 26 of 26 files at r1.
Reviewable status: all files reviewed, 5 unresolved discussions (waiting on @ellis-bigelow, @kimwnasptd, and @vkoukis)
components/jupyter-web-app/jupyter/routes.py, line 4 at r1 (raw file):
from flask import jsonify, render_template, request from kubernetes.client.rest import ApiException from jupyter import app
Is making the top level package named jupyter going to cause problems because it conflicts with the jupyter package?
Should we rename it?
We could make it
from kubeflow.jupyter
I like the idea of beginning to organize our py code into kubeflow sub packages.
Related issue kubeflow/training-operator#914 about better support for namespace packaging.
components/jupyter-web-app/jupyter/server.py, line 21 at r1 (raw file):
def create_client_config(apiserver=APISERVER, ca_cert=CACERT): # By default, use JWT Tokens
What is the CERT being used for?
The JWT should be validated in a side car by ISTIO; so the only thing the web app should need to do is get the identity of the user. is that what this code is doing?
components/jupyter-web-app/jupyter/server.py, line 69 at r1 (raw file):
def create_pvc(body):
So the web app will create PVCs? Not just attach them?
components/jupyter-web-app/jupyter/utils.py, line 24 at r1 (raw file):
# Functions for handling the config from which we load the default values
Default values for what? The notebook? Or for the UI?
components/jupyter-web-app/jupyter/utils.py, line 47 at r1 (raw file):
# Helper functions for the /post-notebook route. def create_notebook_template():
I believe if a CR includes an OpenAPI spec then the client generation tools for K8s can be used to generate python classes for that class. Might be worth file an issue to track that and using that library here.
@kimwnasptd This looks great to me. I mostly looked at the python code. Hopefully someone with more frontend experience like @avdaredevil can take a look at the front end stuff. Is there specific issues or questions that need to be resolved? In other words what's still in progress? If nothing and its ready to review please go ahead and remove WIP from the title to signal to reviewers that its read to be reviewed. If you want to block it from being merged you can put a hold on it. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Yes. currently we've added extra privileges to the webapp's service account to list/edit/get/delete Notebooks, PVCs and Namespces. Then the webapp uses this sa's token for any call to the api server. Is this enough for now or should it use impersonation as described in the design doc?
Also there are some last bits we need to work on the UI to have the same functionality with the current implementation. We'll be ready to remove the WIP label within the week.
Reviewable status: all files reviewed, 5 unresolved discussions (waiting on @ellis-bigelow, @jlewi, and @vkoukis)
components/jupyter-web-app/jupyter/routes.py, line 4 at r1 (raw file):
Previously, jlewi (Jeremy Lewi) wrote…
Is making the top level package named jupyter going to cause problems because it conflicts with the jupyter package?
Should we rename it?
We could make it
from kubeflow.jupyter
I like the idea of beginning to organize our py code into kubeflow sub packages.
Related issue kubeflow/training-operator#914 about better support for namespace packaging.
I think I'm missing something here. Are you referring to the /kubeflow/jupyter
package? Won't the code for the UIs be under /components/jupyter-web-app/
?
components/jupyter-web-app/jupyter/server.py, line 21 at r1 (raw file):
Previously, jlewi (Jeremy Lewi) wrote…
What is the CERT being used for?
The JWT should be validated in a side car by ISTIO; so the only thing the web app should need to do is get the identity of the user. is that what this code is doing?
No, that cert is used from the client to verify the server's certificate for ssl. Also, the webapp does all the api calls using its own privileged service account token. I was waiting for the ISTIO implementation to progress and then I would update the code to use the identity of the user with impersonation.
components/jupyter-web-app/jupyter/server.py, line 69 at r1 (raw file):
Previously, jlewi (Jeremy Lewi) wrote…
So the web app will create PVCs? Not just attach them?
Yes, this was also present with JupyterHub so I preserved this feature. Should I not?
components/jupyter-web-app/jupyter/utils.py, line 24 at r1 (raw file):
Previously, jlewi (Jeremy Lewi) wrote…
Default values for what? The notebook? Or for the UI?
For the UI. They are default values to put in the form. These are loaded from a file in /kubeflow/jupyter/config.yaml
and are attached as a ConfigMap to the webapp
components/jupyter-web-app/jupyter/utils.py, line 47 at r1 (raw file):
Previously, jlewi (Jeremy Lewi) wrote…
I believe if a CR includes an OpenAPI spec then the client generation tools for K8s can be used to generate python classes for that class. Might be worth file an issue to track that and using that library here.
Thanks for the heads up, I'll look into it right away
Yes this is fine.
What is this functionality? Can we do it in a follow on PR and get the initial version merged? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewable status: all files reviewed, 4 unresolved discussions (waiting on @ellis-bigelow, @kimwnasptd, and @vkoukis)
components/jupyter-web-app/jupyter/routes.py, line 4 at r1 (raw file):
Previously, kimwnasptd (Kimonas Sotirchos) wrote…
I think I'm missing something here. Are you referring to the
/kubeflow/jupyter
package? Won't the code for the UIs be under/components/jupyter-web-app/
?
I'm referring to your python package.
Right now you are treating jupyter as a top level python package; i.e. you import it as
from jupyter import app
i.e. you are adding the directory components/jupyter-web-app to your PYTHONPATH.
This will cause a conflict if someone has the actual jupyter package installed on their path because that would also be imported as
import jupyter
So we should try to come up with a unique enough naming that things won't conflict.
So one way to do that is to organize the code as
component/jupyter-web-app/kubeflow/jupyter
And add
component/jupyter-web-app
to your python path
Then the code would be imported as
import kubeflow.jupyter
which should avoid conflicts with the jupyter package.
components/jupyter-web-app/jupyter/server.py, line 21 at r1 (raw file):
Previously, kimwnasptd (Kimonas Sotirchos) wrote…
No, that cert is used from the client to verify the server's certificate for ssl. Also, the webapp does all the api calls using its own privileged service account token. I was waiting for the ISTIO implementation to progress and then I would update the code to use the identity of the user with impersonation.
What does SSL validation have to do with ISTIO and service accounts?
I don't think you need to verify SSL certificates or generate JWTs.
To talk to the K8s master using the pods service account you should just be able to configure the K8s python client library in a way that it automatically uses the PODs service account.
components/jupyter-web-app/jupyter/server.py, line 69 at r1 (raw file):
Previously, kimwnasptd (Kimonas Sotirchos) wrote…
Yes, this was also present with JupyterHub so I preserved this feature. Should I not?
I think its fine.
components/jupyter-web-app/jupyter/utils.py, line 24 at r1 (raw file):
Previously, kimwnasptd (Kimonas Sotirchos) wrote…
For the UI. They are default values to put in the form. These are loaded from a file in
/kubeflow/jupyter/config.yaml
and are attached as a ConfigMap to the webapp
Can you clarify the comment please.
components/jupyter-web-app/jupyter/utils.py, line 47 at r1 (raw file):
Previously, kimwnasptd (Kimonas Sotirchos) wrote…
Thanks for the heads up, I'll look into it right away
I would suggest filing a follow on issue; lets not block the current PR on it.
@kimwnasptd How's it going? Any idea on when you'll have the next iteration of this PR? |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Took a quick look at the front-end changes. Here are my thoughts.
Let me know if you have any questions/updates :)
|
||
function deleteNotebook(ns, nb) { | ||
$.getJSON(prefix + "/delete-notebook", { namespace:ns, notebook:nb}, function(data, status) { | ||
var innerHTML = $("#error-msgs").html() |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This declaration is redundant, could simply be:
let innerHTML = ''
innerHTML = '<div class="alert alert-warning">'; | ||
innerHTML += '<span class="close" ' | ||
innerHTML += 'onclick="this.parentElement.style.display=\'none\';">×</span>'; | ||
innerHTML += '<strong>Warning! </strong>' + data.log + ' </div>'; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This seems to be XSS prone. Maybe consider doing something like:
innerHTML = `
<div class="alert alert-warning">
<span class="close" onclick="this.parentElement.style.display='none'">×</span>
<strong>Warning!</strong><span class='warning-log'></span>
</div>`
and on line 24:
const $e = $("#error-msgs").html(innerHTML)
$('.warning-log', $e).text(data.log) // This will not fail even if your `innerText` is empty
innerHTML += 'onclick="this.parentElement.style.display=\'none\';">×</span>'; | ||
innerHTML += '<strong>Warning! </strong>' + data.log + ' </div>'; | ||
} | ||
$("#error-msgs").html(innerHTML); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Read above
var innerHTML = $("#error-msgs").html() | ||
if(data.success == true) { | ||
updateNotebooksInNamespace(ns) | ||
innerHTML = ''; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is not needed if the above change is made
}; | ||
|
||
function connectNotebook(ns, nb) { | ||
window.open("/" + ns + "/" + nb, "_blank"); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
[nitpick] This could be written as:
window.open(`/${ns}/${nb}`, "_blank");
updateNotebooksInNamespace(ns); | ||
|
||
// Change the function for the CREATE NOTEBOOK button | ||
$("#create-nb-btn").attr("onclick", "createNotebook('" + ns + "')") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Perhaps go with:
$("#create-nb-btn").click(_ => createNotebook(ns))
OR
$("#create-nb-btn").click(createNotebook.bind(0, ns))
updateNotebooksInNamespace(ns); | ||
|
||
// Change the function for the CREATE NOTEBOOK button | ||
$("#create-nb-btn").attr("onclick", "createNotebook('" + ns + "')") |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Same as review above
0617c83
to
75c9112
Compare
So there's good news and bad news. 👍 The good news is that everyone that needs to sign a CLA (the pull request submitter and all commit authors) have done so. Everything is all good there. 😕 The bad news is that it appears that one or more commits were authored or co-authored by someone other than the pull request submitter. We need to confirm that all authors are ok with their commits being contributed to this project. Please have them confirm that here in the pull request. Note to project maintainer: This is a terminal state, meaning the |
Add the webapp's code to /components/jupyter-web-app. It is a basic Flask application with HTML/CSS/jQuery. Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>
Add the jsonnet files in /kubeflow/jupyter. Currently the webapp is used alongside JupyterHub. The webapp is under the prefix /jupyter. The image param of the jupyter-web-app must be configured before deploying. Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>
Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>
75c9112
to
06b0a1b
Compare
CLAs look good, thanks! |
/lgtm |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Once its committed we'll push the image to the registry. We want to limit who has access to our public image registry.
Reviewed 39 of 41 files at r2.
Reviewable status: 33 of 35 files reviewed, 10 unresolved discussions (waiting on @avdaredevil, @ellis-bigelow, @kimwnasptd, and @vkoukis)
components/jupyter-web-app/rok/kubeflow/rokui/server.py, line 3 at r2 (raw file):
import json from kubernetes import client, config from kubernetes.config import ConfigException
Why do we need to duplicate this file for the rokui and the non rokui?
Maybe file an issue to reduce this code duplication?
components/jupyter-web-app/rok/kubeflow/rokui/utils.py, line 6 at r2 (raw file):
# Functions for handling the JWT token
Same question about code duplication with non rokui
kubeflow/jupyter/prototypes/notebooks.jsonnet, line 11 at r2 (raw file):
// @optionalParam registry string gcr.io The docker image registry for notebook // @optionalParam repoName string kubeflow-images-public The repository name for the notebook // @optionalParam disks string null Comma separated list of Google persistent disks to attach to notebook environments.
Did you mean to change this file?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewable status: 33 of 35 files reviewed, 10 unresolved discussions (waiting on @avdaredevil, @ellis-bigelow, @jlewi, and @vkoukis)
components/jupyter-web-app/rok/kubeflow/rokui/server.py, line 3 at r2 (raw file):
Previously, jlewi (Jeremy Lewi) wrote…
Why do we need to duplicate this file for the rokui and the non rokui?
Maybe file an issue to reduce this code duplication?
Good point. We were planning to file a follow up issue and PR to refactor the python code in order to have a Class-based approach, which would be more extendable.
Although this will happen after the demo, but I'll file the issue to keep track of it
components/jupyter-web-app/rok/kubeflow/rokui/utils.py, line 6 at r2 (raw file):
Previously, jlewi (Jeremy Lewi) wrote…
Same question about code duplication with non rokui
same as above
kubeflow/jupyter/prototypes/notebooks.jsonnet, line 11 at r2 (raw file):
Previously, jlewi (Jeremy Lewi) wrote…
Did you mean to change this file?
Don't think so, I can't see this file being changed locally or in the commits in GitHub. Is it shown as edited?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Reviewable status: 33 of 35 files reviewed, 7 unresolved discussions (waiting on @avdaredevil, @ellis-bigelow, and @vkoukis)
/approve |
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: jlewi The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
* Introduce jupyter-web-app's code under /components Add the webapp's code to /components/jupyter-web-app. It is a basic Flask application with HTML/CSS/jQuery. Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * Add jupyter-web-app component in jupyter pkg Add the jsonnet files in /kubeflow/jupyter. Currently the webapp is used alongside JupyterHub. The webapp is under the prefix /jupyter. The image param of the jupyter-web-app must be configured before deploying. Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * UI fixes Fixes noted from avdaredevil: * Fix XSS vulnerability * Use template literals for HTML inside JS Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * Add webapp to init scripts Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>
* Introduce jupyter-web-app's code under /components Add the webapp's code to /components/jupyter-web-app. It is a basic Flask application with HTML/CSS/jQuery. Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * Add jupyter-web-app component in jupyter pkg Add the jsonnet files in /kubeflow/jupyter. Currently the webapp is used alongside JupyterHub. The webapp is under the prefix /jupyter. The image param of the jupyter-web-app must be configured before deploying. Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * UI fixes Fixes noted from avdaredevil: * Fix XSS vulnerability * Use template literals for HTML inside JS Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * Add webapp to init scripts Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>
Description
This PR provides the initial version of the new Web UI to natively spawn Jupyter Notebooks in a Kubeflow environment. The new Jupyter UI aims to replace the existing
jupyter
component and disengage Kubeflow from the JupyterHub API.Please note that the new Jupyter UI is still under development, so there will be various fixes/improvements to its appearance and functionality until it’s ready to be merged. This WIP PR will also be updated soon to support the existing 3rd party alternative Jupyter UIs and not just the default one.
To quickly test the new Jupyter UI in your K8s cluster we provide a series of deployment steps. We have already tested those on Minikube, bare metal K8s and GKE. Of course, we welcome feedback and encourage discussion on issues that might arise.
Closes
#1995
#2156
#2232
Depends
#2349
Τhis PR has been updated and now depends on #2456
Main features of the new Jupyter UI
Future/Related Work
Integration with the new login UI component, as well as @kam’s Profiles when multiple (read/write) ServiceAccounts are supported. The new Jupyter web application will use its own privileged ServiceAccount to dispatch all API requests. In a later iteration it will be using Impersonation.
Jupyter CR E2E test #2174
Integrate Jupyter e2e test into CI #2333
Create an E2E test for the Jupyter custom resource. #2323
centraldashboard
component of Kubeflow to replace “JupyterHub” link with “Notebooks” and route to the new Jupyter UI. As a side change, we would also need to update the corresponding public Docker image on gcr.io for 0.5.Deployment Steps
(Minor tweaks might be needed depending on your development environment)
feature-jupyter-web-app
branchgit clone -b feature-jupyter-web-app https://www.github.com/arrikto/kubeflow.git
docker build -t jupyter-web-app:v0.1 kubeflow/components/jupyter-web-app
kubeflow/scripts/kfctl.sh init kf
cd kf && ../kubeflow/scripts/kfctl.sh generate all && ../kubeflow/scripts/kfctl.sh apply all
jupyter-web-app
KF component to use the image you previously builtcd ks_app && ks param set jupyter-web-app image jupyter-web-app:v0.1
jupyter-web-app
KF componentks apply default -c jupyter-web-app
http://<KUBEFLOW_IP>/jupyter/
(trailing slash). You should be able to access the dashboard of the new Jupyter UI.kubeflow/components/jupyter-web-app/README.md
to create, connect to and delete Jupyter Notebooks.This change is