Jupyter UI that manages Notebook CRs (kubeflow#2357)

* Introduce jupyter-web-app's code under /components Add the webapp's code to /components/jupyter-web-app. It is a basic Flask application with HTML/CSS/jQuery. Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * Add jupyter-web-app component in jupyter pkg Add the jsonnet files in /kubeflow/jupyter. Currently the webapp is used alongside JupyterHub. The webapp is under the prefix /jupyter. The image param of the jupyter-web-app must be configured before deploying. Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * UI fixes Fixes noted from avdaredevil: * Fix XSS vulnerability * Use template literals for HTML inside JS Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com> * Add webapp to init scripts Signed-off-by: Kimonas Sotirchos <kimwnasptd@arrikto.com>
StatCan · Feb 14, 2019 · c7e3de9 · c7e3de9
1 parent 7f9ae3f
commit c7e3de9
Show file tree

Hide file tree

Showing 35 changed files with 4,395 additions and 1 deletion.
diff --git a/Dockerfile b/Dockerfile
@@ -0,0 +1,19 @@
+FROM ubuntu:16.04
+
+RUN apt-get update -y && \
+    apt-get install -y apt-utils  build-essential curl \
+	python-dev python3-pip \
+    libssl-dev libffi-dev python3-bcrypt
+
+# We copy just the requirements.txt first to leverage Docker cache
+COPY ./requirements.txt /app/requirements.txt
+
+RUN pip3 install -r /app/requirements.txt
+
+COPY default /app/default
+COPY rok /app/rok
+
+WORKDIR /app/default
+
+ENTRYPOINT ["python3"]
+CMD ["run.py"]
diff --git a/components/jupyter-web-app/Makefile b/components/jupyter-web-app/Makefile
@@ -0,0 +1,38 @@
+IMG = gcr.io/kubeflow-images-public/jupyter-web-app
+
+# List any changed  files. We only include files in the notebooks directory.
+# because that is the code in the docker image.
+# In particular we exclude changes to the ksonnet configs.
+CHANGED_FILES := $(shell git diff-files --relative=components/jupyter-web-app)
+
+ifeq ($(strip $(CHANGED_FILES)),)
+# Changed files is empty; not dirty
+# Don't include --dirty because it could be dirty if files outside the ones we care
+# about changed.
+GIT_VERSION := $(shell git describe --always)
+else
+GIT_VERSION := $(shell git describe --always)-dirty-$(shell git diff | shasum -a256 | cut -c -6)
+endif
+
+TAG := $(shell date +v%Y%m%d)-$(GIT_VERSION)
+all: build
+
+# To build without the cache set the environment variable
+# export DOCKER_BUILD_OPTS=--no-cache
+build:
+	docker build ${DOCKER_BUILD_OPTS} -t $(IMG):$(TAG) . \
+	  --build-arg kubeflowversion=$(shell git describe --abbrev=0 --tags) \
+      --label=git-verions=$(GIT_VERSION)
+	docker tag $(IMG):$(TAG) $(IMG):latest
+	@echo Built $(IMG):latest
+	@echo Built $(IMG):$(TAG)
+
+# Build but don't attach the latest tag. This allows manual testing/inspection of the image
+# first.
+push: build
+	gcloud docker -- push $(IMG):$(TAG)
+	@echo Pushed $(IMG) with  :$(TAG) tags
+
+push-latest: push
+	gcloud container images add-tag --quiet $(IMG):$(TAG) $(IMG):latest --verbosity=info
+	echo created $(IMG):latest
diff --git a/components/jupyter-web-app/README.md b/components/jupyter-web-app/README.md
@@ -0,0 +1,21 @@
+## Goals
+- replace JupyterHub Spawner UI with a new Jupyter UI that natively manages Notebook CRs
+- allow Users to create, connect to and delete Notebooks by specifying custom resources
+
+## Design
+The new Jupyter UI uses [Python Flask](http://flask.pocoo.org/) for the backend and HTML/jQuery/Material Design Lite for the frontend. A privileged `ServiceAccount` along with proper `RBAC` resources are associated with the Pod hosting the Flask server. In this manner, the `jupyter-web-app` Pod is allowed to manage Notebook CRs and PVCs in the `kubeflow` namespace. 
+
+Please note that as soon as the Profile Controller supports automatic creation of read/write ServiceAccounts for each `Profile`, the new Jupyter UI will be updated to use the respective JWTs and perform all K8s API requests via [K8s Impersonation](https://kubernetes.io/docs/reference/access-authn-authz/authentication/#user-impersonation). This will ultimately provide isolation of resources between Users and avoid any possible conflicts. For more information about User authentication and e2e workflow see the [Jupyter design doc](http://bit.ly/kf_jupyter_design_doc)
+
+
+## User Interaction
+As soon as the User points his/her browser to http://<KUBEFLOW_IP>/jupyter/ he/she will be directed to the new Jupyter UI. By default the UI will try to list the Notebooks in the namespace of its `ServiceAccount`, currently `kubeflow`, as mentioned above. If something goes wrong the UI will notify the user with an appropriate message containing info about the API call error. 
+
+From the Noteooks table he/she can either click the `+` button to create a new Notebook or perform `Delete`/`Connect` actions to an existing one. The UI only performs requests regarding the Notebook CRs to the K8s API server. The management of all child resources(`Service`, `Deployment`) is performed by the Notebook CR Controller. 
+
+By pressing the `+` button to create a Notebook the user is redirected to a form that allows him to configure the `PodTemplateSpec` params of the new Notebook. The User can specify the following options regarding his/her Jupyter Notebook: `name`, `namespace`, `cpu`, `memory`, `workspace volume`, `data volumes`, `extra resources`. Notably, he/she can create new Volumes from scratch (type set to `New`) or mount existing ones (type set to `Existing`). By clicking the `SPAWN` button, a new Notebook with the aforementioned options will be created in the `kubeflow` namespace. If some option is not specified, a default value will be used.
+
+**NOTE:**
+Please wait for the Notebook Pod to be successfully created and reach Ready state before trying to connect to it.
+Otherwise, Ambassador won't be able to route traffic to the correct endpoint and will fail
+with "upstream connect error or disconnect/reset before headers".
diff --git a/components/jupyter-web-app/default/kubeflow/jupyter/__init__.py b/components/jupyter-web-app/default/kubeflow/jupyter/__init__.py
@@ -0,0 +1,5 @@
+from flask import Flask, url_for, jsonify
+
+app = Flask(__name__)
+
+from kubeflow.jupyter import routes
diff --git a/components/jupyter-web-app/default/kubeflow/jupyter/routes.py b/components/jupyter-web-app/default/kubeflow/jupyter/routes.py
@@ -0,0 +1,178 @@
+import json
+from flask import jsonify, render_template, request
+from kubernetes.client.rest import ApiException
+from kubeflow.jupyter import app
+from kubeflow.jupyter.server import parse_error, \
+            get_namespaces, \
+            get_notebooks, \
+            delete_notebook, \
+            create_notebook, \
+            create_pvc
+from kubeflow.jupyter.utils import create_notebook_template, \
+            create_pvc_template, \
+            set_notebook_names, \
+            set_notebook_image, \
+            set_notebook_cpu_ram, \
+            add_notebook_volume, \
+            spawner_ui_config
+
+
+# Helper function for getting the prefix of the webapp
+def prefix():
+    if request.headers.get("x-forwarded-prefix"):
+        return request.headers.get("x-forwarded-prefix")
+    else:
+        return ""
+
+
+@app.route("/post-notebook", methods=['POST'])
+def post_notebook_route():
+    data = {"success": True, "log": ""}
+    body = request.form
+
+    # Template
+    notebook = create_notebook_template()
+    notebook_cont = notebook["spec"]['template']['spec']['containers'][0]
+
+    # Set Name and Namespace
+    set_notebook_names(notebook, body)
+
+    # Set Image
+    set_notebook_image(notebook, body)
+
+    # CPU/RAM
+    set_notebook_cpu_ram(notebook, body)
+
+    # Workspacae Volume
+    if body["ws_type"] == "New":
+        pvc = create_pvc_template()
+        pvc['metadata']['name'] = body['ws_name']
+        pvc['metadata']['namespace'] = body['ns']
+        pvc['spec']['accessModes'].append(body['ws_access_modes'])
+        pvc['spec']['resources']['requests']['storage'] = \
+            body['ws_size'] + 'Gi'
+
+        try:
+            create_pvc(pvc)
+        except ApiException as e:
+            data["success"] = False
+            data["log"] = parse_error(e)
+            return jsonify(data)
+
+    # Create the Workspace Volume in the Pod
+    if body["ws_type"] != "None":
+        add_notebook_volume(notebook,
+                            "volume-" + body["nm"],
+                            body["ws_name"],
+                            "/home/jovyan",)
+
+    # Add the Data Volumes
+    counter = 1
+    while ("vol_name" + str(counter)) in body:
+        i = str(counter)
+        vol_nm = 'data-volume-' + i
+        pvc_nm = body['vol_name' + i]
+        mnt = body['vol_mount_path' + i]
+
+        # Create a PVC if its a new Data Volume
+        if body["vol_type" + i] == "New":
+            size = body['vol_size' + i] + 'Gi'
+            mode = body['vol_access_modes' + i]
+            pvc = create_pvc_template()
+
+            pvc['metadata']['name'] = pvc_nm
+            pvc['metadata']['namespace'] = body['ns']
+            pvc['spec']['accessModes'].append(mode)
+            pvc['spec']['resources']['requests']['storage'] = size
+
+            try:
+                create_pvc(pvc)
+            except ApiException as e:
+                data["success"] = False
+                data["log"] = parse_error(e)
+                return jsonify(data)
+
+        add_notebook_volume(notebook, vol_nm, pvc_nm, mnt)
+        counter += 1
+
+    # Add Extra Resources
+    try:
+        extra = json.loads(body["extraResources"])
+    except Exception as e:
+        data["success"] = False
+        data["log"] = parse_error(e)
+        return jsonify(data)
+
+    notebook_cont['resources']['limits'] = extra
+
+    # If all the parameters are given, then we try to create the notebook
+    # return
+    try:
+        create_notebook(notebook)
+    except ApiException as e:
+        data["success"] = False
+        data["log"] = parse_error(e)
+        return jsonify(data)
+
+    return jsonify(data)
+
+
+@app.route("/add-notebook", methods=['GET'])
+def add_notebook_route():
+    # A default value for the namespace to add the notebook
+    if request.args.get("namespace"):
+        ns = request.args.get("namespace")
+    else:
+        ns = "kubeflow"
+
+    form_defaults = spawner_ui_config("notebook")
+    return render_template('add_notebook.html', prefix=prefix(), ns=ns,
+                           form_defaults=form_defaults)
+
+
+@app.route("/delete-notebook", methods=['GET', 'POST'])
+def del_notebook_route():
+    nb = request.args.get("notebook")
+    ns = request.args.get("namespace")
+
+    # try to delete the notebook
+    data = {"success": True, "log": ""}
+    try:
+        delete_notebook(nb, ns)
+    except ApiException as e:
+        data["success"] = False
+        data["log"] = parse_error(e)
+
+    return jsonify(data)
+
+
+@app.route("/list-notebooks")
+def list_notebooks_route():
+    ns = request.args.get("namespace")
+
+    # Get the list of Notebooks in the given Namespace
+    data = {"notebooks": [], "success": True}
+    try:
+        data['notebooks'] = get_notebooks(ns)
+    except ApiException as e:
+        data['notebooks'] = []
+        data['success'] = False
+        data["log"] = parse_error(e)
+
+    return jsonify(data)
+
+
+@app.route("/")
+@app.route("/home")
+@app.route("/notebooks")
+def notebooks_route():
+    base_ns = "kubeflow"
+
+    # Get the namespaces the token can see
+    try:
+        nmsps = get_namespaces()
+    except ApiException:
+        nmsps = [base_ns]
+
+    return render_template('notebooks.html', prefix=prefix(),
+                           title='Notebooks', namespaces=nmsps)
diff --git a/components/jupyter-web-app/default/kubeflow/jupyter/server.py b/components/jupyter-web-app/default/kubeflow/jupyter/server.py
@@ -0,0 +1,64 @@
+import json
+from kubernetes import client, config
+from kubernetes.config import ConfigException
+
+
+try:
+    # Load configuration inside the Pod
+    config.load_incluster_config()
+except ConfigException:
+    # Load configuration for testing
+    config.load_kube_config()
+
+# Create the Apis
+v1_core = client.CoreV1Api()
+custom_api = client.CustomObjectsApi()
+
+
+def parse_error(e):
+    try:
+        err = json.loads(e.body)['message']
+    except json.JSONDecodeError:
+        err = str(e)
+    except KeyError:
+        err = str(e)
+
+    return err
+
+
+def get_secret(nm, ns):
+    return v1_core.read_namespaced_secret(nm, ns)
+
+
+def get_namespaces():
+    nmsps = v1_core.list_namespace()
+    return [ns.metadata.name for ns in nmsps.items]
+
+
+def get_notebooks(ns):
+    custom_api = client.CustomObjectsApi()
+
+    notebooks = \
+        custom_api.list_namespaced_custom_object("kubeflow.org", "v1alpha1",
+                                                 ns, "notebooks")
+    return [nb['metadata']['name'] for nb in notebooks['items']]
+
+
+def delete_notebook(nb, ns):
+    body = client.V1DeleteOptions()
+
+    return \
+        custom_api.delete_namespaced_custom_object("kubeflow.org", "v1alpha1",
+                                                   ns, "notebooks", nb, body)
+
+
+def create_notebook(body):
+    ns = body['metadata']['namespace']
+    return \
+        custom_api.create_namespaced_custom_object("kubeflow.org", "v1alpha1",
+                                                   ns, "notebooks", body)
+
+
+def create_pvc(body):
+    ns = body['metadata']['namespace']
+    return v1_core.create_namespaced_persistent_volume_claim(ns, body)