/priority p1
/area jupyter

priority p2

/priority p2

/priority p1

I can see a lot of opened PRs that introduce many frameworks alongside different Notebook Servers, so I'll comment in this issue instead of each opened PR.

As mentioned in the Community Meetings and in the weekly WG meets supporting different types of Notebook servers for 1.3 is best effort. So there won't be timely reviews until we ensure the manifests for the WG's artifacts and the CI/CD processes are in place for 1.3.

Lastly we haven't yet finalized the design doc around how we will handle the Notebook images, where will they live, which images there are etc. So until the design doc effort is finalized we can't move forward with reviewing the opened PRs. There should be more traction on the design doc around the Notebook images once we've dealt with the more pressing issues I mentioned above.

While there are many PRs here, half of those require essentially no review as all they do is install a package with pip or conda. Also, while the design doc is not yet finalized, the PRs for the images do follow the intention of the design doc (at least how I interpreted the discussions) and the suggested changes I have made to it. The PRs that need the most attention will be those in stage 1, followed by those in stage 2. After that, the only PR that requires some attention is TensorFlow with CUDA due to needing to install CUDA. Me and @thesuperzapper are trying to coordinate going through these PRs so they would be ready for 1.3.

To go along with these images, I have actually started work on an Image Builder web app, that will allow users to easily extend (these) images with whatever pip and conda packages they need. However, I do not think this will be ready for 1.3 (even though all that is needed is some very simple front end development.

Now that we have set up the CI/CD #5482 I can start allocating more cycles to reviewing this effort. I'll also be updating our design doc as we go along step by step. I'm not expecting us to have everything documented in that doc from the beginning, but lets use this issue to write down our process, assumptions and next steps. In parallel, as we progress, I'll try to keep our design doc up to date.

@davidspek I see a lot of opened PRs for different Notebook-servers/web-IDEs right now, which IIUC are all grouped under this umbrella issue. The way I'll be reviewing this effort is by going over the PRs based on the server they use [ Jupyter, R-Studio, VSCode etc ], starting from Jupyter.

So the first thing I'll be looking at will be the Tensorflow CPU ones #5627 #5623, which we could use as our example Notebook image for running Jupyter with Kubeflow.

@kimwnasptd we are about to raise a new proposal (like in 30+ min) which simplifies this.

I expect we will just edit the text of this issue.

We will also refactor the PR's so dont review them yet.

Thanks for the heads up! I'll keep an eye on the issue then. Although most probably the order in which I'll look at the PRs will be as described above

@kimwnasptd can you please give us your thoughts on the updated proposal above (and specifically on the container names so we can get @PatrickXYS to create the required ECR folders)

NOTE: I am only referring to the proposal at this time, not the PRs themselves

Maybe it is possible to have the images for the notebooks server live under public.ecr.aws/j1r0q0g6/notebook-servers/***:tag, similar to how kfserving has stuff under public.ecr.aws/j1r0q0g6/kfserving/***:tag. This would keep them a bit more organized given the amount of images that will be created.

Thanks for updating the proposal! First off I'm OK with creating upstream images for all the above, but with some changes on the names.

The goal of the following naming scheme is to just further align the proposal with our design doc and the out of scope section.
Since we don't want to be providing images for different frameworks and sets of packages we should only select one image for each server/web-ide and mark them as the "official" example Notebook images. Then we can include, and build/publish, all the other ones and mark them as contrib ones.

So the only changes to the naming scheme I would propose are:

1. Have example-notebook-server-{jupyter,rstudio,vscode} images [ I guess we could also append the underlying framework, i.e. example-notebook-server-jupyter-tensorflow ]
2. For the others to be contrib-notebook-server-{jupyter,rstudio,vscode}-<any-suffix-you-want>

cc @StefanoFioravanzo @elikatsis

@kimwnasptd I'd like to suggest expanding the "official" images to the ones that don't include a * at the moment. Esentially, these are the base images for Jupyter, RStudio and VSCode, along with PyTorch and TensorFlow both with and without CUDA. I think this is necessary purely because TensorFlow with CUDA requires some some more steps to get working, and providing only TensorFlow CUDA without the CPU equivalent and the equivalent for PyTorch seems weird. Also, there the maintenance on the TensorFlow CPU and PytTorch base images should be essentially non-existent.

Also, I thought it was decided that all the images should be seen as examples so I am not sure what the distinction is you are trying to make between example and contrib. It will also make the image names very long, which I think might start causing problems in the UI (or maybe even kubernetes, no idea if there is a limit for this).

To help unblock us, let's start from minimal, I can't see there's any blockers in getting started from minimal changes.

Haste makes waste, let's start with example-notebook-server-jupyter-tensorflow, but keep the code with enough extensibility. Later on, we can move forward with other IDEs and DL framework.

@PatrickXYS That notebook would still require the notebook-server-jupyter and notebook-server-base images. I'm not sure if you've looked at the Dockerfiles, but (other than TensorFlow with CUDA), this is what that Dockerfile looks like:

FROM notebook-server-jupyter:<tag>

COPY requirements-cpu.txt .
RUN python3 -m pip install -r \
    requirements-cpu.txt --quiet --no-cache-dir \
    && rm -f requirements-cpu.txt

Contents of requirements-cpu.txt:
tensorflow==2.4.1

This is what almost all of the Jupyter Dockerfiles look like as well. There really is not much too them.

@StefanoFioravanzo @kimwnasptd

As @davidspek stated, we are more than happy to maintain these images, so there will be no extra work from your behalf.

The "contrib" idea, that makes very little sense, as these images are literally not contrib, they are examples
(NOTE: We will make it clear in the README.md of the components/notebook-servers folder that they are examples)

Given the above, I don't see any reason not to move forwards ASAP, especially since all the code is is ready to go.

PS: We have publicly committed to supporting RStudio/VSCode in 1.3, and there is nothing stopping us except this unproductive back-and-forwarth. Unless you are going to propose and code another way to support RStudio/VSCode in the next 4 days, we are going to have to go with this proposal.

Why call it Notebooks Working Group if there is no desire to actually create notebook images.

I'll add my thoughts as well, since this is a very fundamental matter.

@davidspek I understand you point of view, but I think this sentence is making some wrong assumptions. Notebooks WG does not aim to establish a generic central repository and maintenance for a wide variety of images. As @StefanoFioravanzo mentioned above, our main focus should be to provide example images that focus on setting up the runtime environments, so that end-users can build their own images.

The first step to achieve this is to standardize on how popular Notebook servers can be run on Kubeflow. Of course, after we are satisfied with our basic example images and documentation, we can discuss about having more sophisticated ML layers.
This effort should happen organically though and with well defined processes. We shouldn't try to do it all at once just a few days before the first release cut.

As @davidspek stated, we are more than happy to maintain these images, so there will be no extra work from your behalf.

It’s not a matter of responsibility, but a matter of process and consensus. It's a matter of documenting what we want to do and what the user expectations should be. And this is why we have the design document.

The "contrib" idea, that makes very little sense, as these images are literally not contrib, they are examples

The example notebook images, which is also the title of the design doc, are the ones that are meant to be used as examples on how people can configure/run the different servers. Not ones that expose different frameworks and sets of packages, as per the in/out of scope sections and the iterations we had over the past month.

But because we don't want to exclude the work put in the other images we want to name them contrib-* and pulish them as well. The contrib- prefix means that while an image does not yet completely align with what was established from our process/design-doc [ extra packages/frameworks ] we still want to hear user's feedback. Which is also a good next step for a migration process.

Please treat this proposal as a replacement to the previous design doc

@thesuperzapper I don't understand why you made this argument. We have been iterating on this design document at least a month now, so I can't go forth with a proposal that tosses out of the window all this effort just a few days before the release.

@kimwnasptd

It’s not a matter of responsibility, but a matter of process and consensus. It's a matter of documenting what we want to do and what the user expectations should be. And this is why we have the design document.

You committed to providing these images in the WG-Notebooks charter, which your design doc is undermining. The design doc that you agreed on outside of the working group is therefore not valid and should not be taken into consideration in this discussion.

@kimwnasptd @elikatsis that design doc was not accepted (in fact we have no current method for accepting any proposal)

@kimwnasptd can you please explain what you are trying to achieve by blocking this Proposal and is associated PRs?

@kimwnasptd I will begin reviewing the non-image PRs (as they are not controversial), and are needed to meet the agreed target of RStudio/VSCode support for 1.3:

1. Add annotations to notebook-controller (notebooks.kubeflow.org/http-rewrite-uri, notebooks.kubeflow.org/http-headers-request-set)
   • PR (Notebook-controller): Add http-rewrite-uri and http-headers-request-set annotations #5660
2. Update the jupyter WebUI to support setting those annotations
   • PR (JWA): Add server type selector for jupterlab, vs-code and r-studio #5646

@kimwnasptd @elikatsis that design doc was not accepted (in fact we have no current method for accepting any proposal)

Three of the WG's tech leads, @StefanoFioravanzo , @elikatsis and I, have authored this public document and we've been iterating on it over the last month in the WG's meetings. Although I understand the argument that we don't have yet established processes on accepting proposals. This is something I was planning to tackle after 1.3 by having our design docs in github, but this is a different discussion on its own.

@kimwnasptd can you please explain what you are trying to achieve by blocking this Proposal and is associated PRs?

Nothing. I want us to have example notebook images as much as you do, but I want us to do this in a documented way, by establishing a process and by making it clear what the goals of the WG around these images are.

@kimwnasptd I will begin reviewing the non-image PRs (as they are not controversial), and are needed to meet the agreed target of RStudio/VSCode support for 1.3

Thank you! I was also planning to start reviewing them asap since this can be done in parallel.

From community side, there are a few points I can help add:

1. Closer release date is not a valid reason we have to agree upon a proposal / design doc in a rush, if we failed to achieve what WG said w.r.t 1.3 release, then we fail and has to let it fail in some degree, we can continue our work by doing minor release like 1.3.1.
2. I agree we should establish the standardization and move forward with a formal design doc / proposal, get WG's consensus, if we fail to have consensus, we need to establish mechanism such as voting to have majority tech leads agree upon the proposal (this is based on the fact that no single company employees occupy majority seats in the WG)

@PatrickXYS

Re 1: in general I agree, but I strongly believe we can make this work (as there is very little actual work left, and the only thing we disagree on is the names of images)

Re 2: in the Notebook WG, 4 of 5 members are from the same company (not sure if other WGs have the same issue)

Re 1: in general I agree, but I strongly believe we can make this work (as there is very little actual work left, and the only thing we disagree on is the names of images)

I think the common goal of us is to help WG notebook get to a better state.
All the comments around image naming itself, it's not an issue of "As long as we have one more folk agree upon the proposal", it's the issue of "As a whole group, we can't get a consensus in standardized way".

Again, we all want to help notebook WG comes to a unified artifacts management/naming proposal, to get to that stage, can you comment in the design doc for the improvements that you think the doc need to be done?

If you think the doc is not acceptable from your side. Can you help bring up another design doc on your behalf?

@PatrickXYS While I appreciate you trying to help get this solved, that design doc is not the way this will get solved. The moment this design doc was presented by @kimwnasptd I took the time to go through it thoroughly and I highlighted all the sections that were contradicting each other or that didn't make sense. Then, after a period of no response, it was brought up in one of the meetings and a consensus was formed what the actual intention and plans were that were described in the design doc. Directly following this meeting, I took the time to rewrite most of the design doc according to what was discussed and agreed upon in that meeting. After this, the design doc was not touched for a whole month. Then, when my comments were being resolved (while they actually weren't) and my changes were being removed, I instantly tried to start the dialogue and practically begged @kimwnasptd to chat (both on slack as well as in the document he was actively editing), but my request to actually discuss the design doc was not even acknowledged. After this session of removing my changes without so much as a discussion, what came in it's place were more statements that made no sense, and actually breach the charter of the working group.

In the following Australia friendly WG meeting, where once again only @kimwnasptd and @thesuperzapper as leads of the WG were present (and myself), again a consensus was formed and we agreed on the approach that would be taken. Part of this agreement was that @thesuperzapper and I would create a short issue describing the needed changes that followed what was agreed upon in the meeting so that @kimwnasptd could quickly agree with the high level overview and we could continue. However, as you can see, the needed changes are once again being blocked the picture is painted that the goal of the working group surrounding these images was not decided upon. While in actual fact, the goal and how the intentions of the working group will be communicated to the public has been discussed and agreed upon multiple times in the working group meetings.

Furthermore, I find the most worrying part of this entire discussion that the leverage being used to block this is caused by leads of the working group that have not taken part in these discussions in the meetings (as they haven't been to the meetings in a long time) or in the discussion I tried to start in the design doc.

Even disregarding all of the above. As I highlighted in my earlier comment, it is the responsibility of the Notebooks Working Group to provide these images how they are presented in this issue. Other than @thesuperzapper who is tried to fulfill his commitment and obligation to the working group and, by extent, the Kubeflow project, the other leads of the Notebooks Working Group are failing or actively blocking their commitment and obligations. With that said, I would also like to highlight the following statement in the working group governance document:
Leads of a role SHOULD remove any other leads or roles that ... or are not fulfilling their documented responsibilities for more than 1 month.

I would also like to highlight that the blog post for the 1.3 release from the Notebooks Working Group will be basically empty if this gets blocked any longer, as my PRs are the only actual new features that would be released as a part of this working group. Which I think will severely damage the reputation of the working group.

Summing up our plan around the Notebook images and the next steps to wrap up the remaining work around 1.3, based on our latest discussions.

We will move forward with the above images and names. At the same time we will need to update the docs to note that these images only aim to show people how to configure and run the Notebook servers. For the more technical stuff, like folder names and READMEs we can discuss in the PRs. Moving forward we will also need to define our story and assumptions around the images' frameworks and start including CI/CD tests for these assumptions.

So the next steps for 1.3 are:

1. Review and merge the needed PRs to support more Notebook servers (Notebook-controller): Add http-rewrite-uri and http-headers-request-set annotations #5660 (JWA): Add server type selector for jupterlab, vs-code and r-studio #5646
2. Start merging the Notebook Dockerfile PRs. @thesuperzapper @davidspek the order for this should be the one I see in the issue description above right?
3. Create a PR to Kubeflow's docs to note the context of the images and give a heads up to people about the new images
4. Delete the old Notebook images https://github.com/kubeflow/kubeflow/tree/master/components/tensorflow-notebook-image
5. Extend the CI/CD for building the images [both in post and presubmit like we do for the other components]
6. Extend the current config of the Jupyter web app to include the new images
7. ...
8. Celebrate

If I forgot any step feel free to comment and I'll include it. At the same time I'll try to keep our design doc up to date with our latest progress on the technical decisions.

And with that said, I'm glad we've reached consensus and really looking forward to what we can do!

cc @StefanoFioravanzo @elikatsis @yanniszark @thesuperzapper @davidspek

@kimwnasptd The order above is correct, there are still a few TBD, but those will not take long to make so I will do that once the open ones are merged.
For point 3 (also regarding the READMEs) I'll also get started on a PR.
For point 5, I have started including the CI/CD in the PRs for the notebook images. This way, once it is merged the image is instantly built and the resulting image artifact can be used downstream.
Depending on the timeline of when #5646 gets merged compared to the images, point 6 can be included in #5646.

Thanks @kimwnasptd for the summary! I'll help drafting and reviewing the docs part as well, @davidspek looking forwards to the PR. If you need more assistance in the reviews and other blocking issues, don't hesitate to ping me

@kimwnasptd @StefanoFioravanzo Regarding point 6, I am almost finished pushing initial images of the existing PRs to ECR to test the CI/CD for the images. Once those are done I'll add them to the spawner_ui_config.yaml in #5646 so that the images on my personal dockerhub can be removed from that PR.

@thesuperzapper how is the review for the Notebook images going? Are there any issues?

We will need to cut a release branch around Monday so we will need to start wrapping up. The Jupyter web app and Notebooks controller PRs #5646 #5660 are almost finished, so this should be the only remaining work.

@thesuperzapper @davidspek what is the state of this?

Could you also provide me a list of images that are merged and have a built image ready? I'd like to

1. update the config of Jupyter web app to use the latest images
2. keep track of the images we will publish

Only RStudio still needs to be done. Here is a list of the images for the spawner_ui_config.yaml. Here is a snippet for the latest images for Jupyter and VS-Code.

  image:
    # The container Image for the user's Jupyter Notebook
    value: public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/jupyter-scipy:master-78840844
    # The list of available standard container Images
    options:
      - public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/jupyter-scipy:master-78840844
      - public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/jupyter-pytorch-full:master-de7c3f00
      - public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/jupyter-pytorch-cuda-full:master-de7c3f00
      - public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/jupyter-tensorflow-full:master-a9821baa
      - public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/jupyter-tensorflow-cuda-full:master-a9821baa
  imageVSCode:
    # The container Image for the user's VS-Code Server
    value: public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/codeserver-python:master-c99f886b
    # The list of available standard container Images
    options:
      - public.ecr.aws/j1r0q0g6/notebooks/notebook-servers/codeserver-python:master-c99f886b

@thesuperzapper @davidspek could you provide a list of pending PRs?

Only RStudio still needs to be done

Is there an opened PR for this? What's the effort required?

PR's are #5624 and #5711 (this one is not much work). We were already quite a ways through the RStudio image but Mathew left for a dinner break.