I sure took me some time to figure this one out. The Littlest JupyterHub install itself was a breeze but the Julia side of things was not as it was tricky to find the documentation. Thanks to @fredrikekre for the help!

These are notes to myself plus a script to do the install on a Linode.com server running Ubuntu. But the script should work on any Ubuntu server and many Linux servers. It maybe useful to others too. If you have questions or corrections, open an issue/pull-request.

Also, I figured out a setup to run TLJH within a virtual machine on a (real) server we have and expose it to the internet, see here.

NOTE: only run this on a dedicated server (or virtual-machine) as it will mess with settings you probably don’t want messed with! For security reasons, this is also the recommendation from the TLJH-folks (link).

TL;DR

• clone this repo to your server: git clone https://github.com/mauro3/JupyterHubWithJulia.git
• copy settings_tljh_julia_TEMPLATE.sh to settings_tljh_julia.sh
• edit settings_tljh_julia.sh to your liking
• optional but needed for HTTPS: give the server a domain name by setting the DNS records
• run ./tljh_install.sh (as root). This will take about 10-20min.
• login on your JupyterHub website as root and make user accounts
• done

TODO:

• make and use a system-image for Julia

There are two scripts in this repo (and they are the authoritative source, although I will try to keep this README up to date):

• settings-template is settings_tljh_julia_TEMPLATE.sh which needs to be filled in and renamed to settings_tljh_julia.sh.
• install script tljh_install.sh which does the full install

In the following I will describe what these scipts do. If you execute the commands by hand:

• do it as root user
• first source . ./settings_tljh_julia.sh

All global settings are contained within settings_tljh_julia.sh which is derived from editing ./settings_tljh_julia_TEMPLATE.sh. These settings are used throughout the install, so it is assumed that this file is sourced before executing the commands listed below (it is sourced automatically in tljh_install.sh).

Linode.com offers Linux servers in the cloud. Currently there are quite a few $100-credit offers going around, that’s why I started here. But I think the script should work on any Ubuntu 20.04 system.

NOTE: again, only run these scripts on a dedicated server! Both for security reasons (TLJH-people recommendation) and because these scripts will likely mess with your setup.

Install:

• Ubuntu 20.04 LTS
• choose at least 2GB of RAM (but much more for multi-user)

Boot the server. Then, if you want HTTPS, set the domain record next.

Having an actual url is needed for HTTPS access.

If you have a Domain at Linode, otherwise see below:

• Go to https://cloud.linode.com/domains/
• click “Add an A/AAAA Record”
• Settings Hostname: whatever-your-hostname-is IP address: copy from your server TTL: 5min (this sets on how quickly records propagate through DNS on changes)
• check in a few seconds/minutes whether the record propagated far enough for Let’s Encrypt to pick it up with https://letsdebug.net/ (note that the error about no web-server running is ok. It just needs to be able to resolve the domain name.)

For the domain transfer to Linode I followed https://merelycurious.me/post/connecting-namecheap-domain-to-linode; but you don’t need to have your domain at Linode.

The Let’s Encrypt setup I do via The Littelest JuptyterHub install, which is super easy. But here some notes on how to do it without using the TLJH machinery:

Let’s encrypt: docs

• root@localhost:/opt/letsencrypt# ./letsencrypt-auto certonly –standalone -d example.com -d jhub.example.com -d docker.example.com -> does not work
• loosly follow instead https://www.rosehosting.com/blog/how-to-install-lets-encrypt-on-ubuntu-20-04-with-apache/
  • apt install software-properties-common
  • apt update
  • apt install certbot
  • certbot certonly –standalone -d jhub.example.com
  • keys are in /etc/letencrypt

The install is done within the section Server (Linode) base install of the install script tljh_install.sh. It does:

• update system via apt
• set time-zone
• set networking stuff by hand (this is disabled in the script as it does not work in general)
• disable SSH password login, if there is a root.ssh/authorized_keys file (i.e. assuming there is a key for passwordless login). Note this potentially locks you out of the system, if so disable it.
• setup ufw firewall

Up-front note: there are two terminals once TLJH is running:

• the normal shell (via ssh)
• the terminal in the web-interface

The latter has some special environment variables set, namely the $PATH. The setup described here uses the normal shell exclusively (by setting the $PATH), this is a bit different to the docs on https://tljh.jupyter.org/en/latest/index.html.

This part of the install is done in the section The Littlest JupyterHub install of tljh_install.sh.

The install follows https://tljh.jupyter.org/en/latest/install/custom-server.html

curl -L https://tljh.jupyter.org/bootstrap.py | python3 - --admin $jupyteradmin

Don’t create users yet.

The install-script only does this if the $email4letsencrypt variable is set.

• Using Let’s Encrypt
• this only works once the domain is set in DNS manager (see above). Once set, it should be very quick (seconds) for Let’s Encrypt to be able to resolve the DNS. You can check with https://letsdebug.net/ whether it can resolve it.
• It’s done through TLJH https://tljh.jupyter.org/en/latest/howto/admin/https.html#howto-admin-https
• If there are problems, the website https://letsdebug.net/ might help.

tljh-config set https.enabled true
tljh-config set https.letsencrypt.email $email4letsencrypt
tljh-config add-item https.letsencrypt.domains $fqdn

Check and reload:

tljh-config show
tljh-config reload proxy

Set kernel shutdown time

The Jupyter-kernel of each user will shut down after some idle time, 10min by default. Probably increase this as Julia takes time to startup, so a shutdown is annoying. Ref: https://tljh.jupyter.org/en/latest/topic/idle-culler.html?highlight=timeout

At the root-shell:

tljh-config set services.cull.timeout 3600
tljh-config reload

Limit CPU & RAM https://tljh.jupyter.org/en/latest/topic/tljh-config.html?highlight=environment#user-server-limits

Note that Julia is quite memory hungry with one Julia notebook taking up at least 400MB and more once packages are used (Python notebooks start at 130MB). Thus, set at least 1GB of memory per user, better 2GB.

tljh-config set limits.memory 2G
tljh-config set limits.cpu 1
tljh-config reload

The default install is that each user sets their password on the first login: https://tljh.jupyter.org/en/latest/howto/auth/firstuse.html

• change your own password on https://jhub.example.com/hub/auth/change-password
• admin can reset user password with https://tljh.jupyter.org/en/latest/install/custom-server.html
• admin can reset admin password with https://tljh.jupyter.org/en/latest/howto/admin/admin-users.html

This was the tricky bit to figure out, or more precisely, the Julia side of it is a bit tricky.

If you follow along by hand, maybe backup your server now. On Linode there is the “Manual Snapshot” option in the “Backups” tab, probably best to first shutdown the server though. That way you can get back to the good install. For me, the size of the Linode backup was 32GB.

This part of the install is done in the section Python and Julia package installs (system-wide) of tljh_install.sh which calls out to julia_install.sh for the heavy lifting.

See https://tljh.jupyter.org/en/latest/howto/env/user-environment.html, but note that we don’t execute the commands at the web-terminal, thus dropping the sudo -E.

pip install numpy
pip install matplotlib
pip install scipy

This will spew some warnings about “WARNING: The directory ’home/jupyter-admin.cache/pip/http’ or its parent directory is not owned by the current user”; as far as I can tell, those are ok.

This is in julia_install.sh.

Essentially downloads Julia binaries, puts them into the right place and adds some sym-links. See setion ## Download and unpack Julia of julia_install.sh.

This is where it gets a bit dicey. I do the following:

• the root user installs the system-wide packages using the depot path DEPOT_PATH[2] (this is a variable defined within running Julia) and using an environment DEPOT_PATH[2]/environments/<julia version>/.
• install the packages:
  • IJulia to actually make the Jupyter notebooks work
  • any other packages
• To make these packages available to users the LOAD_PATH of the users needs to be set accordingly.
• Ideally, a sysimage would then be created with the installed packages for speedy startup: Julia Sysimage (WIP). But this I haven’t tried yet.

Make special environment and global depot-folder:

# the packages are installed into this depot:
export julia_global_depot=$(julia -e 'print(DEPOT_PATH[2])')
mkdir -p $julia_global_depot

# The corresponding environment is (another one could be chosen):
export julia_global_env_dir=$(julia -e 'using Pkg; print(Pkg.envdir(DEPOT_PATH[2]))')
export julia_global_env=$julia_global_env_dir/v$julia_version_short
mkdir -p $julia_global_env
touch $julia_global_env/Project.toml

The Julia kernel needs to be copied to the location where TLJH can use it.

julia --project=$julia_global_env -e 'deleteat!(DEPOT_PATH, [1,3]); using Pkg; Pkg.update(); Pkg.add("IJulia"); Pkg.precompile()'
cp -r ~/.local/share/jupyter/kernels/julia-$julia_version_short /opt/tljh/user/share/jupyter/kernels/

Adapted from https://github.com/dclong/docker-jupyterhub-julia/blob/master/Dockerfile; note that the two `chmod` in that docker file are not needed here (in fact are bad, because global package updates then fail).

Install more Julia packages as specified in the settings variable $julia_packages:

julia --project=$julia_global_env -e 'deleteat!(DEPOT_PATH, [1,3]); using Pkg; Pkg.update(); Pkg.add.(split(ENV["julia_packages"], '\'':'\'')); Pkg.precompile()'

Note, the precompilation is usable for all users.

The installed packages are availabe to all users now but they don’t have their own environment at the moment, give it to them:

mkdir -p /etc/skel/.julia/environments/v1.4
touch /etc/skel/.julia/environments/v1.4/Project.toml

This uses the /etc/skel directory which is used as template (by Linux) when a user is created.

But now their own environment shadows the global one, to rectify this the global one needs to be specified explicitly:

export julia_local_env_dir=$(julia -e 'using Pkg; print(Pkg.envdir("/etc/skel/.julia/"))')
export julia_local_env=$julia_local_env_dir/v$julia_version_short
mkdir -p $julia_local_env
touch $julia_local_env/Project.toml
mkdir -p /etc/skel/.julia/config
echo "# Add load-path to globally installed packages" > /etc/skel/.julia/config/startup.jl
echo "push!(LOAD_PATH, "\"$julia_global_env\"")" >> /etc/skel/.julia/config/startup.jl

This is work in progress. If I get to run it, I’ll update here. Create a sysimage with the globally installed packages.

https://julialang.github.io/PackageCompiler.jl/dev/sysimages/

Precompile script tmp.jl:

using ... # installed packages

# execute what is normally executed to make sysimage-compilation pick it up

notebook()

All in all:

create_sysimage([packages...], sysimage_path="/tmp/sysimg2.so", precompile_execution_file="tmp.jl")

The script tljh_install.sh finished with the last section. Here some additional and/or extra steps.

Login as admin user on the web-page and go to the “Admin” panel in the web interface.

Note that the corresponding unix users will only be created upon their first login.

See tljh_extras.sh.

• how to deploy notebooks?
  • See nbgitpuller
  • add them to /etc/skel
  • copy directories/files to all users with copy_to_users.sh script
• how to let the users collaborate:

mkdir -p /srv/scratch
chown  root:jupyterhub-users /srv/scratch
chmod 777 /srv/scratch
chmod g+s /srv/scratch
ln -s /srv/scratch /etc/skel/scratch