Implementation of Kolmogorov-Arnold Networks with PyTorch.
What is new in this project :
- Extension to 2D convolutions (classification networks).
- Extension to Hermite polynomials, in progress !
What is being done at this moment :
- benchmark on images classification datasets
- thinking on new features / evolutions (like using Hermite polynomials)
$ cd /path/to/KolmogorovArnoldNetworks
$ # install venv
$ python -m venv venv
$ source ./venv/bin/activate
$ # install requirements
$ pip install -r requirements.txt
$ # Optional : install pre-commit (needs "git init" command)
$ pre-commit installTrain on Cifar10 :
$ cd /path/to/KolmogorovArnoldNetworks
$ # with hermite of degree 5
$ python -m kan "[(3,8),(8,16),(16,32),(32,64)]" "[(256,512),(512,10)]" -k 3 -s 2 -p 1 -r "silu" -a "hermite" -a "n=5" train "cifar10" "./out/cifar10" "./out/train_hermite_cifar10" -lr 1e-4 -b 64 -e 100 --train-ratio 0.7 --cuda --save-every 256
$ # with spline of degree 2 and grid size 8
$ python -m kan "[(3,8),(8,16),(16,32),(32,64)]" "[(256,512),(512,10)]" -k 3 -s 2 -p 1 -r "silu" -a "b-spline" -a "degree=2" -a "grid_size=8" train "cifar10" "./out/cifar10" "./out/train_bspline_cifar10" -lr 1e-4 -b 64 -e 100 --train-ratio 0.7 --cuda --save-every 256Train on ImageNet :
$ # 1. download data on kaggle : https://www.kaggle.com/c/imagenet-object-localization-challenge
$ # 2. remember about the path of ImageNet dataset
$ # 3. ready for training :
$ cd /path/to/KolmogorovArnoldNetworks
$ # small network with hermite of degree 5
$ python -m kan "[(3,8),(8,16),(16,32),(32,64)]" "[(256,1000)]" -r "silu" -k 5 -s 4 -p 2 -a "hermite" -ao "n=5" --cuda train imagenet "/path/to/imagenet-object-localization-challenge/ILSVRC/Data/CLS-LOC/train" "./out/train_imagenet" -e 1000 -b 64 -lr 1e-4 --save-every 4096 --train-ratio 0.7[1] KAN: Kolmogorov-Arnold Networks - Ziming Liu, Yixuan Wang, Sachin Vaidya, Fabian Ruehle, James Halverson, Marin Soljačić, Thomas Y. Hou, Max Tegmark - 2 May 2024
[2] Wikipedia.org