Merge moe layers and make a smaller model
install the script by:
git clone https://github.com/isEmmanuelOlowe/moe_merger
cd moe_merger
pip install -r requirements.txtThis will convert Jamba 16 experts into 4 experts.
The notebook is available here here.
python main.py --model_id "ai21labs/Jamba-v0.1" --temp_dir "tmp/temp" --save_dir "tmp/Jamba-4xMoE_slerp" --num_experts_per_tok=2 --num_local_experts=4This script is inspired by work on Mistral Merger availabele here.