Inference code for the Mistral's "mixtral" 8x7B mixture of experts model. Largely based on the Mistral 7B inference repository. Requires ~100GB of VRAM.

PyTorch, SentencePiece, and xformers.

pip install -r requirements.txt

Assumes you have 8 CUDA devices. You can modify this near the bottom of main.py.

python main.py