In this project, we investigated the effectiveness of different game of Amazons representations in AlphaZero and MuZero algorithms. To accelerate training, we proposed a novel board game representation that considerably reduces the action space to the size of the board game in the neural network's policy head. Our experimental results demonstrated that the new representation significantly improves the training efficiency and performance of MuZero and AlphaZero compared to conventional representation. The new representation results in less memory usage and a reduced number of learning parameters. The approach is also readily applicable to any game with a fixed board, such as chess. With this new representation, the policy space size reduces from O(P(n)) to O(n), where P(n) represents some polynomial in n.

A commented and documented implementation of MuZero based on the Google DeepMind paper (Schrittwieser et al., Nov 2019) and the associated pseudocode. It is designed to be easily adaptable for every games or reinforcement learning environments (like gym). You only need to add a game file with the hyperparameters and the game class. Please refer to the documentation and the example. This implementation is primarily for educational purpose.
Explanatory video of MuZero

MuZero is a state of the art RL algorithm for board games (Chess, Go, ...) and Atari games. It is the successor to AlphaZero but without any knowledge of the environment underlying dynamics. MuZero learns a model of the environment and uses an internal representation that contains only the useful information for predicting the reward, value, policy and transitions. MuZero is also close to Value prediction networks. See How it works.

• Residual Network and Fully connected network in PyTorch
• Multi-Threaded/Asynchronous/Cluster with Ray
• Multi GPU support for the training and the selfplay
• TensorBoard real-time monitoring
• Model weights automatically saved at checkpoints
• Single and two player mode
• Commented and documented
• Easily adaptable for new games
• Examples of board games, Gym and Atari games (See list of implemented games)
• Pretrained weights available
• Windows support (Experimental / Workaround: Use the notebook in Google Colab)

Here is a list of features which could be interesting to add but which are not in MuZero's paper. We are open to contributions and other ideas.

• Hyperparameter search
• Continuous action space
• Tool to understand the learned model
• Batch MCTS
• Support of more than two player games

All performances are tracked and displayed in real time in TensorBoard :

Testing Lunar Lander :

• Cartpole (Tested with the fully connected network)
• Lunar Lander (Tested in deterministic mode with the fully connected network)
• Gridworld (Tested with the fully connected network)
• Tic-tac-toe (Tested with the fully connected network and the residual network)
• Connect4 (Slightly tested with the residual network)
• Gomoku
• Twenty-One / Blackjack (Tested with the residual network)
• Atari Breakout

Tests are done on Ubuntu with 16 GB RAM / Intel i7 / GTX 1050Ti Max-Q. We make sure to obtain a progression and a level which ensures that it has learned. But we do not systematically reach a human level. For certain environments, we notice a regression after a certain time. The proposed configurations are certainly not optimal and we do not focus for now on the optimization of hyperparameters. Any help is welcome.

Network summary:

git clone https://github.com/werner-duvaud/muzero-general.git
cd muzero-general

pip install -r requirements.txt

python muzero.py

To visualize the training results, run in a new terminal:

tensorboard --logdir ./results

You can adapt the configurations of each game by editing the MuZeroConfig class of the respective file in the games folder.

• EfficientZero (Weirui Ye, Shaohuai Liu, Thanard Kurutach, Pieter Abbeel, Yang Gao)
• Sampled MuZero (Thomas Hubert, Julian Schrittwieser, Ioannis Antonoglou, Mohammadamin Barekatain, Simon Schmitt, David Silver)

• Werner Duvaud
• Aurèle Hainaut
• Paul Lenoir
• Contributors

Please use this bibtex if you want to cite this repository (master branch) in your publications:

@misc{muzero-general,
  author       = {Werner Duvaud, Aurèle Hainaut},
  title        = {MuZero General: Open Reimplementation of MuZero},
  year         = {2019},
  publisher    = {GitHub},
  journal      = {GitHub repository},
  howpublished = {\url{https://github.com/werner-duvaud/muzero-general}},
}

• GitHub Issues: For reporting bugs.
• Pull Requests: For submitting code contributions.
• Discord server: For discussions about development or any general questions.