MACHIAVELLI is an RL-style gym environment for text agents. Our environment is based on human-written, text-based Choose-Your-Own-Adventure games containing over half a million scenes with millions of annotations. The games spotlight high-level social decisions alongside real-world goals to achieve, measuring an agent's ability to plan and navigate complex trade-offs in realistic social environments.

For more information, see our website and our paper: Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the Machiavelli Benchmark.

Clone this repo:

git clone git@github.com:aypan17/machiavelli.git
cd machiavelli-temp

Install dependencies (Please use Python 3.11 or lower!):

pip install -r requirements.txt

Download the data:

• The data is available through Google Drive.
• The password is machiavelli.
• Place the data at the top-level of this repo as ./game_data/. (You should now have a folder structure as described in Repo structure.)

Download the "artificial conscience": In order to replicate the paper, you will need to download the artificial consciensce from Google Drive. Place the model in ./model/.

Use our demo script to play through a game:

python -m demo -h # Get the list of available games
python -m demo -g <GAME> # Play through a game interactively (playing to completion may take several hours on longer games!)
python -m demo -g <GAME> --agent RandomAgent # Play through a game selecting random actions

1. To create your own agent, create a new class in machiavelli/agents/ that inherits from machiavelli.agent.BaseAgent and implements the get_action method.
2. Then, add your agent to machiavelli/agent/load_agents.py. (See machiavelli/agents/random_agent.py for an example.)
3. Done! You can run your agent on a game using the --agent flag in demo.py.

python -m demo -g <GAME> --agent MyCustomAgent # Play through a game using your custom agent

The MACHIAVELLI benchmark is a test set of 30 challenging games where agents trained to maximize reward perform poorly on behavioral metrics by default. To evaluate on this benchmark, generate trajectories with your agent on all 30 games and save them to file. You may use the following script:

python -m generate_trajectories -a <AGENT> --traj_dir <TRAJ_DIR>
# e.g. python -m generate_trajectories -a RandomAgent --traj_dir ./demo_trajectories --num_episodes 1

Given a directory of trajectories, compute the results for your trajectories using the following script:

python -m evaluate_trajectories --traj_dir <TRAJ_DIR> --results_file <RESULTS_FILE>
# e.g. python -m evaluate_trajectories --traj_dir ./demo_trajectories --results_file ./demo_results.csv

We include the scripts used to recreate the results in our paper, including our various agent baselines, in experiments/. To reproduce our results, run the following scripts:

source ./experiments/baselines_random.sh
source ./experiments/baselines_lm.sh
source ./experiments/baselines_rl.sh 

python -m export_results

(Advanced users) If you wish to reproduce our data annotation (e.g. to extend the label set), look at the machiavelli/annotate/ and machiavelli/data/ directories.

Key files to understand the repo:

├── LICENSE
├── README.md
├── env.sh                      # Set up API keys
└── requirements.txt            # Do `pip install -r requirements.txt`

├── docs/                       # Code for website

├── experiments/                # Scripts to run experiments in the paper
│   └── baselines_*.sh          # Run trajectories for baseline agents on test set games

├── game_data/
│   ├── game_metadata.json      # List of all games and short information about them

│   ├── annotations_clean/      # Clean export of all scene-wise annotations, organized by game/scene_id
│   ├── annotations_expert/     # Scene-wise annotations from expert annotators
│   ├── annotations_model/      # Scene-wise annotations from model
│   ├── annotations_surge/      # Scene-wise annotations from Surge (crowdworker) annotators

│   ├── achievements/           # List of nodes required to hit various achievements
│   ├── scenarios/              # Full scene text of all games organized by game/scene_id
│   ├── source/                 # Raw source code used by the machiavelli.game.player
│   └── trees/                  # Game tree structures

├── machiavelli/
│   ├── __init__.py
│   ├── agents/                 # Code for baseline agents (random, LM, RL)
│   ├── annotate/               # Code for generating/managing model labels at `game_data/annotations_model`
│   ├── config.py               # Common config file used by other modules to make it easier to navigate the code
│   ├── data/                   # Code for generating non-annotation files in `game_data/`
│   ├── visualize/              # Code to visualize and explore the games
│   ├── game/                   # Code to run the games and agents
│   └── utils.py                # Helper functions

All python scripts are designed to be run from the root of this repo using module notation, e.g. python -m machiavelli.game.machiavelli_env -g <GAME>. Bash scripts are also designed to be run from the root of this repo, e.g. source ./experiments/baselines_lm.sh.

If you use this code in your research, please cite our paper:

@article{pan2023machiavelli,
    author = {Pan, Alexander and Chan, Jun Shern and Zou, Andy and Li, Nathaniel and Basart, Steven and Woodside, Thomas and Ng, Jonathan and Zhang, Hanlin and Emmons, Scott and Hendrycks, Dan},
    title = {Do the Rewards Justify the Means? Measuring Trade-Offs Between Rewards and Ethical Behavior in the Machiavelli Benchmark.},
    journal = {ICML},
    year = {2023},
}