This is not an officially supported Google product.
This repository contains source code and data for Instruction Following Evaluation for Large Language Models
Please make sure that all required python packages are installed via:
pip3 install -r requirements.txt
You need to create a jsonl file with two entries: prompt and response.
Then, call evaluation_main from the parent folder of
instruction_following_eval. For example:
# Content of `--input_response_data` should be like:
# {"prompt": "Write a 300+ word summary ...", "response": "PUT YOUR MODEL RESPONSE HERE"}
# {"prompt": "I am planning a trip to ...", "response": "PUT YOUR MODEL RESPONSE HERE"}
# ...
python3 -m instruction_following_eval.evaluation_main \
--input_data=./instruction_following_eval/data/input_data.jsonl \
--input_response_data=./instruction_following_eval/data/input_response_data_gpt4_20231107_145030.jsonl \
--output_dir=./instruction_following_eval/data/If you use our work, please consider citing our preprint:
@article{zhou2023instruction,
title={Instruction-Following Evaluation for Large Language Models},
author={Zhou, Jeffrey and Lu, Tianjian and Mishra, Swaroop and Brahma, Siddhartha and Basu, Sujoy and Luan, Yi and Zhou, Denny and Hou, Le},
journal={arXiv preprint arXiv:2311.07911},
year={2023}
}