This repo presents OpenP5, an open-source library for benchmarking foundation models for recommendation under the Pre-train, Personalized Prompt and Predict Paradigm (P5).
Paper: OpenP5: Benchmarking Foundation Models for Recommendation
Paper link: https://arxiv.org/pdf/2203.13366.pdf
A relevant repo regarding how to create item ID for recommendation foundation models is available here:
Paper: How to Index Item IDs for Recommendation Foundation Models
Paper link: https://arxiv.org/pdf/2305.06569.pdf
GitHub link: https://github.com/Wenyueh/LLMforRS_item_representation
Environment requirements can be found in ./environment.txt
The statistics of the selected ten datasets can be found below:
| Datasets | ML-1M | Yelp | LastFM | Beauty | ML-100K |
|---|---|---|---|---|---|
| #Users | 6,040 | 277,631 | 1,090 | 22,363 | 943 |
| #Items | 3,416 | 112,394 | 3,646 | 12,101 | 1,349 |
| #Interactions | 999,611 | 4,250,483 | 52,551 | 198,502 | 99,287 |
| Sparsity | 95.16% | 99.99% | 98.68% | 99.93% | 92.20% |
| Datasets | Clothing | CDs | Movies | Taobao | Electronics |
| #Users | 39,387 | 75,258 | 123,960 | 6,104 | 192,403 |
| #Items | 23,033 | 64,443 | 50,052 | 4,192 | 63,001 |
| #Interactions | 278,677 | 1,697,533 | 1,697,533 | 46,337 | 1,689,188 |
| Sparsity | 99.97% | 99.96% | 99.97% | 99.82% | 99.99% |
More results on various datasets can be found in ./OpenP5_more_results.pdf
Download the data from Google Drive link, and put them into ./data folder.
The training command can be found in ./command folder. Run the command such as
cd command
sh ML1M_random.sh
Download the checkpoint from Google Drive Link, and put them into ./checkpoint folder.
The evaluation command can be found in ./test_command folder. Run the command such as
cd ./test_command
sh ML1M_random.sh
Please cite the following papers corresponding to the repository:
@article{xu2023openp5,
title={OpenP5: Benchmarking Foundation Models for Recommendation},
author={Shuyuan Xu and Wenyue Hua and Yongfeng Zhang},
journal={arXiv:2306.11134},
year={2023}
}
@article{hua2023index,
title={How to Index Item IDs for Recommendation Foundation Models},
author={Hua, Wenyue and Xu, Shuyuan and Ge, Yingqiang and Zhang, Yongfeng},
journal={arXiv:2305.06569},
year={2023}
}