The state-of-the-art query performance prediction methods rely on the fine-tuning of contextual language models to estimate retrieval effectiveness on a per-query basis. Our work in this paper builds on this strong foundation and proposes to learn rich query representations by learning the interactions between the query and two important contextual information, namely the set of documents retrieved by that query, and the set of similar historical queries with known retrieval effectiveness. We propose that such contextualized query representations can be more accurate estimators of query performance as they embed the performance of past similar queries and the semantics of the documents retrieved by the query. We perform extensive experiments on the MSMARCO collection and its accompanying query sets including MSMARCO Dev set and TREC Deep Learning tracks of 2019, 2020, 2021, and DL-Hard. Our experiments reveal that our proposed method shows robust and effective performance compared to state-of-the-art baselines.

first, you need to clone the repository:

git clone https://github.com/sadjadeb/Nearest-Neighbor-QPP.git

Then, you need to create a virtual environment and install the requirements:

cd Nearest-Neighbor-QPP/
sudo apt-get install virtualenv
virtualenv venv
source venv/bin/activate
pip install -r requirements.txt

You need the MSMARCO dataset to run the code. You can download the dataset from here and here. After downloading the dataset, you need to extract the files and put them in the data directory. Here is the list of the files you need to put in the data directory:

• collection.tsv

• queries.train.tsv

• qrels.train.tsv

• top1000.train.tar.gz

• queries.dev.small.tsv

• qrels.dev.small.tsv

• top1000.dev.tar.gz

• msmarco-test2019-queries.tsv

• 2019qrels-pass.txt

• msmarco-passagetest2019-top1000.tsv

• msmarco-test2020-queries.tsv

• 2020qrels-pass.txt

• msmarco-passagetest2020-top1000.tsv

• dl_hard-passage.qrels

• topics.tsv (from DL-Hard)

• bm25.run (from DL-Hard)

To create a dictionary which maps each query to its actual performance by BM25 (i.e. MRR@10), you need to run the following command:

python extract_metrics_per_query.py --run /path/to/run/file --qrels /path/to/qrels/file --qrels /path/to/qrels/file

It will create a file named run-file-name_evaluation-per-query.json in the data/eval_per_query directory.

Then you need to create a file which contains the most similar query from train-set(a.k.a. historical queries with known retrieval effectiveness) to each query. To do so, you need to run the following command:

python find_most_similar_query.py --base_queries /path/to/train-set/queries --base_queries /path/to/train-set/queries --base_queries /path/to/train-set/queries --target_queries /path/to/desired/queries --target_queries /path/to/desired/queries --target_queries /path/to/desired/queries --model_name /name/of/the/model --model_name /name/of/the/model --model_name /name/of/the/language/model --hits /number/of/hits --hits /number/of/hits --hits /number/of/hits

Finally, to gather all data in a file to make it easier to load the data, you need to run the following commands:

python create_train_pkl_file.py
python create_test_pkl_file.py

To train the model, you need to run the following command:

python train.py

You can change the hyperparameters of the model by changing the values in the lines 9-12 of the train.py file.

To test the model, you need to run the following command:

python test.py

To evaluate the model, you need to run the following command:

python evaluation.py --actual /path/to/actual/performance/file --predicted /path/to/predicted/performance/file --target_metric /target/metric

The below table shows the results of our proposed method (NN-QPP) compared to the baselines over four datasets.

We also conducted an ablation study to investigate the impact of size of the Query-Store on the performance of the model. The below figure shows the results of the study.

If you use this code, please cite our paper:

@inproceedings{ebrahimi2024estimating,
    title = {Estimating Query Performance Through Rich Contextualized Query Representations},
    author = {Ebrahimi, Sajad and Khodabakhsh, Maryam and Arabzadeh, Negar and Bagheri, Ebrahim},
    year = {2024},
    month = {03},
    pages = {49-58},
    booktitle={European Conference on Information Retrieval},
    organization={Springer},
    isbn = {978-3-031-56065-1},
    doi = {10.1007/978-3-031-56066-8_6}
}