Skip to content

Commit

Permalink
Add Rocchio negative feedback results (castorini#1902)
Browse files Browse the repository at this point in the history 
* Add -useNegative flag for Rocchio script
* Add Rocchio negative feedback results for msmarco-passage / msmarco-doc / msmarco-doc-segment on topic dl19 / dl20 and dev
* Add Rocchio negative feedback results for robust04 / msmarco-passage-docTTTTquery
  • Loading branch information
@yuki617
yuki617 authored Jun 13, 2022
1 parent dc07344 commit 0648819
Show file tree
Hide file tree
Showing 24 changed files with 836 additions and 135 deletions.
66 changes: 56 additions & 10 deletions docs/regressions-disk45.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -100,6 +100,25 @@ target/appassembler/bin/SearchCollection \
-output runs/run.disk45.bm25+rocchio.topics.robust04.txt \
-bm25 -rocchio &
target/appassembler/bin/SearchCollection \
-index indexes/lucene-index.disk45/ \
-topics src/main/resources/topics-and-qrels/topics.adhoc.351-400.txt \
-topicreader Trec \
-output runs/run.disk45.bm25+rocchio*.topics.adhoc.351-400.txt \
-bm25 -rocchio -rocchio.useNegative &
target/appassembler/bin/SearchCollection \
-index indexes/lucene-index.disk45/ \
-topics src/main/resources/topics-and-qrels/topics.adhoc.401-450.txt \
-topicreader Trec \
-output runs/run.disk45.bm25+rocchio*.topics.adhoc.401-450.txt \
-bm25 -rocchio -rocchio.useNegative &
target/appassembler/bin/SearchCollection \
-index indexes/lucene-index.disk45/ \
-topics src/main/resources/topics-and-qrels/topics.robust04.txt \
-topicreader Trec \
-output runs/run.disk45.bm25+rocchio*.topics.robust04.txt \
-bm25 -rocchio -rocchio.useNegative &
target/appassembler/bin/SearchCollection \
-index indexes/lucene-index.disk45/ \
-topics src/main/resources/topics-and-qrels/topics.adhoc.351-400.txt \
Expand Down Expand Up @@ -176,6 +195,25 @@ target/appassembler/bin/SearchCollection \
-output runs/run.disk45.qld+rocchio.topics.robust04.txt \
-qld -rocchio &
target/appassembler/bin/SearchCollection \
-index indexes/lucene-index.disk45/ \
-topics src/main/resources/topics-and-qrels/topics.adhoc.351-400.txt \
-topicreader Trec \
-output runs/run.disk45.qld+rocchio*.topics.adhoc.351-400.txt \
-qld -rocchio -rocchio.useNegative &
target/appassembler/bin/SearchCollection \
-index indexes/lucene-index.disk45/ \
-topics src/main/resources/topics-and-qrels/topics.adhoc.401-450.txt \
-topicreader Trec \
-output runs/run.disk45.qld+rocchio*.topics.adhoc.401-450.txt \
-qld -rocchio -rocchio.useNegative &
target/appassembler/bin/SearchCollection \
-index indexes/lucene-index.disk45/ \
-topics src/main/resources/topics-and-qrels/topics.robust04.txt \
-topicreader Trec \
-output runs/run.disk45.qld+rocchio*.topics.robust04.txt \
-qld -rocchio -rocchio.useNegative &
target/appassembler/bin/SearchCollection \
-index indexes/lucene-index.disk45/ \
-topics src/main/resources/topics-and-qrels/topics.adhoc.351-400.txt \
Expand Down Expand Up @@ -211,6 +249,10 @@ tools/eval/trec_eval.9.0.4/trec_eval -m map -m P.30 src/main/resources/topics-an
tools/eval/trec_eval.9.0.4/trec_eval -m map -m P.30 src/main/resources/topics-and-qrels/qrels.adhoc.401-450.txt runs/run.disk45.bm25+rocchio.topics.adhoc.401-450.txt
tools/eval/trec_eval.9.0.4/trec_eval -m map -m P.30 src/main/resources/topics-and-qrels/qrels.robust04.txt runs/run.disk45.bm25+rocchio.topics.robust04.txt
tools/eval/trec_eval.9.0.4/trec_eval -m map -m P.30 src/main/resources/topics-and-qrels/qrels.adhoc.351-400.txt runs/run.disk45.bm25+rocchio*.topics.adhoc.351-400.txt
tools/eval/trec_eval.9.0.4/trec_eval -m map -m P.30 src/main/resources/topics-and-qrels/qrels.adhoc.401-450.txt runs/run.disk45.bm25+rocchio*.topics.adhoc.401-450.txt
tools/eval/trec_eval.9.0.4/trec_eval -m map -m P.30 src/main/resources/topics-and-qrels/qrels.robust04.txt runs/run.disk45.bm25+rocchio*.topics.robust04.txt
tools/eval/trec_eval.9.0.4/trec_eval -m map -m P.30 src/main/resources/topics-and-qrels/qrels.adhoc.351-400.txt runs/run.disk45.bm25+ax.topics.adhoc.351-400.txt
tools/eval/trec_eval.9.0.4/trec_eval -m map -m P.30 src/main/resources/topics-and-qrels/qrels.adhoc.401-450.txt runs/run.disk45.bm25+ax.topics.adhoc.401-450.txt
tools/eval/trec_eval.9.0.4/trec_eval -m map -m P.30 src/main/resources/topics-and-qrels/qrels.robust04.txt runs/run.disk45.bm25+ax.topics.robust04.txt
Expand All @@ -227,6 +269,10 @@ tools/eval/trec_eval.9.0.4/trec_eval -m map -m P.30 src/main/resources/topics-an
tools/eval/trec_eval.9.0.4/trec_eval -m map -m P.30 src/main/resources/topics-and-qrels/qrels.adhoc.401-450.txt runs/run.disk45.qld+rocchio.topics.adhoc.401-450.txt
tools/eval/trec_eval.9.0.4/trec_eval -m map -m P.30 src/main/resources/topics-and-qrels/qrels.robust04.txt runs/run.disk45.qld+rocchio.topics.robust04.txt
tools/eval/trec_eval.9.0.4/trec_eval -m map -m P.30 src/main/resources/topics-and-qrels/qrels.adhoc.351-400.txt runs/run.disk45.qld+rocchio*.topics.adhoc.351-400.txt
tools/eval/trec_eval.9.0.4/trec_eval -m map -m P.30 src/main/resources/topics-and-qrels/qrels.adhoc.401-450.txt runs/run.disk45.qld+rocchio*.topics.adhoc.401-450.txt
tools/eval/trec_eval.9.0.4/trec_eval -m map -m P.30 src/main/resources/topics-and-qrels/qrels.robust04.txt runs/run.disk45.qld+rocchio*.topics.robust04.txt
tools/eval/trec_eval.9.0.4/trec_eval -m map -m P.30 src/main/resources/topics-and-qrels/qrels.adhoc.351-400.txt runs/run.disk45.ql+ax.topics.adhoc.351-400.txt
tools/eval/trec_eval.9.0.4/trec_eval -m map -m P.30 src/main/resources/topics-and-qrels/qrels.adhoc.401-450.txt runs/run.disk45.ql+ax.topics.adhoc.401-450.txt
tools/eval/trec_eval.9.0.4/trec_eval -m map -m P.30 src/main/resources/topics-and-qrels/qrels.robust04.txt runs/run.disk45.ql+ax.topics.robust04.txt
Expand All @@ -236,18 +282,18 @@ tools/eval/trec_eval.9.0.4/trec_eval -m map -m P.30 src/main/resources/topics-an

With the above commands, you should be able to reproduce the following results:

| MAP | BM25 | +RM3 | +Rocchio | +Ax | QL | +RM3 | +Rocchio | +Ax |
|:-------------------------------------------------------------------------------------------------------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
| [TREC-7 Ad Hoc Topics](../src/main/resources/topics-and-qrels/topics.adhoc.351-400.txt) | 0.1862 | 0.2354 | 0.2397 | 0.2431 | 0.1843 | 0.2168 | 0.2189 | 0.2298 |
| [TREC-8 Ad Hoc Topics](../src/main/resources/topics-and-qrels/topics.adhoc.401-450.txt) | 0.2515 | 0.2750 | 0.2743 | 0.2812 | 0.2460 | 0.2702 | 0.2709 | 0.2647 |
| [TREC 2004 Robust Track Topics](../src/main/resources/topics-and-qrels/topics.robust04.txt) | 0.2531 | 0.2903 | 0.2924 | 0.2896 | 0.2467 | 0.2747 | 0.2756 | 0.2774 |
| MAP | BM25 | +RM3 | +Rocchio | +Rocchio* | +Ax | QL | +RM3 | +Rocchio | +Rocchio* | +Ax |
|:-------------------------------------------------------------------------------------------------------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
| [TREC-7 Ad Hoc Topics](../src/main/resources/topics-and-qrels/topics.adhoc.351-400.txt) | 0.1862 | 0.2354 | 0.2397 | 0.2385 | 0.2431 | 0.1843 | 0.2168 | 0.2189 | 0.2178 | 0.2298 |
| [TREC-8 Ad Hoc Topics](../src/main/resources/topics-and-qrels/topics.adhoc.401-450.txt) | 0.2515 | 0.2750 | 0.2743 | 0.2753 | 0.2812 | 0.2460 | 0.2702 | 0.2709 | 0.2708 | 0.2647 |
| [TREC 2004 Robust Track Topics](../src/main/resources/topics-and-qrels/topics.robust04.txt) | 0.2531 | 0.2903 | 0.2924 | 0.2931 | 0.2896 | 0.2467 | 0.2747 | 0.2756 | 0.2754 | 0.2774 |


| P30 | BM25 | +RM3 | +Rocchio | +Ax | QL | +RM3 | +Rocchio | +Ax |
|:-------------------------------------------------------------------------------------------------------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
| [TREC-7 Ad Hoc Topics](../src/main/resources/topics-and-qrels/topics.adhoc.351-400.txt) | 0.3093 | 0.3447 | 0.3527 | 0.3287 | 0.3073 | 0.3307 | 0.3293 | 0.3193 |
| [TREC-8 Ad Hoc Topics](../src/main/resources/topics-and-qrels/topics.adhoc.401-450.txt) | 0.3560 | 0.3760 | 0.3707 | 0.3753 | 0.3480 | 0.3680 | 0.3640 | 0.3500 |
| [TREC 2004 Robust Track Topics](../src/main/resources/topics-and-qrels/topics.robust04.txt) | 0.3102 | 0.3365 | 0.3365 | 0.3333 | 0.3079 | 0.3232 | 0.3237 | 0.3229 |
| P30 | BM25 | +RM3 | +Rocchio | +Rocchio* | +Ax | QL | +RM3 | +Rocchio | +Rocchio* | +Ax |
|:-------------------------------------------------------------------------------------------------------------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|-----------|
| [TREC-7 Ad Hoc Topics](../src/main/resources/topics-and-qrels/topics.adhoc.351-400.txt) | 0.3093 | 0.3447 | 0.3527 | 0.3467 | 0.3287 | 0.3073 | 0.3307 | 0.3293 | 0.3320 | 0.3193 |
| [TREC-8 Ad Hoc Topics](../src/main/resources/topics-and-qrels/topics.adhoc.401-450.txt) | 0.3560 | 0.3760 | 0.3707 | 0.3687 | 0.3753 | 0.3480 | 0.3680 | 0.3640 | 0.3667 | 0.3500 |
| [TREC 2004 Robust Track Topics](../src/main/resources/topics-and-qrels/topics.robust04.txt) | 0.3102 | 0.3365 | 0.3365 | 0.3365 | 0.3333 | 0.3079 | 0.3232 | 0.3237 | 0.3249 | 0.3229 |

## Reproduction Log[*](reproducibility.md)

Expand Down
Loading

0 comments on commit 0648819

Please sign in to comment.