update regression docs for hgf and mrtydi Te regressions (#1983)

castorini · Sep 27, 2022 · b5ecc5a · b5ecc5a
1 parent 95b9f06
commit b5ecc5a
Show file tree

Hide file tree

Showing 5 changed files with 30 additions and 30 deletions.
diff --git a/docs/regressions-dl19-doc-hgf-wp.md b/docs/regressions-dl19-doc-hgf-wp.md
@@ -47,19 +47,19 @@ After indexing has completed, you should be able to perform retrieval as follows
 ```
 target/appassembler/bin/SearchCollection \
   -index indexes/lucene-index.msmarco-doc-hgf-wp/ \
-  -topics src/main/resources/topics-and-qrels/topics.dl19-doc.wp.tsv.gz \
+  -topics src/main/resources/topics-and-qrels/topics.dl19-doc.txt \
   -topicreader TsvInt \
-  -output runs/run.msmarco-doc.bm25-default.topics.dl19-doc.wp.txt \
+  -output runs/run.msmarco-doc.bm25-default.topics.dl19-doc.txt \
   -bm25 -analyzeWithHuggingFaceTokenizer  bert-base-uncased &
 ```
 
 Evaluation can be performed using `trec_eval`:
 
 ```
-tools/eval/trec_eval.9.0.4/trec_eval -c -M 100 -m map src/main/resources/topics-and-qrels/qrels.dl19-doc.txt runs/run.msmarco-doc.bm25-default.topics.dl19-doc.wp.txt
-tools/eval/trec_eval.9.0.4/trec_eval -c -m ndcg_cut.10 src/main/resources/topics-and-qrels/qrels.dl19-doc.txt runs/run.msmarco-doc.bm25-default.topics.dl19-doc.wp.txt
-tools/eval/trec_eval.9.0.4/trec_eval -c -m recall.100 src/main/resources/topics-and-qrels/qrels.dl19-doc.txt runs/run.msmarco-doc.bm25-default.topics.dl19-doc.wp.txt
-tools/eval/trec_eval.9.0.4/trec_eval -c -m recall.1000 src/main/resources/topics-and-qrels/qrels.dl19-doc.txt runs/run.msmarco-doc.bm25-default.topics.dl19-doc.wp.txt
+tools/eval/trec_eval.9.0.4/trec_eval -c -M 100 -m map src/main/resources/topics-and-qrels/qrels.dl19-doc.txt runs/run.msmarco-doc.bm25-default.topics.dl19-doc.txt
+tools/eval/trec_eval.9.0.4/trec_eval -c -m ndcg_cut.10 src/main/resources/topics-and-qrels/qrels.dl19-doc.txt runs/run.msmarco-doc.bm25-default.topics.dl19-doc.txt
+tools/eval/trec_eval.9.0.4/trec_eval -c -m recall.100 src/main/resources/topics-and-qrels/qrels.dl19-doc.txt runs/run.msmarco-doc.bm25-default.topics.dl19-doc.txt
+tools/eval/trec_eval.9.0.4/trec_eval -c -m recall.1000 src/main/resources/topics-and-qrels/qrels.dl19-doc.txt runs/run.msmarco-doc.bm25-default.topics.dl19-doc.txt
 ```
 
 ## Effectiveness

diff --git a/docs/regressions-dl19-passage-hgf-wp.md b/docs/regressions-dl19-passage-hgf-wp.md
@@ -47,19 +47,19 @@ After indexing has completed, you should be able to perform retrieval as follows
 ```
 target/appassembler/bin/SearchCollection \
   -index indexes/lucene-index.msmarco-passage-hgf-wp/ \
-  -topics src/main/resources/topics-and-qrels/topics.dl19-passage.wp.tsv.gz \
+  -topics src/main/resources/topics-and-qrels/topics.dl19-passage.txt \
   -topicreader TsvInt \
-  -output runs/run.msmarco-passage.bm25-default.topics.dl19-passage.wp.txt \
+  -output runs/run.msmarco-passage.bm25-default.topics.dl19-passage.txt \
   -bm25 -analyzeWithHuggingFaceTokenizer  bert-base-uncased &
 ```
 
 Evaluation can be performed using `trec_eval`:
 
 ```
-tools/eval/trec_eval.9.0.4/trec_eval -m map -c -l 2 src/main/resources/topics-and-qrels/qrels.dl19-passage.txt runs/run.msmarco-passage.bm25-default.topics.dl19-passage.wp.txt
-tools/eval/trec_eval.9.0.4/trec_eval -m ndcg_cut.10 -c src/main/resources/topics-and-qrels/qrels.dl19-passage.txt runs/run.msmarco-passage.bm25-default.topics.dl19-passage.wp.txt
-tools/eval/trec_eval.9.0.4/trec_eval -m recall.100 -c -l 2 src/main/resources/topics-and-qrels/qrels.dl19-passage.txt runs/run.msmarco-passage.bm25-default.topics.dl19-passage.wp.txt
-tools/eval/trec_eval.9.0.4/trec_eval -m recall.1000 -c -l 2 src/main/resources/topics-and-qrels/qrels.dl19-passage.txt runs/run.msmarco-passage.bm25-default.topics.dl19-passage.wp.txt
+tools/eval/trec_eval.9.0.4/trec_eval -m map -c -l 2 src/main/resources/topics-and-qrels/qrels.dl19-passage.txt runs/run.msmarco-passage.bm25-default.topics.dl19-passage.txt
+tools/eval/trec_eval.9.0.4/trec_eval -m ndcg_cut.10 -c src/main/resources/topics-and-qrels/qrels.dl19-passage.txt runs/run.msmarco-passage.bm25-default.topics.dl19-passage.txt
+tools/eval/trec_eval.9.0.4/trec_eval -m recall.100 -c -l 2 src/main/resources/topics-and-qrels/qrels.dl19-passage.txt runs/run.msmarco-passage.bm25-default.topics.dl19-passage.txt
+tools/eval/trec_eval.9.0.4/trec_eval -m recall.1000 -c -l 2 src/main/resources/topics-and-qrels/qrels.dl19-passage.txt runs/run.msmarco-passage.bm25-default.topics.dl19-passage.txt
 ```
 
 ## Effectiveness

diff --git a/docs/regressions-dl20-doc-hgf-wp.md b/docs/regressions-dl20-doc-hgf-wp.md
@@ -47,19 +47,19 @@ After indexing has completed, you should be able to perform retrieval as follows
 ```
 target/appassembler/bin/SearchCollection \
   -index indexes/lucene-index.msmarco-doc-hgf-wp/ \
-  -topics src/main/resources/topics-and-qrels/topics.dl20.wp.tsv.gz \
+  -topics src/main/resources/topics-and-qrels/topics.dl20.txt \
   -topicreader TsvInt \
-  -output runs/run.msmarco-doc.bm25-default.topics.dl20.wp.txt \
+  -output runs/run.msmarco-doc.bm25-default.topics.dl20.txt \
   -bm25 -analyzeWithHuggingFaceTokenizer  bert-base-uncased &
 ```
 
 Evaluation can be performed using `trec_eval`:
 
 ```
-tools/eval/trec_eval.9.0.4/trec_eval -c -M 100 -m map src/main/resources/topics-and-qrels/qrels.dl20-doc.txt runs/run.msmarco-doc.bm25-default.topics.dl20.wp.txt
-tools/eval/trec_eval.9.0.4/trec_eval -c -m ndcg_cut.10 src/main/resources/topics-and-qrels/qrels.dl20-doc.txt runs/run.msmarco-doc.bm25-default.topics.dl20.wp.txt
-tools/eval/trec_eval.9.0.4/trec_eval -c -m recall.100 src/main/resources/topics-and-qrels/qrels.dl20-doc.txt runs/run.msmarco-doc.bm25-default.topics.dl20.wp.txt
-tools/eval/trec_eval.9.0.4/trec_eval -c -m recall.1000 src/main/resources/topics-and-qrels/qrels.dl20-doc.txt runs/run.msmarco-doc.bm25-default.topics.dl20.wp.txt
+tools/eval/trec_eval.9.0.4/trec_eval -c -M 100 -m map src/main/resources/topics-and-qrels/qrels.dl20-doc.txt runs/run.msmarco-doc.bm25-default.topics.dl20.txt
+tools/eval/trec_eval.9.0.4/trec_eval -c -m ndcg_cut.10 src/main/resources/topics-and-qrels/qrels.dl20-doc.txt runs/run.msmarco-doc.bm25-default.topics.dl20.txt
+tools/eval/trec_eval.9.0.4/trec_eval -c -m recall.100 src/main/resources/topics-and-qrels/qrels.dl20-doc.txt runs/run.msmarco-doc.bm25-default.topics.dl20.txt
+tools/eval/trec_eval.9.0.4/trec_eval -c -m recall.1000 src/main/resources/topics-and-qrels/qrels.dl20-doc.txt runs/run.msmarco-doc.bm25-default.topics.dl20.txt
 ```
 
 ## Effectiveness

diff --git a/docs/regressions-dl20-passage-hgf-wp.md b/docs/regressions-dl20-passage-hgf-wp.md
@@ -47,19 +47,19 @@ After indexing has completed, you should be able to perform retrieval as follows
 ```
 target/appassembler/bin/SearchCollection \
   -index indexes/lucene-index.msmarco-passage-hgf-wp/ \
-  -topics src/main/resources/topics-and-qrels/topics.dl20.wp.tsv.gz \
+  -topics src/main/resources/topics-and-qrels/topics.dl20.txt \
   -topicreader TsvInt \
-  -output runs/run.msmarco-passage.bm25-default.topics.dl20.wp.txt \
+  -output runs/run.msmarco-passage.bm25-default.topics.dl20.txt \
   -bm25 -analyzeWithHuggingFaceTokenizer  bert-base-uncased &
 ```
 
 Evaluation can be performed using `trec_eval`:
 
 ```
-tools/eval/trec_eval.9.0.4/trec_eval -m map -c -l 2 src/main/resources/topics-and-qrels/qrels.dl20-passage.txt runs/run.msmarco-passage.bm25-default.topics.dl20.wp.txt
-tools/eval/trec_eval.9.0.4/trec_eval -m ndcg_cut.10 -c src/main/resources/topics-and-qrels/qrels.dl20-passage.txt runs/run.msmarco-passage.bm25-default.topics.dl20.wp.txt
-tools/eval/trec_eval.9.0.4/trec_eval -m recall.100 -c -l 2 src/main/resources/topics-and-qrels/qrels.dl20-passage.txt runs/run.msmarco-passage.bm25-default.topics.dl20.wp.txt
-tools/eval/trec_eval.9.0.4/trec_eval -m recall.1000 -c -l 2 src/main/resources/topics-and-qrels/qrels.dl20-passage.txt runs/run.msmarco-passage.bm25-default.topics.dl20.wp.txt
+tools/eval/trec_eval.9.0.4/trec_eval -m map -c -l 2 src/main/resources/topics-and-qrels/qrels.dl20-passage.txt runs/run.msmarco-passage.bm25-default.topics.dl20.txt
+tools/eval/trec_eval.9.0.4/trec_eval -m ndcg_cut.10 -c src/main/resources/topics-and-qrels/qrels.dl20-passage.txt runs/run.msmarco-passage.bm25-default.topics.dl20.txt
+tools/eval/trec_eval.9.0.4/trec_eval -m recall.100 -c -l 2 src/main/resources/topics-and-qrels/qrels.dl20-passage.txt runs/run.msmarco-passage.bm25-default.topics.dl20.txt
+tools/eval/trec_eval.9.0.4/trec_eval -m recall.1000 -c -l 2 src/main/resources/topics-and-qrels/qrels.dl20-passage.txt runs/run.msmarco-passage.bm25-default.topics.dl20.txt
 ```
 
 ## Effectiveness

diff --git a/docs/regressions-mrtydi-v1.1-te.md b/docs/regressions-mrtydi-v1.1-te.md
@@ -67,10 +67,10 @@ With the above commands, you should be able to reproduce the following results:
 
 | **MRR@100**                                                                                                  | **BM25**  |
 |:-------------------------------------------------------------------------------------------------------------|-----------|
-| [Mr. TyDi (Telugu): train](https://github.com/castorini/mr.tydi)                                             | 0.2847    |
-| [Mr. TyDi (Telugu): dev](https://github.com/castorini/mr.tydi)                                               | 0.2737    |
-| [Mr. TyDi (Telugu): test](https://github.com/castorini/mr.tydi)                                              | 0.3434    |
+| [Mr. TyDi (Telugu): train](https://github.com/castorini/mr.tydi)                                             | 0.4204    |
+| [Mr. TyDi (Telugu): dev](https://github.com/castorini/mr.tydi)                                               | 0.4269    |
+| [Mr. TyDi (Telugu): test](https://github.com/castorini/mr.tydi)                                              | 0.5283    |
 | **R@100**                                                                                                    | **BM25**  |
-| [Mr. TyDi (Telugu): train](https://github.com/castorini/mr.tydi)                                             | 0.7049    |
-| [Mr. TyDi (Telugu): dev](https://github.com/castorini/mr.tydi)                                               | 0.7040    |
-| [Mr. TyDi (Telugu): test](https://github.com/castorini/mr.tydi)                                              | 0.7577    |
+| [Mr. TyDi (Telugu): train](https://github.com/castorini/mr.tydi)                                             | 0.8229    |
+| [Mr. TyDi (Telugu): dev](https://github.com/castorini/mr.tydi)                                               | 0.8362    |
+| [Mr. TyDi (Telugu): test](https://github.com/castorini/mr.tydi)                                              | 0.8971    |