diff --git a/docs/regressions-dl19-doc-hgf-wp.md b/docs/regressions-dl19-doc-hgf-wp.md index bcf3b4da32..7ea57a7dd0 100644 --- a/docs/regressions-dl19-doc-hgf-wp.md +++ b/docs/regressions-dl19-doc-hgf-wp.md @@ -47,19 +47,19 @@ After indexing has completed, you should be able to perform retrieval as follows ``` target/appassembler/bin/SearchCollection \ -index indexes/lucene-index.msmarco-doc-hgf-wp/ \ - -topics src/main/resources/topics-and-qrels/topics.dl19-doc.wp.tsv.gz \ + -topics src/main/resources/topics-and-qrels/topics.dl19-doc.txt \ -topicreader TsvInt \ - -output runs/run.msmarco-doc.bm25-default.topics.dl19-doc.wp.txt \ + -output runs/run.msmarco-doc.bm25-default.topics.dl19-doc.txt \ -bm25 -analyzeWithHuggingFaceTokenizer bert-base-uncased & ``` Evaluation can be performed using `trec_eval`: ``` -tools/eval/trec_eval.9.0.4/trec_eval -c -M 100 -m map src/main/resources/topics-and-qrels/qrels.dl19-doc.txt runs/run.msmarco-doc.bm25-default.topics.dl19-doc.wp.txt -tools/eval/trec_eval.9.0.4/trec_eval -c -m ndcg_cut.10 src/main/resources/topics-and-qrels/qrels.dl19-doc.txt runs/run.msmarco-doc.bm25-default.topics.dl19-doc.wp.txt -tools/eval/trec_eval.9.0.4/trec_eval -c -m recall.100 src/main/resources/topics-and-qrels/qrels.dl19-doc.txt runs/run.msmarco-doc.bm25-default.topics.dl19-doc.wp.txt -tools/eval/trec_eval.9.0.4/trec_eval -c -m recall.1000 src/main/resources/topics-and-qrels/qrels.dl19-doc.txt runs/run.msmarco-doc.bm25-default.topics.dl19-doc.wp.txt +tools/eval/trec_eval.9.0.4/trec_eval -c -M 100 -m map src/main/resources/topics-and-qrels/qrels.dl19-doc.txt runs/run.msmarco-doc.bm25-default.topics.dl19-doc.txt +tools/eval/trec_eval.9.0.4/trec_eval -c -m ndcg_cut.10 src/main/resources/topics-and-qrels/qrels.dl19-doc.txt runs/run.msmarco-doc.bm25-default.topics.dl19-doc.txt +tools/eval/trec_eval.9.0.4/trec_eval -c -m recall.100 src/main/resources/topics-and-qrels/qrels.dl19-doc.txt runs/run.msmarco-doc.bm25-default.topics.dl19-doc.txt +tools/eval/trec_eval.9.0.4/trec_eval -c -m recall.1000 src/main/resources/topics-and-qrels/qrels.dl19-doc.txt runs/run.msmarco-doc.bm25-default.topics.dl19-doc.txt ``` ## Effectiveness diff --git a/docs/regressions-dl19-passage-hgf-wp.md b/docs/regressions-dl19-passage-hgf-wp.md index 7dea92549a..cd303d33fc 100644 --- a/docs/regressions-dl19-passage-hgf-wp.md +++ b/docs/regressions-dl19-passage-hgf-wp.md @@ -47,19 +47,19 @@ After indexing has completed, you should be able to perform retrieval as follows ``` target/appassembler/bin/SearchCollection \ -index indexes/lucene-index.msmarco-passage-hgf-wp/ \ - -topics src/main/resources/topics-and-qrels/topics.dl19-passage.wp.tsv.gz \ + -topics src/main/resources/topics-and-qrels/topics.dl19-passage.txt \ -topicreader TsvInt \ - -output runs/run.msmarco-passage.bm25-default.topics.dl19-passage.wp.txt \ + -output runs/run.msmarco-passage.bm25-default.topics.dl19-passage.txt \ -bm25 -analyzeWithHuggingFaceTokenizer bert-base-uncased & ``` Evaluation can be performed using `trec_eval`: ``` -tools/eval/trec_eval.9.0.4/trec_eval -m map -c -l 2 src/main/resources/topics-and-qrels/qrels.dl19-passage.txt runs/run.msmarco-passage.bm25-default.topics.dl19-passage.wp.txt -tools/eval/trec_eval.9.0.4/trec_eval -m ndcg_cut.10 -c src/main/resources/topics-and-qrels/qrels.dl19-passage.txt runs/run.msmarco-passage.bm25-default.topics.dl19-passage.wp.txt -tools/eval/trec_eval.9.0.4/trec_eval -m recall.100 -c -l 2 src/main/resources/topics-and-qrels/qrels.dl19-passage.txt runs/run.msmarco-passage.bm25-default.topics.dl19-passage.wp.txt -tools/eval/trec_eval.9.0.4/trec_eval -m recall.1000 -c -l 2 src/main/resources/topics-and-qrels/qrels.dl19-passage.txt runs/run.msmarco-passage.bm25-default.topics.dl19-passage.wp.txt +tools/eval/trec_eval.9.0.4/trec_eval -m map -c -l 2 src/main/resources/topics-and-qrels/qrels.dl19-passage.txt runs/run.msmarco-passage.bm25-default.topics.dl19-passage.txt +tools/eval/trec_eval.9.0.4/trec_eval -m ndcg_cut.10 -c src/main/resources/topics-and-qrels/qrels.dl19-passage.txt runs/run.msmarco-passage.bm25-default.topics.dl19-passage.txt +tools/eval/trec_eval.9.0.4/trec_eval -m recall.100 -c -l 2 src/main/resources/topics-and-qrels/qrels.dl19-passage.txt runs/run.msmarco-passage.bm25-default.topics.dl19-passage.txt +tools/eval/trec_eval.9.0.4/trec_eval -m recall.1000 -c -l 2 src/main/resources/topics-and-qrels/qrels.dl19-passage.txt runs/run.msmarco-passage.bm25-default.topics.dl19-passage.txt ``` ## Effectiveness diff --git a/docs/regressions-dl20-doc-hgf-wp.md b/docs/regressions-dl20-doc-hgf-wp.md index 20447bbe85..4ec0aa3554 100644 --- a/docs/regressions-dl20-doc-hgf-wp.md +++ b/docs/regressions-dl20-doc-hgf-wp.md @@ -47,19 +47,19 @@ After indexing has completed, you should be able to perform retrieval as follows ``` target/appassembler/bin/SearchCollection \ -index indexes/lucene-index.msmarco-doc-hgf-wp/ \ - -topics src/main/resources/topics-and-qrels/topics.dl20.wp.tsv.gz \ + -topics src/main/resources/topics-and-qrels/topics.dl20.txt \ -topicreader TsvInt \ - -output runs/run.msmarco-doc.bm25-default.topics.dl20.wp.txt \ + -output runs/run.msmarco-doc.bm25-default.topics.dl20.txt \ -bm25 -analyzeWithHuggingFaceTokenizer bert-base-uncased & ``` Evaluation can be performed using `trec_eval`: ``` -tools/eval/trec_eval.9.0.4/trec_eval -c -M 100 -m map src/main/resources/topics-and-qrels/qrels.dl20-doc.txt runs/run.msmarco-doc.bm25-default.topics.dl20.wp.txt -tools/eval/trec_eval.9.0.4/trec_eval -c -m ndcg_cut.10 src/main/resources/topics-and-qrels/qrels.dl20-doc.txt runs/run.msmarco-doc.bm25-default.topics.dl20.wp.txt -tools/eval/trec_eval.9.0.4/trec_eval -c -m recall.100 src/main/resources/topics-and-qrels/qrels.dl20-doc.txt runs/run.msmarco-doc.bm25-default.topics.dl20.wp.txt -tools/eval/trec_eval.9.0.4/trec_eval -c -m recall.1000 src/main/resources/topics-and-qrels/qrels.dl20-doc.txt runs/run.msmarco-doc.bm25-default.topics.dl20.wp.txt +tools/eval/trec_eval.9.0.4/trec_eval -c -M 100 -m map src/main/resources/topics-and-qrels/qrels.dl20-doc.txt runs/run.msmarco-doc.bm25-default.topics.dl20.txt +tools/eval/trec_eval.9.0.4/trec_eval -c -m ndcg_cut.10 src/main/resources/topics-and-qrels/qrels.dl20-doc.txt runs/run.msmarco-doc.bm25-default.topics.dl20.txt +tools/eval/trec_eval.9.0.4/trec_eval -c -m recall.100 src/main/resources/topics-and-qrels/qrels.dl20-doc.txt runs/run.msmarco-doc.bm25-default.topics.dl20.txt +tools/eval/trec_eval.9.0.4/trec_eval -c -m recall.1000 src/main/resources/topics-and-qrels/qrels.dl20-doc.txt runs/run.msmarco-doc.bm25-default.topics.dl20.txt ``` ## Effectiveness diff --git a/docs/regressions-dl20-passage-hgf-wp.md b/docs/regressions-dl20-passage-hgf-wp.md index 64a04c3d69..681d38ec7b 100644 --- a/docs/regressions-dl20-passage-hgf-wp.md +++ b/docs/regressions-dl20-passage-hgf-wp.md @@ -47,19 +47,19 @@ After indexing has completed, you should be able to perform retrieval as follows ``` target/appassembler/bin/SearchCollection \ -index indexes/lucene-index.msmarco-passage-hgf-wp/ \ - -topics src/main/resources/topics-and-qrels/topics.dl20.wp.tsv.gz \ + -topics src/main/resources/topics-and-qrels/topics.dl20.txt \ -topicreader TsvInt \ - -output runs/run.msmarco-passage.bm25-default.topics.dl20.wp.txt \ + -output runs/run.msmarco-passage.bm25-default.topics.dl20.txt \ -bm25 -analyzeWithHuggingFaceTokenizer bert-base-uncased & ``` Evaluation can be performed using `trec_eval`: ``` -tools/eval/trec_eval.9.0.4/trec_eval -m map -c -l 2 src/main/resources/topics-and-qrels/qrels.dl20-passage.txt runs/run.msmarco-passage.bm25-default.topics.dl20.wp.txt -tools/eval/trec_eval.9.0.4/trec_eval -m ndcg_cut.10 -c src/main/resources/topics-and-qrels/qrels.dl20-passage.txt runs/run.msmarco-passage.bm25-default.topics.dl20.wp.txt -tools/eval/trec_eval.9.0.4/trec_eval -m recall.100 -c -l 2 src/main/resources/topics-and-qrels/qrels.dl20-passage.txt runs/run.msmarco-passage.bm25-default.topics.dl20.wp.txt -tools/eval/trec_eval.9.0.4/trec_eval -m recall.1000 -c -l 2 src/main/resources/topics-and-qrels/qrels.dl20-passage.txt runs/run.msmarco-passage.bm25-default.topics.dl20.wp.txt +tools/eval/trec_eval.9.0.4/trec_eval -m map -c -l 2 src/main/resources/topics-and-qrels/qrels.dl20-passage.txt runs/run.msmarco-passage.bm25-default.topics.dl20.txt +tools/eval/trec_eval.9.0.4/trec_eval -m ndcg_cut.10 -c src/main/resources/topics-and-qrels/qrels.dl20-passage.txt runs/run.msmarco-passage.bm25-default.topics.dl20.txt +tools/eval/trec_eval.9.0.4/trec_eval -m recall.100 -c -l 2 src/main/resources/topics-and-qrels/qrels.dl20-passage.txt runs/run.msmarco-passage.bm25-default.topics.dl20.txt +tools/eval/trec_eval.9.0.4/trec_eval -m recall.1000 -c -l 2 src/main/resources/topics-and-qrels/qrels.dl20-passage.txt runs/run.msmarco-passage.bm25-default.topics.dl20.txt ``` ## Effectiveness diff --git a/docs/regressions-mrtydi-v1.1-te.md b/docs/regressions-mrtydi-v1.1-te.md index c3b8129464..b7b4fe149e 100644 --- a/docs/regressions-mrtydi-v1.1-te.md +++ b/docs/regressions-mrtydi-v1.1-te.md @@ -67,10 +67,10 @@ With the above commands, you should be able to reproduce the following results: | **MRR@100** | **BM25** | |:-------------------------------------------------------------------------------------------------------------|-----------| -| [Mr. TyDi (Telugu): train](https://github.com/castorini/mr.tydi) | 0.2847 | -| [Mr. TyDi (Telugu): dev](https://github.com/castorini/mr.tydi) | 0.2737 | -| [Mr. TyDi (Telugu): test](https://github.com/castorini/mr.tydi) | 0.3434 | +| [Mr. TyDi (Telugu): train](https://github.com/castorini/mr.tydi) | 0.4204 | +| [Mr. TyDi (Telugu): dev](https://github.com/castorini/mr.tydi) | 0.4269 | +| [Mr. TyDi (Telugu): test](https://github.com/castorini/mr.tydi) | 0.5283 | | **R@100** | **BM25** | -| [Mr. TyDi (Telugu): train](https://github.com/castorini/mr.tydi) | 0.7049 | -| [Mr. TyDi (Telugu): dev](https://github.com/castorini/mr.tydi) | 0.7040 | -| [Mr. TyDi (Telugu): test](https://github.com/castorini/mr.tydi) | 0.7577 | +| [Mr. TyDi (Telugu): train](https://github.com/castorini/mr.tydi) | 0.8229 | +| [Mr. TyDi (Telugu): dev](https://github.com/castorini/mr.tydi) | 0.8362 | +| [Mr. TyDi (Telugu): test](https://github.com/castorini/mr.tydi) | 0.8971 |