updates

castorini · lintool · Jun 14, 2021 · Jun 2, 2021 · Jun 4, 2021 · Jun 4, 2021
commit 7c3cb669146e575d2eb3f8aa2a76b1a898b2dc96
diff --git a/docs/regressions-dl19-doc-docTTTTTquery-per-doc.md b/docs/regressions-dl19-doc-docTTTTTquery-per-doc.md
@@ -54,12 +54,12 @@ nohup target/appassembler/bin/SearchCollection -index indexes/lucene-index.msmar
 nohup target/appassembler/bin/SearchCollection -index indexes/lucene-index.msmarco-doc-docTTTTTquery-per-doc.pos+docvectors+raw \
  -topicreader TsvInt -topics src/main/resources/topics-and-qrels/topics.dl19-doc.txt \
  -output runs/run.msmarco-doc-docTTTTTquery-per-doc.bm25-tuned.topics.dl19-doc.txt \
- -bm25 -bm25.k1 3.44 -bm25.b 0.87 -hits 100 &
+ -bm25 -bm25.k1 4.68 -bm25.b 0.87 -hits 100 &
 
 nohup target/appassembler/bin/SearchCollection -index indexes/lucene-index.msmarco-doc-docTTTTTquery-per-doc.pos+docvectors+raw \
  -topicreader TsvInt -topics src/main/resources/topics-and-qrels/topics.dl19-doc.txt \
  -output runs/run.msmarco-doc-docTTTTTquery-per-doc.bm25-tuned+rm3.topics.dl19-doc.txt \
- -bm25 -bm25.k1 3.44 -bm25.b 0.87 -rm3 -hits 100 &
+ -bm25 -bm25.k1 4.68 -bm25.b 0.87 -rm3 -hits 100 &
 ```
 
 Evaluation can be performed using `trec_eval`:
@@ -80,22 +80,22 @@ With the above commands, you should be able to reproduce the following results:
 
 MAP                                     | BM25 (default)| +RM3      | BM25 (tuned)| +RM3      |
 :---------------------------------------|-----------|-----------|-----------|-----------|
-[DL19 (Doc)](https://trec.nist.gov/data/deep2019.html)| 0.2699    | 0.3044    | 0.2662    | 0.2880    |
+[DL19 (Doc)](https://trec.nist.gov/data/deep2019.html)| 0.2699    | 0.3044    | 0.2620    | 0.2812    |
 
 
 R@100                                   | BM25 (default)| +RM3      | BM25 (tuned)| +RM3      |
 :---------------------------------------|-----------|-----------|-----------|-----------|
-[DL19 (Doc)](https://trec.nist.gov/data/deep2019.html)| 0.4198    | 0.4465    | 0.4041    | 0.4202    |
+[DL19 (Doc)](https://trec.nist.gov/data/deep2019.html)| 0.4198    | 0.4465    | 0.3992    | 0.4119    |
 
 
 NDCG@10                                 | BM25 (default)| +RM3      | BM25 (tuned)| +RM3      |
 :---------------------------------------|-----------|-----------|-----------|-----------|
-[DL19 (Doc)](https://trec.nist.gov/data/deep2019.html)| 0.5968    | 0.5895    | 0.5917    | 0.5973    |
+[DL19 (Doc)](https://trec.nist.gov/data/deep2019.html)| 0.5968    | 0.5895    | 0.5967    | 0.6075    |
 
 Explanation of settings:
 
 + The setting "default" refers the default BM25 settings of `k1=0.9`, `b=0.4`.
-+ The setting "tuned" refers to `k1=3.44`, `b=0.87`, tuned using the MS MARCO document sparse judgments on 2019/06. Tuning was performed on _on the original documents_, not the expanded documents.
++ The setting "tuned" refers to `k1=4.68`, `b=0.87`, tuned using the MS MARCO document sparse judgments to optimize for recall@100 (i.e., for first-stage retrieval) on 2019/12.
 
 Settings tuned on the MS MARCO document sparse judgments _may not_ work well on the TREC dense judgments.
 

diff --git a/docs/regressions-dl20-doc-docTTTTTquery-per-doc.md b/docs/regressions-dl20-doc-docTTTTTquery-per-doc.md
@@ -100,7 +100,7 @@ R@100                                   | BM25 (default)| +RM3      | BM25 (tune
 Explanation of settings:
 
 + The setting "default" refers the default BM25 settings of `k1=0.9`, `b=0.4`.
-+ The setting "tuned" refers to `k1=4.68`, `b=0.87`, tuned to optimize for recall@100 (i.e., for first-stage retrieval) on 2019/12.
++ The setting "tuned" refers to `k1=4.68`, `b=0.87`, tuned using the MS MARCO document sparse judgments to optimize for recall@100 (i.e., for first-stage retrieval) on 2019/12.
 
 Settings tuned on the MS MARCO document sparse judgments _may not_ work well on the TREC dense judgments.
 

diff --git a/src/main/resources/docgen/templates/dl19-doc-docTTTTTquery-per-doc.template b/src/main/resources/docgen/templates/dl19-doc-docTTTTTquery-per-doc.template
@@ -54,7 +54,7 @@ ${effectiveness}
 Explanation of settings:
 
 + The setting "default" refers the default BM25 settings of `k1=0.9`, `b=0.4`.
-+ The setting "tuned" refers to `k1=3.44`, `b=0.87`, tuned using the MS MARCO document sparse judgments on 2019/06. Tuning was performed on _on the original documents_, not the expanded documents.
++ The setting "tuned" refers to `k1=4.68`, `b=0.87`, tuned using the MS MARCO document sparse judgments to optimize for recall@100 (i.e., for first-stage retrieval) on 2019/12.
 
 Settings tuned on the MS MARCO document sparse judgments _may not_ work well on the TREC dense judgments.
 

diff --git a/src/main/resources/docgen/templates/dl20-doc-docTTTTTquery-per-doc.template b/src/main/resources/docgen/templates/dl20-doc-docTTTTTquery-per-doc.template
@@ -54,7 +54,7 @@ ${effectiveness}
 Explanation of settings:
 
 + The setting "default" refers the default BM25 settings of `k1=0.9`, `b=0.4`.
-+ The setting "tuned" refers to `k1=4.68`, `b=0.87`, tuned to optimize for recall@100 (i.e., for first-stage retrieval) on 2019/12.
++ The setting "tuned" refers to `k1=4.68`, `b=0.87`, tuned using the MS MARCO document sparse judgments to optimize for recall@100 (i.e., for first-stage retrieval) on 2019/12.
 
 Settings tuned on the MS MARCO document sparse judgments _may not_ work well on the TREC dense judgments.
 

diff --git a/src/main/resources/regression/dl19-doc-docTTTTTquery-per-doc.yaml b/src/main/resources/regression/dl19-doc-docTTTTTquery-per-doc.yaml
@@ -87,28 +87,28 @@ models:
     display: BM25 (tuned)
     params:
       - -bm25
-      - -bm25.k1 3.44
+      - -bm25.k1 4.68
       - -bm25.b 0.87
       - -hits 100  # Note, this is different DL 2019 passage ranking!
     results:
       map:
-        - 0.2662
+        - 0.2620
       NDCG@10:
-        - 0.5917
+        - 0.5967
       R@100:
-        - 0.4041
+        - 0.3992
   - name: bm25-tuned+rm3
     display: +RM3
     params:
       - -bm25
-      - -bm25.k1 3.44
+      - -bm25.k1 4.68
       - -bm25.b 0.87
       - -rm3
       - -hits 100  # Note, this is different DL 2019 passage ranking!
     results:
       map:
-        - 0.2880
+        - 0.2812
       NDCG@10:
-        - 0.5973
+        - 0.6075
       R@100:
-        - 0.4202
+        - 0.4119