Skip to content

Commit

Permalink
Update MS MARCO V1 pre-built indexes to Lucene 9 (#1295)
Browse files Browse the repository at this point in the history 
+ Fix associated test cases
+ Fix flaky test case for Mr.TyDi
  • Loading branch information
@lintool
lintool authored Oct 9, 2022
1 parent 2673031 commit 9c759cf
Show file tree
Hide file tree
Showing 15 changed files with 491 additions and 471 deletions.
134 changes: 67 additions & 67 deletions docs/2cr/msmarco-v1-doc.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -370,16 +370,16 @@ <h1 class="mb-3">MS MARCO V1 Document</h1>
<td class="expand-button"></td>
<td style="min-width: 85px">[<a href="#" data-mdb-toggle="tooltip" title="Ma et al. (SIGIR 2021) Document Expansions and Learned Sparse Lexical Representations for MS MARCO V1 and V2.">1</a>] &mdash; (1c)</td>
<td style="min-width: 400px">BM25+RM3 doc (<i>k<sub><small>1</small></sub></i>=0.9, <i>b</i>=0.4)</td>
<td>0.2774</td>
<td>0.5170</td>
<td>0.7503</td>
<td>0.2773</td>
<td>0.5174</td>
<td>0.7507</td>
<td></td>
<td>0.4014</td>
<td>0.5225</td>
<td>0.8257</td>
<td>0.4015</td>
<td>0.5254</td>
<td>0.8259</td>
<td></td>
<td>0.1622</td>
<td>0.8791</td>
<td>0.1618</td>
<td>0.8783</td>
</tr>
<tr class="hide-table-padding">
<td></td>
Expand Down Expand Up @@ -471,16 +471,16 @@ <h1 class="mb-3">MS MARCO V1 Document</h1>
<td class="expand-button"></td>
<td style="min-width: 85px">[<a href="#" data-mdb-toggle="tooltip" title="Ma et al. (SIGIR 2021) Document Expansions and Learned Sparse Lexical Representations for MS MARCO V1 and V2.">1</a>] &mdash; (1d)</td>
<td style="min-width: 400px">BM25+RM3 doc segmented (<i>k<sub><small>1</small></sub></i>=0.9, <i>b</i>=0.4)</td>
<td>0.2884</td>
<td>0.5764</td>
<td>0.7384</td>
<td>0.2892</td>
<td>0.5684</td>
<td>0.7368</td>
<td></td>
<td>0.3774</td>
<td>0.5179</td>
<td>0.8041</td>
<td>0.3792</td>
<td>0.5202</td>
<td>0.8023</td>
<td></td>
<td>0.2412</td>
<td>0.9355</td>
<td>0.2413</td>
<td>0.9351</td>
</tr>
<tr class="hide-table-padding">
<td></td>
Expand Down Expand Up @@ -681,7 +681,7 @@ <h1 class="mb-3">MS MARCO V1 Document</h1>
<td>0.5226</td>
<td>0.8102</td>
<td></td>
<td>0.2449</td>
<td>0.2447</td>
<td>0.9351</td>
</tr>
<tr class="hide-table-padding">
Expand Down Expand Up @@ -783,7 +783,7 @@ <h1 class="mb-3">MS MARCO V1 Document</h1>
<td>0.5061</td>
<td>0.7776</td>
<td></td>
<td>0.2768</td>
<td>0.2767</td>
<td>0.9357</td>
</tr>
<tr class="hide-table-padding">
Expand Down Expand Up @@ -977,16 +977,16 @@ <h1 class="mb-3">MS MARCO V1 Document</h1>
<td class="expand-button"></td>
<td style="min-width: 85px"></td>
<td style="min-width: 400px">BM25+RM3 doc (<i>k<sub><small>1</small></sub></i>=4.46, <i>b</i>=0.82)</td>
<td>0.2643</td>
<td>0.2638</td>
<td>0.5526</td>
<td>0.7189</td>
<td>0.7188</td>
<td></td>
<td>0.3619</td>
<td>0.5238</td>
<td>0.3610</td>
<td>0.5195</td>
<td>0.8180</td>
<td></td>
<td>0.2231</td>
<td>0.9305</td>
<td>0.2227</td>
<td>0.9303</td>
</tr>
<tr class="hide-table-padding">
<td></td>
Expand Down Expand Up @@ -1078,16 +1078,16 @@ <h1 class="mb-3">MS MARCO V1 Document</h1>
<td class="expand-button"></td>
<td style="min-width: 85px"></td>
<td style="min-width: 400px">BM25+RM3 doc segmented (<i>k<sub><small>1</small></sub></i>=2.16, <i>b</i>=0.61)</td>
<td>0.2658</td>
<td>0.5405</td>
<td>0.7030</td>
<td>0.2655</td>
<td>0.5392</td>
<td>0.7037</td>
<td></td>
<td>0.3472</td>
<td>0.4979</td>
<td>0.8049</td>
<td>0.3471</td>
<td>0.5030</td>
<td>0.8056</td>
<td></td>
<td>0.2443</td>
<td>0.9363</td>
<td>0.2448</td>
<td>0.9359</td>
</tr>
<tr class="hide-table-padding">
<td></td>
Expand Down Expand Up @@ -1188,7 +1188,7 @@ <h1 class="mb-3">MS MARCO V1 Document</h1>
<td>0.8217</td>
<td></td>
<td>0.2242</td>
<td>0.9316</td>
<td>0.9314</td>
</tr>
<tr class="hide-table-padding">
<td></td>
Expand Down Expand Up @@ -1280,15 +1280,15 @@ <h1 class="mb-3">MS MARCO V1 Document</h1>
<td class="expand-button"></td>
<td style="min-width: 85px"></td>
<td style="min-width: 400px">BM25+Rocchio doc segmented (<i>k<sub><small>1</small></sub></i>=2.16, <i>b</i>=0.61)</td>
<td>0.2677</td>
<td>0.5424</td>
<td>0.2672</td>
<td>0.5421</td>
<td>0.7115</td>
<td></td>
<td>0.3521</td>
<td>0.4997</td>
<td>0.8042</td>
<td></td>
<td>0.2476</td>
<td>0.2475</td>
<td>0.9395</td>
</tr>
<tr class="hide-table-padding">
Expand Down Expand Up @@ -1585,15 +1585,15 @@ <h1 class="mb-3">MS MARCO V1 Document</h1>
<td style="min-width: 85px">[<a href="#" data-mdb-toggle="tooltip" title="Ma et al. (SIGIR 2021) Document Expansions and Learned Sparse Lexical Representations for MS MARCO V1 and V2.">1</a>] &mdash; (2c)</td>
<td style="min-width: 400px">BM25+RM3 w/ doc2query-T5 doc (<i>k<sub><small>1</small></sub></i>=0.9, <i>b</i>=0.4)</td>
<td>0.3045</td>
<td>0.5897</td>
<td>0.7738</td>
<td>0.5904</td>
<td>0.7737</td>
<td></td>
<td>0.4229</td>
<td>0.5407</td>
<td>0.8596</td>
<td>0.4230</td>
<td>0.5427</td>
<td>0.8631</td>
<td></td>
<td>0.1831</td>
<td>0.9128</td>
<td>0.1834</td>
<td>0.9126</td>
</tr>
<tr class="hide-table-padding">
<td></td>
Expand Down Expand Up @@ -1685,16 +1685,16 @@ <h1 class="mb-3">MS MARCO V1 Document</h1>
<td class="expand-button"></td>
<td style="min-width: 85px">[<a href="#" data-mdb-toggle="tooltip" title="Ma et al. (SIGIR 2021) Document Expansions and Learned Sparse Lexical Representations for MS MARCO V1 and V2.">1</a>] &mdash; (2d)</td>
<td style="min-width: 400px">BM25+RM3 w/ doc2query-T5 doc segmented (<i>k<sub><small>1</small></sub></i>=0.9, <i>b</i>=0.4)</td>
<td>0.3021</td>
<td>0.6297</td>
<td>0.7481</td>
<td>0.3030</td>
<td>0.6290</td>
<td>0.7483</td>
<td></td>
<td>0.4268</td>
<td>0.5850</td>
<td>0.8270</td>
<td>0.4271</td>
<td>0.5851</td>
<td>0.8266</td>
<td></td>
<td>0.2818</td>
<td>0.9547</td>
<td>0.2803</td>
<td>0.9551</td>
</tr>
<tr class="hide-table-padding">
<td></td>
Expand Down Expand Up @@ -1989,16 +1989,16 @@ <h1 class="mb-3">MS MARCO V1 Document</h1>
<td class="expand-button"></td>
<td style="min-width: 85px"></td>
<td style="min-width: 400px">BM25+RM3 w/ doc2query-T5 doc (<i>k<sub><small>1</small></sub></i>=4.68, <i>b</i>=0.87)</td>
<td>0.2814</td>
<td>0.6080</td>
<td>0.7177</td>
<td>0.2813</td>
<td>0.6091</td>
<td>0.7184</td>
<td></td>
<td>0.4104</td>
<td>0.5743</td>
<td>0.8240</td>
<td>0.4100</td>
<td>0.5745</td>
<td>0.8238</td>
<td></td>
<td>0.2621</td>
<td>0.9524</td>
<td>0.2623</td>
<td>0.9522</td>
</tr>
<tr class="hide-table-padding">
<td></td>
Expand Down Expand Up @@ -2090,16 +2090,16 @@ <h1 class="mb-3">MS MARCO V1 Document</h1>
<td class="expand-button"></td>
<td style="min-width: 85px"></td>
<td style="min-width: 400px">BM25+RM3 w/ doc2query-T5 doc segmented (<i>k<sub><small>1</small></sub></i>=2.56, <i>b</i>=0.59)</td>
<td>0.2893</td>
<td>0.6239</td>
<td>0.7066</td>
<td>0.2892</td>
<td>0.6247</td>
<td>0.7069</td>
<td></td>
<td>0.4025</td>
<td>0.5724</td>
<td>0.8172</td>
<td>0.4016</td>
<td>0.5711</td>
<td>0.8156</td>
<td></td>
<td>0.2985</td>
<td>0.9567</td>
<td>0.2973</td>
<td>0.9563</td>
</tr>
<tr class="hide-table-padding">
<td></td>
Expand Down
80 changes: 40 additions & 40 deletions docs/2cr/msmarco-v1-passage.html
View file
Original file line number Diff line number Diff line change
Expand Up @@ -270,15 +270,15 @@ <h1 class="mb-3">MS MARCO V1 Passage</h1>
<td class="expand-button"></td>
<td style="min-width: 85px">[<a href="#" data-mdb-toggle="tooltip" title="Ma et al. (SIGIR 2021) Document Expansions and Learned Sparse Lexical Representations for MS MARCO V1 and V2.">1</a>] &mdash; (1b)</td>
<td style="min-width: 400px">BM25+RM3 (<i>k<sub><small>1</small></sub></i>=0.9, <i>b</i>=0.4)</td>
<td>0.3390</td>
<td>0.5180</td>
<td>0.7998</td>
<td>0.3416</td>
<td>0.5216</td>
<td>0.8136</td>
<td></td>
<td>0.3019</td>
<td>0.4821</td>
<td>0.8217</td>
<td>0.3006</td>
<td>0.4896</td>
<td>0.8236</td>
<td></td>
<td>0.1564</td>
<td>0.1566</td>
<td>0.8606</td>
</tr>
<tr class="hide-table-padding">
Expand Down Expand Up @@ -375,11 +375,11 @@ <h1 class="mb-3">MS MARCO V1 Passage</h1>
<td>0.5275</td>
<td>0.8007</td>
<td></td>
<td>0.3102</td>
<td>0.4893</td>
<td>0.3115</td>
<td>0.4910</td>
<td>0.8156</td>
<td></td>
<td>0.1597</td>
<td>0.1595</td>
<td>0.8620</td>
</tr>
<tr class="hide-table-padding">
Expand Down Expand Up @@ -574,16 +574,16 @@ <h1 class="mb-3">MS MARCO V1 Passage</h1>
<td class="expand-button"></td>
<td style="min-width: 85px"></td>
<td style="min-width: 400px">BM25+RM3 (<i>k<sub><small>1</small></sub></i>=0.82, <i>b</i>=0.68)</td>
<td>0.3377</td>
<td>0.5231</td>
<td>0.7792</td>
<td>0.3339</td>
<td>0.5147</td>
<td>0.7950</td>
<td></td>
<td>0.3056</td>
<td>0.4808</td>
<td>0.8286</td>
<td>0.3017</td>
<td>0.4924</td>
<td>0.8292</td>
<td></td>
<td>0.1668</td>
<td>0.8687</td>
<td>0.1646</td>
<td>0.8704</td>
</tr>
<tr class="hide-table-padding">
<td></td>
Expand Down Expand Up @@ -675,15 +675,15 @@ <h1 class="mb-3">MS MARCO V1 Passage</h1>
<td class="expand-button"></td>
<td style="min-width: 85px"></td>
<td style="min-width: 400px">BM25+Rocchio (<i>k<sub><small>1</small></sub></i>=0.82, <i>b</i>=0.68)</td>
<td>0.3394</td>
<td>0.5271</td>
<td>0.7969</td>
<td>0.3396</td>
<td>0.5275</td>
<td>0.7948</td>
<td></td>
<td>0.3110</td>
<td>0.4901</td>
<td>0.3120</td>
<td>0.4908</td>
<td>0.8327</td>
<td></td>
<td>0.1685</td>
<td>0.1684</td>
<td>0.8726</td>
</tr>
<tr class="hide-table-padding">
Expand Down Expand Up @@ -878,16 +878,16 @@ <h1 class="mb-3">MS MARCO V1 Passage</h1>
<td class="expand-button"></td>
<td style="min-width: 85px">[<a href="#" data-mdb-toggle="tooltip" title="Ma et al. (SIGIR 2021) Document Expansions and Learned Sparse Lexical Representations for MS MARCO V1 and V2.">1</a>] &mdash; (2b)</td>
<td style="min-width: 400px">BM25+RM3 w/ doc2query-T5 (<i>k<sub><small>1</small></sub></i>=0.9, <i>b</i>=0.4)</td>
<td>0.4485</td>
<td>0.6548</td>
<td>0.8861</td>
<td>0.4483</td>
<td>0.6586</td>
<td>0.8863</td>
<td></td>
<td>0.4295</td>
<td>0.6172</td>
<td>0.8699</td>
<td>0.4286</td>
<td>0.6131</td>
<td>0.8700</td>
<td></td>
<td>0.2140</td>
<td>0.9463</td>
<td>0.2139</td>
<td>0.9460</td>
</tr>
<tr class="hide-table-padding">
<td></td>
Expand Down Expand Up @@ -1182,15 +1182,15 @@ <h1 class="mb-3">MS MARCO V1 Passage</h1>
<td class="expand-button"></td>
<td style="min-width: 85px"></td>
<td style="min-width: 400px">BM25+RM3 w/ doc2query-T5 (<i>k<sub><small>1</small></sub></i>=2.18, <i>b</i>=0.86)</td>
<td>0.4360</td>
<td>0.6528</td>
<td>0.8424</td>
<td>0.4377</td>
<td>0.6537</td>
<td>0.8443</td>
<td></td>
<td>0.4347</td>
<td>0.6232</td>
<td>0.8609</td>
<td>0.4348</td>
<td>0.6235</td>
<td>0.8605</td>
<td></td>
<td>0.2374</td>
<td>0.2382</td>
<td>0.9528</td>
</tr>
<tr class="hide-table-padding">
Expand Down Expand Up @@ -1291,7 +1291,7 @@ <h1 class="mb-3">MS MARCO V1 Passage</h1>
<td>0.6224</td>
<td>0.8641</td>
<td></td>
<td>0.2396</td>
<td>0.2395</td>
<td>0.9535</td>
</tr>
<tr class="hide-table-padding">
Expand Down
Loading

0 comments on commit 9c759cf

Please sign in to comment.