Skip to content

Commit

Permalink
Add bindings to Lucene's flat dense searcher: single thread ONNX on B…
Browse files Browse the repository at this point in the history 
…EIR (#1973)
  • Loading branch information
@lintool
lintool authored Aug 31, 2024
1 parent 29b6683 commit 0dd5fa7
Show file tree
Hide file tree
Showing 11 changed files with 478 additions and 305 deletions.
126 changes: 125 additions & 1 deletion docs/prebuilt-indexes.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -793,7 +793,7 @@ Detailed configuration information for the prebuilt indexes are stored in [`pyse
</details>


## Lucene Standard Impact Indexes
## Lucene Impact Indexes
<details>
<summary>MS MARCO</summary>
<dl>
Expand Down Expand Up @@ -1125,6 +1125,130 @@ Detailed configuration information for the prebuilt indexes are stored in [`pyse
</details>


## Lucene Flat Indexes
<details>
<summary>BEIR</summary>
<dl>
<dt></dt><b><code>beir-v1.0.0-trec-covid.bge-base-en-v1.5.flat</code></b>
[<a href="../pyserini/resources/index-metadata/lucene-flat.beir-v1.0.0.bge-base-en-v1.5.20240618.6cf601.README.md">readme</a>]
<dd>Lucene flat index of BEIR collection 'trec-covid' encoded by BGE-base-en-v1.5.
</dd>
<dt></dt><b><code>beir-v1.0.0-bioasq.bge-base-en-v1.5.flat</code></b>
[<a href="../pyserini/resources/index-metadata/lucene-flat.beir-v1.0.0.bge-base-en-v1.5.20240618.6cf601.README.md">readme</a>]
<dd>Lucene flat index of BEIR collection 'bioasq' encoded by BGE-base-en-v1.5.
</dd>
<dt></dt><b><code>beir-v1.0.0-nfcorpus.bge-base-en-v1.5.flat</code></b>
[<a href="../pyserini/resources/index-metadata/lucene-flat.beir-v1.0.0.bge-base-en-v1.5.20240618.6cf601.README.md">readme</a>]
<dd>Lucene flat index of BEIR collection 'nfcorpus' encoded by BGE-base-en-v1.5.
</dd>
<dt></dt><b><code>beir-v1.0.0-nq.bge-base-en-v1.5.flat</code></b>
[<a href="../pyserini/resources/index-metadata/lucene-flat.beir-v1.0.0.bge-base-en-v1.5.20240618.6cf601.README.md">readme</a>]
<dd>Lucene flat index of BEIR collection 'nq' encoded by BGE-base-en-v1.5.
</dd>
<dt></dt><b><code>beir-v1.0.0-hotpotqa.bge-base-en-v1.5.flat</code></b>
[<a href="../pyserini/resources/index-metadata/lucene-flat.beir-v1.0.0.bge-base-en-v1.5.20240618.6cf601.README.md">readme</a>]
<dd>Lucene flat index of BEIR collection 'hotpotqa' encoded by BGE-base-en-v1.5.
</dd>
<dt></dt><b><code>beir-v1.0.0-fiqa.bge-base-en-v1.5.flat</code></b>
[<a href="../pyserini/resources/index-metadata/lucene-flat.beir-v1.0.0.bge-base-en-v1.5.20240618.6cf601.README.md">readme</a>]
<dd>Lucene flat index of BEIR collection 'fiqa' encoded by BGE-base-en-v1.5.
</dd>
<dt></dt><b><code>beir-v1.0.0-signal1m.bge-base-en-v1.5.flat</code></b>
[<a href="../pyserini/resources/index-metadata/lucene-flat.beir-v1.0.0.bge-base-en-v1.5.20240618.6cf601.README.md">readme</a>]
<dd>Lucene flat index of BEIR collection 'signal1m' encoded by BGE-base-en-v1.5.
</dd>
<dt></dt><b><code>beir-v1.0.0-trec-news.bge-base-en-v1.5.flat</code></b>
[<a href="../pyserini/resources/index-metadata/lucene-flat.beir-v1.0.0.bge-base-en-v1.5.20240618.6cf601.README.md">readme</a>]
<dd>Lucene flat index of BEIR collection 'trec-news' encoded by BGE-base-en-v1.5.
</dd>
<dt></dt><b><code>beir-v1.0.0-robust04.bge-base-en-v1.5.flat</code></b>
[<a href="../pyserini/resources/index-metadata/lucene-flat.beir-v1.0.0.bge-base-en-v1.5.20240618.6cf601.README.md">readme</a>]
<dd>Lucene flat index of BEIR collection 'robust04' encoded by BGE-base-en-v1.5.
</dd>
<dt></dt><b><code>beir-v1.0.0-arguana.bge-base-en-v1.5.flat</code></b>
[<a href="../pyserini/resources/index-metadata/lucene-flat.beir-v1.0.0.bge-base-en-v1.5.20240618.6cf601.README.md">readme</a>]
<dd>Lucene flat index of BEIR collection 'arguana' encoded by BGE-base-en-v1.5.
</dd>
<dt></dt><b><code>beir-v1.0.0-webis-touche2020.bge-base-en-v1.5.flat</code></b>
[<a href="../pyserini/resources/index-metadata/lucene-flat.beir-v1.0.0.bge-base-en-v1.5.20240618.6cf601.README.md">readme</a>]
<dd>Lucene flat index of BEIR collection 'webis-touche2020' encoded by BGE-base-en-v1.5.
</dd>
<dt></dt><b><code>beir-v1.0.0-cqadupstack-android.bge-base-en-v1.5.flat</code></b>
[<a href="../pyserini/resources/index-metadata/lucene-flat.beir-v1.0.0.bge-base-en-v1.5.20240618.6cf601.README.md">readme</a>]
<dd>Lucene flat index of BEIR collection 'cqadupstack-android' encoded by BGE-base-en-v1.5.
</dd>
<dt></dt><b><code>beir-v1.0.0-cqadupstack-english.bge-base-en-v1.5.flat</code></b>
[<a href="../pyserini/resources/index-metadata/lucene-flat.beir-v1.0.0.bge-base-en-v1.5.20240618.6cf601.README.md">readme</a>]
<dd>Lucene flat index of BEIR collection 'cqadupstack-english' encoded by BGE-base-en-v1.5.
</dd>
<dt></dt><b><code>beir-v1.0.0-cqadupstack-gaming.bge-base-en-v1.5.flat</code></b>
[<a href="../pyserini/resources/index-metadata/lucene-flat.beir-v1.0.0.bge-base-en-v1.5.20240618.6cf601.README.md">readme</a>]
<dd>Lucene flat index of BEIR collection 'cqadupstack-gaming' encoded by BGE-base-en-v1.5.
</dd>
<dt></dt><b><code>beir-v1.0.0-cqadupstack-gis.bge-base-en-v1.5.flat</code></b>
[<a href="../pyserini/resources/index-metadata/lucene-flat.beir-v1.0.0.bge-base-en-v1.5.20240618.6cf601.README.md">readme</a>]
<dd>Lucene flat index of BEIR collection 'cqadupstack-gis' encoded by BGE-base-en-v1.5.
</dd>
<dt></dt><b><code>beir-v1.0.0-cqadupstack-mathematica.bge-base-en-v1.5.flat</code></b>
[<a href="../pyserini/resources/index-metadata/lucene-flat.beir-v1.0.0.bge-base-en-v1.5.20240618.6cf601.README.md">readme</a>]
<dd>Lucene flat index of BEIR collection 'cqadupstack-mathematica' encoded by BGE-base-en-v1.5.
</dd>
<dt></dt><b><code>beir-v1.0.0-cqadupstack-physics.bge-base-en-v1.5.flat</code></b>
[<a href="../pyserini/resources/index-metadata/lucene-flat.beir-v1.0.0.bge-base-en-v1.5.20240618.6cf601.README.md">readme</a>]
<dd>Lucene flat index of BEIR collection 'cqadupstack-physics' encoded by BGE-base-en-v1.5.
</dd>
<dt></dt><b><code>beir-v1.0.0-cqadupstack-programmers.bge-base-en-v1.5.flat</code></b>
[<a href="../pyserini/resources/index-metadata/lucene-flat.beir-v1.0.0.bge-base-en-v1.5.20240618.6cf601.README.md">readme</a>]
<dd>Lucene flat index of BEIR collection 'cqadupstack-programmers' encoded by BGE-base-en-v1.5.
</dd>
<dt></dt><b><code>beir-v1.0.0-cqadupstack-stats.bge-base-en-v1.5.flat</code></b>
[<a href="../pyserini/resources/index-metadata/lucene-flat.beir-v1.0.0.bge-base-en-v1.5.20240618.6cf601.README.md">readme</a>]
<dd>Lucene flat index of BEIR collection 'cqadupstack-stats' encoded by BGE-base-en-v1.5.
</dd>
<dt></dt><b><code>beir-v1.0.0-cqadupstack-tex.bge-base-en-v1.5.flat</code></b>
[<a href="../pyserini/resources/index-metadata/lucene-flat.beir-v1.0.0.bge-base-en-v1.5.20240618.6cf601.README.md">readme</a>]
<dd>Lucene flat index of BEIR collection 'cqadupstack-tex' encoded by BGE-base-en-v1.5.
</dd>
<dt></dt><b><code>beir-v1.0.0-cqadupstack-unix.bge-base-en-v1.5.flat</code></b>
[<a href="../pyserini/resources/index-metadata/lucene-flat.beir-v1.0.0.bge-base-en-v1.5.20240618.6cf601.README.md">readme</a>]
<dd>Lucene flat index of BEIR collection 'cqadupstack-unix' encoded by BGE-base-en-v1.5.
</dd>
<dt></dt><b><code>beir-v1.0.0-cqadupstack-webmasters.bge-base-en-v1.5.flat</code></b>
[<a href="../pyserini/resources/index-metadata/lucene-flat.beir-v1.0.0.bge-base-en-v1.5.20240618.6cf601.README.md">readme</a>]
<dd>Lucene flat index of BEIR collection 'cqadupstack-webmasters' encoded by BGE-base-en-v1.5.
</dd>
<dt></dt><b><code>beir-v1.0.0-cqadupstack-wordpress.bge-base-en-v1.5.flat</code></b>
[<a href="../pyserini/resources/index-metadata/lucene-flat.beir-v1.0.0.bge-base-en-v1.5.20240618.6cf601.README.md">readme</a>]
<dd>Lucene flat index of BEIR collection 'cqadupstack-wordpress' encoded by BGE-base-en-v1.5.
</dd>
<dt></dt><b><code>beir-v1.0.0-quora.bge-base-en-v1.5.flat</code></b>
[<a href="../pyserini/resources/index-metadata/lucene-flat.beir-v1.0.0.bge-base-en-v1.5.20240618.6cf601.README.md">readme</a>]
<dd>Lucene flat index of BEIR collection 'quora' encoded by BGE-base-en-v1.5.
</dd>
<dt></dt><b><code>beir-v1.0.0-dbpedia-entity.bge-base-en-v1.5.flat</code></b>
[<a href="../pyserini/resources/index-metadata/lucene-flat.beir-v1.0.0.bge-base-en-v1.5.20240618.6cf601.README.md">readme</a>]
<dd>Lucene flat index of BEIR collection 'dbpedia-entity' encoded by BGE-base-en-v1.5.
</dd>
<dt></dt><b><code>beir-v1.0.0-scidocs.bge-base-en-v1.5.flat</code></b>
[<a href="../pyserini/resources/index-metadata/lucene-flat.beir-v1.0.0.bge-base-en-v1.5.20240618.6cf601.README.md">readme</a>]
<dd>Lucene flat index of BEIR collection 'scidocs' encoded by BGE-base-en-v1.5.
</dd>
<dt></dt><b><code>beir-v1.0.0-fever.bge-base-en-v1.5.flat</code></b>
[<a href="../pyserini/resources/index-metadata/lucene-flat.beir-v1.0.0.bge-base-en-v1.5.20240618.6cf601.README.md">readme</a>]
<dd>Lucene flat index of BEIR collection 'fever' encoded by BGE-base-en-v1.5.
</dd>
<dt></dt><b><code>beir-v1.0.0-climate-fever.bge-base-en-v1.5.flat</code></b>
[<a href="../pyserini/resources/index-metadata/lucene-flat.beir-v1.0.0.bge-base-en-v1.5.20240618.6cf601.README.md">readme</a>]
<dd>Lucene flat index of BEIR collection 'climate-fever' encoded by BGE-base-en-v1.5.
</dd>
<dt></dt><b><code>beir-v1.0.0-scifact.bge-base-en-v1.5.flat</code></b>
[<a href="../pyserini/resources/index-metadata/lucene-flat.beir-v1.0.0.bge-base-en-v1.5.20240618.6cf601.README.md">readme</a>]
<dd>Lucene flat index of BEIR collection 'scifact' encoded by BGE-base-en-v1.5.
</dd>
</dl>
</details>


## Faiss Indexes
<details>
<summary>MS MARCO</summary>
Expand Down
Loading

0 comments on commit 0dd5fa7

Please sign in to comment.