Skip to content

Commit

Permalink
Install Cohere embed-english-v3.0 pre-built indexes, add relevant bin…
Browse files Browse the repository at this point in the history 
…dings (#2401)

Additionally, rename v3 to v3.0 to be consistent with BGE v1.5.
  • Loading branch information
@lintool
lintool authored Mar 1, 2024
1 parent 7077608 commit d674915
Show file tree
Hide file tree
Showing 27 changed files with 351 additions and 302 deletions.
100 changes: 50 additions & 50 deletions README.md
View file

Large diffs are not rendered by default.

12 changes: 6 additions & 6 deletions docs/regressions.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -55,8 +55,8 @@ nohup python src/main/python/run_regression.py --index --verify --search --regre
nohup python src/main/python/run_regression.py --index --verify --search --regression msmarco-passage-cos-dpr-distil-lexlsh >& logs/log.msmarco-passage-cos-dpr-distil-lexlsh &
nohup python src/main/python/run_regression.py --index --verify --search --regression msmarco-passage-bge-base-en-v1.5-hnsw >& logs/log.msmarco-passage-bge-base-en-v1.5-hnsw &
nohup python src/main/python/run_regression.py --index --verify --search --regression msmarco-passage-bge-base-en-v1.5-hnsw-int8 >& logs/log.msmarco-passage-bge-base-en-v1.5-hnsw-int8 &
nohup python src/main/python/run_regression.py --index --verify --search --regression msmarco-passage-cohere-embed-english-v3-hnsw >& logs/log.msmarco-passage-cohere-embed-english-v3-hnsw &
nohup python src/main/python/run_regression.py --index --verify --search --regression msmarco-passage-cohere-embed-english-v3-hnsw-int8 >& logs/log.msmarco-passage-cohere-embed-english-v3-hnsw-int8 &
nohup python src/main/python/run_regression.py --index --verify --search --regression msmarco-passage-cohere-embed-english-v3.0-hnsw >& logs/log.msmarco-passage-cohere-embed-english-v3.0-hnsw &
nohup python src/main/python/run_regression.py --index --verify --search --regression msmarco-passage-cohere-embed-english-v3.0-hnsw-int8 >& logs/log.msmarco-passage-cohere-embed-english-v3.0-hnsw-int8 &
nohup python src/main/python/run_regression.py --index --verify --search --regression msmarco-passage-openai-ada2 >& logs/log.msmarco-passage-openai-ada2 &

nohup python src/main/python/run_regression.py --index --verify --search --regression msmarco-passage-splade-pp-ed-onnx >& logs/log.msmarco-passage-splade-pp-ed-onnx &
Expand Down Expand Up @@ -95,8 +95,8 @@ nohup python src/main/python/run_regression.py --index --verify --search --regre
nohup python src/main/python/run_regression.py --index --verify --search --regression dl19-passage-bge-base-en-v1.5-hnsw >& logs/log.dl19-passage-bge-base-en-v1.5-hnsw &
nohup python src/main/python/run_regression.py --index --verify --search --regression dl19-passage-bge-base-en-v1.5-hnsw-int8 >& logs/log.dl19-passage-bge-base-en-v1.5-hnsw-int8 &
nohup python src/main/python/run_regression.py --index --verify --search --regression dl19-passage-openai-ada2 >& logs/log.dl19-passage-openai-ada2 &
nohup python src/main/python/run_regression.py --index --verify --search --regression dl19-passage-cohere-embed-english-v3-hnsw >& logs/log.dl19-passage-cohere-embed-english-v3-hnsw &
nohup python src/main/python/run_regression.py --index --verify --search --regression dl19-passage-cohere-embed-english-v3-hnsw-int8 >& logs/log.dl19-passage-cohere-embed-english-v3-hnsw-int8 &
nohup python src/main/python/run_regression.py --index --verify --search --regression dl19-passage-cohere-embed-english-v3.0-hnsw >& logs/log.dl19-passage-cohere-embed-english-v3.0-hnsw &
nohup python src/main/python/run_regression.py --index --verify --search --regression dl19-passage-cohere-embed-english-v3.0-hnsw-int8 >& logs/log.dl19-passage-cohere-embed-english-v3.0-hnsw-int8 &

nohup python src/main/python/run_regression.py --index --verify --search --regression dl19-passage-splade-pp-ed-onnx >& logs/log.dl19-passage-splade-pp-ed-onnx &
nohup python src/main/python/run_regression.py --index --verify --search --regression dl19-passage-splade-pp-sd-onnx >& logs/log.dl19-passage-splade-pp-sd-onnx &
Expand Down Expand Up @@ -134,8 +134,8 @@ nohup python src/main/python/run_regression.py --index --verify --search --regre
nohup python src/main/python/run_regression.py --index --verify --search --regression dl20-passage-bge-base-en-v1.5-hnsw >& logs/log.dl20-passage-bge-base-en-v1.5-hnsw &
nohup python src/main/python/run_regression.py --index --verify --search --regression dl20-passage-bge-base-en-v1.5-hnsw-int8 >& logs/log.dl20-passage-bge-base-en-v1.5-hnsw-int8 &
nohup python src/main/python/run_regression.py --index --verify --search --regression dl20-passage-openai-ada2 >& logs/log.dl20-passage-openai-ada2 &
nohup python src/main/python/run_regression.py --index --verify --search --regression dl20-passage-cohere-embed-english-v3-hnsw >& logs/log.dl20-passage-cohere-embed-english-v3-hnsw &
nohup python src/main/python/run_regression.py --index --verify --search --regression dl20-passage-cohere-embed-english-v3-hnsw-int8 >& logs/log.dl20-passage-cohere-embed-english-v3-hnsw-int8 &
nohup python src/main/python/run_regression.py --index --verify --search --regression dl20-passage-cohere-embed-english-v3.0-hnsw >& logs/log.dl20-passage-cohere-embed-english-v3.0-hnsw &
nohup python src/main/python/run_regression.py --index --verify --search --regression dl20-passage-cohere-embed-english-v3.0-hnsw-int8 >& logs/log.dl20-passage-cohere-embed-english-v3.0-hnsw-int8 &

nohup python src/main/python/run_regression.py --index --verify --search --regression dl20-passage-splade-pp-ed-onnx >& logs/log.dl20-passage-splade-pp-ed-onnx &
nohup python src/main/python/run_regression.py --index --verify --search --regression dl20-passage-splade-pp-sd-onnx >& logs/log.dl20-passage-splade-pp-sd-onnx &
Expand Down
52 changes: 26 additions & 26 deletions ...sage-cohere-embed-english-v3-hnsw-int8.md → ...ge-cohere-embed-english-v3.0-hnsw-int8.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -6,21 +6,21 @@ This page describes regression experiments, integrated into Anserini's regressio

In these experiments, we are using pre-encoded queries (i.e., cached results of query encoding).

The exact configurations for these regressions are stored in [this YAML file](../../src/main/resources/regression/dl19-passage-cohere-embed-english-v3-hnsw-int8.yaml).
Note that this page is automatically generated from [this template](../../src/main/resources/docgen/templates/dl19-passage-cohere-embed-english-v3-hnsw-int8.template) as part of Anserini's regression pipeline, so do not modify this page directly; modify the template instead and then run `bin/build.sh` to rebuild the documentation.
The exact configurations for these regressions are stored in [this YAML file](../../src/main/resources/regression/dl19-passage-cohere-embed-english-v3.0-hnsw-int8.yaml).
Note that this page is automatically generated from [this template](../../src/main/resources/docgen/templates/dl19-passage-cohere-embed-english-v3.0-hnsw-int8.template) as part of Anserini's regression pipeline, so do not modify this page directly; modify the template instead and then run `bin/build.sh` to rebuild the documentation.

From one of our Waterloo servers (e.g., `orca`), the following command will perform the complete regression, end to end:

```bash
python src/main/python/run_regression.py --index --verify --search --regression dl19-passage-cohere-embed-english-v3-hnsw-int8
python src/main/python/run_regression.py --index --verify --search --regression dl19-passage-cohere-embed-english-v3.0-hnsw-int8
```

We make available a version of the MS MARCO Passage Corpus that has already been encoded with Cohere embed-english-v3.0.

From any machine, the following command will download the corpus and perform the complete regression, end to end:

```bash
python src/main/python/run_regression.py --download --index --verify --search --regression dl19-passage-cohere-embed-english-v3-hnsw-int8
python src/main/python/run_regression.py --download --index --verify --search --regression dl19-passage-cohere-embed-english-v3.0-hnsw-int8
```

The `run_regression.py` script automates the following steps, but if you want to perform each step manually, simply copy/paste from the commands below and you'll obtain the same regression results.
Expand All @@ -30,16 +30,16 @@ The `run_regression.py` script automates the following steps, but if you want to
Download the corpus and unpack into `collections/`:

```bash
wget https://rgw.cs.uwaterloo.ca/pyserini/data/msmarco-passage-cohere-embed-english-v3.tar -P collections/
tar xvf collections/msmarco-passage-cohere-embed-english-v3.tar -C collections/
wget https://rgw.cs.uwaterloo.ca/pyserini/data/msmarco-passage-cohere-embed-english-v3.0.tar -P collections/
tar xvf collections/msmarco-passage-cohere-embed-english-v3.0.tar -C collections/
```

To confirm, `msmarco-passage-cohere-embed-english-v3.tar` is 38 GB and has MD5 checksum `6b7d9795806891b227378f6c290464a9`.
To confirm, `msmarco-passage-cohere-embed-english-v3.0.tar` is 38 GB and has MD5 checksum `06a6e38a0522850c6aa504db7b2617f5`.
With the corpus downloaded, the following command will perform the remaining steps below:

```bash
python src/main/python/run_regression.py --index --verify --search --regression dl19-passage-cohere-embed-english-v3-hnsw-int8 \
--corpus-path collections/msmarco-passage-cohere-embed-english-v3
python src/main/python/run_regression.py --index --verify --search --regression dl19-passage-cohere-embed-english-v3.0-hnsw-int8 \
--corpus-path collections/msmarco-passage-cohere-embed-english-v3.0
```

## Indexing
Expand All @@ -49,14 +49,14 @@ Sample indexing command, building HNSW indexes:
```bash
target/appassembler/bin/IndexHnswDenseVectors \
-collection JsonDenseVectorCollection \
-input /path/to/msmarco-passage-cohere-embed-english-v3 \
-input /path/to/msmarco-passage-cohere-embed-english-v3.0 \
-generator HnswDenseVectorDocumentGenerator \
-index indexes/lucene-hnsw.msmarco-passage-cohere-embed-english-v3-int8/ \
-index indexes/lucene-hnsw.msmarco-passage-cohere-embed-english-v3.0-int8/ \
-threads 16 -M 16 -efC 100 -noMerge -quantize.int8 \
>& logs/log.msmarco-passage-cohere-embed-english-v3 &
>& logs/log.msmarco-passage-cohere-embed-english-v3.0 &
```

The path `/path/to/msmarco-passage-cohere-embed-english-v3/` should point to the corpus downloaded above.
The path `/path/to/msmarco-passage-cohere-embed-english-v3.0/` should point to the corpus downloaded above.
Upon completion, we should have an index with 8,841,823 documents.

Note that here we are explicitly using Lucene's `NoMergePolicy` merge policy, which suppresses any merging of index segments.
Expand All @@ -72,39 +72,39 @@ After indexing has completed, you should be able to perform retrieval as follows

```bash
target/appassembler/bin/SearchHnswDenseVectors \
-index indexes/lucene-hnsw.msmarco-passage-cohere-embed-english-v3-int8/ \
-topics tools/topics-and-qrels/topics.dl19-passage.cohere-embed-english-v3.jsonl.gz \
-index indexes/lucene-hnsw.msmarco-passage-cohere-embed-english-v3.0-int8/ \
-topics tools/topics-and-qrels/topics.dl19-passage.cohere-embed-english-v3.0.jsonl.gz \
-topicReader JsonIntVector \
-output runs/run.msmarco-passage-cohere-embed-english-v3.cohere-embed-english-v3.topics.dl19-passage.cohere-embed-english-v3.jsonl.txt \
-output runs/run.msmarco-passage-cohere-embed-english-v3.0.cohere-embed-english-v3.0.topics.dl19-passage.cohere-embed-english-v3.0.jsonl.txt \
-generator VectorQueryGenerator -topicField vector -threads 16 -hits 1000 -efSearch 1000 &
```

Evaluation can be performed using `trec_eval`:

```bash
target/appassembler/bin/trec_eval -m map -c -l 2 tools/topics-and-qrels/qrels.dl19-passage.txt runs/run.msmarco-passage-cohere-embed-english-v3.cohere-embed-english-v3.topics.dl19-passage.cohere-embed-english-v3.jsonl.txt
target/appassembler/bin/trec_eval -m ndcg_cut.10 -c tools/topics-and-qrels/qrels.dl19-passage.txt runs/run.msmarco-passage-cohere-embed-english-v3.cohere-embed-english-v3.topics.dl19-passage.cohere-embed-english-v3.jsonl.txt
target/appassembler/bin/trec_eval -m recall.100 -c -l 2 tools/topics-and-qrels/qrels.dl19-passage.txt runs/run.msmarco-passage-cohere-embed-english-v3.cohere-embed-english-v3.topics.dl19-passage.cohere-embed-english-v3.jsonl.txt
target/appassembler/bin/trec_eval -m recall.1000 -c -l 2 tools/topics-and-qrels/qrels.dl19-passage.txt runs/run.msmarco-passage-cohere-embed-english-v3.cohere-embed-english-v3.topics.dl19-passage.cohere-embed-english-v3.jsonl.txt
target/appassembler/bin/trec_eval -m map -c -l 2 tools/topics-and-qrels/qrels.dl19-passage.txt runs/run.msmarco-passage-cohere-embed-english-v3.0.cohere-embed-english-v3.0.topics.dl19-passage.cohere-embed-english-v3.0.jsonl.txt
target/appassembler/bin/trec_eval -m ndcg_cut.10 -c tools/topics-and-qrels/qrels.dl19-passage.txt runs/run.msmarco-passage-cohere-embed-english-v3.0.cohere-embed-english-v3.0.topics.dl19-passage.cohere-embed-english-v3.0.jsonl.txt
target/appassembler/bin/trec_eval -m recall.100 -c -l 2 tools/topics-and-qrels/qrels.dl19-passage.txt runs/run.msmarco-passage-cohere-embed-english-v3.0.cohere-embed-english-v3.0.topics.dl19-passage.cohere-embed-english-v3.0.jsonl.txt
target/appassembler/bin/trec_eval -m recall.1000 -c -l 2 tools/topics-and-qrels/qrels.dl19-passage.txt runs/run.msmarco-passage-cohere-embed-english-v3.0.cohere-embed-english-v3.0.topics.dl19-passage.cohere-embed-english-v3.0.jsonl.txt
```

## Effectiveness

With the above commands, you should be able to reproduce the following results:

| **AP@1000** | **cohere-embed-english-v3**|
| **AP@1000** | **cohere-embed-english-v3.0**|
|:-------------------------------------------------------------------------------------------------------------|-----------|
| [DL19 (Passage)](https://trec.nist.gov/data/deep2020.html) | 0.487 |
| **nDCG@10** | **cohere-embed-english-v3**|
| **nDCG@10** | **cohere-embed-english-v3.0**|
| [DL19 (Passage)](https://trec.nist.gov/data/deep2020.html) | 0.690 |
| **R@100** | **cohere-embed-english-v3**|
| **R@100** | **cohere-embed-english-v3.0**|
| [DL19 (Passage)](https://trec.nist.gov/data/deep2020.html) | 0.647 |
| **R@1000** | **cohere-embed-english-v3**|
| **R@1000** | **cohere-embed-english-v3.0**|
| [DL19 (Passage)](https://trec.nist.gov/data/deep2020.html) | 0.850 |

Note that due to the non-deterministic nature of HNSW indexing, results may differ slightly between each experimental run.
Nevertheless, scores are generally within 0.005 of the reference values recorded in [our YAML configuration file](../../src/main/resources/regression/dl19-passage-cohere-embed-english-v3-hnsw-int8.yaml).
Nevertheless, scores are generally within 0.005 of the reference values recorded in [our YAML configuration file](../../src/main/resources/regression/dl19-passage-cohere-embed-english-v3.0-hnsw-int8.yaml).

## Reproduction Log[*](../../docs/reproducibility.md)

To add to this reproduction log, modify [this template](../../src/main/resources/docgen/templates/dl19-passage-cohere-embed-english-v3-hnsw-int8.template) and run `bin/build.sh` to rebuild the documentation.
To add to this reproduction log, modify [this template](../../src/main/resources/docgen/templates/dl19-passage-cohere-embed-english-v3.0-hnsw-int8.template) and run `bin/build.sh` to rebuild the documentation.
Loading

0 comments on commit d674915

Please sign in to comment.