Update eval, switch from ssh to https, modify instructions (castorini#47

) * remove evaluate * update anserini-eval * change to https * update docs * update docs and readme
leungjch · Jun 1, 2020 · 13e099b · 13e099b
1 parent f204e4b
commit 13e099b
Show file tree

Hide file tree

Showing 5 changed files with 19 additions and 191 deletions.
diff --git a/.gitmodules b/.gitmodules
@@ -1,3 +1,3 @@
 [submodule "eval"]
 	path = eval
-	url = git@github.com:castorini/anserini-eval.git
+	url = https://github.com/castorini/anserini-eval.git
diff --git a/README.md b/README.md
@@ -25,7 +25,7 @@ Currently, this repo contains implementations of the rerankers for [CovidQA](htt
 
 ## Additional Instructions
 
-0. Clone the repo with `git clone git@github.com:castorini/pygaggle.git`
+0. Clone the repo with `git clone --recursive https://github.com/castorini/pygaggle.git`
 
 0. Make you sure you have an installation of [Python 3.6+](https://www.python.org/downloads/). All `python` commands below refer to this.
 

diff --git a/docs/experiments-msmarco-passage.md b/docs/experiments-msmarco-passage.md
@@ -11,8 +11,19 @@ Since it can take many hours to run these models on all of the 6980 queries from
 Running these instructions with the entire MS MARCO dev set should give about the same results as that in the corresponding paper. 
 
 Note 1: Run the following instructions at root of this repo.
-Note 2: Installation must have been done from source.
-Note 3: Make sure that you have access to a GPU
+Note 2: Make sure that you have access to a GPU
+Note 3: Installation must have been done from source and make sure the [anserini-eval](https://github.com/castorini/anserini-eval) submodule is pulled. 
+To do this, first clone the repository recursively.
+
+```
+git clone --recursive https://github.com/castorini/pygaggle.git
+```
+
+Then install PyGaggle using:
+
+```
+pip install pygaggle/
+```
 
 ## Models
 
@@ -38,7 +49,7 @@ unzip data/msmarco_ans_small.zip -d data
 As a sanity check, we can evaluate the first-stage retrieved documents using the official MS MARCO evaluation script.
 
 ```
-python evaluate/msmarco/msmarco_eval.py data/msmarco_ans_small/qrels.dev.small.tsv data/msmarco_ans_small/run.dev.small.tsv
+python eval/msmarco_eval.py data/msmarco_ans_small/qrels.dev.small.tsv data/msmarco_ans_small/run.dev.small.tsv
 ```
 
 The output should be:
@@ -94,7 +105,7 @@ The re-ranked run file `run.monobert.ans_small.dev.tsv` will also be available i
 We can use the official MS MARCO evaluation script to verify the MRR@10:
 
 ```
-python evaluate/msmarco/msmarco_eval.py data/msmarco_ans_small/qrels.dev.small.tsv runs/run.monobert.ans_small.dev.tsv
+python eval/msmarco_eval.py data/msmarco_ans_small/qrels.dev.small.tsv runs/run.monobert.ans_small.dev.tsv
 ```
 
 You should see the same result. Great, let's move on to monoT5!
@@ -134,7 +145,7 @@ Upon completion, the re-ranked run file `run.monot5.ans_small.dev.tsv` will be a
 We can use the official MS MARCO evaluation script to verify the MRR@10:
 
 ```
-python evaluate/msmarco/msmarco_eval.py data/msmarco_ans_small/qrels.dev.small.tsv runs/run.monot5.ans_small.dev.tsv
+python eval/msmarco_eval.py data/msmarco_ans_small/qrels.dev.small.tsv runs/run.monot5.ans_small.dev.tsv
 ```
 
 You should see the same result.

diff --git a/eval b/eval
diff --git a/evaluate/msmarco/msmarco_eval.py b/evaluate/msmarco/msmarco_eval.py