Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Overhaul of regressions for MS MARCO {passage, doc} and DL {19, 20} #1559

Merged
merged 36 commits into from
Jun 14, 2021
Merged

Overhaul of regressions for MS MARCO {passage, doc} and DL {19, 20} #1559

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
36 commits
Select commit Hold shift + click to select a range
c6ce6e3
Missing regressions on TREC 2019.
lintool Jun 2, 2021
b3d46a1
More tweaks.
lintool Jun 4, 2021
5704803
Merge branch 'master' into regression
lintool Jun 4, 2021
b8cc706
Merge branch 'master' into regression
lintool Jun 4, 2021
7ca8741
updates
lintool Jun 4, 2021
f1cdf09
updateS
lintool Jun 5, 2021
05bfce0
updates
lintool Jun 5, 2021
76d1850
updates.
lintool Jun 5, 2021
4a35f21
more tweaks.
lintool Jun 5, 2021
ad3e08d
updates.
lintool Jun 5, 2021
4bd32b6
Minor tweaks.
lintool Jun 5, 2021
ef7b29e
minor tweaks.
lintool Jun 5, 2021
4340432
minor tweaks.
lintool Jun 5, 2021
623db8e
minor tweaks.
lintool Jun 5, 2021
8a89de9
minor tweaks.
lintool Jun 5, 2021
971fe4c
minor tweaks.
lintool Jun 5, 2021
7d48ae9
updates
lintool Jun 5, 2021
8fd890c
updates
lintool Jun 5, 2021
6299900
Updates.
lintool Jun 8, 2021
c949c0d
updates.
lintool Jun 8, 2021
7c8e9f3
update.
lintool Jun 8, 2021
86eab75
updates.
lintool Jun 8, 2021
525826b
update
lintool Jun 8, 2021
a8daeb0
update
lintool Jun 8, 2021
c7cea30
update
lintool Jun 8, 2021
dededfe
updates
lintool Jun 9, 2021
2280f7d
update
lintool Jun 9, 2021
2d31cfa
update
lintool Jun 9, 2021
085f38e
update.
lintool Jun 9, 2021
f643ab1
updates
lintool Jun 9, 2021
a88571e
updates
lintool Jun 9, 2021
7c3cb66
updates
lintool Jun 9, 2021
a7beb16
updates.
lintool Jun 9, 2021
77b48f6
Merge branch 'master' into regression
lintool Jun 9, 2021
23597b2
Merge branch 'master' into regression
lintool Jun 14, 2021
cff9f7e
updates
lintool Jun 14, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
updates
  • Loading branch information
@lintool
lintool committed Jun 14, 2021
commit cff9f7e1610ecf974925ce59aaa96910178ba0e4
2 changes: 1 addition & 1 deletion docs/regressions-dl19-doc-docTTTTTquery-per-passage.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Note that there are four different regression conditions for this task, and this
+ **Indexing Condition:** each MS MARCO document is first segmented into passages, each passage is treated as a unit of indexing
+ **Expansion Condition:** doc2query-T5

In the passage indexing condition, we select the highest-scoring passage from a document as the score for that document to produce a document ranking; this is known as the MaxP technique.
In the passage indexing condition, we select the score of the highest-scoring passage from a document as the score for that document to produce a document ranking; this is known as the MaxP technique.
All four conditions are described in detail [here](https://github.com/castorini/docTTTTTquery#reproducing-ms-marco-document-ranking-results-with-anserini), in the context of doc2query-T5.

The exact configurations for these regressions are stored in [this YAML file](../src/main/resources/regression/dl19-doc-docTTTTTquery-per-passage.yaml).
Expand Down
2 changes: 1 addition & 1 deletion docs/regressions-dl19-doc-per-passage.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Note that there are four different regression conditions for this task, and this
+ **Indexing Condition:** each MS MARCO document is first segmented into passages, each passage is treated as a unit of indexing
+ **Expansion Condition:** none

In the passage indexing condition, we select the highest-scoring passage from a document as the score for that document to produce a document ranking; this is known as the MaxP technique.
In the passage indexing condition, we select the score of the highest-scoring passage from a document as the score for that document to produce a document ranking; this is known as the MaxP technique.
All four conditions are described in detail [here](https://github.com/castorini/docTTTTTquery#reproducing-ms-marco-document-ranking-results-with-anserini), in the context of doc2query-T5.

The exact configurations for these regressions are stored in [this YAML file](../src/main/resources/regression/dl19-doc-per-passage.yaml).
Expand Down
2 changes: 1 addition & 1 deletion docs/regressions-dl20-doc-docTTTTTquery-per-passage.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Note that there are four different regression conditions for this task, and this
+ **Indexing Condition:** each MS MARCO document is first segmented into passages, each passage is treated as a unit of indexing
+ **Expansion Condition:** doc2query-T5

In the passage indexing condition, we select the highest-scoring passage from a document as the score for that document to produce a document ranking; this is known as the MaxP technique.
In the passage indexing condition, we select the score of the highest-scoring passage from a document as the score for that document to produce a document ranking; this is known as the MaxP technique.
All four conditions are described in detail [here](https://github.com/castorini/docTTTTTquery#reproducing-ms-marco-document-ranking-results-with-anserini), in the context of doc2query-T5.

The exact configurations for these regressions are stored in [this YAML file](../src/main/resources/regression/dl19-doc-docTTTTTquery-per-passage.yaml).
Expand Down
2 changes: 1 addition & 1 deletion docs/regressions-dl20-doc-per-passage.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Note that there are four different regression conditions for this task, and this
+ **Indexing Condition:** each MS MARCO document is first segmented into passages, each passage is treated as a unit of indexing
+ **Expansion Condition:** none

In the passage indexing condition, we select the highest-scoring passage from a document as the score for that document to produce a document ranking; this is known as the MaxP technique.
In the passage indexing condition, we select the score of the highest-scoring passage from a document as the score for that document to produce a document ranking; this is known as the MaxP technique.
All four conditions are described in detail [here](https://github.com/castorini/docTTTTTquery#reproducing-ms-marco-document-ranking-results-with-anserini), in the context of doc2query-T5.

The exact configurations for these regressions are stored in [this YAML file](../src/main/resources/regression/dl20-doc-per-passage.yaml).
Expand Down
2 changes: 1 addition & 1 deletion docs/regressions-msmarco-doc-docTTTTTquery-per-passage.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ Note that there are four different regression conditions for this task, and this
+ **Indexing Condition:** each MS MARCO document is first segmented into passages, each passage is treated as a unit of indexing
+ **Expansion Condition:** doc2query-T5

In the passage indexing condition, we select the highest-scoring passage from a document as the score for that document to produce a document ranking; this is known as the MaxP technique.
In the passage indexing condition, we select the score of the highest-scoring passage from a document as the score for that document to produce a document ranking; this is known as the MaxP technique.
All four conditions are described in detail [here](https://github.com/castorini/docTTTTTquery#reproducing-ms-marco-document-ranking-results-with-anserini), in the context of doc2query-T5.

The exact configurations for these regressions are stored in [this YAML file](../src/main/resources/regression/msmarco-doc-docTTTTTquery-per-passage.yaml).
Expand Down
2 changes: 1 addition & 1 deletion docs/regressions-msmarco-doc-per-passage.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ Note that there are four different regression conditions for this task, and this
+ **Indexing Condition:** each MS MARCO document is first segmented into passages, each passage is treated as a unit of indexing
+ **Expansion Condition:** none

In the passage indexing condition, we select the highest-scoring passage from a document as the score for that document to produce a document ranking; this is known as the MaxP technique.
In the passage indexing condition, we select the score of the highest-scoring passage from a document as the score for that document to produce a document ranking; this is known as the MaxP technique.
All four conditions are described in detail [here](https://github.com/castorini/docTTTTTquery#reproducing-ms-marco-document-ranking-results-with-anserini), in the context of doc2query-T5.

The exact configurations for these regressions are stored in [this YAML file](../src/main/resources/regression/msmarco-doc-per-passage.yaml).
Expand Down
4 changes: 2 additions & 2 deletions docs/regressions.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -62,7 +62,7 @@ nohup python src/main/python/run_regression.py --collection msmarco-doc-docTTTTT
nohup python src/main/python/run_regression.py --collection msmarco-doc-docTTTTTquery-per-passage >& logs/log.msmarco-doc-docTTTTTquery-per-passage &

nohup python src/main/python/run_regression.py --collection dl19-passage >& logs/log.dl19-passage &
nohup python src/main/python/run_regression.py --collection dl19-passage-docTTTTTquery >& logs/dl19-passage-docTTTTTquery &
nohup python src/main/python/run_regression.py --collection dl19-passage-docTTTTTquery >& logs/log.dl19-passage-docTTTTTquery &
nohup python src/main/python/run_regression.py --collection dl19-doc >& logs/log.dl19-doc &
nohup python src/main/python/run_regression.py --collection dl19-doc-per-passage >& logs/log.dl19-doc-per-passage &
nohup python src/main/python/run_regression.py --collection dl19-doc-docTTTTTquery-per-doc >& logs/log.dl19-doc-docTTTTTquery-per-doc &
Expand Down Expand Up @@ -121,7 +121,7 @@ nohup python src/main/python/run_regression.py --index --collection msmarco-doc-
nohup python src/main/python/run_regression.py --index --collection msmarco-doc-docTTTTTquery-per-passage >& logs/log.msmarco-doc-docTTTTTquery-per-passage &

nohup python src/main/python/run_regression.py --index --collection dl19-passage >& logs/log.dl19-passage &
nohup python src/main/python/run_regression.py --index --collection dl19-passage-docTTTTTquery >& logs/dl19-passage-docTTTTTquery &
nohup python src/main/python/run_regression.py --index --collection dl19-passage-docTTTTTquery >& logs/log.dl19-passage-docTTTTTquery &
nohup python src/main/python/run_regression.py --index --collection dl19-doc >& logs/log.dl19-doc &
nohup python src/main/python/run_regression.py --index --collection dl19-doc-per-passage >& logs/log.dl19-doc-per-passage &
nohup python src/main/python/run_regression.py --index --collection dl19-doc-docTTTTTquery-per-doc >& logs/log.dl19-doc-docTTTTTquery-per-doc &
Expand Down
2 changes: 1 addition & 1 deletion src/main/resources/docgen/templates/dl19-doc-docTTTTTquery-per-passage.template
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Note that there are four different regression conditions for this task, and this
+ **Indexing Condition:** each MS MARCO document is first segmented into passages, each passage is treated as a unit of indexing
+ **Expansion Condition:** doc2query-T5

In the passage indexing condition, we select the highest-scoring passage from a document as the score for that document to produce a document ranking; this is known as the MaxP technique.
In the passage indexing condition, we select the score of the highest-scoring passage from a document as the score for that document to produce a document ranking; this is known as the MaxP technique.
All four conditions are described in detail [here](https://github.com/castorini/docTTTTTquery#reproducing-ms-marco-document-ranking-results-with-anserini), in the context of doc2query-T5.

The exact configurations for these regressions are stored in [this YAML file](../src/main/resources/regression/dl19-doc-docTTTTTquery-per-passage.yaml).
Expand Down
2 changes: 1 addition & 1 deletion src/main/resources/docgen/templates/dl19-doc-per-passage.template
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Note that there are four different regression conditions for this task, and this
+ **Indexing Condition:** each MS MARCO document is first segmented into passages, each passage is treated as a unit of indexing
+ **Expansion Condition:** none

In the passage indexing condition, we select the highest-scoring passage from a document as the score for that document to produce a document ranking; this is known as the MaxP technique.
In the passage indexing condition, we select the score of the highest-scoring passage from a document as the score for that document to produce a document ranking; this is known as the MaxP technique.
All four conditions are described in detail [here](https://github.com/castorini/docTTTTTquery#reproducing-ms-marco-document-ranking-results-with-anserini), in the context of doc2query-T5.

The exact configurations for these regressions are stored in [this YAML file](../src/main/resources/regression/dl19-doc-per-passage.yaml).
Expand Down
2 changes: 1 addition & 1 deletion src/main/resources/docgen/templates/dl20-doc-docTTTTTquery-per-passage.template
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Note that there are four different regression conditions for this task, and this
+ **Indexing Condition:** each MS MARCO document is first segmented into passages, each passage is treated as a unit of indexing
+ **Expansion Condition:** doc2query-T5

In the passage indexing condition, we select the highest-scoring passage from a document as the score for that document to produce a document ranking; this is known as the MaxP technique.
In the passage indexing condition, we select the score of the highest-scoring passage from a document as the score for that document to produce a document ranking; this is known as the MaxP technique.
All four conditions are described in detail [here](https://github.com/castorini/docTTTTTquery#reproducing-ms-marco-document-ranking-results-with-anserini), in the context of doc2query-T5.

The exact configurations for these regressions are stored in [this YAML file](../src/main/resources/regression/dl19-doc-docTTTTTquery-per-passage.yaml).
Expand Down
2 changes: 1 addition & 1 deletion src/main/resources/docgen/templates/dl20-doc-per-passage.template
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@ Note that there are four different regression conditions for this task, and this
+ **Indexing Condition:** each MS MARCO document is first segmented into passages, each passage is treated as a unit of indexing
+ **Expansion Condition:** none

In the passage indexing condition, we select the highest-scoring passage from a document as the score for that document to produce a document ranking; this is known as the MaxP technique.
In the passage indexing condition, we select the score of the highest-scoring passage from a document as the score for that document to produce a document ranking; this is known as the MaxP technique.
All four conditions are described in detail [here](https://github.com/castorini/docTTTTTquery#reproducing-ms-marco-document-ranking-results-with-anserini), in the context of doc2query-T5.

The exact configurations for these regressions are stored in [this YAML file](../src/main/resources/regression/dl20-doc-per-passage.yaml).
Expand Down
2 changes: 1 addition & 1 deletion src/main/resources/docgen/templates/msmarco-doc-docTTTTTquery-per-passage.template
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ Note that there are four different regression conditions for this task, and this
+ **Indexing Condition:** each MS MARCO document is first segmented into passages, each passage is treated as a unit of indexing
+ **Expansion Condition:** doc2query-T5

In the passage indexing condition, we select the highest-scoring passage from a document as the score for that document to produce a document ranking; this is known as the MaxP technique.
In the passage indexing condition, we select the score of the highest-scoring passage from a document as the score for that document to produce a document ranking; this is known as the MaxP technique.
All four conditions are described in detail [here](https://github.com/castorini/docTTTTTquery#reproducing-ms-marco-document-ranking-results-with-anserini), in the context of doc2query-T5.

The exact configurations for these regressions are stored in [this YAML file](../src/main/resources/regression/msmarco-doc-docTTTTTquery-per-passage.yaml).
Expand Down
2 changes: 1 addition & 1 deletion src/main/resources/docgen/templates/msmarco-doc-per-passage.template
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,7 +6,7 @@ Note that there are four different regression conditions for this task, and this
+ **Indexing Condition:** each MS MARCO document is first segmented into passages, each passage is treated as a unit of indexing
+ **Expansion Condition:** none

In the passage indexing condition, we select the highest-scoring passage from a document as the score for that document to produce a document ranking; this is known as the MaxP technique.
In the passage indexing condition, we select the score of the highest-scoring passage from a document as the score for that document to produce a document ranking; this is known as the MaxP technique.
All four conditions are described in detail [here](https://github.com/castorini/docTTTTTquery#reproducing-ms-marco-document-ranking-results-with-anserini), in the context of doc2query-T5.

The exact configurations for these regressions are stored in [this YAML file](../src/main/resources/regression/msmarco-doc-per-passage.yaml).
Expand Down