Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Refactor regressions #1671

Merged
merged 15 commits into from
Oct 13, 2023
Merged

Refactor regressions #1671

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
f7f2d3c
Refactor regressions
lintool Oct 6, 2023
f3e391b
more refactoring
lintool Oct 6, 2023
d5d781d
odqa
lintool Oct 6, 2023
45df5e1
more refactoring
lintool Oct 6, 2023
dbcd54f
more refactoring
lintool Oct 6, 2023
a613e4d
Merge branch 'master' into regression-refactoring
lintool Oct 7, 2023
c8054cc
More tweaks.
lintool Oct 7, 2023
b3a2f08
minor tweaks.
lintool Oct 8, 2023
d6036d3
Fixed typos.
lintool Oct 8, 2023
535ebf7
Merge branch 'master' into regression-refactoring
lintool Oct 8, 2023
46b2b6d
Tweak to script.
lintool Oct 10, 2023
8622ae3
Compressed spaces.
lintool Oct 10, 2023
12dc352
Merge branch 'master' into regression-refactoring
lintool Oct 12, 2023
d5885ab
Merge branch 'regression-refactoring' of github.com:castorini/pyserin…
lintool Oct 12, 2023
8b75b4d
Final tweaks, hopefully.
lintool Oct 12, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
More tweaks.
  • Loading branch information
@lintool
lintool committed Oct 7, 2023
commit c8054cc0ab542ee22001e704ddb54166e70d2b9d
2 changes: 1 addition & 1 deletion docs/2cr/msmarco-v1-passage.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2447,7 +2447,7 @@ <h1 class="mb-3">MS MARCO V1 Passage</h1>
<td>0.7280</td>
<td>0.9069</td>
<td></td>
<td>0.3300</td>
<td>0.3301</td>
<td>0.9811</td>
</tr>
<tr class="hide-table-padding">
Expand Down
4 changes: 2 additions & 2 deletions docs/2cr/msmarco-v2-passage.html
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1249,8 +1249,8 @@ <h1 class="mb-3">MS MARCO V2 Passage</h1>
<td>0.2193</td>
<td>0.5756</td>
<td>0.6991</td>
<td>0.4246</td>
<td>0.6897</td>
<td>0.4247</td>
<td>0.6893</td>
<td></td>
<td>0.1342</td>
<td>0.7010</td>
Expand Down
25 changes: 10 additions & 15 deletions pyserini/2cr/miracl.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -389,19 +389,15 @@ def run_conditions(args):
trec_eval_metric_definitions[metric], runfile))
if math.isclose(score, float(expected[metric])):
result_str = ok_str
# Flaky tests
elif (name == 'bm25-mdpr-tied-pft-msmarco-hybrid.zh'
# Flaky on Jimmy's Mac Studio (Apple M1 Ultra), nDCG@10: 0.5255 -> expected 0.5254
and split == 'dev' and metric == 'nDCG@10'
and math.isclose(score, float(expected[metric]), abs_tol=2e-4)) or \
(name == 'bm25-mdpr-tied-pft-msmarco-hybrid.te'
# Flaky on Jimmy's Mac Studio (Apple M1 Ultra), nDCG@10: 0.6000 -> expected 0.5999
and split == 'train' and metric == 'nDCG@10'
and math.isclose(score, float(expected[metric]), abs_tol=2e-4)) or \
(name == 'mcontriever-tied-pft-msmarco.id'
# Flaky on Jimmy's Mac Studio (Apple M1 Ultra), nDCG@10: 0.3749 -> expected 0.3748
and split == 'train' and metric == 'nDCG@10'
and math.isclose(score, float(expected[metric]), abs_tol=1e-4)):
# Flaky on Jimmy's Mac Studio (Apple M1 Ultra), nDCG@10: 0.5255 -> expected 0.5254
elif name == 'bm25-mdpr-tied-pft-msmarco-hybrid.zh' \
and split == 'dev' and metric == 'nDCG@10' \
and math.isclose(score, float(expected[metric]), abs_tol=2e-4):
result_str = okish_str
# Flaky on Jimmy's Mac Studio (Apple M1 Ultra), nDCG@10: 0.3749 -> expected 0.3748
elif name == 'mcontriever-tied-pft-msmarco.id' \
and split == 'train' and metric == 'nDCG@10' \
and math.isclose(score, float(expected[metric]), abs_tol=1e-4):
result_str = okish_str
else:
result_str = fail_str + f' expected {expected[metric]:.4f}'
Expand All @@ -428,8 +424,7 @@ def run_conditions(args):

if __name__ == '__main__':
parser = argparse.ArgumentParser(description='Generate regression matrix for MIRACL.')
parser.add_argument('--condition', type=str,
help='Condition to run', required=False)
parser.add_argument('--condition', type=str, help='Condition to run', required=False)
# To list all conditions
parser.add_argument('--list-conditions', action='store_true', default=False, help='List available conditions.')
# For generating reports
Expand Down