Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add colbert version of IBM reranker #918

Merged
merged 20 commits into from
Jan 3, 2022
Merged

Conversation

yuki617
Copy link
Member

@yuki617 yuki617 commented Dec 30, 2021

We add a max-sim(colbert) operator option for ibm reranker.

@yuki617
Copy link
Member Author

yuki617 commented Dec 30, 2021

@stephaniewhoo The result I got on my laptop for alpha=0.3, max-sim operator =True running is here:
map:0.2236 ndcg:0.2908 mrr@10:0.2155.
Could you help on its replication? Thanks a lot!

@stephaniewhoo
Copy link
Member

@stephaniewhoo The result I got on my laptop for alpha=0.3, max-sim operator =True running is here:
map:0.2236 ndcg:0.2908 mrr@10:0.2155.
Could you help on its replication? Thanks a lot!

can you send me your new ibm model so that we can keep setup same

@yuki617
Copy link
Member Author

yuki617 commented Dec 31, 2021

@stephaniewhoo The result I got on my laptop for alpha=0.3, max-sim operator =True running is here:
map:0.2236 ndcg:0.2908 mrr@10:0.2155.
Could you help on its replication? Thanks a lot!

can you send me your new ibm model so that we can keep setup same

Please just use the last ibm model I sent you, the text_bert_tok_raw one is enough.

@stephaniewhoo
Copy link
Member

@stephaniewhoo The result I got on my laptop for alpha=0.3, max-sim operator =True running is here:
map:0.2236 ndcg:0.2908 mrr@10:0.2155.
Could you help on its replication? Thanks a lot!

can you send me your new ibm model so that we can keep setup same

Please just use the last ibm model I sent you, the text_bert_tok_raw one is enough.

can you try this setup except self_tran is 0.1? As I have this result under self_tran is 0.1

{"map": "0.2240", "ndcg": "0.2923"}```

@yuki617
Copy link
Member Author

yuki617 commented Dec 31, 2021

@stephaniewhoo The result I got on my laptop for alpha=0.3, max-sim operator =True running is here:
map:0.2236 ndcg:0.2908 mrr@10:0.2155.
Could you help on its replication? Thanks a lot!

can you send me your new ibm model so that we can keep setup same

Please just use the last ibm model I sent you, the text_bert_tok_raw one is enough.

can you try this setup except self_tran is 0.1? As I have this result under self_tran is 0.1

{"map": "0.2240", "ndcg": "0.2923"}```

Yes, I got the same score as yours. I kept the selfTrans score to be 0.35 just for simplicity as our default ibm one is 0.35.

@yuki617 yuki617 merged commit 937ec63 into castorini:master Jan 3, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants