Description
Hello! My friend.
I am trying to reproduce the HBMP model(trained on SNLI) on SentEval tool.But i found my result(83.15%) is always 1.0-1.5% lower than the state of the art 84.6% on CR from Conneau's paper. And the result on MR is the same. other settings on model, code are same with your paper.
Here are some details:
1.results(just compared to Infersent)
Infersent results from anthors'paper: CR(84.6%), MR(79.9%) (on SNLI)
HBMP(on SNLI) reproduced results :
2021-06-09 10:17:48,408 : ***** Transfer task : CR *****
2021-06-09 10:18:03,835 : Generated sentence embeddings
2021-06-09 10:18:03,836 : Training pytorch-MLP-nhid0-adam,lr=0.01-bs64 with (inner) 10-fold cross-validation
2021-06-09 10:21:59,686 : Best param found at split 1: l2reg = 0.001 with score 84.84
2021-06-09 10:24:56,416 : Best param found at split 2: l2reg = 0.01 with score 84.37
2021-06-09 10:27:09,470 : Best param found at split 3: l2reg = 1e-05 with score 84.16
2021-06-09 10:29:28,407 : Best param found at split 4: l2reg = 0.01 with score 84.75
2021-06-09 10:31:51,597 : Best param found at split 5: l2reg = 0.01 with score 84.69
2021-06-09 10:34:45,882 : Best param found at split 6: l2reg = 0.001 with score 84.49
2021-06-09 10:38:23,245 : Best param found at split 7: l2reg = 1e-05 with score 84.58
2021-06-09 10:42:02,684 : Best param found at split 8: l2reg = 0.01 with score 84.17
2021-06-09 10:46:16,299 : Best param found at split 9: l2reg = 0.01 with score 85.02
2021-06-09 10:50:04,303 : Best param found at split 10: l2reg = 1e-05 with score 84.43
2021-06-09 10:50:11,270 : Dev acc : 84.55 Test acc : 83.15
2021-06-09 10:50:11,274 : ***** Transfer task : MR *****
2021-06-09 10:50:40,868 : Generated sentence embeddings
2021-06-09 10:50:40,870 : Training pytorch-MLP-nhid0-adam,lr=0.01-bs64 with (inner) 10-fold cross-validation
2021-06-09 11:00:27,195 : Best param found at split 1: l2reg = 0.001 with score 79.22
2021-06-09 11:07:01,380 : Best param found at split 2: l2reg = 0.001 with score 78.97
2021-06-09 11:13:29,129 : Best param found at split 3: l2reg = 0.01 with score 78.56
2021-06-09 11:20:28,882 : Best param found at split 4: l2reg = 0.01 with score 78.66
2021-06-09 11:27:06,542 : Best param found at split 5: l2reg = 0.01 with score 78.62
2021-06-09 11:33:35,779 : Best param found at split 6: l2reg = 0.001 with score 78.68
2021-06-09 11:39:22,995 : Best param found at split 7: l2reg = 0.01 with score 78.54
2021-06-09 11:45:15,899 : Best param found at split 8: l2reg = 0.001 with score 78.65
2021-06-09 11:52:09,794 : Best param found at split 9: l2reg = 0.001 with score 78.73
2021-06-09 11:59:05,532 : Best param found at split 10: l2reg = 0.001 with score 79.23
2021-06-09 11:59:19,889 : Dev acc : 78.79 Test acc : 77.85
2.My senteval settings:
params_senteval = {'task_path': PATH_TRANSFER_TASKS, 'usepytorch': True, 'kfold': 10}
params_senteval['classifier'] = {'nhid': 0, 'optim': 'adam,lr=0.01', 'batch_size': 64,
'tenacity': 5, 'epoch_size': 4}
How could i do for reproducing the results from paper?
change the senteval settings?