02/25/2023, 00:24:00: Namespace(seed=2022, data_dir='data/', logging_dir='log/pretrain_t5_small_beauty_no_tokenization.log', model_dir='model/pretrain_t5_small_beauty_no_tokenization.pt', task='beauty', max_history=20, sequential_num=10, negative_sample=2, yes_no_sample=5, direct_item_proportion=2, train_sequential_item_batch=80, train_sequential_yesno_batch=48, train_direct_yesno_batch=48, train_direct_candidate_batch=12, train_direct_straightforward_batch=48, meta_title_batch=128, meta_description_batch=12, meta_category_batch=128, meta_epochs=10, meta_lr=0.0001, review_user_batch=16, review_item_batch=16, review_epochs=2, review_lr=0.0001, model_type='t5-small', epochs=20, lr=0.001, clip=1.0, logging_step=1000, warmup_prop=0.05, gradient_accumulation_steps=1, weight_decay=0.01, adam_eps=1e-06, dropout=0.1, alpha=2, multiGPU=True, distributed=True, gpu='4,5,6,7', local_rank=0, evaluation_method='sequential_item', evaluation_template_id=0, item_representation='no_tokenization', resolution=2, overlap=0, base=10, random_initialization_embedding=False, no_pretrain=False, use_meta_data=False, use_review_data=False, remove_last_item=False, remove_first_item=False, eval_only=False)

02/25/2023, 00:24:02: Process Launching at GPU 3

02/25/2023, 00:24:03: Process Launching at GPU 0

02/25/2023, 00:24:03: Process Launching at GPU 2

02/25/2023, 00:24:03: Process Launching at GPU 1

02/25/2023, 00:24:03: Building train loader at GPU 2

02/25/2023, 00:24:03: Building train loader at GPU 0

02/25/2023, 00:24:03: Building train loader at GPU 3

02/25/2023, 00:24:03: loading data ...

02/25/2023, 00:24:03: Building train loader at GPU 1

02/25/2023, 00:24:04: finished loading data

02/25/2023, 00:24:04: length of training data is 4107

02/25/2023, 00:24:04: loading model ...

02/25/2023, 00:24:04: using only sequential data, and all possible sequences are here

02/25/2023, 00:24:04: use pretrained T5

02/25/2023, 00:24:13: Batch per epoch: 4107

02/25/2023, 00:24:13: Total steps: 82140

02/25/2023, 00:24:13: Warmup proportion:

02/25/2023, 00:24:13: Warm up steps: 4107

02/25/2023, 00:24:13: model dataparallel set

02/25/2023, 00:24:13: start training

02/25/2023, 00:24:13: example train datapoint

02/25/2023, 00:24:14: The user_13438 has bought items : item_<extra_id_11745>, item_<extra_id_2649>, item_<extra_id_1513>, item_<extra_id_5687>, item_<extra_id_6781>, What else do you think is necessary for the user?

02/25/2023, 00:24:14: item_<extra_id_12>

02/25/2023, 00:24:14: example val datapoint

02/25/2023, 00:24:14: User_8756 has purchased item_<extra_id_8164>, item_<extra_id_4186>, item_<extra_id_6057>, item_<extra_id_9283>, item_<extra_id_8449>, item_<extra_id_6394>, item_<extra_id_8907>, item_<extra_id_3187>, item_<extra_id_6593>, item_<extra_id_9540>, item_<extra_id_9531>, item_<extra_id_7834>, item_<extra_id_7587>, item_<extra_id_3151>, predict next possible item to be bought by the user?

02/25/2023, 00:24:14: item_<extra_id_6752>

02/25/2023, 00:24:14: example test datapoint

02/25/2023, 00:24:14: User_8756 has purchased item_<extra_id_8164>, item_<extra_id_4186>, item_<extra_id_6057>, item_<extra_id_9283>, item_<extra_id_8449>, item_<extra_id_6394>, item_<extra_id_8907>, item_<extra_id_3187>, item_<extra_id_6593>, item_<extra_id_9540>, item_<extra_id_9531>, item_<extra_id_7834>, item_<extra_id_7587>, item_<extra_id_3151>, item_<extra_id_6752>, predict next possible item to be bought by the user?

02/25/2023, 00:24:14: item_<extra_id_9670>

02/25/2023, 00:24:14: ---------- training epoch 0 ----------

02/25/2023, 00:27:17: total loss for 1000 steps : 3251.328221321106

02/25/2023, 00:30:09: total loss for 2000 steps : 2348.6389968395233

02/25/2023, 00:33:03: total loss for 3000 steps : 2307.3097448349

02/25/2023, 00:35:56: total loss for 4000 steps : 2251.5146536827087

02/25/2023, 00:36:14: ---------- start evaluation after epoch 0 ----------

02/25/2023, 00:39:04: validation hit @ 1 is 0.0008942944017170453

02/25/2023, 00:39:04: validation hit @ 5 is 0.006081201931675908

02/25/2023, 00:39:04: validation hit @ 10 is 0.01717045251296727

02/25/2023, 00:39:05: validation hit @ 1 is 0.0017885888034340906

02/25/2023, 00:39:05: validation hit @ 5 is 0.008048649615453407

02/25/2023, 00:39:05: validation hit @ 10 is 0.011446968341978178

02/25/2023, 00:39:06: validation hit @ 1 is 0.0014308710427472723

02/25/2023, 00:39:06: validation hit @ 5 is 0.009837238418887497

02/25/2023, 00:39:06: validation hit @ 10 is 0.016097299230906813

02/25/2023, 00:39:08: validation hit @ 1 is 0.0014308710427472723

02/25/2023, 00:39:08: validation hit @ 5 is 0.007154355213736362

02/25/2023, 00:39:08: validation hit @ 10 is 0.014487569307816132

02/25/2023, 00:41:53: test hit @ 1 is 0.0012520121624038634

02/25/2023, 00:41:53: test hit @ 5 is 0.004471472008585226

02/25/2023, 00:41:53: test hit @ 10 is 0.00965837953854409

02/25/2023, 00:41:53: recall increases from 0 ----> 0.0008942944017170453 at epoch 0

02/25/2023, 00:41:57: test hit @ 1 is 0.003040600965837954

02/25/2023, 00:41:57: test hit @ 5 is 0.007333214094079771

02/25/2023, 00:41:57: test hit @ 10 is 0.010016097299230907

02/25/2023, 00:41:57: recall increases from 0 ----> 0.0017885888034340906 at epoch 0

02/25/2023, 00:41:57: test hit @ 1 is 0.0008942944017170453

02/25/2023, 00:41:57: test hit @ 5 is 0.006438919692362726

02/25/2023, 00:41:57: test hit @ 10 is 0.009837238418887497

02/25/2023, 00:41:57: recall increases from 0 ----> 0.0014308710427472723 at epoch 0

02/25/2023, 00:41:59: test hit @ 1 is 0.0005365766410302271

02/25/2023, 00:41:59: test hit @ 5 is 0.006081201931675908

02/25/2023, 00:41:59: test hit @ 10 is 0.01126810946163477

02/25/2023, 00:41:59: recall increases from 0 ----> 0.0014308710427472723 at epoch 0

02/25/2023, 00:42:00: ---------- training epoch 1 ----------

02/25/2023, 00:44:34: total loss for 5000 steps : 2213.3618173599243

02/25/2023, 00:47:27: total loss for 6000 steps : 2185.251570701599

02/25/2023, 00:50:19: total loss for 7000 steps : 2159.59495472908

02/25/2023, 00:53:10: total loss for 8000 steps : 2142.160880088806

02/25/2023, 00:53:46: ---------- start evaluation after epoch 1 ----------

02/25/2023, 00:56:42: validation hit @ 1 is 0.00375603648721159

02/25/2023, 00:56:42: validation hit @ 5 is 0.014666428188159542

02/25/2023, 00:56:42: validation hit @ 10 is 0.025934537649794313

02/25/2023, 00:56:43: validation hit @ 1 is 0.0033983187265247718

02/25/2023, 00:56:43: validation hit @ 5 is 0.01520300482918977

02/25/2023, 00:56:43: validation hit @ 10 is 0.027365408692541584

02/25/2023, 00:56:44: validation hit @ 1 is 0.004471472008585226

02/25/2023, 00:56:44: validation hit @ 5 is 0.011804686102664998

02/25/2023, 00:56:44: validation hit @ 10 is 0.021820783401895905

02/25/2023, 00:56:46: validation hit @ 1 is 0.003040600965837954

02/25/2023, 00:56:46: validation hit @ 5 is 0.013950992666785907

02/25/2023, 00:56:46: validation hit @ 10 is 0.022893936683956357

02/25/2023, 00:59:37: test hit @ 1 is 0.0023251654444643175

02/25/2023, 00:59:37: test hit @ 5 is 0.008048649615453407

02/25/2023, 00:59:37: test hit @ 10 is 0.015739581470219997

02/25/2023, 00:59:37: recall increases from 0.0008942944017170453 ----> 0.00375603648721159 at epoch 1

02/25/2023, 00:59:39: test hit @ 1 is 0.0017885888034340906

02/25/2023, 00:59:39: test hit @ 5 is 0.010731532820604543

02/25/2023, 00:59:39: test hit @ 10 is 0.017885888034340904

02/25/2023, 00:59:39: recall increases from 0.0014308710427472723 ----> 0.0033983187265247718 at epoch 1

02/25/2023, 00:59:40: test hit @ 1 is 0.0019674476837774997

02/25/2023, 00:59:40: test hit @ 5 is 0.00930066177785727

02/25/2023, 00:59:40: test hit @ 10 is 0.01699159363262386

02/25/2023, 00:59:40: recall increases from 0.0017885888034340906 ----> 0.004471472008585226 at epoch 1

02/25/2023, 00:59:43: test hit @ 1 is 0.003934895367554999

02/25/2023, 00:59:43: test hit @ 5 is 0.010731532820604543

02/25/2023, 00:59:43: test hit @ 10 is 0.015918440350563407

02/25/2023, 00:59:43: recall increases from 0.0014308710427472723 ----> 0.003040600965837954 at epoch 1

02/25/2023, 00:59:44: ---------- training epoch 2 ----------

02/25/2023, 01:01:57: total loss for 9000 steps : 2126.426302433014

02/25/2023, 01:04:35: total loss for 10000 steps : 2112.841304421425

02/25/2023, 01:07:12: total loss for 11000 steps : 2102.8777426481247

02/25/2023, 01:09:49: total loss for 12000 steps : 2090.187928676605

02/25/2023, 01:10:39: ---------- start evaluation after epoch 2 ----------

02/25/2023, 01:13:25: validation hit @ 1 is 0.003934895367554999

02/25/2023, 01:13:25: validation hit @ 5 is 0.020211053478805224

02/25/2023, 01:13:25: validation hit @ 10 is 0.03308889286353067

02/25/2023, 01:13:27: validation hit @ 1 is 0.0033983187265247718

02/25/2023, 01:13:27: validation hit @ 5 is 0.014308710427472724

02/25/2023, 01:13:27: validation hit @ 10 is 0.02825970309425863

02/25/2023, 01:13:28: validation hit @ 1 is 0.00375603648721159

02/25/2023, 01:13:28: validation hit @ 5 is 0.016097299230906813

02/25/2023, 01:13:28: validation hit @ 10 is 0.030584868538722946

02/25/2023, 01:13:28: validation hit @ 1 is 0.004292613128241817

02/25/2023, 01:13:28: validation hit @ 5 is 0.014666428188159542

02/25/2023, 01:13:28: validation hit @ 10 is 0.027186549812198174

02/25/2023, 01:16:09: test hit @ 1 is 0.0033983187265247718

02/25/2023, 01:16:09: test hit @ 5 is 0.011983544983008406

02/25/2023, 01:16:09: test hit @ 10 is 0.021105347880522266

02/25/2023, 01:16:09: recall increases from 0.00375603648721159 ----> 0.003934895367554999 at epoch 2

02/25/2023, 01:16:14: test hit @ 1 is 0.0033983187265247718

02/25/2023, 01:16:14: test hit @ 5 is 0.011804686102664998

02/25/2023, 01:16:14: test hit @ 10 is 0.023072795564299767

02/25/2023, 01:16:14: recall increases from 0.003040600965837954 ----> 0.0033983187265247718 at epoch 2

02/25/2023, 01:16:14: test hit @ 1 is 0.003219459846181363

02/25/2023, 01:16:14: test hit @ 5 is 0.01108925058129136

02/25/2023, 01:16:14: test hit @ 10 is 0.022893936683956357

02/25/2023, 01:16:14: recall increases from 0.0033983187265247718 ----> 0.00375603648721159 at epoch 2

02/25/2023, 01:16:14: test hit @ 1 is 0.0021463065641209086

02/25/2023, 01:16:14: test hit @ 5 is 0.010910391700947953

02/25/2023, 01:16:14: test hit @ 10 is 0.02038991235914863

02/25/2023, 01:16:15: ---------- training epoch 3 ----------

02/25/2023, 01:18:03: total loss for 13000 steps : 2077.8172768354416

02/25/2023, 01:20:47: total loss for 14000 steps : 2061.465958595276

02/25/2023, 01:23:31: total loss for 15000 steps : 2051.0591694116592

02/25/2023, 01:26:13: total loss for 16000 steps : 2036.062689781189

02/25/2023, 01:27:20: ---------- start evaluation after epoch 3 ----------

02/25/2023, 01:30:05: validation hit @ 1 is 0.005008048649615454

02/25/2023, 01:30:05: validation hit @ 5 is 0.027007690931854768

02/25/2023, 01:30:05: validation hit @ 10 is 0.041316401359327494

02/25/2023, 01:30:08: validation hit @ 1 is 0.005902343051332499

02/25/2023, 01:30:08: validation hit @ 5 is 0.024682525487390448

02/25/2023, 01:30:08: validation hit @ 10 is 0.039885530316580216

02/25/2023, 01:30:08: validation hit @ 1 is 0.0053657664103022714

02/25/2023, 01:30:08: validation hit @ 5 is 0.021284206760865676

02/25/2023, 01:30:08: validation hit @ 10 is 0.0375603648721159

02/25/2023, 01:30:08: validation hit @ 1 is 0.004113754247898408

02/25/2023, 01:30:08: validation hit @ 5 is 0.021284206760865676

02/25/2023, 01:30:08: validation hit @ 10 is 0.03648721159005545

02/25/2023, 01:32:47: test hit @ 1 is 0.003219459846181363

02/25/2023, 01:32:47: test hit @ 5 is 0.01681273475228045

02/25/2023, 01:32:47: test hit @ 10 is 0.027365408692541584

02/25/2023, 01:32:47: recall increases from 0.003934895367554999 ----> 0.005008048649615454 at epoch 3

02/25/2023, 01:32:53: test hit @ 1 is 0.00554462529064568

02/25/2023, 01:32:56: test hit @ 1 is 0.003040600965837954

02/25/2023, 01:32:56: test hit @ 1 is 0.003577177606868181

02/25/2023, 01:32:59: test hit @ 5 is 0.01681273475228045

02/25/2023, 01:32:59: test hit @ 5 is 0.014129851547129315

02/25/2023, 01:32:59: test hit @ 10 is 0.027544267572884994

02/25/2023, 01:32:59: test hit @ 10 is 0.027186549812198174

02/25/2023, 01:32:59: test hit @ 5 is 0.01699159363262386

02/25/2023, 01:32:59: recall increases from 0.0033983187265247718 ----> 0.0053657664103022714 at epoch 3

02/25/2023, 01:32:59: recall increases from 0.00375603648721159 ----> 0.005902343051332499 at epoch 3

02/25/2023, 01:32:59: test hit @ 10 is 0.02808084421391522

02/25/2023, 01:33:01: ---------- training epoch 4 ----------

02/25/2023, 01:34:33: total loss for 17000 steps : 2027.653459906578

02/25/2023, 01:37:10: total loss for 18000 steps : 2014.7500373125076

02/25/2023, 01:39:52: total loss for 19000 steps : 2004.5052527189255

02/25/2023, 01:42:36: total loss for 20000 steps : 1991.5390355587006

02/25/2023, 01:44:04: ---------- start evaluation after epoch 4 ----------

02/25/2023, 01:47:00: validation hit @ 1 is 0.006438919692362726

02/25/2023, 01:47:00: validation hit @ 5 is 0.02986943301734931

02/25/2023, 01:47:00: validation hit @ 10 is 0.0473976032910034

02/25/2023, 01:47:01: validation hit @ 1 is 0.006260060812019317

02/25/2023, 01:47:01: validation hit @ 5 is 0.026828832051511358

02/25/2023, 01:47:01: validation hit @ 10 is 0.04811303881237704

02/25/2023, 01:47:04: validation hit @ 1 is 0.005723484170989089

02/25/2023, 01:47:04: validation hit @ 5 is 0.02074763011983545

02/25/2023, 01:47:04: validation hit @ 10 is 0.04077982471829726

02/25/2023, 01:47:05: validation hit @ 1 is 0.0051869075299588625

02/25/2023, 01:47:05: validation hit @ 5 is 0.023967089966016812

02/25/2023, 01:47:05: validation hit @ 10 is 0.038812377034519764

02/25/2023, 01:49:54: test hit @ 1 is 0.00375603648721159

02/25/2023, 01:49:54: test hit @ 5 is 0.01717045251296727

02/25/2023, 01:49:54: test hit @ 10 is 0.027544267572884994

02/25/2023, 01:49:54: recall increases from 0.005008048649615454 ----> 0.006438919692362726 at epoch 4

02/25/2023, 01:49:58: test hit @ 1 is 0.0033983187265247718

02/25/2023, 01:49:58: test hit @ 5 is 0.01895904131640136

02/25/2023, 01:49:58: test hit @ 10 is 0.03487748166696476

02/25/2023, 01:49:58: recall increases from 0.005902343051332499 ----> 0.006260060812019317 at epoch 4

02/25/2023, 01:50:01: test hit @ 1 is 0.003219459846181363

02/25/2023, 01:50:01: test hit @ 5 is 0.014845287068502952

02/25/2023, 01:50:01: test hit @ 10 is 0.027723126453228403

02/25/2023, 01:50:01: recall increases from 0.004471472008585226 ----> 0.005723484170989089 at epoch 4

02/25/2023, 01:50:01: test hit @ 1 is 0.004113754247898408

02/25/2023, 01:50:01: test hit @ 5 is 0.01913790019674477

02/25/2023, 01:50:01: test hit @ 10 is 0.028975138615632265

02/25/2023, 01:50:01: ---------- training epoch 5 ----------

02/25/2023, 01:51:18: total loss for 21000 steps : 1993.023049235344

02/25/2023, 01:54:03: total loss for 22000 steps : 1977.319176197052

02/25/2023, 01:56:46: total loss for 23000 steps : 1971.3403948545456

02/25/2023, 01:59:26: total loss for 24000 steps : 1965.4754384756088

02/25/2023, 02:01:11: ---------- start evaluation after epoch 5 ----------

02/25/2023, 02:03:59: validation hit @ 1 is 0.006081201931675908

02/25/2023, 02:03:59: validation hit @ 5 is 0.027723126453228403

02/25/2023, 02:03:59: validation hit @ 10 is 0.04721874441065999

02/25/2023, 02:04:01: validation hit @ 1 is 0.006438919692362726

02/25/2023, 02:04:01: validation hit @ 5 is 0.026113396530137722

02/25/2023, 02:04:01: validation hit @ 10 is 0.0434627079234484

02/25/2023, 02:04:03: validation hit @ 1 is 0.006260060812019317

02/25/2023, 02:04:03: validation hit @ 5 is 0.029153997495975675

02/25/2023, 02:04:03: validation hit @ 10 is 0.05276336970130567

02/25/2023, 02:04:07: validation hit @ 1 is 0.006438919692362726

02/25/2023, 02:04:07: validation hit @ 5 is 0.025397961008764087

02/25/2023, 02:04:07: validation hit @ 10 is 0.045787873367912714

02/25/2023, 02:06:44: test hit @ 1 is 0.005008048649615454

02/25/2023, 02:06:44: test hit @ 5 is 0.019495617957431585

02/25/2023, 02:06:44: test hit @ 10 is 0.03308889286353067

02/25/2023, 02:06:47: test hit @ 1 is 0.005008048649615454

02/25/2023, 02:06:47: test hit @ 5 is 0.019495617957431585

02/25/2023, 02:06:47: test hit @ 10 is 0.03273117510284385

02/25/2023, 02:06:47: recall increases from 0.0053657664103022714 ----> 0.006438919692362726 at epoch 5

02/25/2023, 02:06:49: test hit @ 1 is 0.006260060812019317

02/25/2023, 02:06:49: test hit @ 5 is 0.02271507780361295

02/25/2023, 02:06:49: test hit @ 10 is 0.039170094795206584

02/25/2023, 02:06:55: test hit @ 1 is 0.004471472008585226

02/25/2023, 02:06:55: test hit @ 5 is 0.021284206760865676

02/25/2023, 02:06:55: test hit @ 10 is 0.035056340547308176

02/25/2023, 02:06:55: recall increases from 0.005723484170989089 ----> 0.006438919692362726 at epoch 5

02/25/2023, 02:06:55: ---------- training epoch 6 ----------

02/25/2023, 02:07:52: total loss for 25000 steps : 1957.606458067894

02/25/2023, 02:10:29: total loss for 26000 steps : 1952.0779161453247

02/25/2023, 02:13:07: total loss for 27000 steps : 1946.669568657875

02/25/2023, 02:15:57: total loss for 28000 steps : 1940.5048180818558

02/25/2023, 02:18:04: ---------- start evaluation after epoch 6 ----------

02/25/2023, 02:21:16: validation hit @ 1 is 0.00947952065820068

02/25/2023, 02:21:16: validation hit @ 5 is 0.03487748166696476

02/25/2023, 02:21:16: validation hit @ 10 is 0.05562511178680021

02/25/2023, 02:21:18: validation hit @ 1 is 0.008764085136827044

02/25/2023, 02:21:18: validation hit @ 5 is 0.03201573958147022

02/25/2023, 02:21:18: validation hit @ 10 is 0.05455195850473976

02/25/2023, 02:21:19: validation hit @ 1 is 0.00965837953854409

02/25/2023, 02:21:19: validation hit @ 5 is 0.03523519942765158

02/25/2023, 02:21:19: validation hit @ 10 is 0.0572348417098909

02/25/2023, 02:21:20: validation hit @ 1 is 0.008227508495796816

02/25/2023, 02:21:20: validation hit @ 5 is 0.02825970309425863

02/25/2023, 02:21:20: validation hit @ 10 is 0.048291897692720444

02/25/2023, 02:24:08: test hit @ 1 is 0.005008048649615454

02/25/2023, 02:24:08: test hit @ 5 is 0.023072795564299767

02/25/2023, 02:24:08: test hit @ 10 is 0.03738150599177249

02/25/2023, 02:24:08: recall increases from 0.006438919692362726 ----> 0.00947952065820068 at epoch 6

02/25/2023, 02:24:14: test hit @ 1 is 0.004829189769272045

02/25/2023, 02:24:14: test hit @ 5 is 0.023788231085673402

02/25/2023, 02:24:14: test hit @ 10 is 0.04113754247898408

02/25/2023, 02:24:14: recall increases from 0.006438919692362726 ----> 0.008764085136827044 at epoch 6

02/25/2023, 02:24:15: test hit @ 1 is 0.006260060812019317

02/25/2023, 02:24:15: test hit @ 5 is 0.027007690931854768

02/25/2023, 02:24:15: test hit @ 10 is 0.040243248077267035

02/25/2023, 02:24:15: recall increases from 0.006260060812019317 ----> 0.00965837953854409 at epoch 6

02/25/2023, 02:24:16: test hit @ 1 is 0.006975496333392953

02/25/2023, 02:24:16: test hit @ 5 is 0.02199964228223931

02/25/2023, 02:24:16: test hit @ 10 is 0.038454659273832945

02/25/2023, 02:24:16: recall increases from 0.006438919692362726 ----> 0.008227508495796816 at epoch 6

02/25/2023, 02:24:18: ---------- training epoch 7 ----------

02/25/2023, 02:25:01: total loss for 29000 steps : 1936.133904337883

02/25/2023, 02:27:46: total loss for 30000 steps : 1924.8340290784836

02/25/2023, 02:30:31: total loss for 31000 steps : 1920.2022117376328

02/25/2023, 02:33:14: total loss for 32000 steps : 1916.7522938251495

02/25/2023, 02:35:35: ---------- start evaluation after epoch 7 ----------

02/25/2023, 02:38:28: validation hit @ 1 is 0.00965837953854409

02/25/2023, 02:38:28: validation hit @ 5 is 0.03648721159005545

02/25/2023, 02:38:28: validation hit @ 10 is 0.06260060812019316

02/25/2023, 02:38:30: validation hit @ 1 is 0.00930066177785727

02/25/2023, 02:38:30: validation hit @ 5 is 0.03863351815417636

02/25/2023, 02:38:30: validation hit @ 10 is 0.06582006796637453

02/25/2023, 02:38:31: validation hit @ 1 is 0.008764085136827044

02/25/2023, 02:38:31: validation hit @ 5 is 0.03308889286353067

02/25/2023, 02:38:31: validation hit @ 10 is 0.05312108746199249

02/25/2023, 02:38:34: validation hit @ 1 is 0.00947952065820068

02/25/2023, 02:38:34: validation hit @ 5 is 0.03398318726524772

02/25/2023, 02:38:34: validation hit @ 10 is 0.05795027723126453

02/25/2023, 02:41:23: test hit @ 1 is 0.006438919692362726

02/25/2023, 02:41:23: test hit @ 5 is 0.02450366660704704

02/25/2023, 02:41:23: test hit @ 10 is 0.04077982471829726

02/25/2023, 02:41:23: recall increases from 0.00947952065820068 ----> 0.00965837953854409 at epoch 7

02/25/2023, 02:41:27: test hit @ 1 is 0.007154355213736362

02/25/2023, 02:41:27: test hit @ 5 is 0.030763727419066356

02/25/2023, 02:41:27: test hit @ 10 is 0.0454301556072259

02/25/2023, 02:41:28: test hit @ 1 is 0.007690931854766589

02/25/2023, 02:41:28: test hit @ 5 is 0.023251654444643177

02/25/2023, 02:41:28: test hit @ 10 is 0.04042210695761045

02/25/2023, 02:41:28: recall increases from 0.008227508495796816 ----> 0.008764085136827044 at epoch 7

02/25/2023, 02:41:32: test hit @ 1 is 0.006438919692362726

02/25/2023, 02:41:32: test hit @ 5 is 0.02450366660704704

02/25/2023, 02:41:32: test hit @ 10 is 0.04042210695761045

02/25/2023, 02:41:32: recall increases from 0.008764085136827044 ----> 0.00947952065820068 at epoch 7

02/25/2023, 02:41:32: ---------- training epoch 8 ----------

02/25/2023, 02:41:55: total loss for 33000 steps : 1908.3649505376816

02/25/2023, 02:44:38: total loss for 34000 steps : 1896.2488067150116

02/25/2023, 02:47:21: total loss for 35000 steps : 1898.4285827875137

02/25/2023, 02:49:59: total loss for 36000 steps : 1891.9000817537308

02/25/2023, 02:52:37: ---------- start evaluation after epoch 8 ----------

02/25/2023, 02:55:35: validation hit @ 1 is 0.012698980504382043

02/25/2023, 02:55:35: validation hit @ 5 is 0.038812377034519764

02/25/2023, 02:55:35: validation hit @ 10 is 0.06564120908603112

02/25/2023, 02:55:36: validation hit @ 1 is 0.01323555714541227

02/25/2023, 02:55:36: validation hit @ 5 is 0.04185297800035772

02/25/2023, 02:55:36: validation hit @ 10 is 0.06617778572706134

02/25/2023, 02:55:38: validation hit @ 1 is 0.00947952065820068

02/25/2023, 02:55:38: validation hit @ 5 is 0.038454659273832945

02/25/2023, 02:55:38: validation hit @ 10 is 0.057771418350921123

02/25/2023, 02:55:38: validation hit @ 1 is 0.00947952065820068

02/25/2023, 02:55:38: validation hit @ 5 is 0.03577177606868181

02/25/2023, 02:55:38: validation hit @ 10 is 0.06242174923984976

02/25/2023, 02:58:31: test hit @ 1 is 0.005723484170989089

02/25/2023, 02:58:31: test hit @ 5 is 0.025397961008764087

02/25/2023, 02:58:31: test hit @ 10 is 0.04364156680379181

02/25/2023, 02:58:31: recall increases from 0.00965837953854409 ----> 0.012698980504382043 at epoch 8

02/25/2023, 02:58:34: test hit @ 1 is 0.00930066177785727

02/25/2023, 02:58:34: test hit @ 5 is 0.032910033983187266

02/25/2023, 02:58:34: test hit @ 10 is 0.04990162761581113

02/25/2023, 02:58:34: recall increases from 0.00965837953854409 ----> 0.01323555714541227 at epoch 8

02/25/2023, 02:58:37: test hit @ 1 is 0.007154355213736362

02/25/2023, 02:58:37: test hit @ 5 is 0.025219102128420677

02/25/2023, 02:58:37: test hit @ 10 is 0.044535861205508856

02/25/2023, 02:58:37: recall increases from 0.008764085136827044 ----> 0.00947952065820068 at epoch 8

02/25/2023, 02:58:37: test hit @ 1 is 0.007333214094079771

02/25/2023, 02:58:37: test hit @ 5 is 0.02647111429082454

02/25/2023, 02:58:37: test hit @ 10 is 0.04650330888928635

02/25/2023, 02:58:38: ---------- training epoch 9 ----------

02/25/2023, 02:58:44: total loss for 37000 steps : 1885.088766336441

02/25/2023, 03:01:31: total loss for 38000 steps : 1879.7391834259033

02/25/2023, 03:04:17: total loss for 39000 steps : 1874.8780479431152

02/25/2023, 03:07:06: total loss for 40000 steps : 1865.8304654359818

02/25/2023, 03:09:54: total loss for 41000 steps : 1859.3068789243698

02/25/2023, 03:10:06: ---------- start evaluation after epoch 9 ----------

02/25/2023, 03:13:03: validation hit @ 1 is 0.011446968341978178

02/25/2023, 03:13:03: validation hit @ 5 is 0.04525129672688249

02/25/2023, 03:13:03: validation hit @ 10 is 0.06653550348774817

02/25/2023, 03:13:04: validation hit @ 1 is 0.013056698265068861

02/25/2023, 03:13:04: validation hit @ 5 is 0.04721874441065999

02/25/2023, 03:13:04: validation hit @ 10 is 0.07208012877839384

02/25/2023, 03:13:06: validation hit @ 1 is 0.012520121624038634

02/25/2023, 03:13:06: validation hit @ 5 is 0.042031836880701126

02/25/2023, 03:13:06: validation hit @ 10 is 0.06957610445358613

02/25/2023, 03:13:07: validation hit @ 1 is 0.010910391700947953

02/25/2023, 03:13:07: validation hit @ 5 is 0.03702378823108567

02/25/2023, 03:13:07: validation hit @ 10 is 0.06009658379538544

02/25/2023, 03:16:00: test hit @ 1 is 0.007154355213736362

02/25/2023, 03:16:00: test hit @ 5 is 0.027007690931854768

02/25/2023, 03:16:00: test hit @ 10 is 0.04507243784653908

02/25/2023, 03:16:05: test hit @ 1 is 0.010552673940261133

02/25/2023, 03:16:05: test hit @ 5 is 0.032552316222500446

02/25/2023, 03:16:05: test hit @ 10 is 0.05294222858164908

02/25/2023, 03:16:06: test hit @ 1 is 0.0067966374530495435

02/25/2023, 03:16:06: test hit @ 5 is 0.02843856197460204

02/25/2023, 03:16:06: test hit @ 10 is 0.047576462171346805

02/25/2023, 03:16:06: recall increases from 0.00947952065820068 ----> 0.012520121624038634 at epoch 9

02/25/2023, 03:16:10: test hit @ 1 is 0.008406367376140225

02/25/2023, 03:16:10: test hit @ 5 is 0.030763727419066356

02/25/2023, 03:16:10: test hit @ 10 is 0.04990162761581113

02/25/2023, 03:16:10: recall increases from 0.00947952065820068 ----> 0.010910391700947953 at epoch 9

02/25/2023, 03:16:11: ---------- training epoch 10 ----------

02/25/2023, 03:18:44: total loss for 42000 steps : 1853.5869921445847

02/25/2023, 03:21:29: total loss for 43000 steps : 1848.077343583107

02/25/2023, 03:24:12: total loss for 44000 steps : 1846.4168255329132

02/25/2023, 03:26:55: total loss for 45000 steps : 1834.2860246896744

02/25/2023, 03:27:24: ---------- start evaluation after epoch 10 ----------

02/25/2023, 03:30:21: validation hit @ 1 is 0.014308710427472724

02/25/2023, 03:30:21: validation hit @ 5 is 0.045787873367912714

02/25/2023, 03:30:21: validation hit @ 10 is 0.0711858343766768

02/25/2023, 03:30:24: validation hit @ 1 is 0.012162403863351816

02/25/2023, 03:30:24: validation hit @ 5 is 0.04006438919692363

02/25/2023, 03:30:24: validation hit @ 10 is 0.06492577356465748

02/25/2023, 03:30:25: validation hit @ 1 is 0.013772133786442497

02/25/2023, 03:30:25: validation hit @ 5 is 0.0434627079234484

02/25/2023, 03:30:25: validation hit @ 10 is 0.07404757646217135

02/25/2023, 03:30:27: validation hit @ 1 is 0.014308710427472724

02/25/2023, 03:30:27: validation hit @ 5 is 0.05061706313718476

02/25/2023, 03:30:27: validation hit @ 10 is 0.07744589518869611

02/25/2023, 03:33:17: test hit @ 1 is 0.0051869075299588625

02/25/2023, 03:33:17: test hit @ 5 is 0.031121445179753175

02/25/2023, 03:33:17: test hit @ 10 is 0.04900733321409408

02/25/2023, 03:33:17: recall increases from 0.012698980504382043 ----> 0.014308710427472724 at epoch 10

02/25/2023, 03:33:23: test hit @ 1 is 0.007690931854766589

02/25/2023, 03:33:23: test hit @ 5 is 0.030584868538722946

02/25/2023, 03:33:23: test hit @ 10 is 0.050795922017528174

02/25/2023, 03:33:23: recall increases from 0.010910391700947953 ----> 0.012162403863351816 at epoch 10

02/25/2023, 03:33:24: test hit @ 1 is 0.008764085136827044

02/25/2023, 03:33:24: test hit @ 5 is 0.02986943301734931

02/25/2023, 03:33:24: test hit @ 10 is 0.05061706313718476

02/25/2023, 03:33:24: recall increases from 0.012520121624038634 ----> 0.013772133786442497 at epoch 10

02/25/2023, 03:33:24: test hit @ 1 is 0.009837238418887497

02/25/2023, 03:33:25: test hit @ 5 is 0.03577177606868181

02/25/2023, 03:33:25: test hit @ 10 is 0.05276336970130567

02/25/2023, 03:33:25: recall increases from 0.01323555714541227 ----> 0.014308710427472724 at epoch 10

02/25/2023, 03:33:25: ---------- training epoch 11 ----------

02/25/2023, 03:35:43: total loss for 46000 steps : 1831.2914363145828

02/25/2023, 03:38:30: total loss for 47000 steps : 1825.6867792606354

02/25/2023, 03:41:17: total loss for 48000 steps : 1815.876898765564

02/25/2023, 03:44:03: total loss for 49000 steps : 1811.285516858101

02/25/2023, 03:44:50: ---------- start evaluation after epoch 11 ----------

02/25/2023, 03:47:49: validation hit @ 1 is 0.014308710427472724

02/25/2023, 03:47:49: validation hit @ 5 is 0.04686102664997317

02/25/2023, 03:47:49: validation hit @ 10 is 0.07637274190663566

02/25/2023, 03:47:49: validation hit @ 1 is 0.015918440350563407

02/25/2023, 03:47:49: validation hit @ 5 is 0.051511357538901806

02/25/2023, 03:47:49: validation hit @ 10 is 0.07816133071006975

02/25/2023, 03:47:51: validation hit @ 1 is 0.011983544983008406

02/25/2023, 03:47:51: validation hit @ 5 is 0.044178143444822036

02/25/2023, 03:47:51: validation hit @ 10 is 0.06975496333392953

02/25/2023, 03:47:52: validation hit @ 1 is 0.012162403863351816

02/25/2023, 03:47:52: validation hit @ 5 is 0.050795922017528174

02/25/2023, 03:47:52: validation hit @ 10 is 0.07619388302629225

02/25/2023, 03:50:44: test hit @ 1 is 0.00751207297442318

02/25/2023, 03:50:44: test hit @ 5 is 0.03022715077803613

02/25/2023, 03:50:44: test hit @ 10 is 0.04775532105169022

02/25/2023, 03:50:48: test hit @ 1 is 0.010552673940261133

02/25/2023, 03:50:48: test hit @ 5 is 0.03523519942765158

02/25/2023, 03:50:48: test hit @ 10 is 0.05383652298336612

02/25/2023, 03:50:48: recall increases from 0.014308710427472724 ----> 0.015918440350563407 at epoch 11

02/25/2023, 03:50:52: test hit @ 1 is 0.008227508495796816

02/25/2023, 03:50:52: test hit @ 5 is 0.03130030406009658

02/25/2023, 03:50:52: test hit @ 10 is 0.05204793417993203

02/25/2023, 03:50:52: test hit @ 1 is 0.009837238418887497

02/25/2023, 03:50:52: test hit @ 5 is 0.0336254695045609

02/25/2023, 03:50:52: test hit @ 10 is 0.057055982829547484

02/25/2023, 03:50:52: ---------- training epoch 12 ----------

02/25/2023, 03:52:52: total loss for 50000 steps : 1807.095207452774

02/25/2023, 03:55:39: total loss for 51000 steps : 1801.871783733368

02/25/2023, 03:58:21: total loss for 52000 steps : 1799.398987531662

02/25/2023, 04:01:04: total loss for 53000 steps : 1790.077452659607

02/25/2023, 04:02:07: ---------- start evaluation after epoch 12 ----------

02/25/2023, 04:04:55: validation hit @ 1 is 0.016276158111250223

02/25/2023, 04:04:55: validation hit @ 5 is 0.05097478089787158

02/25/2023, 04:04:55: validation hit @ 10 is 0.07494187086388839

02/25/2023, 04:04:58: validation hit @ 1 is 0.015560722589876588

02/25/2023, 04:04:58: validation hit @ 5 is 0.0513324986585584

02/25/2023, 04:04:58: validation hit @ 10 is 0.07851904847075657

02/25/2023, 04:04:59: validation hit @ 1 is 0.014666428188159542

02/25/2023, 04:04:59: validation hit @ 5 is 0.04489357896619567

02/25/2023, 04:04:59: validation hit @ 10 is 0.06742979788946521

02/25/2023, 04:04:59: validation hit @ 1 is 0.017885888034340904

02/25/2023, 04:04:59: validation hit @ 5 is 0.05473081738508317

02/25/2023, 04:04:59: validation hit @ 10 is 0.08030763727419066

02/25/2023, 04:07:45: test hit @ 1 is 0.00947952065820068

02/25/2023, 04:07:45: test hit @ 5 is 0.03434090502593454

02/25/2023, 04:07:45: test hit @ 10 is 0.05204793417993203

02/25/2023, 04:07:45: recall increases from 0.014308710427472724 ----> 0.016276158111250223 at epoch 12

02/25/2023, 04:07:52: test hit @ 1 is 0.00930066177785727

02/25/2023, 04:07:52: test hit @ 5 is 0.031836880701126814

02/25/2023, 04:07:52: test hit @ 10 is 0.05490967626542658

02/25/2023, 04:07:52: recall increases from 0.013772133786442497 ----> 0.015560722589876588 at epoch 12

02/25/2023, 04:07:54: test hit @ 1 is 0.010731532820604543

02/25/2023, 04:07:54: test hit @ 5 is 0.0336254695045609

02/25/2023, 04:07:54: test hit @ 10 is 0.05669826506886067

02/25/2023, 04:07:54: recall increases from 0.012162403863351816 ----> 0.014666428188159542 at epoch 12

02/25/2023, 04:07:55: test hit @ 1 is 0.01502414594884636

02/25/2023, 04:07:55: test hit @ 5 is 0.03791808263280272

02/25/2023, 04:07:55: test hit @ 10 is 0.05634054730817385

02/25/2023, 04:07:55: recall increases from 0.015918440350563407 ----> 0.017885888034340904 at epoch 12

02/25/2023, 04:07:56: ---------- training epoch 13 ----------

02/25/2023, 04:09:37: total loss for 54000 steps : 1782.2618337869644

02/25/2023, 04:12:24: total loss for 55000 steps : 1778.347601532936

02/25/2023, 04:15:10: total loss for 56000 steps : 1768.845697760582

02/25/2023, 04:17:56: total loss for 57000 steps : 1766.2803300619125

02/25/2023, 04:19:19: ---------- start evaluation after epoch 13 ----------

02/25/2023, 04:22:16: validation hit @ 1 is 0.01699159363262386

02/25/2023, 04:22:16: validation hit @ 5 is 0.05097478089787158

02/25/2023, 04:22:16: validation hit @ 10 is 0.07690931854766589

02/25/2023, 04:22:17: validation hit @ 1 is 0.014129851547129315

02/25/2023, 04:22:17: validation hit @ 5 is 0.051153639778214986

02/25/2023, 04:22:17: validation hit @ 10 is 0.07994991951350384

02/25/2023, 04:22:19: validation hit @ 1 is 0.01502414594884636

02/25/2023, 04:22:19: validation hit @ 5 is 0.04954390985512431

02/25/2023, 04:22:19: validation hit @ 10 is 0.07279556429976748

02/25/2023, 04:22:20: validation hit @ 1 is 0.018422464675371134

02/25/2023, 04:22:20: validation hit @ 5 is 0.055267394026113394

02/25/2023, 04:22:20: validation hit @ 10 is 0.08048649615453407

02/25/2023, 04:25:16: test hit @ 1 is 0.00930066177785727

02/25/2023, 04:25:16: test hit @ 5 is 0.03380432838490431

02/25/2023, 04:25:16: test hit @ 10 is 0.0532999463423359

02/25/2023, 04:25:16: recall increases from 0.016276158111250223 ----> 0.01699159363262386 at epoch 13

02/25/2023, 04:25:17: test hit @ 1 is 0.010910391700947953

02/25/2023, 04:25:17: test hit @ 5 is 0.033267751743874086

02/25/2023, 04:25:17: test hit @ 10 is 0.0534788052226793

02/25/2023, 04:25:19: test hit @ 1 is 0.011983544983008406

02/25/2023, 04:25:19: test hit @ 5 is 0.03577177606868181

02/25/2023, 04:25:19: test hit @ 10 is 0.058844571632981575

02/25/2023, 04:25:19: recall increases from 0.014666428188159542 ----> 0.01502414594884636 at epoch 13

02/25/2023, 04:25:21: test hit @ 1 is 0.013772133786442497

02/25/2023, 04:25:21: test hit @ 5 is 0.0375603648721159

02/25/2023, 04:25:21: test hit @ 10 is 0.05687712394920408

02/25/2023, 04:25:21: recall increases from 0.017885888034340904 ----> 0.018422464675371134 at epoch 13

02/25/2023, 04:25:21: ---------- training epoch 14 ----------

02/25/2023, 04:26:46: total loss for 58000 steps : 1761.3368334770203

02/25/2023, 04:29:41: total loss for 59000 steps : 1754.5861903429031

02/25/2023, 04:32:34: total loss for 60000 steps : 1749.1633738279343

02/25/2023, 04:35:25: total loss for 61000 steps : 1741.1012853384018

02/25/2023, 04:37:04: ---------- start evaluation after epoch 14 ----------

02/25/2023, 04:39:54: validation hit @ 1 is 0.015381863709533178

02/25/2023, 04:39:54: validation hit @ 5 is 0.051511357538901806

02/25/2023, 04:39:54: validation hit @ 10 is 0.0792344839921302

02/25/2023, 04:39:55: validation hit @ 1 is 0.015381863709533178

02/25/2023, 04:39:55: validation hit @ 5 is 0.05294222858164908

02/25/2023, 04:39:55: validation hit @ 10 is 0.08066535503487748

02/25/2023, 04:39:55: validation hit @ 1 is 0.02092648900017886

02/25/2023, 04:39:55: validation hit @ 5 is 0.05544625290645681

02/25/2023, 04:39:55: validation hit @ 10 is 0.0790556251117868

02/25/2023, 04:40:00: validation hit @ 1 is 0.014666428188159542

02/25/2023, 04:40:00: validation hit @ 5 is 0.0473976032910034

02/25/2023, 04:40:00: validation hit @ 10 is 0.0711858343766768

02/25/2023, 04:42:47: test hit @ 1 is 0.01126810946163477

02/25/2023, 04:42:47: test hit @ 5 is 0.03577177606868181

02/25/2023, 04:42:47: test hit @ 10 is 0.05598282954748703

02/25/2023, 04:42:49: test hit @ 1 is 0.011446968341978178

02/25/2023, 04:42:49: test hit @ 5 is 0.03541405830799499

02/25/2023, 04:42:49: test hit @ 10 is 0.05902343051332499

02/25/2023, 04:42:49: test hit @ 1 is 0.01502414594884636

02/25/2023, 04:42:49: test hit @ 5 is 0.03791808263280272

02/25/2023, 04:42:49: test hit @ 10 is 0.057413700590234304

02/25/2023, 04:42:49: recall increases from 0.018422464675371134 ----> 0.02092648900017886 at epoch 14

02/25/2023, 04:42:52: test hit @ 1 is 0.013414416025755679

02/25/2023, 04:42:52: test hit @ 5 is 0.038096941513146125

02/25/2023, 04:42:52: test hit @ 10 is 0.057771418350921123

02/25/2023, 04:42:52: ---------- training epoch 15 ----------

02/25/2023, 04:43:59: total loss for 62000 steps : 1740.7199091911316

02/25/2023, 04:46:42: total loss for 63000 steps : 1730.8993971347809

02/25/2023, 04:49:23: total loss for 64000 steps : 1722.865222454071

02/25/2023, 04:52:09: total loss for 65000 steps : 1722.6024478673935

02/25/2023, 04:54:06: ---------- start evaluation after epoch 15 ----------

02/25/2023, 04:57:02: validation hit @ 1 is 0.017707029153997494

02/25/2023, 04:57:02: validation hit @ 5 is 0.054015381863709536

02/25/2023, 04:57:02: validation hit @ 10 is 0.08173850831693794

02/25/2023, 04:57:03: validation hit @ 1 is 0.018064746914684314

02/25/2023, 04:57:03: validation hit @ 5 is 0.04721874441065999

02/25/2023, 04:57:03: validation hit @ 10 is 0.07386871758182793

02/25/2023, 04:57:04: validation hit @ 1 is 0.014666428188159542

02/25/2023, 04:57:04: validation hit @ 5 is 0.051869075299588625

02/25/2023, 04:57:04: validation hit @ 10 is 0.08442139152208906

02/25/2023, 04:57:06: validation hit @ 1 is 0.01895904131640136

02/25/2023, 04:57:06: validation hit @ 5 is 0.05508853514576999

02/25/2023, 04:57:06: validation hit @ 10 is 0.08155964943659452

02/25/2023, 04:59:59: test hit @ 1 is 0.010731532820604543

02/25/2023, 04:59:59: test hit @ 5 is 0.036308352709712034

02/25/2023, 04:59:59: test hit @ 10 is 0.055267394026113394

02/25/2023, 04:59:59: recall increases from 0.01699159363262386 ----> 0.017707029153997494 at epoch 15

02/25/2023, 05:00:01: test hit @ 1 is 0.011625827222321588

02/25/2023, 05:00:01: test hit @ 5 is 0.03738150599177249

02/25/2023, 05:00:01: test hit @ 10 is 0.058844571632981575

02/25/2023, 05:00:01: recall increases from 0.01502414594884636 ----> 0.018064746914684314 at epoch 15

02/25/2023, 05:00:02: test hit @ 1 is 0.012341262743695224

02/25/2023, 05:00:02: test hit @ 5 is 0.03577177606868181

02/25/2023, 05:00:02: test hit @ 10 is 0.06116973707744589

02/25/2023, 05:00:05: test hit @ 1 is 0.014129851547129315

02/25/2023, 05:00:05: test hit @ 5 is 0.03970667143623681

02/25/2023, 05:00:05: test hit @ 10 is 0.059560007154355214

02/25/2023, 05:00:05: ---------- training epoch 16 ----------

02/25/2023, 05:00:53: total loss for 66000 steps : 1714.6824326515198

02/25/2023, 05:03:36: total loss for 67000 steps : 1710.340737581253

02/25/2023, 05:06:16: total loss for 68000 steps : 1708.1846976280212

02/25/2023, 05:08:56: total loss for 69000 steps : 1700.9390457868576

02/25/2023, 05:11:09: ---------- start evaluation after epoch 16 ----------

02/25/2023, 05:13:59: validation hit @ 1 is 0.01913790019674477

02/25/2023, 05:13:59: validation hit @ 5 is 0.05473081738508317

02/25/2023, 05:13:59: validation hit @ 10 is 0.08012877839384726

02/25/2023, 05:14:06: validation hit @ 1 is 0.02235736004292613

02/25/2023, 05:14:06: validation hit @ 5 is 0.059917724915042034

02/25/2023, 05:14:06: validation hit @ 10 is 0.08585226256483634

02/25/2023, 05:14:06: validation hit @ 1 is 0.018064746914684314

02/25/2023, 05:14:06: validation hit @ 5 is 0.05508853514576999

02/25/2023, 05:14:06: validation hit @ 10 is 0.08173850831693794

02/25/2023, 05:14:13: validation hit @ 1 is 0.018422464675371134

02/25/2023, 05:14:13: validation hit @ 5 is 0.04990162761581113

02/25/2023, 05:14:13: validation hit @ 10 is 0.0770881774280093

02/25/2023, 05:16:51: test hit @ 1 is 0.011983544983008406

02/25/2023, 05:16:51: test hit @ 5 is 0.0375603648721159

02/25/2023, 05:16:51: test hit @ 10 is 0.057413700590234304

02/25/2023, 05:16:51: recall increases from 0.017707029153997494 ----> 0.01913790019674477 at epoch 16

02/25/2023, 05:17:00: test hit @ 1 is 0.015918440350563407

02/25/2023, 05:17:00: test hit @ 5 is 0.04221069576104453

02/25/2023, 05:17:00: test hit @ 10 is 0.06116973707744589

02/25/2023, 05:17:00: recall increases from 0.02092648900017886 ----> 0.02235736004292613 at epoch 16

02/25/2023, 05:17:05: test hit @ 1 is 0.011625827222321588

02/25/2023, 05:17:05: test hit @ 5 is 0.035056340547308176

02/25/2023, 05:17:05: test hit @ 10 is 0.05848685387229476

02/25/2023, 05:17:05: recall increases from 0.015560722589876588 ----> 0.018064746914684314 at epoch 16

02/25/2023, 05:17:19: test hit @ 1 is 0.01323555714541227

02/25/2023, 05:17:19: test hit @ 5 is 0.03773922375245931

02/25/2023, 05:17:19: test hit @ 10 is 0.057055982829547484

02/25/2023, 05:17:19: recall increases from 0.018064746914684314 ----> 0.018422464675371134 at epoch 16

02/25/2023, 05:17:20: ---------- training epoch 17 ----------

02/25/2023, 05:17:50: total loss for 70000 steps : 1697.6774805784225

02/25/2023, 05:20:34: total loss for 71000 steps : 1692.6146692037582

02/25/2023, 05:23:22: total loss for 72000 steps : 1686.1435638666153

02/25/2023, 05:26:20: total loss for 73000 steps : 1678.323574066162

02/25/2023, 05:29:01: ---------- start evaluation after epoch 17 ----------

02/25/2023, 05:31:50: validation hit @ 1 is 0.021105347880522266

02/25/2023, 05:31:50: validation hit @ 5 is 0.05973886603469862

02/25/2023, 05:31:50: validation hit @ 10 is 0.0869254158468968

02/25/2023, 05:31:52: validation hit @ 1 is 0.017885888034340904

02/25/2023, 05:31:52: validation hit @ 5 is 0.05687712394920408

02/25/2023, 05:31:52: validation hit @ 10 is 0.08173850831693794

02/25/2023, 05:32:16: validation hit @ 1 is 0.019674476837774995

02/25/2023, 05:32:16: validation hit @ 5 is 0.050795922017528174

02/25/2023, 05:32:16: validation hit @ 10 is 0.07637274190663566

02/25/2023, 05:32:28: validation hit @ 1 is 0.01699159363262386

02/25/2023, 05:32:28: validation hit @ 5 is 0.0532999463423359

02/25/2023, 05:32:28: validation hit @ 10 is 0.0849579681631193

02/25/2023, 05:34:44: test hit @ 1 is 0.018064746914684314

02/25/2023, 05:34:44: test hit @ 5 is 0.040600965837953855

02/25/2023, 05:34:44: test hit @ 10 is 0.059917724915042034

02/25/2023, 05:34:45: test hit @ 1 is 0.011446968341978178

02/25/2023, 05:34:45: test hit @ 5 is 0.03595063494902522

02/25/2023, 05:34:45: test hit @ 10 is 0.05759255947057772

02/25/2023, 05:35:21: test hit @ 1 is 0.012877839384725451

02/25/2023, 05:35:21: test hit @ 5 is 0.03863351815417636

02/25/2023, 05:35:21: test hit @ 10 is 0.059560007154355214

02/25/2023, 05:35:21: recall increases from 0.018422464675371134 ----> 0.019674476837774995 at epoch 17

02/25/2023, 05:35:54: test hit @ 1 is 0.012877839384725451

02/25/2023, 05:35:54: test hit @ 5 is 0.03791808263280272

02/25/2023, 05:35:54: test hit @ 10 is 0.05938114827401181

02/25/2023, 05:35:54: ---------- training epoch 18 ----------

02/25/2023, 05:36:07: total loss for 74000 steps : 1682.966084599495

02/25/2023, 05:38:52: total loss for 75000 steps : 1675.7378385066986

02/25/2023, 05:41:38: total loss for 76000 steps : 1672.0766928195953

02/25/2023, 05:44:23: total loss for 77000 steps : 1672.8174179792404

02/25/2023, 05:47:08: total loss for 78000 steps : 1669.0752058029175

02/25/2023, 05:47:13: ---------- start evaluation after epoch 18 ----------

02/25/2023, 05:50:08: validation hit @ 1 is 0.01717045251296727

02/25/2023, 05:50:08: validation hit @ 5 is 0.05490967626542658

02/25/2023, 05:50:08: validation hit @ 10 is 0.08424253264174567

02/25/2023, 05:50:09: validation hit @ 1 is 0.019495617957431585

02/25/2023, 05:50:09: validation hit @ 5 is 0.057413700590234304

02/25/2023, 05:50:09: validation hit @ 10 is 0.08173850831693794

02/25/2023, 05:50:19: validation hit @ 1 is 0.02092648900017886

02/25/2023, 05:50:19: validation hit @ 5 is 0.05240565194061885

02/25/2023, 05:50:19: validation hit @ 10 is 0.0770881774280093

02/25/2023, 05:50:28: validation hit @ 1 is 0.02235736004292613

02/25/2023, 05:50:28: validation hit @ 5 is 0.06081201931675908

02/25/2023, 05:50:28: validation hit @ 10 is 0.08656769808620998

02/25/2023, 05:53:03: test hit @ 1 is 0.013593274906099087

02/25/2023, 05:53:03: test hit @ 5 is 0.03791808263280272

02/25/2023, 05:53:03: test hit @ 10 is 0.0633160436415668

02/25/2023, 05:53:06: test hit @ 1 is 0.012341262743695224

02/25/2023, 05:53:06: test hit @ 5 is 0.0355929171883384

02/25/2023, 05:53:06: test hit @ 10 is 0.05902343051332499

02/25/2023, 05:53:06: recall increases from 0.01913790019674477 ----> 0.019495617957431585 at epoch 18

02/25/2023, 05:53:25: test hit @ 1 is 0.011804686102664998

02/25/2023, 05:53:25: test hit @ 5 is 0.03899123591486317

02/25/2023, 05:53:25: test hit @ 10 is 0.05812913611160794

02/25/2023, 05:53:25: recall increases from 0.019674476837774995 ----> 0.02092648900017886 at epoch 18

02/25/2023, 05:53:48: test hit @ 1 is 0.016276158111250223

02/25/2023, 05:53:48: test hit @ 5 is 0.0414952602396709

02/25/2023, 05:53:48: test hit @ 10 is 0.06009658379538544

02/25/2023, 05:53:48: ---------- training epoch 19 ----------

02/25/2023, 05:56:28: total loss for 79000 steps : 1663.1472301483154

02/25/2023, 05:59:12: total loss for 80000 steps : 1664.2699173688889

02/25/2023, 06:01:55: total loss for 81000 steps : 1659.7333216667175

02/25/2023, 06:04:34: total loss for 82000 steps : 1658.6572173833847

02/25/2023, 06:04:59: ---------- start evaluation after epoch 19 ----------

02/25/2023, 06:07:49: validation hit @ 1 is 0.016455016991593633

02/25/2023, 06:07:49: validation hit @ 5 is 0.05687712394920408

02/25/2023, 06:07:49: validation hit @ 10 is 0.08531568592380612

02/25/2023, 06:07:49: validation hit @ 1 is 0.01931675907708818

02/25/2023, 06:07:49: validation hit @ 5 is 0.05598282954748703

02/25/2023, 06:07:49: validation hit @ 10 is 0.08316937935968521

02/25/2023, 06:07:50: validation hit @ 1 is 0.020032194598461815

02/25/2023, 06:07:50: validation hit @ 5 is 0.05383652298336612

02/25/2023, 06:07:50: validation hit @ 10 is 0.07834018959041317

02/25/2023, 06:07:53: validation hit @ 1 is 0.021641924521552496

02/25/2023, 06:07:53: validation hit @ 5 is 0.06116973707744589

02/25/2023, 06:07:53: validation hit @ 10 is 0.08477910928277589

02/25/2023, 06:10:38: test hit @ 1 is 0.012698980504382043

02/25/2023, 06:10:38: test hit @ 5 is 0.0375603648721159

02/25/2023, 06:10:38: test hit @ 10 is 0.05848685387229476

02/25/2023, 06:10:39: test hit @ 1 is 0.013414416025755679

02/25/2023, 06:10:39: test hit @ 5 is 0.03970667143623681

02/25/2023, 06:10:39: test hit @ 10 is 0.057771418350921123

02/25/2023, 06:10:41: test hit @ 1 is 0.01323555714541227

02/25/2023, 06:10:41: test hit @ 5 is 0.03773922375245931

02/25/2023, 06:10:41: test hit @ 10 is 0.06295832588087999

02/25/2023, 06:10:41: test hit @ 1 is 0.016633875871937043

02/25/2023, 06:10:41: test hit @ 5 is 0.04185297800035772

02/25/2023, 06:10:41: test hit @ 10 is 0.060990878197102485