I found that even finetune from ImageNet pretrain, the loss grows up. I used the ResNet-vd as the baseline arch, and add GC-Block in. I also double check the performance of baseline (w/o GC Block) which is normal (~79% top-1). But when start training (finetuning), loss grows up quickly (~6.xxx) and the val Top-1 Acc is only less than 10% after first epoch. And after 20 epochs training, Top-1 Acc is near 55%. Something wrong?