Thanks for your code
in the paper

• the cls loss is normalized by the number of positive and negative anchors
• the reg term is normalized by the number of positive anchors.
  so may be
  
  it should be changed

M = num_pos.data.sum().float() + num_neg.data.sum().float()
loss_c /= N

also in the paper L =\lambda * L_{cls} + L_{reg}
in the code

loss_l, loss_c = criterion(out, targets)
loss = loss_l + loss_c

may be the two place should be modified