This PR adds a config flag that makes a generation hack from bart's adjust_logits_during_generation, optional. This change (setting the flag to False) improves metrics for bart-large-xsum and mbart-large-en-ro, but not bart-large-cnn (hence the need for the flag).

I remember @patrickvonplaten asked me about this in February, and I ran on CNN and then decided the hack needed to stay, could have been more careful.
cc @patil-suraj

Todo

[x] test xsum
[x] test cnn
[x] test en-ro
[x] update cnn config
[x] test pegasus
[x] update distilbart configs
[] should max_length be changed in xsum config?

TODO:

• check pegasus
• should max_length be 1 lower for configs where hack is removed? Yes.

Metrics

(all on val.source vs. val.target) the first number is runtime over the whole val set. "noforce" means this PR without any config change. (so BOS token is not forced to be generated.)

bart-large-cnn
master: 86:23 {"rouge1": 44.79, "rouge2": 21.64, "rougeL": 31.18}
noforce: 87:40 {'rouge1': 44.26, 'rouge2': 21.22, 'rougeL': 30.72}

bart-large-xsum
master: 41:59, {'rouge1': 45.16, 'rouge2': 21.77, 'rougeL': 36.35}
noforce: 34:12, {'rouge1': 45.45, 'rouge2': 22.38, 'rougeL': 37.25}

mbart-large-en-ro
master: 04:58 BLEU=27.83
noforce: 04:42, BLEU=28.15

pegasus-xsum
master: 56:12 {'rouge1': 46.69, 'rouge2': 24.13, 'rougeL': 38.79}
noforce: 54:15 {'rouge1': 46.98, 'rouge2': 24.43, 'rougeL': 39.11}

Commands

export DATA_DIR=wmt_en_ro
python run_eval.py facebook/mbart-large-en-ro \
    $DATA_DIR/val.source gens/mbart-enro-branch-gens.txt \
    --reference_path $DATA_DIR/val.target \
    --score_path gens/mbart-enro-master-bleu.json \
    --task translation_en_to_ro \
    --device cuda \
    --fp16 \
    --bs 32

export DATA_DIR=$CNN_DIR
python run_eval.py facebook/bart-large-cnn \
    $DATA_DIR/val.source gens/cnn_val_generations_no_force.txt \
    --reference_path $DATA_DIR/val.target \
    --score_path gens/cnn_gen_no_force_rouge.txt \
    --device cuda \
    --fp16 \
    --bs 32


export DATA_DIR=$CNN_DIR
python run_eval.py facebook/bart-large-cnn \
    $DATA_DIR/val.source gens/cnn_val_generations_master.txt \
    --reference_path $DATA_DIR/val.target \
    --score_path gens/cnn_gen_master_rouge.txt \
    --device cuda \
    --fp16 \
    --bs 32