This code is for EMNLP 2019 paper Text Summarization with Pretrained Encoders
Results on CNN/DailyMail (20/8/2019):
| Models | ROUGE-1 | ROUGE-2 | ROUGE-L |
|---|---|---|---|
| Extractive | |||
| TransformerExt | 40.90 | 18.02 | 37.17 |
| BertSumExt | 43.23 | 20.24 | 39.63 |
| BertSumExt (large) | 43.85 | 20.34 | 39.90 |
| Abstractive | |||
| TransformerAbs | 40.21 | 17.76 | 37.09 |
| BertSumAbs | 41.72 | 19.39 | 38.76 |
| BertSumExtAbs | 42.13 | 19.60 | 39.18 |
Python version: This code is in Python3.6
Package Requirements: torch==1.1.0 pytorch_transformers tensorboardX multiprocess pyrouge
Some codes are borrowed from ONMT(https://github.com/OpenNMT/OpenNMT-py)
unzip the zipfile and put all .pt files into bert_data
Download and unzip the stories directories from here for both CNN and Daily Mail. Put all .story files in one directory (e.g. ../raw_stories)
We will need Stanford CoreNLP to tokenize the data. Download it here and unzip it. Then add the following command to your bash_profile:
export CLASSPATH=/path/to/stanford-corenlp-full-2017-06-09/stanford-corenlp-3.8.0.jar
replacing /path/to/ with the path to where you saved the stanford-corenlp-full-2017-06-09 directory.
python preprocess.py -mode tokenize -raw_path RAW_PATH -save_path TOKENIZED_PATH
RAW_PATHis the directory containing story files (../raw_stories),JSON_PATHis the target directory to save the generated json files (../merged_stories_tokenized)
python preprocess.py -mode format_to_lines -raw_path RAW_PATH -save_path JSON_PATH -n_cpus 1 -use_bert_basic_tokenizer false -map_path MAP_PATH
RAW_PATHis the directory containing tokenized files (../merged_stories_tokenized),JSON_PATHis the target directory to save the generated json files (../json_data/cnndm),MAP_PATHis the directory containing the urls files (../urls)
python preprocess.py -mode format_to_bert -raw_path JSON_PATH -save_path BERT_DATA_PATH -lower -n_cpus 1 -log_file ../logs/preprocess.log
JSON_PATHis the directory containing json files (../json_data),BERT_DATA_PATHis the target directory to save the generated binary files (../bert_data)
First run: For the first time, you should use single-GPU, so the code can download the BERT model. Use -visible_gpus -1, after downloading, you could kill the process and rerun the code with multi-GPUs.
python train.py -task ext -mode train -bert_data_path BERT_DATA_PATH -ext_dropout 0.1 -model_path MODEL_PATH -lr 2e-3 -visible_gpus 0,1,2 -report_every 50 -save_checkpoint_steps 1000 -batch_size 3000 -train_steps 50000 -accum_count 2 -log_file ../logs/ext_bert_cnndm -use_interval true -warmup_steps 10000 -max_pos 512
python train.py -task abs -mode train -bert_data_path BERT_DATA_PATH -dec_dropout 0.2 -model_path MODEL_PATH -sep_optim true -lr_bert 0.002 -lr_dec 0.2 -save_checkpoint_steps 2000 -batch_size 140 -train_steps 200000 -report_every 50 -accum_count 5 -use_bert_emb true -use_interval true -warmup_steps_bert 20000 -warmup_steps_dec 10000 -max_pos 512 -visible_gpus 0,1,2,3 -log_file ../logs/abs_bert_cnndm
python train.py -task abs -mode train -bert_data_path BERT_DATA_PATH -dec_dropout 0.2 -model_path MODEL_PATH -sep_optim true -lr_bert 0.002 -lr_dec 0.2 -save_checkpoint_steps 2000 -batch_size 140 -train_steps 200000 -report_every 50 -accum_count 5 -use_bert_emb true -use_interval true -warmup_steps_bert 20000 -warmup_steps_dec 10000 -max_pos 512 -visible_gpus 0,1,2,3 -log_file ../logs/abs_bert_cnndm -load_from_extractive EXT_CKPT
EXT_CKPTis the saved.ptcheckpoint of the extractive model.
python train.py -task abs -mode validate -batch_size 3000 -test_batch_size 500 -bert_data_path BERT_DATA_PATH -log_file ../logs/val_abs_bert_cnndm -model_path MODEL_PATH -sep_optim true -use_interval true -visible_gpus 1 -max_pos 512 -max_length 200 -alpha 0.95 -min_length 50 -result_path ../logs/abs_bert_cnndm
-modecan be {validate, test}, wherevalidatewill inspect the model directory and evaluate the model for each newly saved checkpoint,testneed to be used with-test_from, indicating the checkpoint you want to useMODEL_PATHis the directory of saved checkpoints- use
-mode valiadtewith-test_all, the system will load all saved checkpoints and select the top ones to generate summaries (this will take a while)