The official pytorch implementation of UCPM: Uncertainty-Guided Cross-Modal Retrieval with Partially Mismatched Pairs (submitted to IEEE TIP).
- Python 3.8
- PyTorch 1.20.0
- numpy
- scikit-learn
- Punkt Sentence Tokenizer:
import nltk
nltk.download()
> d punkt
(Optional) if the above download failed, you can manually download it from here. The directory structure is:
/home/username/
├── nltk_data
│ ├── tokenizers
│ ├── punkt
│ ├── czech.pickle
│ ├── french.pickle
│ ├── polish.pickle
│ ├── ......
Our directory structure of data.
data
├── f30k_precomp # pre-computed BUTD region features for Flickr30K, provided by SCAN
│ ├── train_ids.txt
│ ├── train_caps.txt
│ ├── ......
│
├── coco_precomp # pre-computed BUTD region features for COCO, provided by SCAN
│ ├── train_ids.txt
│ ├── train_caps.txt
│ ├── ......
│
├── cc152k_precomp # pre-computed BUTD region features for cc152k, provided by NCR
│ ├── train_ids.txt
│ ├── train_caps.tsv
│ ├── ......
│
└── vocab # vocab files provided by SCAN and NCR
├── f30k_precomp_vocab.json
├── coco_precomp_vocab.json
└── cc152k_precomp_vocab.json
We follow NCR to obtain image features and vocabularies. Download Dataset
If you want to experiment with the same noise index as in this paper, the noise index files can be downloaded from here.
Coming soon
The pre-trained models are available here:
- CC152K model Download
- F30k 20% MRate model Download
- F30k 40% MRate model Download
- F30k 60% MRate model Download
- F30k 80% MRate model Download
- COCO 20% MRate model Download
- COCO 40% MRate model Download
- COCO 60% MRate model Download
- COCO 80% MRate model Download
Modify the data_path, vocab_path, and model_paths in the eval.py file and run it.
python eval.py
