The model was trained with python=3.11.0, pytorch=2.1.0, cuda=11.8 and builds on the DETR-variants DINO/DN/DAB and Deformable-DETR.

1. Clone this repository and create a virtual environment
2. Follow instructions to install a Pytorch version compatible with your system and CUDA version
3. Install other dependencies

   pip install -r requirements.txt

4. Compiling CUDA operators

   python src/models/dino/ops/setup.py build install # 'cuda not available', run => export CUDA_HOME=/usr/local/cuda-<version>
   # unit test (should see all checking is True) # could output an outofmemory error
   python src/models/dino/ops/test.py

Datasets should be placed in the appropriate folder specified in datasets/config.json. We preprocess the images and annotations for the IAM dataset, while all other datasets are used in their original form. For each dataset (except IAM), a charset file (.pkl) is required. Charset files can be found in the folder data.

Handwritten

1. IAM: the official website is here. We preprocess the images and annotation following the instruction in the PyLai Repository. The annotations are stored in data/IAM_new/labels.pkl.
2. RIMES: TEKLIA provide the dataset here. After downloading, place the charset file in the same folder as the dataset.
3. READ: the dataset is available here. After downloading, place the charset file in the same folder as the dataset.

Chinese The official website is here. Images and annotations are provide only in bytes format for these datasets.

1. CASIA v1: Download the dataset in bytes format with the link above and place the charset in the same folder as the dataset.
2. CASIA v2: We provide directly a version of the dataset with images (PNG) and annotations (TXT). Download the dataset here.

Ciphers The ciphers borg and copiale are available here. The charset files are provided in the folder data.

Pretrained checkpoints can be found here. The folder includes the weights of the following pretrained models:

• General model: Trained on random Latin characters. Typically used for finetuning on ciphers.
• English model: Trained on English text with random erasing. Typically used for finetuning on IAM.
• French model: Trained on French text with random erasing. Typically used for finetuning on RIMES.
• German model: Trained on German text with random erasing. Typically used for finetuning on READ.
• Chinese model: Trained on random handwritten Chinese characters from HWDB 1. Typically used for finetuning on HWDB 2.

Finetuned checkpoints can be found here.

Checkpoints should be organized as follows:

  logs/
    └── IAM/
      └── checkpoint.pth
    └── other_model/
      └── checkpoint.pth
    ...

Pretraining scipts are available in scripts/pretraining.

You need to download the folder resources (background, fonts, noises, texts) and place it in the folder dataset.

To train models with random erasing:

sh scripts/pretraining/Synthetic_english_w_masking.sh
sh scripts/pretraining/Synthetic_german_w_masking.sh
sh scripts/pretraining/Synthetic_french_w_masking.sh
sh scripts/pretraining/Synthetic_general.sh

You need the dataset CASIA v1 [here]

To train a model with random erasing

sh scripts/pretraining/Synthetic_english.sh

Then for instances to train a model for chinese with random erasing:

bash scripts/pretraining/Synthetic_chinese_w_masking.sh

Finetuning occurs in two stages. The scripts are available in scripts/finetuning.. For Step 1 it is expected that a model is pretrained is placed in the folder logs/your_model_name.

Use the scripts in scripts/evaluating to evaluate the model on the different datasets.

We provide our N-gran models for RIMES, READ and IAM here. We strongly advice to create a separate environment for the ngram model and to install the libraries in the ngram/mini_guide.md. To run an evalutation with the ngram model:

bash python ngram/clean_gen_ngram_preds.py --config_path ngram_decoder/IAM.yaml
bash python ngram/clean_gen_ngram_preds.py --config_path ngram_decoder/READ.yaml
bash python ngram/clean_gen_ngram_preds.py --config_path ngram_decoder/RIMES.yaml

To train you own ngram model, follow the instructions in the ngram/mini_guide.md