Update: In case you are looking for Wav2Lip, it is in https://github.com/Rudrabha/Wav2Lip
Generate high quality speech from only lip movements. This code is part of the paper: Learning Individual Speaking Styles for Accurate Lip to Speech Synthesis published at CVPR'20.
[Paper] | [Project Page] | [Demo Video]
- Dataset and Pre-trained models for all speakers are released!
- Pre-trained model for multi-speaker word-level Lip2Wav model trained on the LRW dataset is released! (multispeaker branch)
- First work to generate intelligible speech from only lip movements in unconstrained settings.
- Sequence-to-Sequence modelling of the problem.
- Dataset for 5 speakers containing 100+ hrs of video data made available! [Dataset folder of this repo]
- Complete training code and pretrained models made available.
- Inference code to generate results from the pre-trained models.
- Code to calculate metrics reported in the paper is also made available.
🎉 Lip-sync talking face videos to any speech using Wav2Lip: https://github.com/Rudrabha/Wav2Lip
Python 3.7.4
(code has been tested with this version)- ffmpeg:
sudo apt-get install ffmpeg
- Install necessary packages using
pip install -r requirements.txt
- Face detection pre-trained model should be downloaded to
face_detection/detection/sfd/s3fd.pth
. Alternative link if the above does not work.
Speaker | Link to the model |
---|---|
Chemistry Lectures | Link |
Chess Commentary | Link |
Hardware-security Lectures | Link |
Deep-learning Lectures | Link |
Ethical Hacking Lectures | Link |
The dataset is present in the Dataset folder in this repository. The folder Dataset/chem
contains .txt
files for the train, val and test sets.
data_root (Lip2Wav in the below examples)
├── Dataset
| ├── chess, chem, dl (list of speaker-specific folders)
| | ├── train.txt, test.txt, val.txt (each will contain YouTube IDs to download)
To download the complete video data for a specific speaker, just run:
sh download_speaker.sh Dataset/chem
This should create
Dataset
├── chem (or any other speaker-specific folder)
| ├── train.txt, test.txt, val.txt
| ├── videos/ (will contain the full videos)
| ├── intervals/ (cropped 30s segments of all the videos)
python preprocess.py --speaker_root Dataset/chem --speaker chem
Additional options like batch_size
and number of GPUs to use can also be set.
python complete_test_generate.py -d Dataset/chem -r Dataset/chem/test_results \
--preset synthesizer/presets/chem.json --checkpoint <path_to_checkpoint>
#A sample checkpoint_path can be found in hparams.py alongside the "eval_ckpt" param.
This will create:
Dataset/chem/test_results
├── gts/ (cropped ground-truth audio files)
| ├── *.wav
├── wavs/ (generated audio files)
| ├── *.wav
You can calculate the PESQ
, ESTOI
and STOI
scores for the above generated results using score.py
:
python score.py -r Dataset/chem/test_results
python train.py <name_of_run> --data_root Dataset/chem/ --preset synthesizer/presets/chem.json
Additional arguments can also be set or passed through --hparams
, for details: python train.py -h
The software is licensed under the MIT License. Please cite the following paper if you have use this code:
@InProceedings{Prajwal_2020_CVPR,
author = {Prajwal, K R and Mukhopadhyay, Rudrabha and Namboodiri, Vinay P. and Jawahar, C.V.},
title = {Learning Individual Speaking Styles for Accurate Lip to Speech Synthesis},
booktitle = {The IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2020}
}
The repository is modified from this TTS repository. We thank the author for this wonderful code. The code for Face Detection has been taken from the face_alignment repository. We thank the authors for releasing their code and models.