Official code for "Controllable Talking Face Generation by Implicit Facial Keypoints Editing"

ControlTalk: A talking face generation method to control face expression deformation based on driven audio, constructing the head pose and facial expression (lip motion) for both single image or sequential video inputs in a unified manner.

• 2024/10/31: Inference code is now available!

# Create a python 3.10 conda env (you could also use virtualenv)
conda env create -f environment.yml

• Download pretrained models from huggingface detailed guidance.

# Download hubert model
https://huggingface.co/TencentGameMate/chinese-hubert-large

# Download our pretrained model 
https://huggingface.co/Lavivis/ControlTalk

• Put all pretrained models in ./checkpoints, the file structure should be like:

checkpoints
├── audio_encoder.pt
├── lipControlNet.pt
├── 20231128_210236_337a_e0362-checkpoint.pth.tar
├── TencentGameMate
├───└──chinese-hubert-large
├─────────└──config.json
├─────────└──pytorch_model.bin
├─────────└──preprocessor_config.json
└─────────└──chinese-hubert-large-fairseq-ckpt.pt

python inference.py \
        --source_video './data/drive_video.mp4' \
        --source_img_path  './data/example.png' \
        --audio './data/drive_audio.wav' \
        --save_as_video \
        --box -1 0 0 0 \
        # --img_mode   # if you only want to control the face expression

Coming soon!

• chinese_speech_pretrain
• face-vid2vid
• face-vid2vid (Unofficial implementation)

If our work and codebase is useful for you, please cite as:

@article{zhao2024controllable,
  title={Controllable Talking Face Generation by Implicit Facial Keypoints Editing},
  author={Zhao, Dong and Shi, Jiaying and Li, Wenjun and Wang, Shudong and Xu, Shenghui and Pan, Zhaoming},
  journal={arXiv preprint arXiv:2406.02880},
  year={2024}
}

Our code is released under MIT License.