- Home
- AI Video Lip Sync Tools
- Diff2Lip
Diff2Lip
What it is
Diff2Lip is an open-source Linux and self-hosted project that uses an audio-conditioned diffusion model to synchronize speech with face video. It accepts an audio source and face video, producing lip-synced video while the project says generated frames preserve the person’s identity. The repository includes inference scripts for audio-video pairs and for a single video. Cross mode drives a video with separate audio, while reconstruction mode drives the video’s first frame with an audio source. Listed applications include movies, education, and virtual avatars; video conferencing is described as a possible future use. Diff2Lip is not real time yet. The project reports reconstruction and cross-audio-video results on VoxCeleb2 and LRW and identifies the work as a WACV 2024 paper. Setup uses Python 3.9, FFmpeg 5.0.1, and the requirements file. Inference supports one or more GPUs through NUM_GPUS, with values above one running distributed generation. A Google Colab notebook is linked for trying it. The code and text use CC BY-NC 4.0, which excludes commercial use.
For and against
For
- Supports cross and reconstruction inference modes
- Preserves identity according to the project
- Provides scripts for pairs or single videos
- Can run distributed generation across GPUs
- Offers a Google Colab notebook
Against
- It is not real time yet
- Commercial use is not permitted by the license
- Setup requires Python 3.9 and FFmpeg 5.0.1
The verdict
Diff2Lip provides configurable speech-to-video lip synchronization with documented inference modes and a noncommercial license. Its offline, GPU-oriented workflow is suited to experimentation and production pipelines that do not require real-time output or commercial licensing.
Who it suits
Diff2Lip suits developers or researchers working with self-hosted lip-sync generation on Linux. It may fit noncommercial movie, education, or avatar projects that can run offline inference.
On the record
- Access
- The repository links a Google Colab notebook to try Diff2Lip.github.com · read 10 Oct 2026
- Applications
- The project lists movies, education, and virtual avatars as applications, and video conferencing as a possible future application.soumik-kanad.github.io · read 4 Oct 2026
- Commercial use
- The repository's license description says the work may not be used for commercial purposes.github.com · read 4 Oct 2026
- Datasets
- The paper says the model was trained on VoxCeleb2 and reports results on VoxCeleb2 and LRW.arxiv.org · read 10 Oct 2026
- GPU requirement
- The inference instructions include a NUM_GPUS setting and say values greater than one run distributed generation.github.com · read 4 Oct 2026
- Identity preservation
- The project says its generated video frames preserve identity without identity loss.soumik-kanad.github.io · read 4 Oct 2026
- Inference
- The repository includes scripts for lip-sync inference on audio-video pairs and on a single video.github.com · read 4 Oct 2026
- Inference inputs
- The repository includes inference scripts for audio-video pairs and supports running on a single video with specified video, audio, and output paths.github.com · read 10 Oct 2026
- Inference modes
- The repository documents cross mode to drive a video with an audio source and reconstruction mode to drive the video's first frame with an audio source.github.com · read 4 Oct 2026
- Input and output
- The project describes its task as turning arbitrary speech and face videos into high-quality lip-synced video.github.com · read 4 Oct 2026
- Integrations
- The repository links Google Colab for trying the project; it does not list other product integrations.github.com · read 10 Oct 2026
- License
- The repository states that its text and code are under CC BY-NC 4.0, with attribution required and commercial use prohibited.github.com · read 10 Oct 2026
- Method
- It uses an audio-conditioned diffusion model for lip synchronization in the wild.soumik-kanad.github.io · read 10 Oct 2026
- Not real time
- The project states that Diff2Lip is not real time yet.soumik-kanad.github.io · read 4 Oct 2026
- Purpose
- Diff2Lip generates lip-synchronized videos from speech audio and face videos.github.com · read 10 Oct 2026
- Quality goals
- The project describes its approach as preserving identity, pose, emotions, and image quality while generating lip movements.arxiv.org · read 10 Oct 2026
- Requirements
- The setup instructions use Python 3.9 and FFmpeg 5.0.1, and inference can use one or more GPUs.github.com · read 10 Oct 2026
- Research publication
- The repository identifies Diff2Lip as a WACV 2024 paper.github.com · read 4 Oct 2026
