- Home
- AI Video Lip Sync Tools
- DreamTalk
On the recordRead 9 Oct 202624 facts
DreamTalk
On the record
- Checkpoint access
- Public checkpoint downloads have been discontinued; the README asks researchers to request checkpoints by email and states that doing so implies consent to use them solely for academic research.github.com · read 9 Oct 2026
- CPU support
- The repository says the program can run on CPU by adding the --device=cpu argument.github.com · read 4 Oct 2026
- Duration limit
- The max_gen_len setting controls the maximum generated video duration in seconds, and longer input audio is truncated.github.com · read 9 Oct 2026
- Expressions
- Its style predictor predicts a target expression from audio without requiring an expression reference video or text.dreamtalk-project.github.io · read 9 Oct 2026
- Generation approach
- The framework combines a denoising network, a style-aware lip expert, and a style predictor.dreamtalk-project.github.io · read 4 Oct 2026
- How it works
- The framework combines a denoising network, a style-aware lip expert, and a style predictor.dreamtalk-project.github.io · read 9 Oct 2026
- Inference inputs
- Inference accepts WAV, MP3, M4A, or MP4 audio, a portrait image, and reference 3DMM sequences for speaking style and head pose.github.com · read 9 Oct 2026
- Input audio
- The inference instructions list WAV, MP3, M4A, and MP4 files with sound as compatible audio inputs.github.com · read 4 Oct 2026
- Input coverage
- The project reports generalization across songs, noisy audio, out-of-domain portraits, and speech in Chinese, French, German, Italian, Japanese, Korean, and Spanish.dreamtalk-project.github.io · read 9 Oct 2026
- Languages and audio
- The project page lists speech examples in Chinese, French, German, Italian, Japanese, Korean, and Spanish, as well as songs and noisy audio.dreamtalk-project.github.io · read 4 Oct 2026
- License
- The repository includes an MIT license, while its README says the method is intended for research and non-commercial use only.github.com · read 9 Oct 2026
- Maintainers
- The project page lists authors affiliated with Tsinghua University, Alibaba Group, and Huazhong University of Science and Technology.dreamtalk-project.github.io · read 9 Oct 2026
- Motion references
- Speaking-style and head-pose references use 3DMM parameter sequences extracted from videos, and the stated video frame rate is 25 FPS.github.com · read 4 Oct 2026
- Output
- The inference script saves the generated video as an MP4 file in the output_video folder.github.com · read 4 Oct 2026
- Portrait requirements
- The input portrait should be larger than 256×256 pixels; the instructions say frontal portraits usually produce satisfactory results and are cropped to 256×256.github.com · read 4 Oct 2026
- Purpose
- DreamTalk is a diffusion-based framework for generating expressive talking-head videos from audio.dreamtalk-project.github.io · read 9 Oct 2026
- Resolution
- The README says resolution was not considered in the initial design and describes optional super-resolution methods with tradeoffs in speed, temporal consistency, or facial emotion intensity.github.com · read 9 Oct 2026
- Setup
- The repository provides inference code and installation instructions for a Python 3.7 environment with PyTorch 1.8.0 and CUDA 11.1 packages.github.com · read 9 Oct 2026
