- Home
- AI Video Lip Sync Tools
- LatentSync
LatentSync
What it is
LatentSync is an open-source method for aligning video lip movements with audio, built around audio-conditioned latent diffusion models. It uses Whisper to turn mel spectrograms into audio embeddings, which are fed into a U-Net through cross-attention layers. The project offers inference through a Gradio app or command-line interface. Version 1.6 was trained at 512 × 512 resolution to address blurry teeth and lips; stated minimum inference memory is 18 GB of VRAM for that version and 8 GB for version 1.5. Increasing inference steps can improve visual quality but slows generation, while a higher guidance scale can improve lip-sync accuracy and may cause distortion or jitter. The open-source release includes inference code and checkpoints, a data processing pipeline, and training code. U-Net training configurations call for 20–55 GB of VRAM, depending on setup. The documented pipeline uses PySceneDetect for scene detection and InsightFace landmarks to transform faces. LatentSync is free and self-hosted for Linux. Its repository lists Apache-2.0, while the Hugging Face model card lists openrail++; LatentSync 1.6 is not deployed by an Inference Provider.
For and against
For
- Free open-source release includes inference and training code.
- Offers both a Gradio app and command-line inference.
- Version 1.6 targets blurry teeth and lips.
- Documented pipeline includes scene detection and face landmarks.
Against
- Inference requires at least 8 GB VRAM for version 1.5.
- Version 1.6 requires at least 18 GB VRAM for inference.
- U-Net training configurations require 20–55 GB VRAM.
- LatentSync 1.6 is not deployed by an Inference Provider.
The verdict
LatentSync provides inference and training resources for self-hosted lip-sync work, with separate VRAM requirements by version and configuration. Its generation settings involve trade-offs between quality, speed, accuracy, and possible distortion or jitter.
Who it suits
LatentSync suits Linux users who want to run lip-sync inference themselves or work with the released training resources. Its stated inference memory requirements and 20–55 GB training configurations are important to check against available hardware.
Plans and prices
Open-source release
Inference code and checkpoints · data processing pipeline · training code
Read 8 Oct 2026 · github.com
On the record
- Audio conditioning
- Whisper converts mel-spectrograms into audio embeddings that enter the U-Net through cross-attention.raw.githubusercontent.com · read 21 Oct 2026
- Community complaint: GPU compatibility
- Open issues include requests for RTX 5090 support and reports that older PyTorch versions do not work with 50-series Nvidia cards.github.com · read 21 Oct 2026
- Community complaint: output quality
- Open issues include reports of red faces, blurry mouths, face misalignment and visible teeth artifacts.github.com · read 21 Oct 2026
- Community issue volume
- The GitHub issue tracker showed 211 open issues and 16 pull requests when opened.github.com · read 21 Oct 2026
- Data pipeline
- Its processing pipeline resamples video to 25 FPS and audio to 16 kHz, detects scenes, splits clips into 5–10 second segments, aligns faces and filters low-quality samples.raw.githubusercontent.com · read 21 Oct 2026
- Data processing tools
- The documented data pipeline uses PySceneDetect for scene detection and InsightFace landmarks to transform faces.github.com · read 8 Oct 2026
- Dependencies
- Pinned dependencies include PyTorch 2.5.1, Diffusers 0.32.2, Transformers 4.48.0, Gradio 5.24.0 and GPU ONNX Runtime 1.21.0.github.com · read 21 Oct 2026
- Developer extensibility
- The repository exposes scripts for preprocessing, inference, U-Net training and file-list generation, enabling custom training workflows.raw.githubusercontent.com · read 21 Oct 2026
- Efficient training tradeoff
- The 20 GB efficient Stage 2 configuration may slightly reduce visual quality and temporal consistency compared with the 30 GB configuration.raw.githubusercontent.com · read 21 Oct 2026
- Environment
- The setup script creates a Conda environment with Python 3.10.13, installs FFmpeg and requires the Linux package libgl1.github.com · read 21 Oct 2026
- Generation trade-off
- The README says more inference steps can improve visual quality but slow generation, while higher guidance scale can improve lip-sync accuracy but cause distortion or jitter.github.com · read 8 Oct 2026
- Hardware requirement
- The stated minimum inference VRAM is 8 GB for LatentSync 1.5 and 18 GB for LatentSync 1.6.github.com · read 8 Oct 2026
- Hosted API inputs
- Layer's schema requires at least one source-video file and one reference-audio file and supports seed and guidance-scale fields.layer.ai · read 21 Oct 2026
- Hosted API option
- Layer documents a POST inference endpoint that accepts a source video and reference audio through bearer-authenticated JSON requests.layer.ai · read 21 Oct 2026
- Hugging Face availability
- The model page provides Diffusers usage instructions and links to Google Colab and Kaggle notebooks.huggingface.co · read 21 Oct 2026
- Independent description
- ExplainGit describes LatentSync as a ByteDance research tool that replaces a video's lip movements to match a new audio track using Stable Diffusion and Whisper.explaingit.com · read 21 Oct 2026
- Independent setup assessment
- ExplainGit rates setup difficulty as hard and estimates more than one hour to reach a first run.explaingit.com · read 21 Oct 2026
- Inference interfaces
- The repository provides both a local Gradio application and a command-line inference script.raw.githubusercontent.com · read 21 Oct 2026
