ComfyUI-LatentSyncWrapper

Overview

The ComfyUI-LatentSyncWrapper repository provides an implementation of the LatentSync 1.6 model for ComfyUI, enhancing video lip-sync capabilities by synchronizing lip movements with audio inputs. The repository focuses on improving clarity and resolution in lip-sync outputs, resolving issues seen in previous versions. The node is particularly useful for creating high-quality video outputs where precise lip-syncing is required.

Installation

Prerequisites

Before installing the ComfyUI-LatentSyncWrapper, ensure that you have the following software and tools installed:

1. ComfyUI: Make sure you have ComfyUI installed and working correctly. You can find it here.
2. FFmpeg: Install FFmpeg on your system. For Windows, you can download it here and add it to your system PATH.

Installing the Node

Follow these steps to install the ComfyUI-LatentSyncWrapper:

1. Navigate to your ComfyUI/custom_nodes directory.

   cd ComfyUI/custom_nodes
   

2. Clone the repository into the custom_nodes directory.

   git clone https://github.com/ShmuelRonen/ComfyUI-LatentSyncWrapper.git
   

3. Move into the repository directory and install the required dependencies.

   cd ComfyUI-LatentSyncWrapper
   pip install -r requirements.txt
   

Manual Model Downloads

To use this node, you must manually download certain models, as the LatentSync 1.6 models are on a private HuggingFace repository that cannot be accessed automatically.

Option 1: Complete Checkpoints Package (Recommended)

1. Download the complete pre-configured checkpoints folder via Google Drive:

   • Link to download

2. Extract the archive to retrieve the complete checkpoints directory.

3. Replace or merge it with your existing checkpoints folder in the extension directory.

Option 2: Manual Individual Downloads

If preferred, download models individually by following these steps:

1. VAE Model Download

   • Create a vae folder inside your checkpoints directory:

     mkdir checkpoints/vae
     

   • Visit this HuggingFace page and download diffusion_pytorch_model.safetensors and config.json.

   • Place the downloaded files inside the checkpoints/vae/ folder.

2. LatentSync 1.6 Checkpoints Download

   • Ensure you have access to the ByteDance/LatentSync-1.6 HuggingFace repository.

   • Download all files from the repository and place them directly in the checkpoints/ folder.

After downloading models, your checkpoint directory should look like:

./checkpoints/
|-- .cache/
|-- auxiliary/
|-- vae/
|   |-- config.json
|   `-- diffusion_pytorch_model.safetensors
|-- whisper/
|   `-- tiny.pt
|-- config.json
|-- latentsync_unet.pt
|-- stable_syncnet.pt

Node Descriptions

This repository provides the following nodes:

LatentSyncNode

The LatentSyncNode is responsible for synchronizing video lips with audio inputs with high clarity and resolution.

VideoLengthAdjuster

The VideoLengthAdjuster is used to adjust the video length to ensure smooth synchronization.

Features and Capabilities

• Enhanced Resolution Training: Trained on 512×512 resolution videos for improved detail, especially in teeth and lips.
• Improved Visual Quality: Addresses blurriness issues seen in previous versions, enhancing the clarity of outputs.
• Backward Compatibility: Works with both LatentSync 1.5 and 1.6, requiring only checkpoint changes.
• Reduced VRAM Requirements: Compatible with GPUs with 20GB VRAM, such as RTX 3090.
• Flexibility: Allows for adjustments in lip movement intensity and inference steps to balance quality and speed.
• Language Support: Improved support for Chinese language videos.

Usage in ComfyUI Workflows

The ComfyUI-LatentSyncWrapper can be integrated into workflows to produce high-quality lip-synced videos. Here’s how it might be used:

1. Video and Audio Selection: Use ComfyUI’s video and audio loader nodes to select the input files.
2. Parameter Adjustments: Adjust parameters such as lips_expression and inference_steps to suit the specific requirements of the project.
3. Processing: Connect to the LatentSync1.6 node and execute the workflow to obtain the processed video output.
4. Output: The processed video will be saved in ComfyUI's output directory, ready for further use.

This repository is particularly useful for projects requiring precise lip-syncing and high-quality video outputs, such as presentations, video content creation, and more.