LatentSyncNode Documentation

Overview

The LatentSyncNode is a component of the ComfyUI-LatentSyncWrapper, an unofficial implementation that integrates ByteDance's LatentSync 1.6 model into ComfyUI. This node provides advanced lip-sync capabilities, enabling users to synchronize video lips with audio inputs. The enhancements in version 1.6 focus on improved visual quality with clearer and more detailed lip and teeth generation, addressing issues found in previous versions.

Features

• Enhanced Resolution: Utilizes training at 512×512 resolution for sharper details.
• Improved Visual Quality: Reduces blurriness in teeth and lips.
• Backward Compatibility: Compatible with LatentSync 1.5 models.
• Optimized Performance: Offers improved temporal consistency, better support for Chinese videos, and reduced VRAM requirements.

Inputs

The LatentSyncNode accepts the following inputs:

1. Images: A batch of video frames that will be processed for lip-sync.
2. Audio: An audio waveform to synchronize with the video frames.
3. Seed: An integer used for maintaining reproducible results. Default is 1247.
4. Lips Expression: A float value controlling the expressiveness of lip movements. Default is 1.5.
   • Higher values (e.g., 2.0-3.0) provide more pronounced lip movements.
   • Lower values (e.g., 1.0-1.5) result in subtler movements.
5. Inference Steps: The number of denoising steps during inference. Default is 20.
   • Higher values (30-50) offer better quality at the expense of speed.
   • Lower values (10-15) provide faster but possibly lower-quality results.

Outputs

The LatentSyncNode produces the following outputs:

1. Images: A batch of processed video frames synchronized with the audio input.
2. Audio: The same audio input, potentially processed for length adjustment.

Usage in ComfyUI Workflows

To use the LatentSyncNode in ComfyUI workflows, follow these general steps:

1. Prepare Inputs: Select and load video frames and an audio file.
2. Set Parameters: Optionally set the seed, lips_expression, and inference_steps to control output quality and characteristics.
3. Connect to the Node: Integrate the node into the ComfyUI workflow where synchronization between video and audio is needed.
4. Run the Workflow: Execute the workflow, and the node will process the input data, producing synchronized video and audio outputs.

Special Features and Considerations

• Supports High-Resolution Videos: The node is optimized for 512×512 resolution, significantly enhancing the clarity of the output.
• Temporal Consistency: Improved temporal layers ensure stable lip-sync across frames.
• GPU Optimization: Designed to run efficiently on GPUs with 20GB VRAM, such as the RTX 3090.
• Language Support: Enhanced performance for Chinese language videos.
• File Management: Temporary files are automatically managed and cleaned up to prevent storage issues.

Known Limitations

• Face Visibility: Works best with clear, frontal face videos where the face remains visible throughout.
• Frame Rate: Best results are achieved with videos at 25 FPS.
• Anime/Cartoon Faces: Not currently supported for non-photorealistic faces.
• Manual Model Downloads: Models must be manually downloaded due to hosting on a private HuggingFace repository. A pre-configured package is available via Google Drive.

Support

For support or further information, visit the project repository or the official LatentSync page.

This documentation should provide users with a clear understanding of how to utilize the LatentSyncNode within their ComfyUI workflows to achieve advanced lip-sync capability in their video projects.