ComfyUI EchoMimic Custom Nodes

Introduction

The ComfyUI EchoMimic repository provides custom nodes that integrate the EchoMimic and EchoMimic V2 frameworks into ComfyUI. These frameworks enable lifelike audio-driven portrait animations and human animations with semi-body motion through the use of editable landmarks and advanced animation techniques.

Repository URL

https://github.com/smthemex/ComfyUI_EchoMimic

Installation

Step 1: Clone the Repository

Navigate to the ./ComfyUI/custom_node directory and clone the repository using the following command:

git clone https://github.com/smthemex/ComfyUI_EchoMimic.git

Step 2: Install Requirements

Install the required Python packages using the provided requirements.txt:

pip install -r requirements.txt

Additional Installation Notes

• To use EchoMimic V1, you may need to install facenet-pytorch:

  pip install --no-deps facenet-pytorch
  

• If you encounter issues with the installation of this package leading to crashes, follow the troubleshooting steps provided in the README to uninstall and reinstall specific dependencies.
• Install any missing modules manually using pip install <missing_module>.
• Conda users and those using Python >3.12 may need to downgrade Python to 3.11.9 for compatibility.

Node Mappings

This repository contains the following nodes:

• Echo_LoadModel
• Echo_Sampler

Models Required

Download and set up the appropriate models from the following sources:

Shared Models for V1 and V2

• unet: Hugging Face link
• VAE: Hugging Face link
• Audio Models: EchoMimic audio data

V1 Models

Retrieve models specific to EchoMimic V1 from Hugging Face link.

V2 Models

Download EchoMimic V2 models using the link provided in the README or add them manually to your setup as needed.

Additional Models

For advanced functionalities, download YOLO and sapiens pose models from Ultralytics and Facebook.

Features and Capabilities

ComfyUI EchoMimic provides a range of features and capabilities:

• Pose-Driven and Audio-Driven Animations: Use both audio and pose input data to generate lifelike animations.
• Custom Video Drivers: Integrate video drivers for enhanced control over animation output.
• INFER Modes: Utilize multiple inference modes for animations - audio-driven, pose-driven, and reference PKL models.
• Automatic Download: The system can automatically download required model files if internet connectivity is sufficient.
• Low VRAM Mode: A low memory option is available for users with less GPU memory, though it may increase processing times.

Example Usage in ComfyUI Workflows

The EchoMimic nodes can be incorporated into workflows for creating lifelike human animations in ComfyUI. Using the nodes, users can:

• Automate alignment of input images.
• Load and process animations from custom video sources.
• Generate animations based on pose data and PKL models.
• Use motion sync to extract facial features directly from video inputs.
• Choose special options like video saving, frame count, and more to tailor animations to specific needs.

These capabilities make the EchoMimic repository a powerful tool for developers and designers working with animated content in ComfyUI.