ComfyUI ControlNet Auxiliary Preprocessors

Overview

This repository provides a collection of plug-and-play node sets for creating ControlNet hint images within the ComfyUI framework. It supports the creation of various preprocessors for generating hint images like stickman, canny edge, and more. The nodes are designed to integrate seamlessly with the ControlNet architecture and are connected to resources on the 🤗 Hugging Face Hub.

Installation

Using ComfyUI Manager (Recommended)

1. Install the ComfyUI Manager.
2. Follow the steps provided in ComfyUI Manager to install this repository.

Alternative Method

If you cannot use ComfyUI Manager, follow these steps:

1. Ensure /ComfyUI/custom_nodes and the comfyui_controlnet_aux directory have write permissions.

2. For Windows users:

   • Run the install.bat file to install the package. If a portable version is detected, it will install to it; otherwise, it defaults to the system.

3. For Linux or users who can't run the install.bat:

   • Open your terminal or command prompt.
   • Navigate to the /ComfyUI/custom_nodes/ directory.
   • Run git clone https://github.com/Fannovel16/comfyui_controlnet_aux/.
   • Inside the comfyui_controlnet_aux directory:
     • For portable/venv: Run path/to/ComfUI/python_embeded/python.exe -s -m pip install -r requirements.txt
     • For system Python: Run pip install -r requirements.txt
   • Start ComfyUI.

Repository Purpose

The primary purpose of this repository is to offer a suite of nodes for generating hint images for ControlNet, a framework that supports conditional generation in neural networks. It facilitates the creation of images based on structural or textural hints, thereby enhancing the capabilities of the ComfyUI interface.

Nodes Provided

The repository includes a variety of preprocessing nodes, categorized into sections such as Line Extractors, Normal and Depth Estimators, Faces and Poses Estimators, Optical Flow Estimators, Semantic Segmentation, T2IAdapter-only, and Recolor.

Important Node Categories

1. Line Extractors

   • Binary Lines
   • Canny Edge
   • HED Soft-Edge Lines
   • More...

2. Normal and Depth Estimators

   • MiDaS Depth Map
   • LeReS Depth Map
   • Zoe Depth Map
   • More...

3. Faces and Poses Estimators

   • DWPose Estimator
   • OpenPose Estimator
   • More...

4. Optical Flow Estimators

   • Unimatch Optical Flow

5. Semantic Segmentation

   • OneFormer ADE20K Segmentor
   • UniFormer Segmentor

6. Recolor

   • Image Luminance

Special Features

Integration and Flexibility

• The nodes in this repository work seamlessly within ComfyUI and are designed to support a wide range of operations from simple edge detection to complex depth map processing.
• Most preprocessors are integrated into the AIO Aux Preprocessor node, allowing users to switch between them quickly.

GPU Acceleration

• Some nodes, like DWPose and AnimalPose, can leverage GPU acceleration for faster processing via TorchScript checkpoints or ONNXRuntime.

JSON Output for OpenPose

• This repository provides functionality to extract OpenPose-format JSON data, which can be used to create complex workflows in ComfyUI.

Usefulness in ComfyUI Workflows

The nodes provided by this repository are essential for users looking to expand the capabilities of ComfyUI when working with ControlNet. They allow for the generation of complex hint images and enable the integration of additional features like semantic segmentation and pose estimation. By facilitating these diverse functions, the repository enhances the creative and practical possibilities available in ComfyUI workflows.