VideoX-Fun ComfyUI Custom Nodes

Introduction

The VideoX-Fun repository is designed to integrate advanced video generation and transformation capabilities into ComfyUI. It allows users to create text-to-video, image-to-video, and video-to-video transformations using state-of-the-art AI models. The repository also includes tools to train Lora models and apply various control conditions like Canny, Depth, and Pose for enhanced video creation and control.

Installation

To use the custom nodes from the VideoX-Fun repository in ComfyUI, you can clone the repository and follow instructions for setting up your environment. This includes downloading necessary model weights and ensuring dependencies are installed, as specified in the repository.

Steps:

1. Clone the Repository:

   git clone https://github.com/aigc-apps/VideoX-Fun.git
   

2. Enter the Directory:

   cd VideoX-Fun
   

3. Download Necessary Weights: Follow instructions to download the required model weights, either through Hugging Face or Model Scope links provided in the repository.
4. Configure Your Environment: Ensure your environment meets the specified dependencies in terms of Python version, PyTorch, CUDA, CUDNN, etc.
5. Launch ComfyUI with custom nodes integrated from the VideoX-Fun directory.

Purpose

VideoX-Fun serves as a comprehensive video generation pipeline. It allows users to generate AI-based videos and images, train models like Lora, and perform style transformations. The repository supports multiple resolutions, durations, and FPS, making it versatile for different video generation needs.

Nodes Provided

This repository offers a variety of nodes for integration with ComfyUI:

• FunTextBox
• FunRiflex
• FunCompile
• LoadCogVideoXFunModel
• LoadCogVideoXFunLora
• CogVideoXFunT2VSampler
• CogVideoXFunInpaintSampler
• CogVideoXFunV2VSampler
• LoadWanModel
• LoadWanLora
• WanT2VSampler
• WanI2VSampler
• LoadWanFunModel
• LoadWanFunLora
• WanFunT2VSampler
• WanFunInpaintSampler
• WanFunV2VSampler
• VideoToCanny
• VideoToDepth
• VideoToOpenpose
• CreateTrajectoryBasedOnKJNodes
• CameraBasicFromChaoJie
• CameraTrajectoryFromChaoJie
• CameraJoinFromChaoJie
• CameraCombineFromChaoJie
• ImageMaximumNode

Special Features and Capabilities

• Multi-resolution Video Generation: Supports flexible resolutions such as 256x256 to 1024x1024 for diverse video needs.
• Control and Reference Conditions: Use of Canny, Depth, Pose, and more as control conditions, allowing for unique style applications and video manipulations.
• GPU Memory Optimization: Implements several strategies to manage large model operations on consumer-grade GPUs including model_cpu_offload and sequential_cpu_offload.
• Training Capabilities: Provides an infrastructure for training Lora models and custom baseline models for specific video transformation tasks.
• Scenario-Based Node Utilization: Designed nodes to handle various tasks including text-to-video and video control through trajectory creation and camera manipulation.

Utility in ComfyUI Workflows

Integrating VideoX-Fun into ComfyUI enhances the UI's capability to automate video generation workflows. Users can create complex video-based pipelines with pre-trained models, or train and integrate their own models. The repository aids in prototyping and developing advanced video applications by enabling straightforward integration with popular AI frameworks and resources, furthering capabilities for prototyping and AI-driven content creation in user-friendly environments.