ComfyUI-GGUF Custom Nodes Documentation

Overview

The ComfyUI-GGUF repository provides a collection of custom nodes tailored for use with ComfyUI, primarily aimed at supporting the GGUF quantization format. This format, popularized by llama.cpp, is particularly useful for transformer/DiT models, which are less affected by quantization compared to regular UNET models. The repository enables the usage of quantized model files, offering VRAM savings and efficient performance on low-end GPUs. In addition, the repository includes a custom node to load a quantized version of the T5 text encoder.

Installation

To integrate the ComfyUI-GGUF custom nodes into your ComfyUI setup, follow these instructions:

General Installation

1. Ensure your version of ComfyUI is recent enough to support custom operations for UNET-only loading.
2. Navigate to your ComfyUI/custom_nodes directory.
3. Clone the repository using the following command:

   git clone https://github.com/city96/ComfyUI-GGUF
   

4. Install the required dependencies for inference:

   pip install --upgrade gguf
   

Standalone ComfyUI Release

1. Open a command prompt inside the ComfyUI_windows_portable folder (where your run_nvidia_gpu.bat file is located).
2. Execute the following commands:

   git clone https://github.com/city96/ComfyUI-GGUF ComfyUI/custom_nodes/ComfyUI-GGUF
   .\python_embeded\python.exe -s -m pip install -r .\ComfyUI\custom_nodes\ComfyUI-GGUF\requirements.txt
   

Note: On MacOS with sequoia, torch 2.4.1 is recommended as later versions may cause buffer size errors.

Purpose

This repository is designed to extend the functionality of ComfyUI by supporting models stored in the GGUF format. GGUF quantization allows these models to run on lower specifications by reducing the bitrate per weight variable without significantly sacrificing model performance.

Provided Nodes

The repository offers the following custom nodes, each implemented within nodes.py:

• UnetLoaderGGUF
• CLIPLoaderGGUF
• DualCLIPLoaderGGUF
• TripleCLIPLoaderGGUF
• UnetLoaderGGUFAdvanced

Special Features and Capabilities

• Quantization Support: The nodes enable usage of quantized models, which are less memory-intensive, making them suitable for environments with limited GPU resources.
• Flexible Model Loading: The nodes are capable of loading GGUF formatted model files directly into ComfyUI, as well as handling both gguf and traditional safetensors/bin formats.
• T5 Support: Initial support for the quantization of T5 models is provided, broadening the range of models that can be utilized in quantized form.

Integration with ComfyUI Workflows

These custom nodes can be seamlessly integrated into existing ComfyUI workflows. Users can replace standard nodes with the provided GGUF nodes to take advantage of reduced memory overhead without altering the overall workflow logic. The nodes allow for considerable VRAM savings, thus enabling more efficient model deployment on constrained hardware setups.

To use these nodes, simply place the .gguf model files in the designated ComfyUI/models/unet folder and select the appropriate GGUF loader nodes in your ComfyUI workflow.

Pre-Quantized Models

The following pre-quantized models are available for use:

• flux1-dev GGUF
• flux1-schnell GGUF
• stable-diffusion-3.5-large GGUF
• stable-diffusion-3.5-large-turbo GGUF

CLIP Model Handling

Model loaders can handle both GGUF and traditional formats seamlessly, allowing users to deploy their existing models without modification. For CLIP models, continue using your existing CLIP model setup, as the loaders within this repository support integrated usage.

Conclusion

The ComfyUI-GGUF custom nodes repository is an invaluable resource for ComfyUI users looking to optimize model usage on lower-end GPUs by leveraging quantization technologies. The ability to load quantized model files facilitates efficient and flexible integration within diverse ComfyUI workflows.