Documentation for FluxKohyaInferenceSampler Node

Overview

What This Node Does

The FluxKohyaInferenceSampler node is designed for generating images using a model with pre-trained LoRA (Low-Rank Adaptation) weights in ComfyUI. It leverages the ComfyUI framework's integration with Kohya's training scripts to perform inference tasks. The node allows users to apply or merge LoRA weights onto the pre-trained model and generate images based on a provided text prompt.

Inputs

Accepted Inputs

1. flux_models (TRAIN_FLUX_MODELS)

   • This input expects the selection of the pre-trained models used for inference, consisting of transformer, VAE (Variational Autoencoder), Clip, and T5 models.

2. lora_name (STRING)

   • The name of the LoRA file to be used. This file contains the pre-trained LoRA weights and must be stored in the ComfyUI loras folder.

3. lora_method (STRING: apply/merge)

   • Determines whether the LoRA weights should be applied or merged with the model. "Apply" retains the original model and adds the LoRA; "merge" integrates the LoRA weights directly into the model, effectively overwriting the original weights.

4. steps (INT)

   • The number of sampling steps to perform during the inference process. More steps generally increase image quality at the expense of longer processing time.

5. width (INT) and height (INT)

   • The dimensions of the generated image in pixels. They must be multiples of 8.

6. guidance_scale (FLOAT)

   • A scaling factor for guidance. Higher values may yield results that adhere more closely to the prompt, at the cost of potential loss of creativity in the generated images.

7. seed (INT)

   • A seed for random number generation, ensuring reproducibility of the generated results if set to a specific value.

8. use_fp8 (BOOLEAN)

   • Determines whether the inference should use FP8 (Floating Point 8-bit) weights for computation, optimizing for performance and memory usage.

9. apply_t5_attn_mask (BOOLEAN)

   • Flags whether to apply the attention mask of the T5 (Text-To-Text Transfer Transformer) during inference, which can affect text encoding for the model.

10. prompt (STRING)

    • The text prompt that guides the image generation process. This string should describe the intended content or style of the output image.

Outputs

Produced Outputs

1. image (IMAGE)
   • The generated image result from the inference process using the specified models, LoRA weights, and settings.

Usage in ComfyUI Workflows

The FluxKohyaInferenceSampler node is primarily used in workflows focused on image generation from textual prompts within ComfyUI. Within a workflow, this node can be used after model selection and LoRA preparation nodes to perform actual image synthesis. By allowing specific tuning such as the image dimensions, number of steps, and applying/merging LoRA weights, users gain fine control over the inference results. This is particularly useful for artistic applications or automated image generation processes where specific visual results are required based on textual input.

Example Use Case

A user could create a workflow where they initially load a series of models and LoRA weights using other nodes. Then, integrate the FluxKohyaInferenceSampler node, providing it with a descriptive prompt, desired dimensions, and guidance settings to generate high-fidelity image content based on those specifications. The images could further be evaluated within the workflow for quality checks or be stored for analysis or distribution.

Special Features or Considerations

• FP8 Weights:

  • The option to utilize FP8 weights can significantly increase the speed and reduce memory usage during inference. However, it requires hardware compatibility and may lead to precision trade-offs.

• Attention Masking:

  • The application of T5 attention masks can enhance or otherwise alter the influence of the text prompt on the image. Depending on the scenario, turning this feature on or off can yield varied results.

• Reproducibility:

  • By specifying a seed, users can ensure that they receive consistent outputs across multiple executions with the same settings, which is crucial for debugging and testing.

• LoRA Integration:

  • The choice between applying and merging LoRA weights allows practitioners to either experiment with modifications on-the-fly or commit changes for more permanent model adjustments.