OllamaVision Node Documentation

Overview

The OllamaVision node in the ComfyUI Ollama extension is designed to allow users to query input images using the capabilities of a Vision Model. The primary function of this node is to process images through a specified Vision Model running on the Ollama server and generate a descriptive output based on the input query.

Functionality

• Query Processing: Using a Vision Model, the node processes a given image alongside an input query to generate descriptions or other relevant textual output.
• Image Handling: The node converts images into a compatible format for processing with the Vision Model.
• Connection: Facilitates communication with an Ollama server to execute the image-query interaction.

Inputs

The OllamaVision node accepts the following inputs:

• Images: A list of images that the node will process. This input is mandatory and should be provided as an image data type.
• Query: A textual prompt that describes what the node should focus on in the image. It should be a string, and it allows multiline input with a default value "describe the image".
• Debug: A toggle to enable or disable debug mode, with options "enable" or "disable". Debug mode provides additional console output for troubleshooting.
• URL: The address of the Ollama server where the Vision Model is hosted. Its default value is "http://127.0.0.1:11434".
• Model: Specifies the Vision Model to use for processing. It requires a model with vision capabilities.
• Seed: A numerical value used for randomization, ensuring consistent results in repeated executions when unchanged.
• Keep Alive: Sets how long to maintain the server connection. Accepts integer values, with a default of 5 minutes.
• Format: Determines the output format for the response, either "text", "json", or an empty value.

Output

The OllamaVision node generates the following output:

• Description: A string output that provides the Vision Model's response to the image and query input. It entails a description of the image based on the query provided.

Suggested Workflow Integration

Here are some ideas on how you might use the OllamaVision node in ComfyUI workflows:

1. Visual Content Analysis: Use the node to process images automatically and gather descriptive information, which can be invaluable for image-based research or content sorting.
2. Assisted Image Tagging: Integrate the node to assist in generating tags for image databases or repositories, enhancing search capabilities.
3. Interactive Installations: Employ in interactive setups where real-time image analysis and descriptive feedback are valuable, such as exhibitions or educational kiosks.

Special Features and Considerations

• Vision Model Requirement: For effective output, the chosen model should have image processing capabilities, such as those available in the Ollama library, like models with vision abilities.
• Debug Mode: The node's debug feature provides console outputs that are helpful for troubleshooting and understanding the processing flow of the images and queries.
• Server Dependency: The node requires a correctly configured and running Ollama server; otherwise, it will not function correctly.

By providing these inputs, users can automatically generate detailed and relevant textual descriptions for images directly within the ComfyUI environment, expanding the capabilities of image handling workflows.