ComfyUI-productfix
🎨 ComfyUI-productfix
ComfyUI custom node that helps generate images while preserving the text, logos, and details of e-commerce products.
🎬 Demo
AI-generated images of items in my room taken with a smartphone (no color correction).
📌 Index
🚀 Introduction
Images generated with Stable Diffusion are visually natural and high-fidelity, but there is an issue where the input object is deformed during generation. This problem is especially noticeable with elements that have artificial regularity, such as text and brand logos. Such deformation issues are a serious limitation when applied to real products sold in e-commerce environments.
Productfix provides an AI application called Latent Injection, which generates images while preserving the characteristics of the input object (text, logo, details, etc.). It also offers additional nodes that help retain fine details of objects.
With these nodes, it is expected that much of the post-processing work that previously had to be done with design tools (like Photoshop or Illustrator) can be greatly reduced. You can integrate these custom nodes into your workflow in ComfyUI.
💡 Features
Apply Latent Injection
- Hijacks the KSampler node in ComfyUI to perform Latent Injection.
- Restores the original KSampler node after execution.
Get Text Mask
- Node that loads a text mask as a tensor using the Easy OCR package.
- Although an Easy OCR custom node already exists (https://github.com/JaidedAI/EasyOCR), this node is recommended because PIL usage is not stable.
Reset Model Patcher Calculate Weight
- Many custom nodes (e.g., ComfyUI-Easy-Use https://github.com/yolain/ComfyUI-Easy-Use.git) cause errors if another node has injected the calculate weight function of Modelpatcher.
- This node resets it to the original Modelpatcher calculate weight to resolve such issues.
📝 Models and Custom Nodes
Models
Custom nodes
🏃🏻♂️ Application
condition / Input / IC-Light / latent injection($\sigma_{end}$=1.0) / latent injection($\sigma_{end}$=0.5)
prompt: product photo, professional photography, realistic, leaf, outdoors / seed: 42
condition / Input / IC-Light / latent injection($\sigma_{end}$=1.0) / latent injection($\sigma_{end}$=0.5)
prompt: product photo, professional photography, realistic / seed: 42
Latent injection truly shines when used together with IC-Light and IP-Adapter. Try it when compositing template-style images and products!
Items in my room captured with my phone camera
prompt: product photo, professional photography, realistic, water, bubble / seed: 42 / controlnet: depth
prompt: product photo, professional photography, realistic, flowers, outdoors / seed: 42 / controlnet: depth
Items in my room captured with my phone camera
prompt: product photo, professional photography, realistic / seed: 42 / controlnet: depth
prompt: product photo, professional photography, realistic / seed: 42 / controlnet: depth
Input / text condition / image condition(IP-Adapter)
only IC-Light / Latent injection / detail transfer / Text transfer
close up
only IC-Light / Latent injection / detail transfer / Text transfer
close up
Text transfer is a detail transfer application based on OCR text masks, developed to preserve the text of input objects. You can implement it using the GetTextMask node and the DetailTransfer node.
🛠 Approach
Inpainting in diffusion models generates images conditioned on a mask. At each sampling step, the latent space of the original and the generated image is composited based on the mask. This method allows for generation while preserving the input object, but for low-quality input objects (e.g., taken with a smartphone), the output image quality is also limited.
IC-Light is an innovative Adapter UNet that manipulates foreground and background lighting. By relighting the input object, even low-quality objects can be transformed into high-quality output images. However, there are still issues with deformation of object details during foreground generation.
Kandinsky diffusion inpainting differs from typical inpainting. When compositing latent spaces at each sampling step, it uses a latent space with noise added according to the scheduler’s sigma value instead of the original. This approach improves quality through consistent noise.
CLIP Skip is an inference method where text conditioning is not applied until the last sampling step but is stopped midway. This allows for more contextually appropriate results by controlling the conditioning process.
$X_t$ : sample
$M$ : product mask
$P$ : product latent
$CO$ : composition operation(ex: add, overlay, soft light etc.)
To achieve both preservation of object features and meaningful lighting changes, a composite strategy is applied. During the sampling process, latent spaces with added noise are composited to preserve fine object details. Additionally, to reflect the global lighting changes of IC-Light, the initial and final steps of sampling are selectively skipped. This method operates based on the scheduler’s sigma value, ensuring stable performance across various scheduler types. As a result, it is possible to flexibly apply lighting effects while preserving the unique characteristics of the object.
📥 Install
cd custom_nodes
git clone {this repository}
pip install -r requirements.txt🖥 How to use
ComfyUI-workflows
You can download the workflow here.
You can download the workflow here.
Demo Example Assets
📚 Reference
This project is based on research and code from several papers and open-source repositories.