ComfyUI-ZeroShot-MTrans

★ 180

材质迁移零样本照明传递ComfyUI节点

在ComfyUI中实现ZeST的零样本材质迁移：从单张示例图把材质（含光照信息）准确转移到目标图像，支持IPA与ControlNet配合。

💡 将示例材质从单张图迁移到目标图并保留光照

🍴 10 Forks💻 Python🔄 2024-07-04

🔗 GitHub 原文

📦

网盘下载

复制链接后前往夸克网盘下载

https://pan.quark.cn/s/8b0992d318c3

📦 requirements.txt

torch
numpy
pillow

Zero-Shot Material Transfer from a Single Image

📄 README

ComfyUI-ZeroShot-MTrans: Zero-Shot Material Transfer from a Single Image

中文

An unofficial ComfyUI custom node for ZeST(Zero-Shot Material Transfer from a Single Image)

Given an input image (e.g., a photo of an apple) and a single material exemplar image (e.g., a golden bowl), ZeST can transfer the gold material from the exemplar onto the apple with accurate lighting cues while making everything else consistent.

Explains

Refer to zest_workflow.json in this repo for quick start.

This node implenment the red rect area of paper figure. the illumination transfer, and make it an ComfyUI node. You can use any IPA and controlnet node as your wish.

The original ZeST demo code use IPA to extract the style of material, and control for depth information. Then, change the original image with illumination informaion of subject for better lighting style.

install

in ComfyUI Manager or git clone to ComfyUI/custom_nodes

After restart ComfyUI, the following custom node will be available.

dependences

ControlNet

IP-Adaptor

Segment Anything Model

Input/Output

INPUT:

target_image : the original image for inpaint

subject_mask : the mask for inpaint, this mask will be also used as input of inpaint node

brighter : default is 1, means no change

value < 1 , means darker the subject, useful when subject in high light

value > 1, means brighter the subject, useful when subject in dark light

OUTPUT:

IMAGE : image with subject in luminosity(grey) mode. used as input for inpaint node

Tips

for output, the subject in middle grey is prefered. for high light material, dark it; otherwise, brighter it.

for material image, suggest remove all backgroup, just keep the material you want.

use IP-Adaptor strength, controlnet strength, brighter to control the output 1st.

Workflow

You can download the zest_workflow.json from this repo for quickly start.

sample workflow

Image Input

Material Input

Temp Images

Mask

Subject Grayout

Output

Thanks to

ZeST: Zero-Shot Material Transfer from a Single Image Paper

ZeST official DEMO

ZeST site

ZeST official video

@article{cheng2024zest,
  title={ZeST: Zero-Shot Material Transfer from a Single Image},
  author={Cheng, Ta-Ying and Sharma, Prafull and Markham, Andrew and Trigoni, Niki and Jampani, Varun},
  journal={arXiv preprint arXiv:2404.06425},
  year={2024}
}