avatar-graph-comfyui

★ 262

角色绑定实时交互Blender bpy生成式工作流

为ComfyUI提供avatar-graph-comfyui节点，结合sam与bpy操作，利用Blender mesh API与Avatech Shape Flow实时生成并驱动可交互的角色绑定与动画工作流。

💡 在ComfyUI中构建实时交互的角色绑定与动画流程。

🍴 17 Forks💻 Python🔄 2024-05-22

🔗 GitHub 原文

📦

网盘下载

复制链接后前往夸克网盘下载

https://pan.quark.cn/s/51137d50651f

📦 requirements.txt

torchvision
numpy
opencv-python
opencv-contrib-python
einops
bpy==3.6.0
segment-anything
tqdm
python-dotenv
mediapipe
#
-e
git+https://github.com/facebookresearch/segment-anything.git#egg=segment_anything

📄 README

avatar-graph-comfyui

Wanna animate or got a question? Join our Discord

A custom nodes module for creating real-time interactive avatars powered by blender bpy mesh api + Avatech Shape Flow runtime.

WARNING

We are still making changes to the nodes and demo templates, please stay tuned.

Demo

| :——————————————————————————————————————————————————————————————————————————: | :——————————————————————————————————————————————————————————————————————————: | :——————————————————————————————————————————————————————————————————————————: | :——————————————————————————————————————————————————————————————————————————: |

How to?

Basic Rigging Workflow Template

Best Practices for image input

Basic Rigging Workflow Template

1. Creating an eye blink and lipsync avatar

For optimal results, please input a character image with an open mouth and a minimum resolution of 768×768. This higher resolution will enable the tool to accurately recognize and work with facial features.

Download: Save the image, and drag into Comfyui or Simple Shape Flow

2. Creating an eye blink and lipsync emoji avatar

| | |

| :–: | :–: |

Download: Save the image, and drag into Comfyui

| | |

| :–: | :–: |

Download: Save the image, and drag into Comfyui or Dog Workflow

Best practices for image input

1. Generate a new character image

We need a character image with an open mouth and enable the tool to easily recognize facial features, so please add to the prompt:

looking at viewer, detailed face, open mouth, [smile], solo,eye-level angle

Download: Character Gen Template

2. Make existing character image mouth open (Inpaint)

To maintain consistency with the base image, it is recommended to utilize a checkpoint model that aligns with its style.

Download: Mouth Open Inpaint Template.json>)

Inpaint Demonstration

3. Pose Constraints (ControlNet)

Place normal and openpose image with reference to images.

Download: ControlNet Gen

Recommend Checkpoint Model List

Anime Style SD1.5

https://civitai.com/models/35960/flat-2d-animerge

https://civitai.com/models/24149/mistoonanime

Realistic Style SD1.5

https://civitai.com/models/4201/realistic-vision-v51

https://civitai.com/models/49463/am-i-real

https://civitai.com/models/43331/majicmix-realistic

Custom Nodes

Expand to see all the available nodes description.

All Custom Nodes

| Name | Description | Preview |

| ——————– | ———————————————————————————————- | ——————————————————————————————————————————————————————– |

| Segmentation (SAM) | Integrative SAM node allowing you to directly select and create multiple image segment output. | |

| Name | Description | Preview |

| —————————- | ———————————————————————————————– | ——————————————————————————————————————————————————————– |

| Create Mesh Layer | Create a mesh object from the input images (usually a segmented part of the entire image) | |

| Join Meshes | Combine multiple meshes into a single mesh object | |

| Match Texture Aspect Ratio | Since the mesh is created in 1:1 aspect ratio, a re-scale is needed at the end of the operation | |

| Plane Texture Unwrap | Will perform mesh face fill and UV Cube project on the target plane mesh, scaled to bounds. | |

| Name | Description | Preview |

| ———————– | ————————————————————————————————– | ——————————————————————————————————————————————————————– |

| Mesh Modify Shape Key | Given shape key name & target vertex_group, modify the vertex / all vertex’s transform | |

| Create Shape Flow | Create runtime shape flow graph, allowing interactive inputs affecting shape keys value in runtime | |

| Name | Description | Preview |

| ——————– | —————————————————————————————————– | ——————————————————————————————————————————————————————– |

| Avatar Main Output | The primary output of the .ava file. The embedded Avatar View will auto update with this node’s output | |

Shape Flow

Installation

Method 1 – Windows

Download Python environment from here

Unzip it to ComfyUI directory

Run the run_cpu_3.10.bat or run_nvidia_gpu_3.10.bat

Install avatar-graph-comfyui from ComfyUI Manager

Method 2 – macOS/Linux

Make sure your Python environment is 3.10.x as required by the bpy package. Then go to the ComfyUI directory and run:

Suggest using conda for your comfyui python environment

conda create --name comfyui python=3.10

conda activate comfyui

pip install -r requirements.txt

cd custom_nodes

git clone https://github.com/avatechgg/avatar-graph-comfyui.git

cd avatar-graph-comfyui && python -m pip install -r requirements.txt

Restart ComfyUI with enable-cors-header python main.py --enable-cors-header or (for mac) python main.py --force-fp16 --enable-cors-header

Development

If you are interested in contributing

For comfyui frontend extension, frontend js located at avatar-graph-comfyui/js

Web stack used: vanjs tailwindcss

Install deps

pnpm i

Run the dev command to start the tailwindcss watcher

pnpm dev

For each changes, simply refresh the comfyui page to see the changes.

p.s. For tailwind autocomplete, add the following to your vscode settings.json.

{
    "tailwindCSS.experimental.classRegex": [
        ["class\\s?:\\s?([\\s\\S]*)", "(?:\"|')([^\"']*)(?:\"|')"]
    ]
}

Update blender node types

To update blender operations input and output types (stored in blender/input_types.txt), run:

python generate_blender_types.py

FAQ

What is `--enable-cors-header` used for?

It is used to enable communication between ComfyUI and our editor (https://editor.avatech.ai), which is in charge of animating static characters. The only messages exchanged between them are the character data like the meshes of eyes and mouth, and the JSON format of our editor graph.

When you execute the ComfyUI graph, it sends the character data and the JSON graph to our editor for animating. When you modify and save the graph in our editor, it sends the modified graph back to ComfyUI. To validate it, you can open the js/index.js, and log the message in window.addEventListener("message", ...) and postMessage(message).

You can also run ComfyUI *without* the --enable-cors-header: execute the ComfyUI workflow, then download the .GLB or .GLTF format by right clicking the Avatar Main Output node and Save File option. Yet, this will disable the real-time character preview in the top-right corner of ComfyUI. Feel free to view it in other software like Blender.

avatar-graph-comfyui

avatar-graph-comfyui

Demo

How to?

Basic Rigging Workflow Template

1. Creating an eye blink and lipsync avatar

2. Creating an eye blink and lipsync emoji avatar

Best practices for image input

1. Generate a new character image

2. Make existing character image mouth open (Inpaint)

3. Pose Constraints (ControlNet)

Recommend Checkpoint Model List

Anime Style SD1.5

Realistic Style SD1.5

Custom Nodes

Shape Flow

Installation

Method 1 – Windows

Method 2 – macOS/Linux

Development

Install deps

Update blender node types

FAQ

What is --enable-cors-header used for?

What is `--enable-cors-header` used for?