ComfyUI-V-Express

ComfyUI-V-Express
★ 117

人像视频生成弱条件平衡音频驱动ComfyUI节点
单张人像生成视频的ComfyUI节点，通过progressive drop逐步平衡强弱控制信号，使音频等弱条件有效参与生成，提升对姿态与原图的联合控制（需手动下载权重）。
💡 用单张人像与音频、姿态共同生成连贯的人像视频。
🍴 11 Forks💻 Python🔄 2024-06-26
📦
网盘下载
复制链接后前往夸克网盘下载
https://pan.quark.cn/s/79aaff81621b
📦 requirements.txt
diffusers==0.24.0
imageio-ffmpeg==0.4.9
insightface==0.7.3
omegaconf==2.2.3
onnxruntime==1.16.3
safetensors==0.4.2
torch==2.0.1
torchaudio==2.0.2
torchvision==0.15.2
transformers==4.30.2
einops==0.4.1
tqdm==4.66.1
xformers==0.0.22
av==11.0.0
workflow
📄 README

_V-Express: Conditional Dropout for Progressive Training of Portrait Video Generation_

Introduction

In the field of portrait video generation, the use of single images to generate portrait videos has become increasingly prevalent.

A common approach involves leveraging generative models to enhance adapters for controlled generation.

However, control signals can vary in strength, including text, audio, image reference, pose, depth map, etc.

Among these, weaker conditions often struggle to be effective due to interference from stronger conditions, posing a challenge in balancing these conditions.

In our work on portrait video generation, we identified audio signals as particularly weak, often overshadowed by stronger signals such as pose and original image.

However, direct training with weak signals often leads to difficulties in convergence.

To address this, we propose V-Express, a simple method that balances different control signals through a series of progressive drop operations.

Our method gradually enables effective control by weak conditions, thereby achieving generation capabilities that simultaneously take into account pose, input image, and audio.

Workflow

Installation

  • Clone this repo into the Your ComfyUI root directory\ComfyUI\custom_nodes\ and install dependent Python packages from here:

    • “`shell

    cd Your_ComfyUI_root_directory\ComfyUI\custom_nodes\

    git clone https://github.com/tiankuan93/ComfyUI-V-Express

    pip install -r requirements.txt

    “`

    If you are using ComfyUI_windows_portable , you should use .\python_embeded\python.exe -m pip to replace pip for installation.

    If you got error regards insightface, you may find solution here.

  • first, you should download .whl file here
  • then, install it by .\python_embeded\python.exe -m pip install --no-deps --target=\your_path_of\python_embeded\Lib\site-packages [path-to-wheel]

  • Download V-Express models and other needed models:

  • model_ckpts
  • You need to replace the model_ckpts folder with the downloaded V-Express/model_ckpts. Then you should download and put all .bin model to model_ckpts/v-express directory, which includes audio_projection.bin, denoising_unet.bin, motion_module.bin, reference_net.bin, and v_kps_guider.bin. The final model_ckpts folder is as follows:

    • “`text

    ./model_ckpts/

    |– insightface_models

    |– sd-vae-ft-mse

    |– stable-diffusion-v1-5

    |– v-express

    |– wav2vec2-base-960h

    “`

  • You should put the files in input directory into the Your ComfyUI Input root directory\ComfyUI\input\.
  • You need to set output_path as directory\ComfyUI\output\xxx.mp4, otherwise the output video will not be displayed in the ComfyUI.

    • Acknowledgements

    We would like to thank the contributors to the AIFSH/ComfyUI_V-Express, for the open research and exploration.