ComfyUI-MuseTalk_FSH

ComfyUI-MuseTalk_FSH
★ 23

音频驱动视频生成口型同步MuseTalk集成
ComfyUI 自定义节点，集成 MuseTalk，将音频驱动生成口型同步与动态视频，便于在 ComfyUI 流水线中自动化音频到视频制作。
💡 将语音或配乐自动生成口型同步的动态视频素材。
🍴 8 Forks💻 Python🔄 2024-06-14
📦
网盘下载
复制链接后前往夸克网盘下载
https://pan.quark.cn/s/b6135d9bd930
📦 requirements.txt
opencv-python
ffmpeg-python
soundfile
diffusers
pydub
moviepy
accelerate
📄 README

ComfyUI-MuseTalk_FSH

the comfyui custom node of MuseTalk to make audio driven videos!

How to use

make sure ffmpeg is worked in your commandline

for Linux

apt update
apt install ffmpeg

for Windows,you can install ffmpeg by WingetUI automatically

then!

git clone https://github.com/AIFSH/ComfyUI-MuseTalk_FSH.git
cd ComfyUI-MuseTalk_FSH
pip install -r requirements.txt

mmlab packages

pip install --no-cache-dir -U openmim 
mim install mmengine 
mim install "mmcv>=2.0.1" 
mim install "mmdet>=3.1.0" 
mim install "mmpose>=1.1.0"

Download weights

You can download weights manually as follows:

  • Download our trained weights.

  • Download the weights of other components:
  • sd-vae-ft-mse
  • whisper
  • dwpose
  • face-parse-bisent
  • resnet18

    • 或者下载MuseTalk.zip，

    解压后把子文件夹放入ComfyUI-MuseTalk_FSH/models/目录

    Finally, these weights should be organized in models as follows:

    ComfyUI-MuseTalk_FSH/models/
├── musetalk
│   └── musetalk.json
│   └── pytorch_model.bin
├── dwpose
│   └── dw-ll_ucoco_384.pth
├── face-parse-bisent
│   ├── 79999_iter.pth
│   └── resnet18-5c106cde.pth
├── sd-vae-ft-mse
│   ├── config.json
│   └── diffusion_pytorch_model.bin
└── whisper
    └── tiny.pt

    Tutorial

  • Demo on 3060 12GB
  • Demo on 4090 24GB

    • WeChat Group && Donate

    Thanks

  • MuseTalk