diffusers >=0.29.0 opencv-python torch torchaudio torchvision transformers imageio imageio-ffmpeg omegaconf tqdm librosa einops #tqdm ==4.65.2 #librosa ==0.10.2.post1 #einops ==0.7.0 #imageio-ffmpeg ==0.5.1 #gradio==3.50.0 #transformers ==4.43.2 #imageio ==2.31.1 #torchvision ==0.17.1 #torch ==2.2.1 #diffusers >=0.29.0
Sonic is a method about ‘ Shifting Focus to Global Audio Perception in Portrait Animation’,you can use it in comfyUI
In the ./ComfyUI/custom_node directory, run the following:
git clone https://github.com/smthemex/ComfyUI_Sonic.gitpip install -r requirements.txt-- ComfyUI/models/sonic/
|-- audio2bucket.pth
|-- audio2token.pth
|-- unet.pth
|-- yoloface_v5m.pt
|-- whisper-tiny/
|--config.json
|--model.safetensors
|--preprocessor_config.json
|-- RIFE/
|--flownet.pkl-- ComfyUI/models/checkpoints
├── svd_xt.safetensors or svd_xt_1_1.safetensors@article{ji2024sonic,
title={Sonic: Shifting Focus to Global Audio Perception in Portrait Animation},
author={Ji, Xiaozhong and Hu, Xiaobin and Xu, Zhihong and Zhu, Junwei and Lin, Chuming and He, Qingdong and Zhang, Jiangning and Luo, Donghao and Chen, Yi and Lin, Qin and others},
journal={arXiv preprint arXiv:2411.16331},
year={2024}
}
@article{ji2024realtalk,
title={Realtalk: Real-time and realistic audio-driven face generation with 3d facial prior-guided identity alignment network},
author={Ji, Xiaozhong and Lin, Chuming and Ding, Zhonggan and Tai, Ying and Zhu, Junwei and Hu, Xiaobin and Luo, Donghao and Ge, Yanhao and Wang, Chengjie},
journal={arXiv preprint arXiv:2406.18284},
year={2024}
}