ComfyUI-DragNUWA

ComfyUI-DragNUWA
★ 412

视频生成交互式编辑DragNUWA模型权重
将 DragNUWA 集成到 ComfyUI，允许通过拖拽操控图像中的物体或背景并生成对应视频；需将权重文件放至指定路径以运行。
💡 在 ComfyUI 中通过拖拽物体生成短视频
🍴 29 Forks💻 Python🔄 2024-06-14
📦
网盘下载
复制链接后前往夸克网盘下载
https://pan.quark.cn/s/2df45d172dc1
📦 requirements.txt
black==23.7.0
chardet==5.1.0
clip
@
git+https://github.com/openai/CLIP.git
einops>=0.6.1
fairscale>=0.4.13
fire>=0.5.0
fsspec>=2023.6.0
invisible-watermark>=0.2.0
kornia==0.6.9
matplotlib>=3.7.2
natsort>=8.4.0
ninja>=1.11.1
numpy>=1.24.4
omegaconf>=2.3.0
open-clip-torch>=2.20.0
opencv-python==4.6.0.66
pandas>=2.0.3
pillow>=9.5.0
pudb>=2022.1.3
pytorch-lightning==2.1.0
pyyaml>=6.0.1
scipy>=1.10.1
streamlit>=0.73.1
tensorboardx==2.6
timm>=0.9.2
tokenizers>=0.12.1
torchdata>=0.6.1
torchmetrics>=1.0.1
tqdm>=4.65.0
urllib3<1.27,>=1.25.4
wandb>=0.15.6
webdataset>=0.2.33
wheel>=0.41.0
colorlog
deepdish
json_lines
msgpack
configobj
psutil
h5py
imageio
tensorboard
moviepy
pyyaml
pandas
einops
📄 README

This is an implementation of DragNUWA for ComfyUI

DragNUWA: DragNUWA enables users to manipulate backgrounds or objects within images directly, and the model seamlessly translates these actions into camera movements or object motions, generating the corresponding video.

Install

  • Clone this repo into custom_nodes directory of ComfyUI location

  • Run pip install -r requirements.txt

  • Download the weights of DragNUWA drag_nuwa_svd.pth and put it to ComfyUI/models/checkpoints/drag_nuwa_svd.pth

    • For chinese users:drag_nuwa_svd.pth

    smaller and faster fp16 model: dragnuwa-svd-pruned.fp16.safetensors from https://github.com/painebenjamin/app.enfugue.ai

    For chinese users: wget https://hf-mirror.com/benjamin-paine/dragnuwa-pruned-safetensors/resolve/main/dragnuwa-svd-pruned.fp16.safetensors 不能直接在浏览器下载，或者参照 https://hf-mirror.com/ 官方使用说明

    Tools

    Motion Traj Tool Generate motion trajectories

    Examples

  • basic workflow

    • https://github.com/chaojie/ComfyUI-DragNUWA/blob/main/workflow.json

  • InstantCameraMotionBrush & InstantObjectMotionBrush

    • InstantCameraMotionBrush Node Generate zoomin/zoomout/left/right/up/bottom camera motion brush

    InstantObjectMotionBrush Node Generate zoomin/zoomout/left/right/up/bottom object motion brush (by draw mask on object)

    https://github.com/chaojie/ComfyUI-DragNUWA/blob/main/workflow_InstantMotionBrush.json

  • optical flow workflow

    • Thanks for Fannovol16’s Unimatch_ OptFlowPreprocessor

    Thanks for toyxyz’s load optical flow from directory

    https://github.com/chaojie/ComfyUI-DragNUWA/blob/main/workflow_optical_flow.json

  • motion brush

    • need nodes “https://github.com/chaojie/ComfyUI-RAFT”

    https://github.com/chaojie/ComfyUI-DragNUWA/blob/main/workflow_motionbrush.json