AnimateDiff for ComfyUI

AnimateDiff integration for ComfyUI, adapts from sd-webui-animatediff. Please read the original repo README for more information.

How to Use

Clone this repo into custom_nodes folder.

Download motion modules and put them under comfyui-animatediff/models/.

Original modules: Google Drive | HuggingFace | CivitAI | Baidu NetDisk

Community modules: manshoety/AD_Stabilized_Motion | CiaraRowles/TemporalDiff

AnimateDiff v2 mm_sd_v15_v2.ckpt

Update 2023/09/25

Motion LoRA is now supported!

Download motion LoRAs and put them under comfyui-animatediff/loras/ folder.

Note: LoRAs only work with AnimateDiff v2 mm_sd_v15_v2.ckpt module.

New node: AnimateDiffLoraLoader

Example workflow:

Workflow: lora.json

Samples:

Update 2023/09/21

Sliding Window is now available!

The sliding window feature enables you to generate GIFs without a frame length limit. It divides frames into smaller batches with a slight overlap. This feature is activated automatically when generating more than 16 frames. To modify the trigger number and other settings, utilize the SlidingWindowOptions node. See the sample workflow bellow.

Nodes

AnimateDiffLoader

AnimateDiffSampler

Mostly the same with KSampler

motion_module: use AnimateDiffLoader to load the motion module

inject_method: should left default

frame_number: animation length

latent_image: You can pass an EmptyLatentImage

sliding_window_opts: custom sliding window options

AnimateDiffCombine

Combine GIF frames and produce the GIF image

frame_rate: number of frame per second

loop_count: use 0 for infinite loop

save_image: should GIF be saved to disk

format: supports image/gif, image/webp (better compression), video/webm, video/h264-mp4, video/h265-mp4. To use video formats, you’ll need ffmpeg installed and available in PATH

SlidingWindowOptions

Custom sliding window options

context_length: number of frame per _window_. Use 16 to get the best results. Reduce it if you have low VRAM.

context_stride:

1: sampling every frame

2: sampling every frame then every second frame

3: sampling every frame then every second frame then every third frames

…

context_overlap: overlap frames between each window slice

closed_loop: make the GIF a closed loop, will add more sampling step

LoadVideo

Load GIF or video as images. Usefull to load a GIF as ControlNet input.

frame_start: Skip some begining frames and start at frame_start

frame_limit: Only take frame_limit frames

Workflows

Simple txt2gif

Workflow: simple.json

Samples:

Long duration with sliding window

Workflow: sliding-window.json

Samples:

Latent upscale

Upscale latent output using LatentUpscale then do a 2nd pass with AnimateDiffSampler.

Workflow: latent-upscale.json

Samples:

Using with ControlNet

You will need following additional nodes:

Kosinkadink/ComfyUI-Advanced-ControlNet: Apply different weight for each latent in batch

Fannovel16/comfyui_controlnet_aux: ControlNet preprocessors

Animate with starting and ending images

Use LatentKeyframe and TimestampKeyframe from ComfyUI-Advanced-ControlNet to apply diffrent weights for each latent index.

Use 2 controlnet modules for two images with weights reverted.

Workflow: cn-2images.json

Samples:

Using GIF as ControlNet input

Using a GIF (or video, or a list of images) as ControlNet input.

Workflow: cn-vid2vid.json

Samples:

Known Issues

CUDA error: invalid configuration argument

It’s an xformers bug accidentally triggered by the way the original AnimateDiff CrossAttention is passed in. The current workaround is to disable xformers with --disable-xformers when booting ComfyUI.

GIF split into multiple scenes

Work around:

Shorter your prompt and negative prompt

Reduce resolution. AnimateDiff is trained on 512×512 images so it works best with 512×512 output.

Disable xformers with --disable-xformers

GIF has Wartermark (especially when using mm_sd_v15)

See: https://github.com/continue-revolution/sd-webui-animatediff/issues/31

Training data used by the authors of the AnimateDiff paper contained Shutterstock watermarks. Since mm_sd_v15 was finetuned on finer, less drastic movement, the motion module attempts to replicate the transparency of that watermark and does not get blurred away like mm_sd_v14. Try other community finetuned modules.