ComfyUI_StreamingT2V

★ 36

文本到视频流式生成VidXTend低显存

在ComfyUI中集成StreamingT2V文本到视频生成功能，支持VidXTend自动下载并显著降低显存占用，便于流式与工作流使用

💡 在ComfyUI中用低显存模式实时生成视频并集成工作流

🍴 5 Forks💻 Python🔄 2024-06-14

🔗 GitHub 原文

📦

网盘下载

复制链接后前往夸克网盘下载

https://pan.quark.cn/s/2df45d172dc1

📦 requirements.txt

IPython
accelerate
bitsandbytes
transformers
albumentations
av
boto3
decord
einops
gdown
huggingface-hub
omegaconf
pandas
pytorch-lightning
scikit-image
scikit-learn
scipy
seaborn
tqdm
xformers
open-clip-torch
jsonargparse[signatures]>=4.26.1
fairscale
rotary-embedding-torch
easydict
torchsde
imageio[ffmpeg]
addict
yapf
gast
simplejson
sortedcontainers
oss2
datasets
diffusers
raft
ffmpeg
rich
clip
@
git+https://github.com/openai/CLIP.git@a1d071733d7111c9c014f024669f959182114e33
pibble

📄 README

ComfyUI-StreamingT2V

Painebenjamin created the VidXTend model, which reduces the size from the original 25.6GB to 5GB. When using the VidXTend process, there is no need to manually download the model, as it will be downloaded automatically.

If you use the original model, please download the StreamingT2V model and place it in folder ComfyUI/model/checkpoints

VidXTend (<10G VRAM) | Original (<15G VRAM)

workflow

Recommended complete Workflow

Recommended STAGE 2 Workflow

Test prompt travel cyclist Workflow

Open the Auto Queue option, Stop it when you want to.

More Workflow

Step 1 (It’s not necessary.) Generate short video (16x256x256):

You can freely generate a 16x256x256 video without using the process below. You can even use videos downloaded from the internet.

AnimateDiff Option: https://github.com/chaojie/ComfyUI_StreamingT2V/blob/main/workflow/wf_short_ad.json

Modelscope Option: https://github.com/chaojie/ComfyUI_StreamingT2V/blob/main/workflow/wf_short_ms.json

SVD Option: https://github.com/chaojie/ComfyUI_StreamingT2V/blob/main/workflow/wf_short_svd.json

Step 2 Generate long video (nx256x256):

VidXTend Option: https://github.com/chaojie/ComfyUI_StreamingT2V/blob/main/workflow/wf_VidXTend.json

Original Option: https://github.com/chaojie/ComfyUI_StreamingT2V/blob/main/workflow/wf_long.json

Step 3 Generate long enhanced video (nx512x512):

https://github.com/chaojie/ComfyUI_StreamingT2V/blob/main/workflow/wf_enhance.json

You can generate all by one workflow (MORE VRAM)

ModelscopeT2V (T2V)

https://github.com/chaojie/ComfyUI_StreamingT2V/blob/main/workflow/wf.json

AnimateDiff (T2V)

https://github.com/chaojie/ComfyUI_StreamingT2V/blob/main/workflow/wf_ad.json

SVD (I2V)

https://github.com/chaojie/ComfyUI_StreamingT2V/blob/main/workflow/wf_svd.json