ComfyUI-decadetw-spout-syphon-im-vj

ComfyUI-decadetw-spout-syphon-im-vj
★ 13

实时传输Spout/SyphonVJ/现场表演GPU加速
将ComfyUI中实时生成的SD图像通过Spout（Windows）或Syphon（macOS）以GPU加速传到OpenGL/VJ软件，便于现场演出与可视化控制。
💡 将实时生成的SD图像通过Spout/Syphon投放到VJ软件
🍴 3 Forks💻 Python🔄 2024-09-03
📦
网盘下载
复制链接后前往夸克网盘下载
https://pan.quark.cn/s/a1f1f564f19c
📦 requirements.txt
numpy
PyOpenGL
PyOpenGL_accelerate
📄 README

Quick Links

  • I’m SD-VJ. (share SD-generating-process in realtime by gpu)
  • SD-WEB-UI: https://github.com/xlinx/sd-webui-decadetw-spout-syphon-im-vj
  • ComfyUI: https://github.com/xlinx/ComfyUI-decadetw-spout-syphon-im-vj
  • Auto prompt by LLM and LLM-Vision (Trigger more details out inside model)
  • SD-WEB-UI: https://github.com/xlinx/sd-webui-decadetw-auto-prompt-llm
  • ComfyUI: https://github.com/xlinx/ComfyUI-decadetw-auto-prompt-llm
  • Auto msg to ur mobile (LINE | Telegram | Discord)
  • SD-WEB-UI :https://github.com/xlinx/sd-webui-decadetw-auto-messaging-realtime
  • ComfyUI: https://github.com/xlinx/ComfyUI-decadetw-auto-messaging-realtime
  • CivitAI Info|discuss:
  • https://civitai.com/articles/6988/extornode-using-llm-trigger-more-detail-that-u-never-thought
  • https://civitai.com/articles/6989/extornode-sd-image-auto-msg-to-u-mobile-realtime
  • https://civitai.com/articles/7090/share-sd-img-to-3rd-software-gpu-share-memory-realtime-spout-or-syphon

    • SD-WEB-UI | ComfyUI | decadetw-spout-syphon-im-vj

    Update Log

  • [add|202400822] | 🟢 spout support
  • [add|2024008xx] | 🟠 syphon support
  • [add|2024008xx] | 🟠 ofxNDI
  • 🟢 SD-Image(latent space) -> Spout(win)|(mac)Syphon -> GPU Memory -> Other Software (Ultra-fast-access)
  • 🟠 SD-Inpainting use <- GPU Memory <- Spout(win)|(mac)Syphon <- Other Software

    • Motivation💡

  • Live Performances & …and show control cues.
  • QLab https://qlab.app/
  • vjdj | madmapper | processing | JAVA ….etc.,
  • https://madmapper.com/
  • https://processing.org/

  • Enjoy ur party or project
  • I’m VJ(Visual Jockey)
  • Interface, middle-ware between SD and Spout lib (https://spoutgl-site.netlify.app/#CClass:Spout)
  • SD-Image to OpenGL in realtime

    • Web-UI | Forge | ComfyUI preview

    Input

    Output

    Spout / Win

  • Tx
  • SD-Image -> 3rd software
  • Rx
  • 3rd software (maybe webcam) -> SD-Img2Img
    		•

    Syphon / Mac

    @Mac using Syphon (later. finding M series mac ing…)

    Installtion

  • You need install Spout or Syphon Client first.
  • https://github.com/leadedge/Spout2/releases/download/2.007.015/SPOUT_2007-015.zip
  • https://spout.zeal.co/

    • Reference

  • re-compiler for SD python env 3.10 | 3.11
  • SpoutGL https://github.com/jlai/Python-SpoutGL/tree/main/VS2019

    • How to show high-quality previews?

  • Edit the “run_nvidia_gpu.bat” file with “–preview-method auto” on the end.
  • If you have ComfyUI Manager Menu installed and just change on “Preview method”
  • Use –preview-method auto to enable previews.

  • The default installation includes a fast latent preview method that’s low-resolution. To enable higher-quality previews with TAESD, download the taesd_decoder.pth, taesdxl_decoder.pth, taesd3_decoder.pth and taef1_decoder.pth and place them in the models/vae_approx folder. Once they’re installed, restart ComfyUI and launch it with –preview-method taesd to enable high-quality previews.

    • Suggestion software info list

  • https://syphon.github.io/ (Mac)
  • https://github.com/Syphon/Syphon-Framework
  • https://spout.zeal.co/ (Win)
  • https://github.com/leadedge/Spout2
  • https://github.com/leadedge/ofxNDI (Net)

    • Suggestion LLM Model in VJ mode

  • LLM-text (normal, chat, assistant)
  • 4B VRAM<2G
  • CHE-72/Qwen1.5-4B-Chat-Q2_K-GGUF/qwen1.5-4b-chat-q2_k.gguf
  • https://huggingface.co/CHE-72/Qwen1.5-4B-Chat-Q2_K-GGUF
  • 7B VRAM<8G
  • ccpl17/Llama-3-Taiwan-8B-Instruct-GGUF/Llama-3-Taiwan-8B-Instruct.Q2_K.gguf
  • Lewdiculous/L3-8B-Stheno-v3.2-GGUF-IQ-Imatrix/L3-8B-Stheno-v3.2-IQ3_XXS-imat.gguf
  • Google-Gemma
  • https://huggingface.co/bartowski/gemma-2-9b-it-GGUF
  • bartowski/gemma-2-9b-it-GGUF/gemma-2-9b-it-IQ2_M.gguf
  • small and good for SD-Prompt

  • LLM-vision 👀 (work with SDXL, VRAM >=8G is better )
  • https://huggingface.co/xtuner/llava-phi-3-mini-gguf
  • llava-phi-3-mini-mmproj-f16.gguf (600MB,vision adapter)
  • ⭐⭐⭐llava-phi-3-mini-f16.gguf (7G, main model)
  • https://huggingface.co/FiditeNemini/Llama-3.1-Unhinged-Vision-8B-GGUF
  • llava-llama-3.1-8b-mmproj-f16.gguf
  • ⭐⭐⭐Llama-3.1-Unhinged-Vision-8B-Q8.0.gguf
  • https://huggingface.co/Lewdiculous/Eris_PrimeV4-Vision-32k-7B-GGUF-IQ-Imatrix#quantization-information
  • quantization_options = [“Q4_K_M”, “Q4_K_S”, “IQ4_XS”, “Q5_K_M”, “Q5_K_S”,”Q6_K”, “Q8_0”, “IQ3_M”, “IQ3_S”, “IQ3_XXS”]
  • ⭐⭐⭐⭐⭐for low VRAM super small: IQ3_XXS (2.83G)
  • in fact, it’s enough uses.

    • Fixing Spout Post-Build

    By default the spout build config will try to copy to a non-existent Binaries/x64 folder. Fix this by:

    Clicking on SpoutSDK in the Visual Studio Solutions Explorer, and choosing Properties

    Choose BuildEvents->Post-Build Event

    Delete the Command Line option

    Click Apply

    Buy me a Coca cola ☕

    https://buymeacoffee.com/xxoooxx

    Colophon

    Made for fun. I hope if brings you great joy, and perfect hair forever. Contact me with questions and comments, but not

    threats, please. And feel free to contribute! Pull requests and ideas in Discussions or Issues will be taken quite

    seriously!

    — https://decade.tw