ComfyUI 集成混元DIT（comfyui-hydit）

本文主要是介绍ComfyUI 集成混元DIT（comfyui-hydit），希望对大家解决编程问题提供一定的参考价值，需要的开发者们随着小编来一起学习吧！

最近腾讯官方推出了ComfyUI插件comfyui-hydit 。是一个专门为腾讯的 Hunyuan-DiT 模型设计的自定义节点和工作流。本文主要介绍如何通过ComfyUI来运行腾讯新出的支持中文提示词的混元文生图大模型Hunyuan-DiT

环境准备

插件

从腾讯混元DIT 源码库获取插件源码：
https://github.com/Tencent/HunyuanDiT.git
从该仓库中获取：comfyui-hydit

模型

所有模型下载后保存在需要挂载的路径下。

所有模型下载后保存在需要挂载的路径下：/u01/workspace/models/hunyuan

克隆HunyuanDiT模型

有多个模型可选择，我们这里采用了第三个模型：
Tencent-Hunyuan/HunyuanDiT 基础模型
Tencent-Hunyuan/HunyuanDiT-Diffusers 加入Diffusers格式
Tencent-Hunyuan/Distillation 加速版

git clone https://huggingface.co/Tencent-Hunyuan/HunyuanDiT
git clone https://huggingface.co/Tencent-Hunyuan/Distillation

如果想要加速版，以上两个地址的文件都要下载
注意下载完Distillation 模型后，需要拷贝模型pytorch_model_distill.pt到： HunyuanDiT/t2i/model/目录下

下载OpenAI视觉模型

git clone https://hf-mirror.com/openai/clip-vit-large-patch14-336
:::danger
保存位置：/u01/workspace/models/clip-vit-large-patch14-336
源码修改方式请参照混元模型的部署过程。
:::

容器化部署ComfyUI

复制comfyui-hydit插件目录到ComfyUI custom_nodes目录：

├── AIGODLIKE-ComfyUI-Translation
├── comfyui_controlnet_aux
├── ComfyUI_Custom_Nodes_AlekPet
├── ComfyUI-Custom-Scripts
├── comfyui-hydit
├── ComfyUI-Manager
├── ComfyUI_TiledKSampler
├── ComfyUI_UltimateSDUpscale
├── Derfuu_ComfyUI_ModdedNodes
├── efficiency-nodes-comfyui
├── example_node.py.example
├── __pycache__
├── sdxl_prompt_styler
└── websocket_image_save.py

准备Dockerfile文件

准备Dockerfile可以安装预装插件和不预装两种不同的方式，如果不预装插件，可以把插件挂载到宿主机上等启动之在安装插件，本文采用的是预装的方式准备的Dockerfile

FROM pytorch/pytorch:2.3.0-cuda12.1-cudnn8-runtime
ARG DEBIAN_FRONTEND=noninteractive
ENV TZ=Etc/UTC
ENV LANG=zh_CN.UTF-8
WORKDIR /appRUN apt-get update 
RUN apt-get install -y libgl1-mesa-glx libglib2.0-0 libsm6 libxrender1 libxext6 git
RUN rm -rf /var/lib/apt/lists/*
ENV GIT_PYTHON_REFRESH=quiet RUN pip config set global.index-url http://mirrors.aliyun.com/pypi/simple
RUN pip config set install.trusted-host mirrors.aliyun.comCOPY ComfyUI/ /app/ComfyUI/
WORKDIR /app/ComfyUIRUN pip install --use-pep517 -r requirements.txt
RUN pip install -r  custom_nodes/comfyui_controlnet_aux/requirements.txt
RUN pip install -r  custom_nodes/ComfyUI-Manager/requirements.txt
RUN pip install -r  custom_nodes/ComfyUI_Custom_Nodes_AlekPet/ArgosTranslateNode/requirements.txt
RUN pip install -r  custom_nodes/ComfyUI_Custom_Nodes_AlekPet/DeepTranslatorNode/requirements.txt
RUN pip install -r  custom_nodes/ComfyUI_Custom_Nodes_AlekPet/GoogleTranslateNode/requirements.txt
RUN pip install -r  custom_nodes/efficiency-nodes-comfyui/requirements.txt
RUN pip install -r  custom_nodes/comfyui-hydit/requirements.txtEXPOSE 8188
CMD [ "python","main.py ","--port 8188"]

custom_nodes/comfyui-hydit/requirements.txt 修改：

#--extra-index-url https://pypi.ngc.nvidia.com
#--extra-index-url https://download.pytorch.org/whl/cu117
timm
diffusers
peft
protobuf
accelerate
loguru
sentencepiece
cuda-python
polygraphy
pandas
omegaconf
# torch>=2.3.0
# torchvision>=0.18.0
# torchaudio>=2.3.0
xformers>=0.0.26.post1
pytorch_lightning

执行构建

docker buile -t qingcloudtech/comfyui:v1.1 .

启动容器

docker run -it --gpus all \
-p 8188:8188 \
-v /u01/workspace/models/stablediffusion:/app/ComfyUI/models \
-v /u01/workspace/comfyui/output/:/app/ComfyUI/output \
- /u01/workspace/ComfyUI/extra_model_paths.yaml:/app/ComfyUI/extra_model_paths.yaml
qingcloudtech/comfyui:v1.5 python main.py

启动容器（docker-compose方式）

准备配置文件

version: '3.8'
services:comfyui:image: qingcloudtech/comfyui:v1.1container_name: comfyuicommand: python /app/ComfyUI/main.pydeploy:resources:reservations:devices:- driver: nvidiacount: allcapabilities: [gpu]volumes:- /u01/workspace/models/stablediffusion/:/app/ComfyUI/models/- /u01/workspace/comfyui/output/:/app/ComfyUI/output- /u01/workspace/ComfyUI/extra_model_paths.yaml:/app/ComfyUI/extra_model_paths.yamlnetwork_mode: host

执行启动命令：

docker-compose up -d

浏览器访问：

演示访问：

1、浏览器访问：127.0.0.1:8188
2、从web界面中加载如下测试json：

{"last_node_id": 21,"last_link_id": 6,"nodes": [{"id": 19,"type": "DiffusersClipTextEncode","pos": [74,277],"size": {"0": 400,"1": 200},"flags": {},"order": 0,"mode": 0,"outputs": [{"name": "positive","type": "STRINGC","links": [5],"shape": 3,"label": "positive"},{"name": "negative","type": "STRINGC","links": [6],"shape": 3,"label": "negative"}],"properties": {"Node name for S&R": "DiffusersClipTextEncode"},"widgets_values": ["castle,landscape,1girl,loli,silver hair,school_uniform,crystal_earrings,kind_smile,arm_support,in summer,mini_witch_hat,frilled,strappy_heels,book,\n","错误的眼睛，糟糕的人脸，毁容，糟糕的艺术，变形，多余的肢体，模糊的颜色，模糊，重复，病态，残缺，"]},{"id": 16,"type": "DiffusersModelMakeup","pos": [526,393],"size": {"0": 304.79998779296875,"1": 46},"flags": {},"order": 3,"mode": 0,"inputs": [{"name": "pipeline","type": "PIPELINE","link": 2,"label": "pipeline"},{"name": "scheduler","type": "SCHEDULER","link": 3,"label": "scheduler"}],"outputs": [{"name": "MAKED_PIPELINE","type": "MAKED_PIPELINE","links": [4],"shape": 3,"label": "MAKED_PIPELINE"}],"properties": {"Node name for S&R": "DiffusersModelMakeup"}},{"id": 18,"type": "DiffusersSampler","pos": [842,132],"size": [315,454],"flags": {},"order": 4,"mode": 0,"inputs": [{"name": "maked_pipeline","type": "MAKED_PIPELINE","link": 4,"label": "maked_pipeline"},{"name": "positive","type": "STRINGC","link": 5,"label": "positive"},{"name": "negative","type": "STRINGC","link": 6,"label": "negative"}],"outputs": [{"name": "IMAGE","type": "IMAGE","links": [1],"shape": 3,"label": "IMAGE"}],"properties": {"Node name for S&R": "DiffusersSampler"},"widgets_values": [1,1024,1024,30,6,2705109163,"randomize"]},{"id": 6,"type": "PreviewImage","pos": [1186,181],"size": [210,246],"flags": {},"order": 5,"mode": 0,"inputs": [{"name": "images","type": "IMAGE","link": 1,"label": "图像"}],"title": "Preview Image","properties": {"Node name for S&R": "PreviewImage"}},{"id": 21,"type": "DiffusersPipelineLoader","pos": [81,544],"size": {"0": 315,"1": 190},"flags": {},"order": 1,"mode": 0,"outputs": [{"name": "PIPELINE","type": "PIPELINE","links": [2],"shape": 3,"label": "PIPELINE"},{"name": "MODEL","type": "MODEL","links": null,"shape": 3,"label": "MODEL"},{"name": "CLIP","type": "CLIP","links": null,"shape": 3,"label": "CLIP"},{"name": "VAE","type": "VAE","links": null,"shape": 3,"label": "VAE"}],"properties": {"Node name for S&R": "DiffusersPipelineLoader"},"widgets_values": ["ckpts","pytorch_model_distill.pt","disable","diffusers"]},{"id": 15,"type": "DiffusersSchedulerLoader","pos": [73,129],"size": {"0": 315,"1": 58},"flags": {},"order": 2,"mode": 0,"outputs": [{"name": "SCHEDULER","type": "SCHEDULER","links": [3],"shape": 3,"label": "SCHEDULER"}],"properties": {"Node name for S&R": "DiffusersSchedulerLoader"},"widgets_values": ["ddim"]}],"links": [[1,18,0,6,0,"IMAGE"],[2,21,0,16,0,"PIPELINE"],[3,15,0,16,1,"SCHEDULER"],[4,16,0,18,0,"MAKED_PIPELINE"],[5,19,0,18,1,"STRINGC"],[6,19,1,18,2,"STRINGC"]],"groups": [],"config": {},"extra": {"ds": {"scale": 0.9090909090909098,"offset": {"0": 163.4918776991919,"1": 16.755363890844446}}},"version": 0.4
}

3、查看结果：