ComfyUI 集成混元DIT（comfyui-hydit）

作者：代码探险家 | 2024-08-07 22:35:13

踩

hydit

最近腾讯官方推出了ComfyUI插件comfyui-hydit 。是一个专门为腾讯的 Hunyuan-DiT 模型设计的自定义节点和工作流。本文主要介绍如何通过ComfyUI来运行腾讯新出的支持中文提示词的混元文生图大模型Hunyuan-DiT

环境准备

插件

从腾讯混元DIT 源码库获取插件源码：
https://github.com/Tencent/HunyuanDiT.git
从该仓库中获取：comfyui-hydit

模型

所有模型下载后保存在需要挂载的路径下。

所有模型下载后保存在需要挂载的路径下：/u01/workspace/models/hunyuan

克隆HunyuanDiT模型

有多个模型可选择，我们这里采用了第三个模型：
Tencent-Hunyuan/HunyuanDiT 基础模型
Tencent-Hunyuan/HunyuanDiT-Diffusers 加入Diffusers格式
Tencent-Hunyuan/Distillation 加速版

git clone https://huggingface.co/Tencent-Hunyuan/HunyuanDiT
git clone https://huggingface.co/Tencent-Hunyuan/Distillation
1
2

如果想要加速版，以上两个地址的文件都要下载
注意下载完Distillation 模型后，需要拷贝模型pytorch_model_distill.pt到： HunyuanDiT/t2i/model/目录下

下载OpenAI视觉模型

git clone https://hf-mirror.com/openai/clip-vit-large-patch14-336
:::danger
保存位置：/u01/workspace/models/clip-vit-large-patch14-336
源码修改方式请参照混元模型的部署过程。
:::

容器化部署ComfyUI

复制comfyui-hydit插件目录到ComfyUI custom_nodes目录：

├── AIGODLIKE-ComfyUI-Translation
├── comfyui_controlnet_aux
├── ComfyUI_Custom_Nodes_AlekPet
├── ComfyUI-Custom-Scripts
├── comfyui-hydit
├── ComfyUI-Manager
├── ComfyUI_TiledKSampler
├── ComfyUI_UltimateSDUpscale
├── Derfuu_ComfyUI_ModdedNodes
├── efficiency-nodes-comfyui
├── example_node.py.example
├── __pycache__
├── sdxl_prompt_styler
└── websocket_image_save.py

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15

准备Dockerfile文件

准备Dockerfile可以安装预装插件和不预装两种不同的方式，如果不预装插件，可以把插件挂载到宿主机上等启动之在安装插件，本文采用的是预装的方式准备的Dockerfile

FROM pytorch/pytorch:2.3.0-cuda12.1-cudnn8-runtime
ARG DEBIAN_FRONTEND=noninteractive
ENV TZ=Etc/UTC
ENV LANG=zh_CN.UTF-8
WORKDIR /app

RUN apt-get update 
RUN apt-get install -y libgl1-mesa-glx libglib2.0-0 libsm6 libxrender1 libxext6 git
RUN rm -rf /var/lib/apt/lists/*
ENV GIT_PYTHON_REFRESH=quiet 

RUN pip config set global.index-url http://mirrors.aliyun.com/pypi/simple
RUN pip config set install.trusted-host mirrors.aliyun.com

COPY ComfyUI/ /app/ComfyUI/
WORKDIR /app/ComfyUI

RUN pip install --use-pep517 -r requirements.txt
RUN pip install -r  custom_nodes/comfyui_controlnet_aux/requirements.txt
RUN pip install -r  custom_nodes/ComfyUI-Manager/requirements.txt
RUN pip install -r  custom_nodes/ComfyUI_Custom_Nodes_AlekPet/ArgosTranslateNode/requirements.txt
RUN pip install -r  custom_nodes/ComfyUI_Custom_Nodes_AlekPet/DeepTranslatorNode/requirements.txt
RUN pip install -r  custom_nodes/ComfyUI_Custom_Nodes_AlekPet/GoogleTranslateNode/requirements.txt
RUN pip install -r  custom_nodes/efficiency-nodes-comfyui/requirements.txt
RUN pip install -r  custom_nodes/comfyui-hydit/requirements.txt

EXPOSE 8188
CMD [ "python","main.py ","--port 8188"]
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28

custom_nodes/comfyui-hydit/requirements.txt 修改：

#--extra-index-url https://pypi.ngc.nvidia.com
#--extra-index-url https://download.pytorch.org/whl/cu117
timm
diffusers
peft
protobuf
accelerate
loguru
sentencepiece
cuda-python
polygraphy
pandas
omegaconf
# torch>=2.3.0
# torchvision>=0.18.0
# torchaudio>=2.3.0
xformers>=0.0.26.post1
pytorch_lightning

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19

执行构建

docker buile -t qingcloudtech/comfyui:v1.1 .
1

启动容器

docker run -it --gpus all \
-p 8188:8188 \
-v /u01/workspace/models/stablediffusion:/app/ComfyUI/models \
-v /u01/workspace/comfyui/output/:/app/ComfyUI/output \
- /u01/workspace/ComfyUI/extra_model_paths.yaml:/app/ComfyUI/extra_model_paths.yaml
qingcloudtech/comfyui:v1.5 python main.py
1
2
3
4
5
6

启动容器（docker-compose方式）

准备配置文件

version: '3.8'
services:
  comfyui:
    image: qingcloudtech/comfyui:v1.1
    container_name: comfyui
    command: python /app/ComfyUI/main.py
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    volumes:
      - /u01/workspace/models/stablediffusion/:/app/ComfyUI/models/
      - /u01/workspace/comfyui/output/:/app/ComfyUI/output
      - /u01/workspace/ComfyUI/extra_model_paths.yaml:/app/ComfyUI/extra_model_paths.yaml
    network_mode: host
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

执行启动命令：

docker-compose up -d
1

浏览器访问：

演示访问：

1、浏览器访问：127.0.0.1:8188
2、从web界面中加载如下测试json：

{
  "last_node_id": 21,
  "last_link_id": 6,
  "nodes": [
    {
      "id": 19,
      "type": "DiffusersClipTextEncode",
      "pos": [
        74,
        277
      ],
      "size": {
        "0": 400,
        "1": 200
      },
      "flags": {},
      "order": 0,
      "mode": 0,
      "outputs": [
        {
          "name": "positive",
          "type": "STRINGC",
          "links": [
            5
          ],
          "shape": 3,
          "label": "positive"
        },
        {
          "name": "negative",
          "type": "STRINGC",
          "links": [
            6
          ],
          "shape": 3,
          "label": "negative"
        }
      ],
      "properties": {
        "Node name for S&R": "DiffusersClipTextEncode"
      },
      "widgets_values": [
        "castle,landscape,1girl,loli,silver hair,school_uniform,crystal_earrings,kind_smile,arm_support,in summer,mini_witch_hat,frilled,strappy_heels,book,\n",
        "错误的眼睛，糟糕的人脸，毁容，糟糕的艺术，变形，多余的肢体，模糊的颜色，模糊，重复，病态，残缺，"
      ]
    },
    {
      "id": 16,
      "type": "DiffusersModelMakeup",
      "pos": [
        526,
        393
      ],
      "size": {
        "0": 304.79998779296875,
        "1": 46
      },
      "flags": {},
      "order": 3,
      "mode": 0,
      "inputs": [
        {
          "name": "pipeline",
          "type": "PIPELINE",
          "link": 2,
          "label": "pipeline"
        },
        {
          "name": "scheduler",
          "type": "SCHEDULER",
          "link": 3,
          "label": "scheduler"
        }
      ],
      "outputs": [
        {
          "name": "MAKED_PIPELINE",
          "type": "MAKED_PIPELINE",
          "links": [
            4
          ],
          "shape": 3,
          "label": "MAKED_PIPELINE"
        }
      ],
      "properties": {
        "Node name for S&R": "DiffusersModelMakeup"
      }
    },
    {
      "id": 18,
      "type": "DiffusersSampler",
      "pos": [
        842,
        132
      ],
      "size": [
        315,
        454
      ],
      "flags": {},
      "order": 4,
      "mode": 0,
      "inputs": [
        {
          "name": "maked_pipeline",
          "type": "MAKED_PIPELINE",
          "link": 4,
          "label": "maked_pipeline"
        },
        {
          "name": "positive",
          "type": "STRINGC",
          "link": 5,
          "label": "positive"
        },
        {
          "name": "negative",
          "type": "STRINGC",
          "link": 6,
          "label": "negative"
        }
      ],
      "outputs": [
        {
          "name": "IMAGE",
          "type": "IMAGE",
          "links": [
            1
          ],
          "shape": 3,
          "label": "IMAGE"
        }
      ],
      "properties": {
        "Node name for S&R": "DiffusersSampler"
      },
      "widgets_values": [
        1,
        1024,
        1024,
        30,
        6,
        2705109163,
        "randomize"
      ]
    },
    {
      "id": 6,
      "type": "PreviewImage",
      "pos": [
        1186,
        181
      ],
      "size": [
        210,
        246
      ],
      "flags": {},
      "order": 5,
      "mode": 0,
      "inputs": [
        {
          "name": "images",
          "type": "IMAGE",
          "link": 1,
          "label": "图像"
        }
      ],
      "title": "Preview Image",
      "properties": {
        "Node name for S&R": "PreviewImage"
      }
    },
    {
      "id": 21,
      "type": "DiffusersPipelineLoader",
      "pos": [
        81,
        544
      ],
      "size": {
        "0": 315,
        "1": 190
      },
      "flags": {},
      "order": 1,
      "mode": 0,
      "outputs": [
        {
          "name": "PIPELINE",
          "type": "PIPELINE",
          "links": [
            2
          ],
          "shape": 3,
          "label": "PIPELINE"
        },
        {
          "name": "MODEL",
          "type": "MODEL",
          "links": null,
          "shape": 3,
          "label": "MODEL"
        },
        {
          "name": "CLIP",
          "type": "CLIP",
          "links": null,
          "shape": 3,
          "label": "CLIP"
        },
        {
          "name": "VAE",
          "type": "VAE",
          "links": null,
          "shape": 3,
          "label": "VAE"
        }
      ],
      "properties": {
        "Node name for S&R": "DiffusersPipelineLoader"
      },
      "widgets_values": [
        "ckpts",
        "pytorch_model_distill.pt",
        "disable",
        "diffusers"
      ]
    },
    {
      "id": 15,
      "type": "DiffusersSchedulerLoader",
      "pos": [
        73,
        129
      ],
      "size": {
        "0": 315,
        "1": 58
      },
      "flags": {},
      "order": 2,
      "mode": 0,
      "outputs": [
        {
          "name": "SCHEDULER",
          "type": "SCHEDULER",
          "links": [
            3
          ],
          "shape": 3,
          "label": "SCHEDULER"
        }
      ],
      "properties": {
        "Node name for S&R": "DiffusersSchedulerLoader"
      },
      "widgets_values": [
        "ddim"
      ]
    }
  ],
  "links": [
    [
      1,
      18,
      0,
      6,
      0,
      "IMAGE"
    ],
    [
      2,
      21,
      0,
      16,
      0,
      "PIPELINE"
    ],
    [
      3,
      15,
      0,
      16,
      1,
      "SCHEDULER"
    ],
    [
      4,
      16,
      0,
      18,
      0,
      "MAKED_PIPELINE"
    ],
    [
      5,
      19,
      0,
      18,
      1,
      "STRINGC"
    ],
    [
      6,
      19,
      1,
      18,
      2,
      "STRINGC"
    ]
  ],
  "groups": [],
  "config": {},
  "extra": {
    "ds": {
      "scale": 0.9090909090909098,
      "offset": {
        "0": 163.4918776991919,
        "1": 16.755363890844446
      }
    }
  },
  "version": 0.4
}
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
174
175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
205
206
207
208
209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326

3、查看结果：

【Qinghub Studio 】更适合开发人员的低代码开源开发平台
【QingHub企业级应用统一部署】
【QingHub企业级应用开发管理】
【QingHub演示】
【https://qingplus.cn】

声明：本文内容由网友自发贡献，不代表【wpsshop博客】立场，版权归原作者所有，本站不承担相应法律责任。如您发现有侵权的内容，请联系我们。转载请注明出处：https://www.wpsshop.cn/w/代码探险家/article/detail/944813