Stable Diffusion原理_stable diffusion的理论基础

# images in skimage to use and their textual descriptions
descriptions = {
    "page": "a page of text about segmentation",
    "chelsea": "a facial photo of a tabby cat",
    "astronaut": "a portrait of an astronaut with the American flag",
    "rocket": "a rocket standing on a launchpad",
    "motorcycle_right": "a red motorcycle standing in a garage",
    "camera": "a person looking at a camera on a tripod",
    "horse": "a black-and-white silhouette of a horse", 
    "coffee": "a cup of coffee on a saucer"
}
1
2
3
4
5
6
7
8
9
10
11

2.3.4.2、计算余弦相似度

2.3.4.3、Zero-Shot图像分类

from torchvision.datasets import CIFAR100
1

cifar100 = CIFAR100(os.path.expanduser(“~/.cache”), transform=preprocess, download=True)

text_descriptions = [f"This is a photo of a { label}" for label in cifar100.classes]
text_tokens = clip.tokenize(text_descriptions).cuda()

with torch.no_grad():
text_features = model.encode_text(text_tokens).float()
text_features /= text_features.norm(dim=-1, keepdim=True)

text_probs = (100.0 * image_features @ text_features.T).softmax(dim=-1)
top_probs, top_labels = text_probs.cpu().topk(5, dim=-1)

plt.figure(figsize=(16, 16))

for i, image in enumerate(original_images):
plt.subplot(4, 4, 2 * i + 1)
plt.imshow(image)
plt.axis(“off”)

plt<span class="token punctuation">.</span>subplot<span class="token punctuation">(</span><span class="token number">4</span><span class="token punctuation">,</span> <span class="token number">4</span><span class="token punctuation">,</span> <span class="token number">2</span> <span class="token operator">*</span> i <span class="token operator">+</span> <span class="token number">2</span><span class="token punctuation">)</span>
y <span class="token operator">=</span> np<span class="token punctuation">.</span>arange<span class="token punctuation">(</span>top_probs<span class="token punctuation">.</span>shape<span class="token punctuation">[</span><span class="token operator">-</span><span class="token number">1</span><span class="token punctuation">]</span><span class="token punctuation">)</span>
plt<span class="token punctuation">.</span>grid<span class="token punctuation">(</span><span class="token punctuation">)</span>
plt<span class="token punctuation">.</span>barh<span class="token punctuation">(</span>y<span class="token punctuation">,</span> top_probs<span class="token punctuation">[</span>i<span class="token punctuation">]</span><span class="token punctuation">)</span>
plt<span class="token punctuation">.</span>gca<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span>invert_yaxis<span class="token punctuation">(</span><span class="token punctuation">)</span>
plt<span class="token punctuation">.</span>gca<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">.</span>set_axisbelow<span class="token punctuation">(</span><span class="token boolean">True</span><span class="token punctuation">)</span>
plt<span class="token punctuation">.</span>yticks<span class="token punctuation">(</span>y<span class="token punctuation">,</span> <span class="token punctuation">[</span>cifar100<span class="token punctuation">.</span>classes<span class="token punctuation">[</span>index<span class="token punctuation">]</span> <span class="token keyword">for</span> index <span class="token keyword">in</span> top_labels<span class="token punctuation">[</span>i<span class="token punctuation">]</span><span class="token punctuation">.</span>numpy<span class="token punctuation">(</span><span class="token punctuation">)</span><span class="token punctuation">]</span><span class="token punctuation">)</span>
plt<span class="token punctuation">.</span>xlabel<span class="token punctuation">(</span><span class="token string">"probability"</span><span class="token punctuation">)</span>
1
2
3
4
5
6
7
8

plt.subplots_adjust(wspace=0.5)
plt.show()
、

2.4、补充：Stable Diffusion训练的四个主流AI模型

1. Dreambooth：会使用正则化。通常只用少量图片做输入微调，就可以做一些其他扩散模型不能或者不擅长的事情——具备个性化结果的能力，既包括文本到图像模型生成的结果，也包括用户输入的任何图片；

2. text-inversion：通过控制文本到图像的管道，标记特定的单词，在文本提示中使用，以实现对生成图像的细粒度控制；

3. LoRA：大型语言模型的低阶自适应，简化过程降低硬件需求，详情请学习LoRA模型原理；

4. Hypernetwork：这是连接到Stable Diffusion模型上的一个小型神经网络，是噪声预测器U-Net的交叉互视（cross-attention）模块；

四个主流模型的区别：

• Dreambooth最直接但非常复杂占内存大，用的人很多评价好；
• text-inversion很聪明，不用重新创作一个新模型，所有人都可以下载并运用到自己的模型，模型小，存储空间占用小；
• LoRA可以在不做完整模型拷贝的情况下，让模型理解这个概念，速度快；
• Hypernetwork：没有官方论文；

三、补充：四大生成模型对比

GAN生成对抗模型、VAE变微分自动编码器、流模型、DM扩散模型

3.1、GAN生成对抗模型

1. GAN模型要同时训练两个网络，难度较大，多模态分布学习困难；
2. 不容易收敛，不好观察损失；
3. 图像特征多样性较差，容易出现模型坍缩，只关注如何骗过判别器；