图像质量与相似度评估指标 SSIM 和 MS-SSIM 的 Paddle 实现_ms-ssim标准

引入

• 自然图像具有极高的结构性，表现在图像的像素间存在着很强的相关性，尤其是在空间相似的情况下，这些相关性在视觉场景中携带着关于物体结构的重要信息
• 我们假设人类视觉系统主要从可视区域内获取结构信息，人类视觉系统所以通过探测结构信息是否改变来感知图像失真的近似信息
• 大多数的基于误差敏感度（Error Sensitivity）的质量评估方法（如 MSE, PSNR）使用线性变换来分解图像信号，这不会涉及到相关性
• 而接下来要介绍的 SSIM 和 MS-SSIM 方法，便是通过图像结构相关性来衡量图像之间相似度和评估图像质量的两种经典的方法

参考资料

• SSIM 研究主页

• SSIM 论文

• MS-SSIM 论文

• Paddle-MSSSIM

算法介绍

SSIM

• SSIM（Structural SIMilarity）即结构相似性指数，是一种测量两个图像之间相似性的方法

• 假定其中一幅图像具有完美的质量，则 SSIM 指数可以被视为另一幅图像质量的度量。

• SSIM 指数的计算流程如下图所示：

  

• 由 SSIM 测量系统可得相似度的测量可由三种对比模块组成，分别为：亮度（l），对比度（c），结构（s），各个模块的计算公式如下：

  

• 总体的计算公式如下：

  

MS-SSIM

• MS-SSIM（Multi-Scale Structural Similarity）即多尺度结构相似性指数

• 是一种基于多尺度（图片按照一定规则，由大到小缩放）的 SSIM 指数

• 具体的计算公式如下：

  

Paddle 实现

• 基于 Pytorch MS-SSIM 项目开发了一个快速、可微分的 SSIM 和 MS-SSIM 的 Paddle 实现

• 可以通过安装并调用 paddle_msssim 包快速实现 SSIM 和 MS-SSIM 的计算

• Paddle MS-SSIM 与 SKImage、TensorFlow 和 Pytorch MS-SSIM 实现的测试对比结果如下：

  outputs(AMD Ryzen 4600H): 
  
  ===================================
              Test SSIM
  ===================================
  ====> Single Image
  Repeat 10 times
  sigma=0.0 ssim_skimage=1.000000 (247.7732 ms), ssim_tf=1.000000 (277.2696 ms), ssim_paddle=1.000000 (179.4677 ms), ssim_torch=1.000000 (183.6994 ms)
  sigma=10.0 ssim_skimage=0.932399 (226.1620 ms), ssim_tf=0.932640 (257.2435 ms), ssim_paddle=0.932636 (163.2263 ms), ssim_torch=0.932400 (179.1418 ms)
  sigma=20.0 ssim_skimage=0.786023 (224.1826 ms), ssim_tf=0.786032 (279.2126 ms), ssim_paddle=0.786017 (158.3070 ms), ssim_torch=0.786027 (180.0890 ms)
  sigma=30.0 ssim_skimage=0.637174 (237.5582 ms), ssim_tf=0.637183 (267.6092 ms), ssim_paddle=0.637165 (167.9277 ms), ssim_torch=0.637178 (181.7910 ms)
  sigma=40.0 ssim_skimage=0.515865 (221.0388 ms), ssim_tf=0.515876 (264.3230 ms), ssim_paddle=0.515857 (170.7676 ms), ssim_torch=0.515869 (189.0941 ms)
  sigma=50.0 ssim_skimage=0.422551 (222.6846 ms), ssim_tf=0.422558 (273.1971 ms), ssim_paddle=0.422542 (168.3579 ms), ssim_torch=0.422554 (176.7442 ms)
  sigma=60.0 ssim_skimage=0.351337 (215.1536 ms), ssim_tf=0.351340 (270.5560 ms), ssim_paddle=0.351325 (164.3315 ms), ssim_torch=0.351340 (194.6781 ms)
  sigma=70.0 ssim_skimage=0.295752 (210.0273 ms), ssim_tf=0.295756 (272.1814 ms), ssim_paddle=0.295744 (169.3864 ms), ssim_torch=0.295755 (178.9230 ms)
  sigma=80.0 ssim_skimage=0.253164 (239.2978 ms), ssim_tf=0.253169 (260.8894 ms), ssim_paddle=0.253157 (184.7061 ms), ssim_torch=0.253166 (181.4640 ms)
  sigma=90.0 ssim_skimage=0.219240 (224.7329 ms), ssim_tf=0.219245 (270.3727 ms), ssim_paddle=0.219235 (172.3580 ms), ssim_torch=0.219242 (180.5838 ms)
  sigma=100.0 ssim_skimage=0.192630 (238.8582 ms), ssim_tf=0.192634 (261.4317 ms), ssim_paddle=0.192624 (166.0294 ms), ssim_torch=0.192632 (175.7241 ms)
  Pass!
  ====> Batch
  Pass!
  1
  2
  3
  4
  5
  6
  7
  8
  9
  10
  11
  12
  13
  14
  15
  16
  17
  18
  19
  20
  21

  ===================================
              Test MS-SSIM
  ===================================
  ====> Single Image
  Repeat 10 times
  sigma=0.0 msssim_tf=1.000000 (534.9398 ms), msssim_paddle=1.000000 (231.7381 ms), msssim_torch=1.000000 (257.3238 ms)
  sigma=10.0 msssim_tf=0.991148 (525.1758 ms), msssim_paddle=0.991147 (213.8527 ms), msssim_torch=0.991101 (243.9299 ms)
  sigma=20.0 msssim_tf=0.967450 (523.3070 ms), msssim_paddle=0.967447 (217.2415 ms), msssim_torch=0.967441 (253.1073 ms)
  sigma=30.0 msssim_tf=0.934692 (538.5145 ms), msssim_paddle=0.934687 (215.2203 ms), msssim_torch=0.934692 (242.5429 ms)
  sigma=40.0 msssim_tf=0.897363 (558.0346 ms), msssim_paddle=0.897357 (219.1107 ms), msssim_torch=0.897362 (249.1027 ms)
  sigma=50.0 msssim_tf=0.859276 (524.8582 ms), msssim_paddle=0.859267 (232.4189 ms), msssim_torch=0.859275 (263.1328 ms)
  sigma=60.0 msssim_tf=0.820967 (512.8726 ms), msssim_paddle=0.820958 (223.7422 ms), msssim_torch=0.820965 (251.9713 ms)
  sigma=70.0 msssim_tf=0.784204 (529.6149 ms), msssim_paddle=0.784194 (213.1742 ms), msssim_torch=0.784203 (244.9676 ms)
  sigma=80.0 msssim_tf=0.748574 (545.3014 ms), msssim_paddle=0.748563 (222.8581 ms), msssim_torch=0.748572 (261.0413 ms)
  sigma=90.0 msssim_tf=0.715980 (538.3886 ms), msssim_paddle=0.715968 (214.4464 ms), msssim_torch=0.715977 (282.6247 ms)
  sigma=100.0 msssim_tf=0.683882 (540.9150 ms), msssim_paddle=0.683870 (218.5596 ms), msssim_torch=0.683880 (244.1856 ms)
  Pass
  ====> Batch
  Pass
  1
  2
  3
  4
  5
  6
  7
  8
  9
  10
  11
  12
  13
  14
  15
  16
  17
  18
  19

• 具体的安装使用方法如下：

安装

!pip install paddle_msssim
1

计算 SSIM 和 MS-SSIM 指标

• 这里使用如下三张图像来计算他们之间的 SSIM 和 MS-SSIM 指标，结果如下：

  Image
  Simga	0	50	100
  SSIM	1.000000	0.422927	0.192567
  MS-SSIM	1.000000	0.858861	0.684299

• 具体的计算代码如下：

import cv2
import paddle
from paddle_msssim import ssim, ms_ssim

def imread(img_path):
    img = cv2.imread(img_path)
    return paddle.to_tensor(img.transpose(2, 0, 1)[None, ...], dtype=paddle.float32)

simga_0 = imread('./images/simga_0.png')
simga_50 = imread('./images/simga_50.png')
simga_100 = imread('./images/simga_100.png')

ssim_0 = ssim(simga_0, simga_0)
ssim_50 = ssim(simga_0, simga_50)
ssim_100 = ssim(simga_0, simga_100)
print('[SSIM] simga_0: %f simga_50: %f simga_100: %f' % (ssim_0, ssim_50, ssim_100))

ms_ssim_0 = ms_ssim(simga_0, simga_0)
ms_ssim_50 = ms_ssim(simga_0, simga_50)
ms_ssim_100 = ms_ssim(simga_0, simga_100)
print('[MS-SSIM] simga_0: %f simga_50: %f simga_100: %f' % (ms_ssim_0, ms_ssim_50, ms_ssim_100))
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21

[SSIM] simga_0: 1.000000 simga_50: 0.422927 simga_100: 0.192567
[MS-SSIM] simga_0: 1.000000 simga_50: 0.858861 simga_100: 0.684299
1
2

作为损失函数使用

• 随机初始化的一张雪花图像，使用 SSIM 和 MS-SSIM 作为损失函数去拟合目标图像

import os
import sys
import paddle
import numpy as np

from PIL import Image
from paddle.optimizer import Adam
from paddle_msssim import SSIM, MS_SSIM

loss_type = 'ssim'
assert loss_type in ['ssim', 'msssim']

if loss_type == 'ssim':
    loss_obj = SSIM(win_size=11, win_sigma=1.5, data_range=1, size_average=True, channel=3)
else:
    loss_obj = MS_SSIM(win_size=11, win_sigma=1.5, data_range=1, size_average=True, channel=3)


np_img1 = np.array(Image.open("./images/simga_0.png"))

img1 = paddle.to_tensor(np_img1.transpose(2, 0 , 1)).unsqueeze(0) / 255.0
img2 = paddle.rand(img1.shape)

img1 = paddle.to_tensor(img1, stop_gradient=True)
img2 = paddle.to_tensor(img2, stop_gradient=False)

with paddle.no_grad():
    ssim_value = loss_obj(img1, img2).item()
    print("Initial %s: %f:" % (loss_type, ssim_value))

optimizer = Adam(parameters=[img2], learning_rate=0.05)

step = 0
while ssim_value < 0.9999:
    step += 1
    optimizer.clear_grad()
    loss = loss_obj(img1, img2)
    (1 - loss).backward()
    optimizer.step()

    ssim_value = loss.item()
    if step % 10 == 0:
        print('step: %d %s: %f' % (step, loss_type, ssim_value))

img2_ = (img2 * 255.0).squeeze()
np_img2 = img2_.detach().numpy().astype(np.uint8).transpose(1, 2, 0)
results = Image.fromarray(np.concatenate([np_img1, np_img2], 1))
results.save('results_%s.png' % loss_type)
results

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50

Initial ssim: 0.010401:
step: 10 ssim: 0.225660
step: 20 ssim: 0.733606
step: 30 ssim: 0.919254
step: 40 ssim: 0.970057
step: 50 ssim: 0.990348
step: 60 ssim: 0.998122
step: 70 ssim: 0.999767
1
2
3
4
5
6
7
8

[外链图片转存失败,源站可能有防盗链机制,建议将图片保存下来直接上传(img-H0DIo8dz-1646470420843)(output_9_1.png)]

更多

• 更多使用细节和示例可以参考 Paddle-MSSSIM 的 Github 仓库

示例项目（To do…）

• 使用 SSIM 和 MS-SSIM 指标作为损失函数训练一个自编码器（AutoEncoder）模型