GAN的图像修复：多样化补全

点击上方“机器学习与生成对抗网络”，关注"星标"

获取有趣、好玩的前沿干货！

2019 cvpr：Pluralistic Image Completion

https://arxiv.xilesou.top/pdf/1903.04227.pdf

https://github.com/lyndonzheng/Pluralistic-Inpainting

• 对于每个输入（masked input），大多数图像补全方法只能产生一种结果（尽管有许多合理的其它可能结果）。基于学习的方法里，通常每个标签只有一个ground true（目标参照图象GT）。即便从有条件VAE采样、仍然会多样性不足。本文提出了一种多元化图像补全方法。

• 提出了一种具有两个平行路径的概率学习框架。一个是重建路径（网络），它只利用对应一个的GT去获取缺失区域的先验信息，并完成重建；另一个是生成路径（网络），它将条件先验耦合于重建路径所获得的分布。在对抗方式下完成训练，训练完后只用生成路径。

• 提出了一种新的、建模长短区域关系的注意力机制，以提高图像一致性。

• 在建筑、人脸（Celeba-HQ）、ImageNet等数据集上不仅取得了更好的补全效果，在多样性上也合理、令人信服。

方法

• 定义 为原始完整图像， 是被遮挡的（掩masked）图像，则经典的图像补全方法是去学习映射 ，它们是确定性的。

• 本文还定义 表示 的“反”，也就是它仅仅是由被遮挡部分构成，而本文的目标是从 采样去恢复。

---

• 损失函数：

进一步地：

• 分布正则化（参照VAE/CVAE）：

对于重建路径：

对于生成路径：

（这里的KL，个人感觉不应该带负号啊？？

• 图像外表匹配：

对于重建路径，约束重建图像和目标参考GT相似：

对于生成路径，约束生成和GT相似：

• 对抗损失：

在特征层面的约束和LSGAN损失：

---

• 网络结构之attention设计

在前面网络整体的图所示，decoder之前的红色模块即为本文所提出的attention设计：

部分效果

---

公众号近期荐读：

• GAN整整6年了！是时候要来捋捋了！ 

• 新手指南综述 | GAN模型太多，不知道选哪儿个？
  

• 数百篇GAN论文已下载好！配一份生成对抗网络最新综述！

• 图卷积网络GCN的理解与介绍
  

• CVPR2020之MSG-GAN：简单有效的SOTA
  

• CVPR2020之姿势变换GAN：图像里谁都会劈叉？
  

• CVPR2020之多码先验GAN：预训练模型如何使用？
  

• 两幅图像！这样能训练好GAN做图像转换吗？
  

• 有点夸张、有点扭曲！速览这些GAN如何夸张漫画化人脸！
  

• 见微知细之超分辨率GAN！附70多篇论文下载！
  

• 天降斯雨，于我却无！GAN用于去雨如何？
  

• 脸部转正！GAN能否让侧颜杀手、小猪佩奇真容无处遁形？
  

• 容颜渐失！GAN来预测？
  

• 强数据所难！SSL（半监督学习）结合GAN如何？
  

• 弱水三千，只取你标！AL（主动学习）结合GAN如何？
  

• 异常检测，GAN如何gan ?

• 虚拟换衣！速览这几篇最新论文咋做的！
  

• 脸部妆容迁移！速览几篇用GAN来做的论文

• 【1】GAN在医学图像上的生成，今如何？

• 01-GAN公式简明原理之铁甲小宝篇

GAN&CV交流群，无论小白还是大佬，诚挚邀您加入！

一起讨论交流！长按备注【进群】加入：

更多分享、长按关注本公众号：

GAN图像修复/补全生成-相关阅读：

001  (2020-04-8) Attentive Normalization for Conditional Image Generation

     https://arxiv.xilesou.top/pdf/2004.03828.pdf

002  (2020-03-27) GAN-based Priors for Quantifying Uncertainty

     https://arxiv.xilesou.top/pdf/2003.12597.pdf

003  (2020-02-6) Automatic Segmentation and Visualization of Choroid in OCT with Knowledge Infused Deep Learning

     https://arxiv.xilesou.top/pdf/2002.04712.pdf

004  (2020-02-7) Local Facial Attribute Transfer through Inpainting

     https://arxiv.xilesou.top/pdf/2002.03040.pdf

005  (2020-02-5) Domain Embedded Multi-model Generative Adversarial Networks for Image-based Face Inpainting

     https://arxiv.xilesou.top/pdf/2002.02909.pdf

006  (2020-02-6) Image Fine-grained Inpainting

     https://arxiv.xilesou.top/pdf/2002.02609.pdf

007  (2020-02-4) Pixel-wise Conditioned Generative Adversarial Networks for Image Synthesis and Completion

     https://arxiv.xilesou.top/pdf/2002.01281.pdf

008  (2020-01-11) Symmetric Skip Connection Wasserstein GAN for High-Resolution Facial Image Inpainting

     https://arxiv.xilesou.top/pdf/2001.03725.pdf

009  (2019-12-31) Erase and Restore  Simple Accurate and Resilient Detection of $L 2$ Adversarial Examples

     https://arxiv.xilesou.top/pdf/2001.00116.pdf

010  (2019-12-18) Unsupervised Adversarial Image Inpainting

     https://arxiv.xilesou.top/pdf/1912.12164.pdf

011  (2020-02-15) Image Outpainting and Harmonization using Generative Adversarial Networks

     https://arxiv.xilesou.top/pdf/1912.10960.pdf

012  (2020-03-31) Image Processing Using Multi-Code GAN Prior

     https://arxiv.xilesou.top/pdf/1912.07116.pdf

013  (2019-12-9) Environment reconstruction on depth images using Generative Adversarial Networks

     https://arxiv.xilesou.top/pdf/1912.03992.pdf

014  (2019-12-5) Blind Inpainting of Large-scale Masks of Thin Structures with Adversarial and Reinforcement Learning

     https://arxiv.xilesou.top/pdf/1912.02470.pdf

015  (2020-03-19) PiiGAN  Generative Adversarial Networks for Pluralistic Image Inpainting

     https://arxiv.xilesou.top/pdf/1912.01834.pdf

016  (2019-11-2) Pixel-wise Conditioning of Generative Adversarial Networks

     https://arxiv.xilesou.top/pdf/1911.00689.pdf

017  (2019-10-23) Facial Expression Restoration Based on Improved Graph Convolutional Networks

     https://arxiv.xilesou.top/pdf/1910.10344.pdf

018  (2019-09-27) Region-wise Generative Adversarial ImageInpainting for Large Missing Areas

     https://arxiv.xilesou.top/pdf/1909.12507.pdf

019  (2019-12-23) Physics-informed semantic inpainting  Application to geostatistical modeling

     https://arxiv.xilesou.top/pdf/1909.09459.pdf

020  (2019-08-19) Boundless  Generative Adversarial Networks for Image Extension

     https://arxiv.xilesou.top/pdf/1908.07007.pdf

021  (2019-08-19) Fully Automated Image De-fencing using Conditional Generative Adversarial Networks

     https://arxiv.xilesou.top/pdf/1908.06837.pdf

022  (2019-08-16) The Angel is in the Priors  Improving GAN based Image and Sequence Inpainting with Better Noise and Structural Priors

     https://arxiv.xilesou.top/pdf/1908.05861.pdf

023  (2019-08-14) Faster Unsupervised Semantic Inpainting  A GAN Based Approach

     https://arxiv.xilesou.top/pdf/1908.04968.pdf

024  (2019-10-29) Generative Modeling by Estimating Gradients of the Data Distribution

     https://arxiv.xilesou.top/pdf/1907.05600.pdf

025  (2019-06-14) Single-Channel Signal Separation and Deconvolution with Generative Adversarial Networks

     https://arxiv.xilesou.top/pdf/1906.07552.pdf

026  (2019-06-6) How to make a pizza  Learning a compositional layer-based GAN model

     https://arxiv.xilesou.top/pdf/1906.02839.pdf

027  (2019-09-28) Fashion Editing with Adversarial Parsing Learning

     https://arxiv.xilesou.top/pdf/1906.00884.pdf

028  (2019-06-3) GazeCorrection Self-Guided Eye Manipulation in the wild using Self-Supervised Generative Adversarial Networks

     https://arxiv.xilesou.top/pdf/1906.00805.pdf

029  (2019-05-23) Generative Imaging and Image Processing via Generative Encoder

     https://arxiv.xilesou.top/pdf/1905.13300.pdf

030  (2020-01-30) Training Generative Adversarial Networks from Incomplete Observations using Factorised Discriminators

     https://arxiv.xilesou.top/pdf/1905.12660.pdf

031  (2019-09-27) Invertible generative models for inverse problems  mitigating representation error and dataset bias

     https://arxiv.xilesou.top/pdf/1905.11672.pdf

032  (2020-03-6) PEPSI++  Fast and Lightweight Network for Image Inpainting

     https://arxiv.xilesou.top/pdf/1905.09010.pdf

033  (2019-05-6) The Missing Data Encoder  Cross-Channel Image Completion\\with Hide-And-Seek Adversarial Network

     https://arxiv.xilesou.top/pdf/1905.01861.pdf

034  (2019-07-10) Learning Pyramid-Context Encoder Network for High-Quality Image Inpainting

     https://arxiv.xilesou.top/pdf/1904.07475.pdf

035  (2019-04-15) A deep learning framework for quality assessment and restoration in video endoscopy

     https://arxiv.xilesou.top/pdf/1904.07073.pdf

036  (2019-03-28) GANs-NQM  A Generative Adversarial Networks based No Reference Quality Assessment Metric for RGB-D Synthesized Views

     https://arxiv.xilesou.top/pdf/1903.12088.pdf

037  (2019-11-6) Implicit Generation and Generalization in Energy-Based Models

     https://arxiv.xilesou.top/pdf/1903.08689.pdf

038  (2019-03-10) A Sinogram Inpainting Method based on Generative Adversarial Network for Limited-angle Computed Tomography

     https://arxiv.xilesou.top/pdf/1903.03984.pdf

039  (2019-10-23) GAN-based Projector for Faster Recovery with Convergence Guarantees in Linear Inverse Problems

     https://arxiv.xilesou.top/pdf/1902.09698.pdf

040  (2019-02-10) Cross-spectral Face Completion for NIR-VIS Heterogeneous Face Recognition

     https://arxiv.xilesou.top/pdf/1902.03565.pdf

041  (2019-01-25) Diversity-Sensitive Conditional Generative Adversarial Networks

     https://arxiv.xilesou.top/pdf/1901.09024.pdf

042  (2019-01-9) Detecting Overfitting of Deep Generative Networks via Latent Recovery

     https://arxiv.xilesou.top/pdf/1901.03396.pdf

043  (2019-01-11) EdgeConnect  Generative Image Inpainting with Adversarial Edge Learning

     https://arxiv.xilesou.top/pdf/1901.00212.pdf

044  (2019-07-30) Exploiting the Inherent Limitation of L0 Adversarial Examples

     https://arxiv.xilesou.top/pdf/1812.09638.pdf

045  (2019-11-27) Face Completion with Semantic Knowledge and Collaborative Adversarial Learning

     https://arxiv.xilesou.top/pdf/1812.03252.pdf

046  (2018-12-3) Semantic Image Inpainting Through Improved Wasserstein Generative Adversarial Networks

     https://arxiv.xilesou.top/pdf/1812.01071.pdf

047  (2019-04-9) Context Encoding Chest X-rays

     https://arxiv.xilesou.top/pdf/1812.00964.pdf

048  (2019-02-26) Void Filling of Digital Elevation Models with Deep Generative Models

     https://arxiv.xilesou.top/pdf/1811.12693.pdf

049  (2019-01-15) SEIGAN  Towards Compositional Image Generation by Simultaneously Learning to Segment Enhance and Inpaint

     https://arxiv.xilesou.top/pdf/1811.07630.pdf

050  (2020-01-11) On Hallucinating Context and Background Pixels from a Face Mask using Multi-scale GANs

     https://arxiv.xilesou.top/pdf/1811.07104.pdf

051  (2018-10-20) Improved Techniques for GAN based Facial Inpainting

     https://arxiv.xilesou.top/pdf/1810.08774.pdf

052  (2018-10-15) Adversarial Inpainting of Medical Image Modalities

     https://arxiv.xilesou.top/pdf/1810.06621.pdf

053  (2019-02-15) Empty Cities  Image Inpainting for a Dynamic-Object-Invariant Space

     https://arxiv.xilesou.top/pdf/1809.10239.pdf

054  (2018-09-6) Dense Pose Transfer

     https://arxiv.xilesou.top/pdf/1809.01995.pdf

055  (2018-08-29) Chest X-ray Inpainting with Deep Generative Models

     https://arxiv.xilesou.top/pdf/1809.01471.pdf

056  (2018-08-25) Painting Outside the Box  Image Outpainting with GANs

     https://arxiv.xilesou.top/pdf/1808.08483.pdf

057  (2019-07-18) Cross-view image synthesis using geometry-guided conditional GANs

     https://arxiv.xilesou.top/pdf/1808.05469.pdf

058  (2018-08-6) X-GANs  Image Reconstruction Made Easy for Extreme Cases

     https://arxiv.xilesou.top/pdf/1808.04432.pdf

059  (2018-11-2) Latent Convolutional Models

     https://arxiv.xilesou.top/pdf/1806.06284.pdf

060  (2018-06-13) Self-Supervised Feature Learning by Learning to Spot Artifacts

     https://arxiv.xilesou.top/pdf/1806.05024.pdf

061  (2018-07-17) AVID  Adversarial Visual Irregularity Detection

     https://arxiv.xilesou.top/pdf/1805.09521.pdf

062  (2018-05-21) Unsupervised Deep Context Prediction for Background Foreground Separation

     https://arxiv.xilesou.top/pdf/1805.07903.pdf

063  (2018-06-4) An Unsupervised Approach to Solving Inverse Problems using Generative Adversarial Networks

     https://arxiv.xilesou.top/pdf/1805.07281.pdf

064  (2019-07-20) Generative Steganography by Sampling

     https://arxiv.xilesou.top/pdf/1804.10531.pdf

065  (2018-04-25) Generative Model for Heterogeneous Inference

     https://arxiv.xilesou.top/pdf/1804.09858.pdf

066  (2018-05-14) The Reincarnation of Grille Cipher  A Generative Approach

     https://arxiv.xilesou.top/pdf/1804.06514.pdf

067  (2018-04-3) Correlated discrete data generation using adversarial training

     https://arxiv.xilesou.top/pdf/1804.00925.pdf

068  (2018-03-27) Structural inpainting

     https://arxiv.xilesou.top/pdf/1803.10348.pdf

069  (2018-05-10) Digital Cardan Grille  A Modern Approach for Information Hiding

     https://arxiv.xilesou.top/pdf/1803.09219.pdf

070  (2018-03-27) Image Inpainting using Block-wise Procedural Training with Annealed Adversarial Counterpart

     https://arxiv.xilesou.top/pdf/1803.08943.pdf

071  (2018-03-20) Patch-Based Image Inpainting with Generative Adversarial Networks

     https://arxiv.xilesou.top/pdf/1803.07422.pdf

072  (2018-07-5) Generating Realistic Geology Conditioned on Physical Measurements with Generative Adversarial Networks

     https://arxiv.xilesou.top/pdf/1802.03065.pdf

073  (2018-08-30) Deep Structured Energy-Based Image Inpainting

     https://arxiv.xilesou.top/pdf/1801.07939.pdf

074  (2017-12-20) Context-Aware Semantic Inpainting

     https://arxiv.xilesou.top/pdf/1712.07778.pdf

075  (2017-12-11) GibbsNet  Iterative Adversarial Inference for Deep Graphical Models

     https://arxiv.xilesou.top/pdf/1712.04120.pdf

076  (2018-08-6) Discriminative Region Proposal Adversarial Networks for High-Quality Image-to-Image Translation

     https://arxiv.xilesou.top/pdf/1711.09554.pdf

077  (2017-11-26) Semantically Consistent Image Completion with Fine-grained Details

     https://arxiv.xilesou.top/pdf/1711.09345.pdf

078  (2017-11-17) Improving Consistency and Correctness of Sequence Inpainting using Semantically Guided Generative Adversarial Network

     https://arxiv.xilesou.top/pdf/1711.06106.pdf

079  (2019-04-2) Perceptual Adversarial Networks for Image-to-Image Transformation

     https://arxiv.xilesou.top/pdf/1706.09138.pdf

080  (2017-12-6) r-BTN  Cross-domain Face Composite and Synthesis from Limited Facial Patches

     https://arxiv.xilesou.top/pdf/1706.00556.pdf

081  (2017-09-1) Adversarial Inverse Graphics Networks  Learning 2D-to-3D Lifting and Image-to-Image Translation from Unpaired Supervision

     https://arxiv.xilesou.top/pdf/1705.11166.pdf

082  (2017-10-12) CVAE-GAN  Fine-Grained Image Generation through Asymmetric Training

     https://arxiv.xilesou.top/pdf/1703.10155.pdf

083  (2016-11-21) Context Encoders  Feature Learning by Inpainting

     https://arxiv.xilesou.top/pdf/1604.07379.pdf