devbox
Li_阴宅
这个屌丝很懒，什么也没留下！
关注作者
热门标签
热门文章
当前位置:   article > 正文

视觉问答学习（一）——视觉问答的动态记忆网络DMN+（tensorflow实现）_视觉问答模型

作者：Li_阴宅 | 2024-08-17 04:00:24

赞

踩

视觉问答模型

一、背景

本文实现的模型来自于论文：《Dynamic Memory Networks for Visual and Textual Question Answering》

之前做了两个月杂活，最近该上手实验了，这里先从别人的实验开始学习。这篇是视觉问答实验的第一篇。

实验数据比较多，图片用的是COCO的，文本标注是VQA 1.0的，另外还用到了vgg16，所以需要准备的东西也非常多。

二、论文简介

论文的下载链接为：https://arxiv.org/pdf/1603.01417.pdf

先给出论文的摘要：

Neural network architectures with memory and attention mechanisms exhibit certain reasoning capabilities required for question answering. One such architecture, the dynamic memory network (DMN), obtained high accuracy on a variety of language tasks. However, it was not shown whether the architecture achieves strong results for question answering when supporting facts are not marked during training or whether it could be applied to other modalities such as images. Based on an analysis of the DMN, we propose several improvements to its memory and input modules. Together with these changes we introduce a novel input module for images in order to be able to answer visual questions. Our new DMN+ model improves the state of the art on both the Visual Question Answering dataset and the bAbI-10k text question-answering dataset without supporting fact supervision.

对于视觉问答，神经网络的记忆模块和注意力机制展示出了很好的推理能力。一种动态记忆网络（DMN），在多种语言任务中获得了很高的精度。然而，这并未体现出，当支持的事实在训练中没有被标记时，该结构是否在回答问题上取得了准确的结果，或者它是否可以应用于其他模态，比如图像。基于DMN的分析，作者对于它的记忆模块和输入模块提出了多个改进。结合这些改进，作者引入一种新的图像输入模块以实现视觉问答。作者提出的DMN+模型，在没有任何监督信息的条件下，在VQA数据集和bAbI-10k文本问答数据集上都表现出了最好的效果。

记忆机制能够一定程度的处理推理关系，注意力机制也成功用在了机器翻译和看图说话的模型中。而动态记忆网络DMN（Dynamic Memory Network）就是一个同时使用了这两种机制的神经网络模型。本文分了了DMN的组成，主要是输入模块和记忆模块，以改善问答的效果。作者的主要创新在于：

We propose a new input module which uses a two level encoder with a sentence reader and input fusion layer to allow for information flow between sentences. （提出了一个输入模块。该模块使用了由句子阅读器和输入融合层构成的二级编码器，以实现信息在句子间的流动）

For the memory, we propose a modification to gated recurrent units (GRU). The new GRU formulation incorporates attention gates that are computed using global knowledge over the facts.（对于记忆模块，使用了一个修改的GRU单元。新的GRU公式包含了注意力门，该注意力门由信息中的全局知识计算得到）

In addition, we introduce a new input module to represent images.（另外还提出了一个输入模块来表示图像）

这个新的DMN+模型的最大优势在于，训练过程中不需要任何由标签的支持信息（比如回答一个特殊问题的有关信息）。

下图展示了修改的记忆模块对于文本问答和视觉问答的改进：

1. 动态记忆网络DMN（Dynamic Memory Networks）

DMN是一个用于问答的模型，它由多个不同的模块组成，比如输入表示模块，记忆模块等。部分模块如前面的Figure 1所表示的那样：

输入模块（Input Module）：该模块用于处理输入的数据，主要是将提问的问题转化为一组向量，并称为facts，记为F = [f_{1}, ······ , f_{N}]，N表示所有facts的数量，这组向量可以用于后续的模块中。该输入模块在文本问答中是由许多GRU单元组成，若用x_{i}表示第i步的输入，前一步的隐含状态（hidden state）为h_{i-1}，那么当前步的隐含状态h_{i}=GRU(x_{i}, h_{i-1})可以由下式计算出：

问题模块（Question Module）：该模块用于计算问题q的向量表示，其中q是问题中的单词在GRU中的最终隐含状态。

情景记忆模块（Episodic Memory Module）：情景记忆旨在从输入的facts中检索能够回答问题q的信息。为了改进对输入问题的理解，特别是当问题需要涉及推理，情景记忆模块会多次传递输入，每次传递后都会更新情景记忆。情景记忆模块由两个部分组成：注意力机制和记忆更新机制。注意力机制用于产生上下文向量c^{t}和通过问题q和前一次的情景记忆m^{t-1}推断的相关性，其中c^{t}是对传入t的相关输入的总结。记忆更新机制是基于上下文向量c^{t}和前一次的情景记忆m^{t-1}，生成新的情景记忆m^{t}。

回答模块（Answer Module）：回答模块是根据q和m^{t}来生成模型的预测答案。对于一个简单的答案，比如一个单词，使用一个softmax激活的线性层即可。对于复杂答案，比如一个句子，可以用RNN来解码问题q和情景记忆m^{t}链接a=[q;m^{t}]至单词的有序排列。训练过程可以采用交叉熵。

2. 改进的动态记忆网络DMN+（Improved Dynamic Memory Networks: DMN+）

作者提出的模型主要修改了两个地方：一个是输入表示，另一个是注意力机制和输入更新。文本问答（Text QA）和视觉问答（VQA）不同之处仅在于输入表示。

（1）文本问答的输入模块（Input Module for Text QA）

在DMN中，GRU被用于处理文本中的单词，提取句子的表示。但是，该方法在有支持信息（supporting facts）的bAbI-1k数据集上表现较好，在没有支持信息（supporting facts）的bAbI-10k数据集上表现很差。作者推测有两个原因：首先，GRU只允许句子前面的句子有上下文，但不允许后面的有。这组织了信息在后面句子中的传播。其次，支持语句可能在词级相距过于遥远，使得这些遥远的句子无法通过词级GRU相互作用。

输入融合层（Input Fusion Layer）：在DMN+中，我们用两个部分替代了单个GRU。第一个是句子阅读器（sentence reader），用于将单词编码为句子嵌入；第二部分是输入融合层（input fusion layer），能够进行句子间的交互，它类似于分层神经自编码器结构。输入融合层采用双向GRU（bi-directional GRU），它能实现信息在句子前后的传递。因为梯度不需要借助句子间的单词进行传播，融合层也能实现远距离的支持语句（supporting sentences）进行更多的直接交互。

下图的Fig 2表示了一个输入模块，句子阅读器（sentence reader）使用了位置编码器（positional encoder），输入融合曾使用了双向GRU，每一个句子编码f_{i}是使用词特征[w_{1}^{i}, ......, w_{M_{i}}^{i}]的编码输出，其中M_{i}是句子的长度：

句子阅读器（sentence reader）能够使用多种编码机制。这里作者使用了位置编码。由于GRU和LSTM需要更多的计算资源而且在大量任务情况下（比如重构原始语句）容易过拟合，因此没有使用。

对于位置编码机制，句子表示可以利用公式：fi=∑j=1Mlj∘wij生成，其中\circ表示矩阵点乘，l_{j}是一个列向量，且l_{jd}=\left ( 1-j/M \right )-(d/D)(1-2j/M)，d是嵌入的索引，D是嵌入的维度。

对于输入融合层，采用双向GRU，输入facts并能实现他们之间的信息交换：

（2）VQA的输入模块（Input Module for VQA）

将DMN引入视觉问答，作者提出一种新的图像输入模块。该模块将输入图像分成很多小的局部区域，每个小区域都等价于文本中的一个句子。该模块由3部分组成，如下图所示，局部区域特征提取，视觉特征嵌入和输入融合层：

局部区域特征提取（Local region feature extraction）：基于VGG19来提取图像特征。首先将输入图像resize成448*448的，然后拿出最后一个池化层，此时维度为d=512*14*14，池化层将图像分成了14*14的小格网，因此原图对应196个局部区域，每个区域的维度是512。

视觉特征嵌入（Visual feature embedding）：VQA涉及到图像和文本，作者添加了一个tanh激活的线性层，映射局部区域向量到问题向量q使用的文本特征空间。

输入融合层（Input fusion layer）：前面做的局部特征提取，并没有提取全局特征，缺乏全局特征的表示能力是有限的，为了解决这个问题，作者加入了一个输入融合层，首先，生成一个输入的facts F，用蛇形遍历图像；然后在这些输入的facts F上用双向GRU来生成一个全局感知的输入facts，双向GRU能够使得信息在图像相邻的网格中传递信息。

（3）情景记忆模块（The Episodic Memory Module）

情景记忆模块如下图所示：

将注意力集中在这些facts的一个子集上以从输入的facts中检索信息。通过结合单个标量注意力门g_{i}^{t}（attention gate）和传递t中的每一个fact来实现该注意力。它的计算是通过fact和问题表示与情景记忆状态之间的交互实现：

注意力机制：

一旦有了注意力门g_{i}^{t}，我们就能使用注意力机制来提取上下文向量c^{t}。我们主要关注两种类型的注意力，软注意力和GRU注意力。

软注意力（Soft attention）：软注意力生成上下文向量c^{t}，是通过向量\underset{F}{\rightarrow }排序列表相应的注意力门g_{i}^{t}加权求和得到的，即：ct=∑Ni=1gtifi。这种方法有两个优点：第一，易于计算；第二，如果softmax激活是尖峰的，它可以通过仅为上下文向量选择一个fact来近似一个硬注意函数，且它仍然是可微的。

GRU注意力（Attention based GRU）：对于复杂的查询，我们希望注意力对输入fact的位置和次序更敏感，在这种情况下RNN是更好的选择。我们提出了一个修改的GRU结构，它从注意力机制中嵌入信息。公式（1）中的更新门u_{i}决定了隐含状态的每一个维度的多少来保持，和多少来更新输入的x_{i}。因为u_{i}的计算只涉及当前的输入和前一步的隐含状态，它缺乏来自前一个情景记忆或者问题的知识。

因此再公式（4）中用注意力门g_{i}^{t}来取代更新门u_{i}，GRU就可以用注意力来更新它的每一次状态，该过程可以用下图来描述：

用公式表达也就是：

需要重点考虑的是g_{i}^{t}是一个标量，它是softmax激活产生的，与向量u_{i}相反，它是由sigmoid激活产生的。这样使得我们更容易对注意力门如何激活输入进行可视化，结果如下图所示：

情景记忆更新（Episode Memory Updates）：

通过每一次的注意力机制，我们都希望能够根据上下文向量c^{t}更新场景记忆m^{t-1}，生成m^{t}。在DMN中，将初始隐含状态的GRU设置为问题向量q即为了实现该目的。对于传递t，情景记忆可以按照下式计算：

有文献建议，每次传递使用不同的权值更新场景记忆会更好一些。当模型只包含一组权值时，对于所有经过输入的情景，它被称为tied model，如Table 1中的“Mem Weights”行所示：

对于情景记忆的更新，作者使用的是ReLU层，计算新的情景记忆状态可以通过下式：

最后的记忆模块输出会传递到答案模块，这里和DMN的答案模块是一致的。

3. 数据集

实验使用的数据集有3个：

bAbI-10k：该合成数据集包含20种任务，每一个样本包含facts，问题，和答案，还有提示答案的supporting facts。

DAQUAR-ALL visual dataset：The DAtaset for QUestion Answering on Real-world images (DAQUAR)包含795张训练图像和654张测试图像，6795个训练问题和5673个测试问题。

VQA：VQA 1.0数据集

4. 实验

（1）模型分析

这里作者进行了DMN及其模型变种的分析，ODMN为原始的DMN模型；DMN2是用输入融合层取代输入模块；DMN3在DMN2的基础上，用GRU注意力取代了软注意力；DMN+是在DMN3的基础上，在每次传递的过程中，使用线性层ReLU和一组权值来更新记忆。所有的实验结果如Table 1中所示。

（2）基于bAbI-10k数据集的比较

对于该数据集，优化器采用Adam，学习率为0.001，batch size为128，epoch设置为256，如果验证的loss在20个epoch内不改变，则提前结束模型。词嵌入初始化采用随机均匀分布，范围为[-\sqrt{3}, \sqrt{3}]，其余的权值初始化都采用Xavier。hidden size都设置为d=80，所有权值采用l2范数约束。对于dropout来说，保留输入的90%。场景记忆模块中，一共传递3次。不同task及其结果如下图所示：

 

（3）基于VQA的数据集比较

优化器用Adam，学习率0.003，batch size为100，一共256个epochs，如果验证loss在10个epochs内没有改变，则提前结束训练。权值初始化采用随机均匀分布，范围[-0.08 0.08]。hidden size设置为d=512，dropout=0.5。结果主要展示在Fig 6中。

三、实验介绍

本文主要参考代码：https://github.com/DeepRNN/visual_question_answering

所有文件结构为：

  1. |------ base_model.py
  2. |------ config.py
  3. |------ dataset.py
  4. |------ episodic_memory.py
  5. |------ main.py
  6. |------ model.py
  7. |------ vgg16_no_fc.npy                    # 需要额外下载
  8. |------ utils
  9.         |------ vqa
  10.                 |------ vqa.py
  11.                 |------ vqaEval.py
  12.         |------ vocabulary.py
  13.         |------ misc.py
  14.         |------ nn.py
  15. |------ train                              # 需要额外下载
  16.         |------ images
  17.                 |------ image01.jpg
  18.                 |------ image01.jpg
  19.                 |------ ......
  20.         |------ mscoco_train2014_annotations.json
  21.         |------ OpenEnded_mscoco_train2014_questions.json
  22. |------ val                                # 需要额外下载
  23.         |------ images
  24.                 |------ image01.jpg
  25.                 |------ image01.jpg
  26.                 |------ ......
  27.         |------ mscoco_val2014_annotations.json
  28.         |------ OpenEnded_mscoco_val2014_questions.json
  29. |------ test
  30.         |------ images

1. 实验环境

我的运行环境如下：

python 3.7

GPU: GTX 1050TI 4G

tensorflow 1.14

numpy 1.16.2

opencv 3.4.1

Natural Language Toolkit (NLTK) 3.4

Pandas 0.24.2

Matplotlib 3.0.3

tqdm 4.31.1

2. 数据集准备

（1）图像数据

训练的图像数据：http://images.cocodataset.org/zips/train2014.zip

验证的图像数据：http://images.cocodataset.org/zips/val2014.zip

测试的图像数据：http://images.cocodataset.org/zips/test2014.zip

以上链接可以直接打开网页或者迅雷下载，下载好之后解压，放到相应文件夹下的'images/'下

（2）文本数据

打开VQA的官方页面：https://visualqa.org/vqa_v1_download.html

然后找到下面的5个标注进行下载即可：

（3）VGG数据

作者给出的预训练的VGG16模型的下载链接点不开，所以通过其他渠道找到了该文件，我把它上传到自己的网盘上了：链接：https://pan.baidu.com/s/1jPzXKZIXbNnknT7Nubh3yw  提取码：nms0 

另外还从其他地方找到了resnet-101，提取图像特征也可以考虑用这个模型，直接用迅雷或者页面就可以进行下载，下载链接：http://download.tensorflow.org/models/resnet_v1_101_2016_08_28.tar.gz

3. 代码

按照前面的文件结构准备需要的训练数据，就可以开始编写代码进行实验部分，下面会逐一介绍实验中用到的文件：

（1）超参数config.py文件

改文件主要定义了实验的超参数，由于参数比较多，因此直接用类进行封装，类的初始化不需要传入任何参数，如果需要对超参数进行修改的话，可以直接在这个文件里进行操作：

  1. class Config(object):
  2.     """ Wrapper class for various (hyper)parameters. """
  3.     def __init__(self):
  4.         # 模型结构
  5.         self.cnn = 'vgg16'               # 'vgg16' 或 'resnet50'
  6.         self.max_question_length = 30    
  7.         self.dim_embedding = 512        
  8.         self.num_gru_units = 512         
  9.         self.memory_step = 3             
  10.         self.memory_update = 'relu'      # 'gru' 或 'relu'
  11.         self.attention = 'gru'           # 'gru' 或 'soft'，消融实验的时候可以设置为soft
  12.         self.tie_memory_weight = False
  13.         self.question_encoding = 'gru'   # 'gru' 或 'positional'
  14.         self.embed_fact = False
  15.  
  16.         # 权值初始化和正则化
  17.         self.fc_kernel_initializer_scale = 0.08
  18.         self.fc_kernel_regularizer_scale = 1e-6
  19.         self.fc_activity_regularizer_scale = 0.0
  20.         self.conv_kernel_regularizer_scale = 1e-6
  21.         self.conv_activity_regularizer_scale = 0.0
  22.         self.fc_drop_rate = 0.5
  23.         self.gru_drop_rate = 0.3
  24.  
  25.         # 优化
  26.         self.num_epochs = 100
  27.         self.batch_size = 4
  28.         self.optimizer = 'Adam'          # 'Adam', 'RMSProp', 'Momentum' or 'SGD'
  29.         self.initial_learning_rate = 0.0001
  30.         self.learning_rate_decay_factor = 1.0
  31.         self.num_steps_per_decay = 10000
  32.         self.clip_gradients = 10.0
  33.         self.momentum = 0.0
  34.         self.use_nesterov = True
  35.         self.decay = 0.9
  36.         self.centered = True
  37.         self.beta1 = 0.9
  38.         self.beta2 = 0.999
  39.         self.epsilon = 1e-5
  40.  
  41.         # 存储器
  42.         self.save_period = 1000
  43.         self.save_dir = './models/'
  44.         self.summary_dir = './summary/'
  45.  
  46.         # 词汇表
  47.         self.vocabulary_file = './vocabulary.csv'
  48.  
  49.         # 训练
  50.         self.train_image_dir = './train/images/'
  51.         self.train_question_file = './train/OpenEnded_mscoco_train2014_questions.json'
  52.         self.train_answer_file = './train/mscoco_train2014_annotations.json'
  53.         self.temp_train_annotation_file = './train/anns.csv'
  54.         self.temp_train_data_file = './train/data.npy'
  55.  
  56.         # 评价
  57.         self.eval_image_dir = './val/images/'
  58.         self.eval_question_file = './val/OpenEnded_mscoco_val2014_questions.json'
  59.         self.eval_answer_file = './val/mscoco_val2014_annotations.json'
  60.         self.temp_eval_annotation_file = './val/anns.csv'
  61.         self.temp_eval_data_file = './val/data.npy'
  62.         self.eval_result_dir = './val/results/'
  63.         self.eval_result_file = './val/results.json'
  64.         self.save_eval_result_as_image = False
  65.  
  66.         # 测试
  67.         self.test_image_dir = './test/images/'
  68.         self.test_question_file = './test/questions.csv'
  69.         self.temp_test_info_file = './test/info.csv'
  70.         self.test_result_dir = './test/results/'
  71.         self.test_result_file = './test/results.csv'

（2）utils文件夹下的文件

utils文件夹下包含一个vqa子文件夹和其他几个文件，现在逐一来看：

misc.py文件：改文件里面就定义了一个读取图像的类，该类有两个函数，一个是用于读取单张图像，另一个则是用于读取多个图像，下面来看一下该文件的代码：

  1. import numpy as np
  2. import cv2
  3.  
  4. class ImageLoader(object):
  5.     def __init__(self, mean_file):
  6.         self.bgr = True
  7.         self.scale_shape = np.array([224, 224], np.int32)
  8.         self.crop_shape = np.array([224, 224], np.int32)
  9.         self.mean = np.load(mean_file).mean(1).mean(1)
  10.  
  11.     def load_image(self, image_file):
  12.         """ Load and preprocess an image. """
  13.         image = cv2.imread(image_file)
  14.  
  15.         if self.bgr:
  16.             temp = image.swapaxes(0, 2)
  17.             temp = temp[::-1]                    # 变成rgb
  18.             image = temp.swapaxes(0, 2)
  19.  
  20.         image = cv2.resize(image, (self.scale_shape[0], self.scale_shape[1]))
  21.         offset = (self.scale_shape - self.crop_shape) / 2
  22.         offset = offset.astype(np.int32)
  23.         image = image[offset[0]:offset[0]+self.crop_shape[0],
  24.                       offset[1]:offset[1]+self.crop_shape[1]]
  25.         image = image - self.mean
  26.         return image
  27.  
  28.     def load_images(self, image_files):
  29.         """ Load and preprocess a list of images. """
  30.         images = []
  31.         for image_file in image_files:
  32.             images.append(self.load_image(image_file))
  33.         images = np.array(images, np.float32)
  34.         return images

nn.py文件：主要定义了网络中的一些操作，包括GRU，BN，FC，卷积等：

  1. import tensorflow as tf
  2. import tensorflow.contrib.layers as layers
  3.  
  4.  
  5. class NN(object):
  6.     def __init__(self, config):
  7.         self.config = config
  8.         self.is_train = True if config.phase == 'train' else False
  9.         self.train_cnn = self.is_train and config.train_cnn
  10.         self.prepare()
  11.  
  12.     def prepare(self):
  13.         """ Setup the weight initalizers and regularizers. """
  14.         config = self.config
  15.  
  16.         self.conv_kernel_initializer = layers.xavier_initializer()
  17.  
  18.         if self.train_cnn and config.conv_kernel_regularizer_scale > 0:
  19.             self.conv_kernel_regularizer = layers.l2_regularizer(
  20.                 scale = config.conv_kernel_regularizer_scale)
  21.         else:
  22.             self.conv_kernel_regularizer = None
  23.  
  24.         if self.train_cnn and config.conv_activity_regularizer_scale > 0:
  25.             self.conv_activity_regularizer = layers.l1_regularizer(
  26.                 scale = config.conv_activity_regularizer_scale)
  27.         else:
  28.             self.conv_activity_regularizer = None
  29.  
  30.         self.fc_kernel_initializer = tf.random_uniform_initializer(
  31.             minval = -config.fc_kernel_initializer_scale,
  32.             maxval = config.fc_kernel_initializer_scale)
  33.  
  34.         if self.is_train and config.fc_kernel_regularizer_scale > 0:
  35.             self.fc_kernel_regularizer = layers.l2_regularizer(
  36.                 scale = config.fc_kernel_regularizer_scale)
  37.         else:
  38.             self.fc_kernel_regularizer = None
  39.  
  40.         if self.is_train and config.fc_activity_regularizer_scale > 0:
  41.             self.fc_activity_regularizer = layers.l1_regularizer(
  42.                 scale = config.fc_activity_regularizer_scale)
  43.         else:
  44.             self.fc_activity_regularizer = None
  45.  
  46.     def conv2d(self,
  47.                inputs,
  48.                filters,
  49.                kernel_size = (3, 3),
  50.                strides = (1, 1),
  51.                activation = tf.nn.relu,
  52.                use_bias = True,
  53.                name = None):
  54.         """ 2D Convolution layer. """
  55.         if activation is not None:
  56.             activity_regularizer = self.conv_activity_regularizer
  57.         else:
  58.             activity_regularizer = None
  59.         return tf.layers.conv2d(
  60.             inputs = inputs,
  61.             filters = filters,
  62.             kernel_size = kernel_size,
  63.             strides = strides,
  64.             padding='same',
  65.             activation = activation,
  66.             use_bias = use_bias,
  67.             trainable = self.train_cnn,
  68.             kernel_initializer = self.conv_kernel_initializer,
  69.             kernel_regularizer = self.conv_kernel_regularizer,
  70.             activity_regularizer = activity_regularizer,
  71.             name = name)
  72.  
  73.     def max_pool2d(self,
  74.                    inputs,
  75.                    pool_size = (2, 2),
  76.                    strides = (2, 2),
  77.                    name = None):
  78.         """ 2D Pooling layer. """
  79.         return tf.layers.max_pooling2d(
  80.             inputs = inputs,
  81.             pool_size = pool_size,
  82.             strides = strides,
  83.             padding='same',
  84.             name = name)
  85.  
  86.     def dense(self,
  87.               inputs,
  88.               units,
  89.               activation = tf.tanh,
  90.               use_bias = True,
  91.               name = None):
  92.         """ Fully-connected layer. """
  93.         if activation is not None:
  94.             activity_regularizer = self.fc_activity_regularizer
  95.         else:
  96.             activity_regularizer = None
  97.         return tf.layers.dense(
  98.             inputs = inputs,
  99.             units = units,
  100.             activation = activation,
  101.             use_bias = use_bias,
  102.             trainable = self.is_train,
  103.             kernel_initializer = self.fc_kernel_initializer,
  104.             kernel_regularizer = self.fc_kernel_regularizer,
  105.             activity_regularizer = activity_regularizer,
  106.             name = name)
  107.  
  108.     def dropout(self,
  109.                 inputs,
  110.                 name = None):
  111.         """ Dropout layer. """
  112.         return tf.layers.dropout(
  113.             inputs = inputs,
  114.             rate = self.config.fc_drop_rate,
  115.             training = self.is_train)
  116.  
  117.     def batch_norm(self,
  118.                    inputs,
  119.                    name = None):
  120.         """ Batch normalization layer. """
  121.         return tf.layers.batch_normalization(
  122.             inputs = inputs,
  123.             training = self.train_cnn,
  124.             trainable = self.train_cnn,
  125.             name = name
  126.         )
  127.  
  128.     def gru(self):
  129.         """ GRU layer. """
  130.         gru = tf.nn.rnn_cell.GRUCell(
  131.             num_units = self.config.num_gru_units,
  132.             kernel_initializer = self.fc_kernel_initializer)
  133.         if self.is_train:
  134.             gru = tf.nn.rnn_cell.DropoutWrapper(
  135.                 gru,
  136.                 input_keep_prob = 1.0 - self.config.gru_drop_rate,
  137.                 output_keep_prob = 1.0 - self.config.gru_drop_rate,
  138.                 state_keep_prob = 1.0 - self.config.gru_drop_rate)
  139.         return gru

vocabulary.py文件：该文件用于生成词汇表，并保存到csv文件中：

  1. import os
  2. import numpy as np
  3. import pandas as pd
  4. from nltk.tokenize import word_tokenize
  5.  
  6. class Vocabulary(object):
  7.     def __init__(self, save_file = None):
  8.         self.words = []
  9.         self.word2idx = {}
  10.         self.size = 0
  11.         self.word_counts = {}
  12.         self.word_frequencies = []
  13.         if save_file is not None:
  14.             self.load(save_file)
  15.         else:
  16.             self.add_words(["<unknown>"])
  17.  
  18.     def add_words(self, words):
  19.         """ Add new words to the vocabulary. """
  20.         for w in words:
  21.             if w not in self.word2idx.keys():
  22.                 self.words.append(w)
  23.                 self.word2idx[w] = self.size
  24.                 self.size += 1
  25.             self.word_counts[w] = self.word_counts.get(w, 0) + 1
  26.  
  27.     def compute_frequency(self):
  28.         """ Compute the frequency of each word. """
  29.         self.word_frequencies = []
  30.         for w in self.words:
  31.             self.word_frequencies.append(self.word_counts[w])
  32.         self.word_frequencies = np.array(self.word_frequencies, np.float32)
  33.         self.word_frequencies /= np.sum(self.word_frequencies)
  34.         self.word_frequencies = np.log(self.word_frequencies)
  35.         self.word_frequencies -= np.max(self.word_frequencies)
  36.  
  37.     def word_to_idx(self, word):
  38.         """ Translate a word into its index. """
  39.         return self.word2idx[word] if word in self.word2idx.keys() else 0
  40.  
  41.     def process_sentence(self, sentence):
  42.         """ Tokenize a sentence, and translate each token into its index
  43.             in the vocabulary. """
  44.         words = word_tokenize(sentence.lower())
  45.         word_idxs = [self.word_to_idx(w) for w in words]
  46.         return word_idxs
  47.  
  48.     def save(self, save_file):
  49.         """ Save the vocabulary to a file. """
  50.         data = pd.DataFrame({'word': self.words,
  51.                              'index': list(range(self.size)),
  52.                              'frequency': self.word_frequencies})
  53.         data.to_csv(save_file)
  54.  
  55.     def load(self, save_file):
  56.         """ Load the vocabulary from a file. """
  57.         assert os.path.exists(save_file)
  58.         data = pd.read_csv(save_file)
  59.         self.words = data['word'].values
  60.         self.size = len(self.words)
  61.         self.word2idx = {self.words[i]:i for i in range(self.size)}
  62.         self.word_frequencies = data['frequency'].values

最后是vqa文件夹下的两个文件：vqa.py和vqaEval.py。：

"vqa/vqa.py"文件的主要内容：

  1. __author__ = 'aagrawal'
  2. __version__ = '0.9'
  3.  
  4. # Interface for accessing the VQA dataset.
  5.  
  6. # This code is based on the code written by Tsung-Yi Lin for MSCOCO Python API available at the following link:
  7. # (https://github.com/pdollar/coco/blob/master/PythonAPI/pycocotools/coco.py).
  8.  
  9. # The following functions are defined:
  10. #  VQA        - VQA class that loads VQA annotation file and prepares data structures.
  11. #  getQuesIds - Get question ids that satisfy given filter conditions.
  12. #  getImgIds  - Get image ids that satisfy given filter conditions.
  13. #  loadQA     - Load questions and answers with the specified question ids.
  14. #  showQA     - Display the specified questions and answers.
  15. #  loadRes    - Load result file and create result object.
  16.  
  17. # Help on each function can be accessed by: "help(COCO.function)"
  18.  
  19. import json
  20. import datetime
  21. import copy
  22. from tqdm import tqdm
  23. from nltk.tokenize import word_tokenize
  24.  
  25. class VQA:
  26.         def __init__(self, annotation_file=None, question_file=None):
  27.                 """
  28.                 Constructor of VQA helper class for reading and visualizing questions and answers.
  29.                 :param annotation_file (str): location of VQA annotation file
  30.                 :return:
  31.                 """
  32.                 # load dataset
  33.                 self.dataset = {}
  34.                 self.questions = {}
  35.                 self.qa = {}
  36.                 self.qqa = {}
  37.                 self.imgToQA = {}
  38.                 self.max_ques_len = 0
  39.                 if not annotation_file == None and not question_file == None:
  40.                         print('loading VQA annotations and questions into memory...')
  41.                         time_t = datetime.datetime.utcnow()
  42.                         dataset = json.load(open(annotation_file, 'r'))
  43.                         questions = json.load(open(question_file, 'r'))
  44.                         print(datetime.datetime.utcnow() - time_t)
  45.                         self.dataset = dataset
  46.                         self.questions = questions
  47.                         self.process_dataset()
  48.                         self.createIndex()
  49.  
  50.         def createIndex(self):
  51.                 # create index
  52.                 print('creating index...')
  53.                 imgToQA = {ann['image_id']: [] for ann in self.dataset['annotations']}
  54.                 qa =  {ann['question_id']: [] for ann in self.dataset['annotations']}
  55.                 qqa = {ann['question_id']: [] for ann in self.dataset['annotations']}
  56.                 max_ques_len = 0
  57.                 for ann in self.dataset['annotations']:
  58.                         imgToQA[ann['image_id']] += [ann]
  59.                         qa[ann['question_id']] = ann
  60.                 # print(qa)
  61.                 for ques in self.questions['questions']:
  62.                         qqa[ques['question_id']] = ques
  63.                         max_ques_len = max(max_ques_len, len(word_tokenize(ques['question'])))
  64.  
  65.                 # create class members
  66.                 self.qa = qa
  67.                 self.qqa = qqa
  68.                 self.imgToQA = imgToQA
  69.                 self.max_ques_len = max_ques_len
  70.                 # print("11111111111")
  71.  
  72.         def info(self):
  73.                 """
  74.                 Print information about the VQA annotation file.
  75.                 :return:
  76.                 """
  77.                 for key, value in list(self.datset['info'].items()):
  78.                         print('%s: %s'%(key, value))
  79.  
  80.         def getQuesIds(self, imgIds=[], quesTypes=[], ansTypes=[]):
  81.                 """
  82.                 Get question ids that satisfy given filter conditions. default skips that filter
  83.                 :param         imgIds    (int array)   : get question ids for given imgs
  84.                                 quesTypes (str array)   : get question ids for given question types
  85.                                 ansTypes  (str array)   : get question ids for given answer types
  86.                 :return:    ids   (int array)   : integer array of question ids
  87.                 """
  88.                 imgIds           = imgIds    if type(imgIds)    == list else [imgIds]
  89.                 quesTypes = quesTypes if type(quesTypes) == list else [quesTypes]
  90.                 ansTypes  = ansTypes  if type(ansTypes)  == list else [ansTypes]
  91.  
  92.                 if len(imgIds) == len(quesTypes) == len(ansTypes) == 0:
  93.                         anns = self.dataset['annotations']
  94.                 else:
  95.                         if not len(imgIds) == 0:
  96.                                 anns = sum([self.imgToQA[imgId] for imgId in imgIds if imgId in self.imgToQA],[])
  97.                         else:
  98.                                 anns = self.dataset['annotations']
  99.                         anns = anns if len(quesTypes) == 0 else [ann for ann in anns if ann['question_type'] in quesTypes]
  100.                         anns = anns if len(ansTypes)  == 0 else [ann for ann in anns if ann['answer_type'] in ansTypes]
  101.                 ids = [ann['question_id'] for ann in anns]
  102.                 return ids
  103.  
  104.         def getImgIds(self, quesIds=[], quesTypes=[], ansTypes=[]):
  105.                 """
  106.                 Get image ids that satisfy given filter conditions. default skips that filter
  107.                 :param quesIds   (int array)   : get image ids for given question ids
  108.                 quesTypes (str array)   : get image ids for given question types
  109.                 ansTypes  (str array)   : get image ids for given answer types
  110.                 :return: ids     (int array)   : integer array of image ids
  111.                 """
  112.                 quesIds   = quesIds   if type(quesIds)   == list else [quesIds]
  113.                 quesTypes = quesTypes if type(quesTypes) == list else [quesTypes]
  114.                 ansTypes  = ansTypes  if type(ansTypes)  == list else [ansTypes]
  115.  
  116.                 if len(quesIds) == len(quesTypes) == len(ansTypes) == 0:
  117.                         anns = self.dataset['annotations']
  118.                 else:
  119.                         if not len(quesIds) == 0:
  120.                                 anns = sum([self.qa[quesId] for quesId in quesIds if quesId in self.qa],[])
  121.                         else:
  122.                                 anns = self.dataset['annotations']
  123.                         anns = anns if len(quesTypes) == 0 else [ann for ann in anns if ann['question_type'] in quesTypes]
  124.                         anns = anns if len(ansTypes)  == 0 else [ann for ann in anns if ann['answer_type'] in ansTypes]
  125.                 ids = [ann['image_id'] for ann in anns]
  126.                 return ids
  127.  
  128.         def loadQA(self, ids=[]):
  129.                 """
  130.                 Load questions and answers with the specified question ids.
  131.                 :param ids (int array)       : integer ids specifying question ids
  132.                 :return: qa (object array)   : loaded qa objects
  133.                 """
  134.                 if type(ids) == list:
  135.                         return [self.qa[id] for id in ids]
  136.                 elif type(ids) == int:
  137.                         return [self.qa[ids]]
  138.  
  139.         def showQA(self, anns):
  140.                 """
  141.                 Display the specified annotations.
  142.                 :param anns (array of object): annotations to display
  143.                 :return: None
  144.                 """
  145.                 if len(anns) == 0:
  146.                         return 0
  147.                 for ann in anns:
  148.                         quesId = ann['question_id']
  149.                         print("Question: %s" %(self.qqa[quesId]['question']))
  150.                         for ans in ann['answers']:
  151.                                 print("Answer %d: %s" %(ans['answer_id'], ans['answer']))
  152.  
  153.         def loadRes(self, resFile, quesFile):
  154.                 """
  155.                 Load result file and return a result object.
  156.                 :param   resFile (str)     : file name of result file
  157.                 :return: res (obj)         : result api object
  158.                 """
  159.                 res = VQA()
  160.                 res.questions = json.load(open(quesFile))
  161.                 res.dataset['info'] = copy.deepcopy(self.questions['info'])
  162.                 res.dataset['task_type'] = copy.deepcopy(self.questions['task_type'])
  163.                 res.dataset['data_type'] = copy.deepcopy(self.questions['data_type'])
  164.                 res.dataset['data_subtype'] = copy.deepcopy(self.questions['data_subtype'])
  165.                 res.dataset['license'] = copy.deepcopy(self.questions['license'])
  166.  
  167.                 print('Loading and preparing results...     ')
  168.                 time_t = datetime.datetime.utcnow()
  169.                 anns    = json.load(open(resFile))
  170.                 assert type(anns) == list, 'results is not an array of objects'
  171.                 annsQuesIds = [ann['question_id'] for ann in anns]
  172.                 assert set(annsQuesIds) == set(self.getQuesIds()), \
  173.                 'Results do not correspond to current VQA set. Either the results do not have predictions for all question ids in annotation file or there is atleast one question id that does not belong to the question ids in the annotation file.'
  174.                 for ann in anns:
  175.                         quesId                              = ann['question_id']
  176.                         if res.dataset['task_type'] == 'Multiple Choice':
  177.                                 assert ann['answer'] in self.qqa[quesId]['multiple_choices'], 'predicted answer is not one of the multiple choices'
  178.                         qaAnn                = self.qa[quesId]
  179.                         ann['image_id']      = qaAnn['image_id']
  180.                         ann['question_type'] = qaAnn['question_type']
  181.                         ann['answer_type']   = qaAnn['answer_type']
  182.                 print('DONE (t=%0.2fs)'%((datetime.datetime.utcnow() - time_t).total_seconds()))
  183.  
  184.                 res.dataset['annotations'] = anns
  185.                 res.createIndex()
  186.                 return res
  187.  
  188.         def process_dataset(self):
  189.                 for ann in self.dataset['annotations']:
  190.                     count = {}
  191.                     for ans in ann['answers']:
  192.                         ans['answer'] = ans['answer'].lower()                   # 将每个答案中的大写变成小写
  193.                         count[ans['answer']] = count.get(ans['answer'], 0) + 1  # 统计每个答案出现的数量
  194.  
  195.                     sorted_ans = sorted(list(count.items()),
  196.                                         key=lambda x: x[1],
  197.                                         reverse=True)                           # 按答案出现次数排序
  198.                     best_ans, best_ans_count = sorted_ans[0]                    # 记录出现次数最多的答案和次数
  199.                     ann['best_answer'] = best_ans
  200.                     ann['best_answer_count'] = best_ans_count
  201.  
  202.                 for ques in self.questions['questions']:
  203.                     q = ques['question']
  204.                     q = q.replace('?', '')                                      # 用空格将问号取代掉
  205.                     q = q.lower()
  206.                     ques['question'] = q
  207.  
  208.         def filter_by_ques_len(self, max_ques_len):
  209.                 print("Filtering the questions by length...")
  210.                 keep_ques = {}
  211.                 for ques in tqdm(self.questions['questions']):
  212.                     if len(word_tokenize(ques['question'])) <= max_ques_len:
  213.                         keep_ques[ques['question_id']] = \
  214.                             keep_ques.get(ques['question_id'], 0) + 1
  215.  
  216.                 self.dataset['annotations'] = \
  217.                     [ann for ann in self.dataset['annotations'] \
  218.                     if keep_ques.get(ann['question_id'],0)>0]
  219.                 self.questions['questions'] = \
  220.                     [ques for ques in self.questions['questions'] \
  221.                     if keep_ques.get(ques['question_id'],0)>0]
  222.  
  223.                 self.createIndex()
  224.  
  225.         def filter_by_ans_len(self, max_ans_len, min_freq=5):
  226.                 print("Filtering the answers by length...")
  227.                 keep_ques = {}
  228.                 for ann in tqdm(self.dataset['annotations']):
  229.                     if len(word_tokenize(ann['best_answer'])) <= max_ans_len \
  230.                         and ann['best_answer_count']>=min_freq:
  231.                         keep_ques[ann['question_id']] = \
  232.                             keep_ques.get(ann['question_id'], 0) + 1
  233.  
  234.                 self.dataset['annotations'] = \
  235.                     [ann for ann in self.dataset['annotations'] \
  236.                     if keep_ques.get(ann['question_id'],0)>0]
  237.                 self.questions['questions'] = \
  238.                     [ques for ques in self.questions['questions'] \
  239.                     if keep_ques.get(ques['question_id'],0)>0]
  240.  
  241.                 self.createIndex()

"vqa/vqaEval.py"文件的主要内容：

  1. import sys
  2. import re
  3. from tqdm import tqdm
  4.  
  5. class VQAEval:
  6.         def __init__(self, vqa, vqaRes, n=2):
  7.                 self.n            = n
  8.                 self.accuracy     = {}
  9.                 self.evalQA       = {}
  10.                 self.evalQuesType = {}
  11.                 self.evalAnsType  = {}
  12.                 self.vqa          = vqa
  13.                 self.vqaRes       = vqaRes
  14.                 self.params       = {'question_id': vqa.getQuesIds()}
  15.                 self.contractions = {"aint": "ain't", "arent": "aren't", "cant": "can't", "couldve": "could've", "couldnt": "couldn't", \
  16.                                      "couldn'tve": "couldn’t’ve", "couldnt’ve": "couldn’t’ve", "didnt": "didn’t", "doesnt": "doesn’t", "dont": "don’t", "hadnt": "hadn’t", \
  17.                                      "hadnt’ve": "hadn’t’ve", "hadn'tve": "hadn’t’ve", "hasnt": "hasn’t", "havent": "haven’t", "hed": "he’d", "hed’ve": "he’d’ve", \
  18.                                      "he’dve": "he’d’ve", "hes": "he’s", "howd": "how’d", "howll": "how’ll", "hows": "how’s", "Id’ve": "I’d’ve", "I’dve": "I’d’ve", \
  19.                                      "Im": "I’m", "Ive": "I’ve", "isnt": "isn’t", "itd": "it’d", "itd’ve": "it’d’ve", "it’dve": "it’d’ve", "itll": "it’ll", "let’s": "let’s", \
  20.                                      "maam": "ma’am", "mightnt": "mightn’t", "mightnt’ve": "mightn’t’ve", "mightn’tve": "mightn’t’ve", "mightve": "might’ve", \
  21.                                      "mustnt": "mustn’t", "mustve": "must’ve", "neednt": "needn’t", "notve": "not’ve", "oclock": "o’clock", "oughtnt": "oughtn’t", \
  22.                                      "ow’s’at": "’ow’s’at", "’ows’at": "’ow’s’at", "’ow’sat": "’ow’s’at", "shant": "shan’t", "shed’ve": "she’d’ve", "she’dve": "she’d’ve", \
  23.                                      "she’s": "she’s", "shouldve": "should’ve", "shouldnt": "shouldn’t", "shouldnt’ve": "shouldn’t’ve", "shouldn’tve": "shouldn’t’ve", \
  24.                                      "somebody’d": "somebodyd", "somebodyd’ve": "somebody’d’ve", "somebody’dve": "somebody’d’ve", "somebodyll": "somebody’ll", \
  25.                                      "somebodys": "somebody’s", "someoned": "someone’d", "someoned’ve": "someone’d’ve", "someone’dve": "someone’d’ve", \
  26.                                      "someonell": "someone’ll", "someones": "someone’s", "somethingd": "something’d", "somethingd’ve": "something’d’ve", \
  27.                                      "something’dve": "something’d’ve", "somethingll": "something’ll", "thats": "that’s", "thered": "there’d", "thered’ve": "there’d’ve", \
  28.                                      "there’dve": "there’d’ve", "therere": "there’re", "theres": "there’s", "theyd": "they’d", "theyd’ve": "they’d’ve", \
  29.                                      "they’dve": "they’d’ve", "theyll": "they’ll", "theyre": "they’re", "theyve": "they’ve", "twas": "’twas", "wasnt": "wasn’t", \
  30.                                      "wed’ve": "we’d’ve", "we’dve": "we’d’ve", "weve": "we've", "werent": "weren’t", "whatll": "what’ll", "whatre": "what’re", \
  31.                                      "whats": "what’s", "whatve": "what’ve", "whens": "when’s", "whered": "where’d", "wheres": "where's", "whereve": "where’ve", \
  32.                                      "whod": "who’d", "whod’ve": "who’d’ve", "who’dve": "who’d’ve", "wholl": "who’ll", "whos": "who’s", "whove": "who've", "whyll": "why’ll", \
  33.                                      "whyre": "why’re", "whys": "why’s", "wont": "won’t", "wouldve": "would’ve", "wouldnt": "wouldn’t", "wouldnt’ve": "wouldn’t’ve", \
  34.                                      "wouldn’tve": "wouldn’t’ve", "yall": "y’all", "yall’ll": "y’all’ll", "y’allll": "y’all’ll", "yall’d’ve": "y’all’d’ve", \
  35.                                      "y’alld’ve": "y’all’d’ve", "y’all’dve": "y’all’d’ve", "youd": "you’d", "youd’ve": "you’d’ve", "you’dve": "you’d’ve", \
  36.                                      "youll": "you’ll", "youre": "you’re", "youve": "you’ve"}
  37.                 self.manualMap    = {'none': '0',
  38.                                      'zero': '0',
  39.                                      'one': '1',
  40.                                      'two': '2',
  41.                                      'three': '3',
  42.                                      'four': '4',
  43.                                      'five': '5',
  44.                                      'six': '6',
  45.                                      'seven': '7',
  46.                                      'eight': '8',
  47.                                      'nine': '9',
  48.                                      'ten': '10'}
  49.                 self.articles     = ['a', 'an', 'the']
  50.  
  51.  
  52.                 self.periodStrip  = re.compile("(?!<=\d)(\.)(?!\d)")
  53.                 self.commaStrip   = re.compile("(\d)(\,)(\d)")
  54.                 self.punct        = [';', r"/", '[', ']', '"', '{', '}', '(', ')', '=', '+', '\\', '_', '-', '>', '<', '@', '`', ',', '?', '!']
  55.  
  56.         
  57.         def evaluate(self, quesIds=None):
  58.                 if quesIds == None:
  59.                         quesIds = [quesId for quesId in self.params['question_id']]
  60.                 gts = {}
  61.                 res = {}
  62.                 for quesId in quesIds:
  63.                         gts[quesId] = self.vqa.qa[quesId]
  64.                         res[quesId] = self.vqaRes.qa[quesId]
  65.                 
  66.                 # =================================================
  67.                 # Compute accuracy
  68.                 # =================================================
  69.                 accQA       = []
  70.                 accQuesType = {}
  71.                 accAnsType  = {}
  72.                 print("computing accuracy")
  73.                 step = 0
  74.                 for quesId in tqdm(quesIds):
  75.                         resAns      = res[quesId]['answer']
  76.                         resAns      = resAns.replace('\n', ' ')
  77.                         resAns      = resAns.replace('\t', ' ')
  78.                         resAns      = resAns.strip()
  79.                         resAns      = self.processPunctuation(resAns)
  80.                         resAns      = self.processDigitArticle(resAns)
  81.                         gtAcc  = []
  82.                         gtAnswers = [ans['answer'] for ans in gts[quesId]['answers']]
  83.                         if len(set(gtAnswers)) > 1: 
  84.                                 for ansDic in gts[quesId]['answers']:
  85.                                         ansDic['answer'] = self.processPunctuation(ansDic['answer'])
  86.                         for gtAnsDatum in gts[quesId]['answers']:
  87.                                 otherGTAns = [item for item in gts[quesId]['answers'] if item!=gtAnsDatum]
  88.                                 matchingAns = [item for item in otherGTAns if item['answer']==resAns]
  89.                                 acc = min(1, float(len(matchingAns))/3)
  90.                                 gtAcc.append(acc)
  91.                         quesType    = gts[quesId]['question_type']
  92.                         ansType     = gts[quesId]['answer_type']
  93.                         avgGTAcc = float(sum(gtAcc))/len(gtAcc)
  94.                         accQA.append(avgGTAcc)
  95.                         if quesType not in accQuesType:
  96.                                 accQuesType[quesType] = []
  97.                         accQuesType[quesType].append(avgGTAcc)
  98.                         if ansType not in accAnsType:
  99.                                 accAnsType[ansType] = []
  100.                         accAnsType[ansType].append(avgGTAcc)
  101.                         self.setEvalQA(quesId, avgGTAcc)
  102.                         self.setEvalQuesType(quesId, quesType, avgGTAcc)
  103.                         self.setEvalAnsType(quesId, ansType, avgGTAcc)
  104.                         step = step + 1
  105.  
  106.                 self.setAccuracy(accQA, accQuesType, accAnsType)
  107.                 print("Done computing accuracy")
  108.  
  109.                 self.showAccuracy(accQA, accQuesType, accAnsType)
  110.         
  111.         def processPunctuation(self, inText):
  112.                 outText = inText
  113.                 for p in self.punct:
  114.                         if (p + ' ' in inText or ' ' + p in inText) or (re.search(self.commaStrip, inText) != None):
  115.                                 outText = outText.replace(p, '')
  116.                         else:
  117.                                 outText = outText.replace(p, ' ')        
  118.                 outText = self.periodStrip.sub("", outText, re.UNICODE)
  119.                 return outText
  120.         
  121.         def processDigitArticle(self, inText):
  122.                 outText = []
  123.                 tempText = inText.lower().split()
  124.                 for word in tempText:
  125.                         word = self.manualMap.setdefault(word, word)
  126.                         if word not in self.articles:
  127.                                 outText.append(word)
  128.                         else:
  129.                                 pass
  130.                 for wordId, word in enumerate(outText):
  131.                         if word in self.contractions: 
  132.                                 outText[wordId] = self.contractions[word]
  133.                 outText = ' '.join(outText)
  134.                 return outText
  135.  
  136.         def setAccuracy(self, accQA, accQuesType, accAnsType):
  137.                 self.accuracy['overall']         = round(100*float(sum(accQA))/len(accQA), self.n)
  138.                 self.accuracy['perQuestionType'] = {quesType: round(100*float(sum(accQuesType[quesType]))/len(accQuesType[quesType]), self.n) for quesType in accQuesType}
  139.                 self.accuracy['perAnswerType']   = {ansType:  round(100*float(sum(accAnsType[ansType]))/len(accAnsType[ansType]), self.n) for ansType in accAnsType}
  140.  
  141.         def showAccuracy(self, accQA, accQuesType, accAnsType):
  142.                 print("Overall accurarcy = %f" %(self.accuracy['overall']))
  143.  
  144.                 print("Accuracy per question type:")
  145.                 for quesType in accQuesType:
  146.                     print("quesType: %s   accuracy = %f" %(quesType, self.accuracy['perQuestionType'][quesType]))
  147.  
  148.                 print("Accuracy per answer type:")
  149.                 for ansType in accAnsType:
  150.                     print("ansType: %s   accuracy = %f" %(ansType, self.accuracy['perAnswerType'][ansType]))
  151.  
  152.         def setEvalQA(self, quesId, acc):
  153.                 self.evalQA[quesId] = round(100*acc, self.n)
  154.  
  155.         def setEvalQuesType(self, quesId, quesType, acc):
  156.                 if quesType not in self.evalQuesType:
  157.                         self.evalQuesType[quesType] = {}
  158.                 self.evalQuesType[quesType][quesId] = round(100*acc, self.n)
  159.         
  160.         def setEvalAnsType(self, quesId, ansType, acc):
  161.                 if ansType not in self.evalAnsType:
  162.                         self.evalAnsType[ansType] = {}
  163.                 self.evalAnsType[ansType][quesId] = round(100*acc, self.n)
  164.  
  165.         def updateProgress(self, progress):
  166.                 barLength = 20
  167.                 status = ""
  168.                 if isinstance(progress, int):
  169.                         progress = float(progress)
  170.                 if not isinstance(progress, float):
  171.                         progress = 0
  172.                         status = "error: progress var must be float\r\n"
  173.                 if progress < 0:
  174.                         progress = 0
  175.                         status = "Halt...\r\n"
  176.                 if progress >= 1:
  177.                         progress = 1
  178.                         status = "Done...\r\n"
  179.                 block = int(round(barLength*progress))
  180.                 text = "\rFinshed Percent: [{0}] {1}% {2}".format( "#"*block + "-"*(barLength-block), int(progress*100), status)
  181.                 print(text)
  182.

（3）数据的读取加载文件dataset.py

该文件主要是将（图像，问题，答案）三种数据整合起来，如果是第一次做VQA，会觉得整合数据还是非常有难度的，所以就来学习一下这个文件是如何写的：

  1. import os
  2. import numpy as np
  3. import pandas as pd
  4. from tqdm import tqdm
  5. from nltk.tokenize import word_tokenize            # 如果word_tokenize报错了就重新安一下
  6.  
  7. from utils.vocabulary import Vocabulary
  8. from utils.vqa.vqa import VQA
  9.  
  10. # 建立数据类
  11. class DataSet(object):
  12.     def __init__(self,
  13.                  image_files,
  14.                  question_word_idxs,
  15.                  question_lens,
  16.                  question_ids,
  17.                  batch_size,
  18.                  answer_idxs = None,
  19.                  is_train = False,
  20.                  shuffle = False):
  21.         self.image_files = np.array(image_files)
  22.         self.question_word_idxs = np.array(question_word_idxs)
  23.         self.question_lens = np.array(question_lens)
  24.         self.question_ids = np.array(question_ids)
  25.         self.answer_idxs = np.array(answer_idxs)
  26.         self.batch_size = batch_size
  27.         self.is_train = is_train
  28.         self.shuffle = shuffle
  29.         self.setup()                # 初始化传完参数建立数据集
  30.  
  31.     def setup(self):
  32.         """ Setup the dataset. """
  33.         self.count = len(self.question_ids)            # 问题计数
  34.         self.num_batches = int(np.ceil(self.count * 1.0 / self.batch_size)) 
  35.         self.fake_count = self.num_batches * self.batch_size - self.count
  36.         self.idxs = list(range(self.count))
  37.         self.reset()                # 重建
  38.  
  39.     def reset(self):
  40.         """ Reset the dataset. """
  41.         self.current_idx = 0
  42.         if self.shuffle:
  43.             np.random.shuffle(self.idxs)
  44.  
  45.     def next_batch(self):
  46.         """ Fetch the next batch. """
  47.         assert self.has_next_batch()
  48.  
  49.         if self.has_full_next_batch():
  50.             start, end = self.current_idx, self.current_idx + self.batch_size
  51.             current_idxs = self.idxs[start:end]
  52.         else:
  53.             start, end = self.current_idx, self.count
  54.             current_idxs = self.idxs[start:end]
  55.             current_idxs += list(np.random.choice(self.count, self.fake_count))
  56.  
  57.         image_files = self.image_files[current_idxs]
  58.         question_word_idxs = self.question_word_idxs[current_idxs]
  59.         question_lens = self.question_lens[current_idxs]
  60.  
  61.         if self.is_train:
  62.             answer_idxs = self.answer_idxs[current_idxs]
  63.             self.current_idx += self.batch_size
  64.             return image_files, question_word_idxs, question_lens, answer_idxs
  65.         else:
  66.             self.current_idx += self.batch_size
  67.             return image_files, question_word_idxs, question_lens
  68.  
  69.     def has_next_batch(self):
  70.         """ Determine whether there is a batch left. """
  71.         return self.current_idx < self.count
  72.  
  73.     def has_full_next_batch(self):
  74.         """ Determine whether there is a full batch left. """
  75.         return self.current_idx + self.batch_size <= self.count
  76.  
  77.  
  78. # 准备训练数据
  79. def prepare_train_data(config):
  80.     """ Prepare the data for training the model. """
  81.     vqa = VQA(config.train_answer_file, config.train_question_file)
  82.     vqa.filter_by_ques_len(config.max_question_length)
  83.     vqa.filter_by_ans_len(1)
  84.  
  85.     print("Reading the questions and answers...")
  86.     annotations = process_vqa(vqa,
  87.                               'COCO_train2014',
  88.                               config.train_image_dir,
  89.                               config.temp_train_annotation_file)
  90.  
  91.     image_files = annotations['image_file'].values
  92.     questions = annotations['question'].values
  93.     question_ids = annotations['question_id'].values
  94.     answers = annotations['answer'].values
  95.     print("Questions and answers read.")
  96.     print("Number of questions = %d" %(len(question_ids)))
  97.  
  98.     print("Building the vocabulary...")
  99.     vocabulary = Vocabulary()
  100.     if not os.path.exists(config.vocabulary_file):
  101.         for question in tqdm(questions):
  102.             vocabulary.add_words(word_tokenize(question))
  103.         for answer in tqdm(answers):
  104.             vocabulary.add_words(word_tokenize(answer))
  105.         vocabulary.compute_frequency()
  106.         vocabulary.save(config.vocabulary_file)
  107.     else:
  108.         vocabulary.load(config.vocabulary_file)
  109.     print("Vocabulary built.")
  110.     print("Number of words = %d" %(vocabulary.size))
  111.     config.vocabulary_size = vocabulary.size
  112.  
  113.     print("Processing the questions and answers...")
  114.     if not os.path.exists(config.temp_train_data_file):
  115.         question_word_idxs, question_lens = process_questions(questions,
  116.                                                               vocabulary,
  117.                                                               config)
  118.         answer_idxs = process_answers(answers, vocabulary)
  119.         data = {'question_word_idxs': question_word_idxs,
  120.                 'question_lens': question_lens,
  121.                 'answer_idxs': answer_idxs}
  122.         np.save(config.temp_train_data_file, data)
  123.     else:
  124.         data = np.load(config.temp_train_data_file).item()
  125.         question_word_idxs = data['question_word_idxs']
  126.         question_lens = data['question_lens']
  127.         answer_idxs = data['answer_idxs']
  128.     print("Questions and answers processed.")
  129.  
  130.     print("Building the dataset...")
  131.     dataset = DataSet(image_files,
  132.                       question_word_idxs,
  133.                       question_lens,
  134.                       question_ids,
  135.                       config.batch_size,
  136.                       answer_idxs,
  137.                       True,
  138.                       True)
  139.     print("Dataset built.")
  140.     return dataset, config
  141.  
  142.  
  143. # 准备评价数据
  144. def prepare_eval_data(config):
  145.     """ Prepare the data for evaluating the model. """
  146.     vqa = VQA(config.eval_answer_file, config.eval_question_file)
  147.     vqa.filter_by_ques_len(config.max_question_length)
  148.     vqa.filter_by_ans_len(1)
  149.  
  150.     print("Reading the questions...")
  151.     annotations = process_vqa(vqa,
  152.                               'COCO_val2014',
  153.                               config.eval_image_dir,
  154.                               config.temp_eval_annotation_file)
  155.  
  156.     image_files = annotations['image_file'].values
  157.     questions = annotations['question'].values
  158.     question_ids = annotations['question_id'].values
  159.     print("Questions read.")
  160.     print("Number of questions = %d" %(len(question_ids)))
  161.  
  162.     print("Building the vocabulary...")
  163.     if os.path.exists(config.vocabulary_file):
  164.         vocabulary = Vocabulary(config.vocabulary_file)
  165.     else:
  166.         vocabulary = build_vocabulary(config)
  167.     print("Vocabulary built.")
  168.     print("Number of words = %d" %(vocabulary.size))
  169.     config.vocabulary_size = vocabulary.size
  170.  
  171.     print("Processing the questions...")
  172.     if not os.path.exists(config.temp_eval_data_file):
  173.         question_word_idxs, question_lens = process_questions(questions,
  174.                                                               vocabulary,
  175.                                                               config)
  176.         data = {'question_word_idxs': question_word_idxs,
  177.                 'question_lens': question_lens}
  178.         np.save(config.temp_eval_data_file, data)
  179.     else:
  180.         data = np.load(config.temp_eval_data_file).item()
  181.         question_word_idxs = data['question_word_idxs']
  182.         question_lens = data['question_lens']
  183.     print("Questions processed.")
  184.  
  185.     print("Building the dataset...")
  186.     dataset = DataSet(image_files,
  187.                       question_word_idxs,
  188.                       question_lens,
  189.                       question_ids,
  190.                       config.batch_size)
  191.     print("Dataset built.")
  192.     return vqa, dataset, vocabulary, config
  193.  
  194.  
  195. # 准备测试数据
  196. def prepare_test_data(config):
  197.     """ Prepare the data for testing the model. """
  198.     print("Reading the questions...")
  199.     annotations = pd.read_csv(config.test_question_file)
  200.     images = annotations['image'].unique()
  201.     image_files = [os.path.join(config.test_image_dir, f) for f in images]
  202.  
  203.     temp = pd.DataFrame({'image': images, 'image_file': image_files})
  204.     annotations = pd.merge(annotations, temp)
  205.     annotations.to_csv(config.temp_test_info_file)
  206.  
  207.     image_files = annotations['image_file'].values
  208.     questions = annotations['question'].values
  209.     question_ids = annotations['question_id'].values
  210.     print("Questions read.")
  211.     print("Number of questions = %d" %(len(question_ids)))
  212.  
  213.     print("Building the vocabulary...")
  214.     if os.path.exists(config.vocabulary_file):
  215.         vocabulary = Vocabulary(config.vocabulary_file)
  216.     else:
  217.         vocabulary = build_vocabulary(config)
  218.     print("Vocabulary built.")
  219.     print("Number of words = %d" %(vocabulary.size))
  220.     config.vocabulary_size = vocabulary.size
  221.  
  222.     print("Processing the questions...")
  223.     question_word_idxs, question_lens = process_questions(questions,
  224.                                                           vocabulary,
  225.                                                           config)
  226.     print("Questions processed.")
  227.  
  228.     print("Building the dataset...")
  229.     dataset = DataSet(image_files,
  230.                       question_word_idxs,
  231.                       question_lens,
  232.                       question_ids,
  233.                       config.batch_size)
  234.     print("Dataset built.")
  235.     return dataset, vocabulary, config
  236.  
  237.  
  238. # 处理vqa
  239. def process_vqa(vqa, label, image_dir, annotation_file):
  240.     """ Build a temporary annotation file for training or evaluation. """
  241.     question_ids = list(vqa.qa.keys())
  242.     image_ids = [vqa.qa[k]['image_id'] for k in question_ids]
  243.     image_files = [os.path.join(image_dir, label+"_000000"+("%06d" %k)+".jpg")
  244.                    for k in image_ids]
  245.     questions = [vqa.qqa[k]['question'] for k in question_ids]
  246.     answers = [vqa.qa[k]['best_answer'] for k in question_ids]
  247.  
  248.     annotations = pd.DataFrame({'question_id': question_ids,
  249.                                 'image_id': image_ids,
  250.                                 'image_file': image_files,
  251.                                 'question': questions,
  252.                                 'answer': answers})
  253.     annotations.to_csv(annotation_file)
  254.     return annotations
  255.  
  256.  
  257. # 处理问题
  258. def process_questions(questions, vocabulary, config):
  259.     """ Tokenize the questions and translate each token into its index \
  260.         in the vocabulary, and get the number of tokens. """
  261.     question_word_idxs = []
  262.     question_lens = []
  263.     for q in tqdm(questions):
  264.         word_idxs = vocabulary.process_sentence(q)
  265.         current_length = len(word_idxs)
  266.         current_word_idxs = np.zeros((config.max_question_length), np.int32)
  267.         current_word_idxs[:current_length] = np.array(word_idxs)
  268.         question_word_idxs.append(current_word_idxs)
  269.         question_lens.append(current_length)
  270.     return np.array(question_word_idxs), np.array(question_lens)
  271.  
  272.  
  273. # 处理答案
  274. def process_answers(answers, vocabulary):
  275.     """ Translate the answers into their indicies in the vocabulary. """
  276.     answer_idxs = []
  277.     for answer in tqdm(answers):
  278.         answer_idxs.append(vocabulary.word_to_idx(word_tokenize(answer)[0]))
  279.     return np.array(answer_idxs)
  280.  
  281.  
  282. # 建立词汇表
  283. def build_vocabulary(config):
  284.     """ Build the vocabulary from the training data and save it to a file. """
  285.     vqa = VQA(config.train_answer_file, config.train_question_file)
  286.     vqa.filter_by_ques_len(config.max_question_length)
  287.     vqa.filter_by_ans_len(1)
  288.  
  289.     question_ids = list(vqa.qa.keys())
  290.     questions = [vqa.qqa[k]['question'] for k in question_ids]
  291.     answers = [vqa.qa[k]['best_answer'] for k in question_ids]
  292.  
  293.     vocabulary = Vocabulary()
  294.     for question in tqdm(questions):
  295.         vocabulary.add_words(word_tokenize(question))
  296.     for answer in tqdm(answers):
  297.         vocabulary.add_words(word_tokenize(answer))
  298.     vocabulary.compute_frequency()
  299.     vocabulary.save(config.vocabulary_file)
  300.     return vocabulary

（4）基本模型文件base_model.py

该文件主要的功能就是实现模型的存储，加载等：

  1. import os
  2. import numpy as np
  3. import pandas as pd
  4. import tensorflow as tf
  5. import cv2
  6. import matplotlib.pyplot as plt
  7. import pickle as pickle
  8. from tqdm import tqdm
  9. import json
  10. import copy
  11. import string
  12.  
  13. from utils.nn import NN
  14. from utils.misc import ImageLoader
  15. from utils.vqa.vqa import VQA
  16. from utils.vqa.vqaEval import VQAEval
  17.  
  18. class BaseModel(object):
  19.     def __init__(self, config):
  20.         self.config = config
  21.         self.is_train = True if config.phase == 'train' else False
  22.         self.train_cnn = self.is_train and config.train_cnn
  23.         self.image_loader = ImageLoader('./utils/ilsvrc_2012_mean.npy')
  24.         self.image_shape = [224, 224, 3]
  25.         self.global_step = tf.Variable(0,
  26.                                        name = 'global_step',
  27.                                        trainable = False)
  28.         self.nn = NN(config)
  29.         self.build()
  30.  
  31.     def build(self):
  32.         raise NotImplementedError()
  33.  
  34.     def get_feed_dict(self, batch):
  35.         raise NotImplementedError()
  36.  
  37.     def train(self, sess, train_data):
  38.         """ Train the model using the VQA training data. """
  39.         print("Training the model...")
  40.         config = self.config
  41.  
  42.         if not os.path.exists(config.summary_dir):
  43.             os.mkdir(config.summary_dir)
  44.         train_writer = tf.summary.FileWriter(config.summary_dir, sess.graph)
  45.  
  46.         for epoch_no in tqdm(list(range(config.num_epochs)), desc='epoch'):
  47.             for idx in tqdm(list(range(train_data.num_batches)), desc='batch'):
  48.                 batch = train_data.next_batch()
  49.                 feed_dict = self.get_feed_dict(batch)
  50.                 _, summary, global_step = sess.run([self.opt_op,
  51.                                                     self.summary,
  52.                                                     self.global_step],
  53.                                                     feed_dict = feed_dict)
  54.                 if (global_step + 1) % config.save_period == 0:
  55.                     self.save()
  56.                 train_writer.add_summary(summary, global_step)
  57.             train_data.reset()
  58.  
  59.         print("Training complete.")
  60.  
  61.     def eval(self, sess, eval_gt_vqa, eval_data, vocabulary):
  62.         """ Evaluate the model using the VQA validation data. """
  63.         print("Evaluating the model...")
  64.         config = self.config
  65.         if not os.path.exists(config.eval_result_dir):
  66.             os.mkdir(config.eval_result_dir)
  67.  
  68.         question_ids = eval_data.question_ids
  69.         answers = []
  70.  
  71.         # Compute the answers to the questions
  72.         idx = 0
  73.         for k in tqdm(list(range(eval_data.num_batches))):
  74.             batch = eval_data.next_batch()
  75.             image_files, question_word_idxs, question_lens = batch
  76.             feed_dict = self.get_feed_dict(batch)
  77.             result = sess.run(self.prediction, feed_dict = feed_dict)
  78.  
  79.             fake_cnt = 0 if k<eval_data.num_batches-1 \
  80.                          else eval_data.fake_count
  81.             for l in range(eval_data.batch_size-fake_cnt):
  82.                 answer = vocabulary.words[result[l]]
  83.                 answers.append(answer)
  84.  
  85.                 # Save the result in an image file
  86.                 if config.save_eval_result_as_image:
  87.                     image_file = image_files[l]
  88.                     image_name = image_file.split(os.sep)[-1]
  89.                     image_name = os.path.splitext(image_name)[0]
  90.  
  91.                     q_word_idxs = question_word_idxs[l]
  92.                     q_len = question_lens[l]
  93.                     q_words = [vocabulary.words[q_word_idxs[i]] \
  94.                         for i in range(q_len)]
  95.                     if q_words[-1] != '?':
  96.                         q_words.append('?')
  97.                     Q = 'Q: ' + ''.join([' '+w if not w.startswith("'") \
  98.                         and w not in string.punctuation \
  99.                         else w for w in q_words]).strip()
  100.                     A = 'A: ' + answer
  101.  
  102.                     image = plt.imread(image_file)
  103.                     plt.imshow(image)
  104.                     plt.axis('off')
  105.                     plt.title(Q+'\n'+A)
  106.                     plt.savefig(image_name + '_' + str(question_ids[idx]) + '_result.jpg')
  107.  
  108.                 idx += 1
  109.  
  110.         results = [{'question_id': question_id, 'answer': answer} \
  111.                    for question_id, answer in zip(question_ids, answers)]
  112.         fp = open(config.eval_result_file, 'wb')
  113.         json.dump(results, fp)
  114.         fp.close()
  115.  
  116.         # Evaluate these answers
  117.         eval_res_vqa = eval_gt_vqa.loadRes(config.eval_result_file,
  118.                                            config.eval_question_file)
  119.         scorer = VQAEval(eval_gt_vqa, eval_res_vqa)
  120.         scorer.evaluate()
  121.         print("Evaluation complete.")
  122.  
  123.     def test(self, sess, test_data, vocabulary):
  124.         """ Test the model using any given images and questions. """
  125.         print("Testing the model...")
  126.         config = self.config
  127.  
  128.         if not os.path.exists(config.test_result_dir):
  129.             os.mkdir(config.test_result_dir)
  130.  
  131.         question_ids = test_data.question_ids
  132.         answers = []
  133.  
  134.         # Compute the answers to the questions
  135.         idx = 0
  136.         for k in tqdm(list(range(test_data.num_batches))):
  137.             batch = test_data.next_batch()
  138.             image_files, question_word_idxs, question_lens = batch
  139.             feed_dict = self.get_feed_dict(batch)
  140.             result = sess.run(self.prediction, feed_dict = feed_dict)
  141.  
  142.             fake_cnt = 0 if k < test_data.num_batches-1 \
  143.                        else test_data.fake_count
  144.             for l in range(test_data.batch_size-fake_cnt):
  145.                 answer = vocabulary.words[result[l]]
  146.                 answers.append(answer)
  147.  
  148.                 # Save the result in an image file
  149.                 image_file = image_files[l]
  150.                 image_name = image_file.split(os.sep)[-1]
  151.                 image_name = os.path.splitext(image_name)[0]
  152.  
  153.                 q_word_idxs = question_word_idxs[l]
  154.                 q_len = question_lens[l]
  155.                 q_words = [vocabulary.words[q_word_idxs[i]] \
  156.                     for i in range(q_len)]
  157.                 if q_words[-1] != '?':
  158.                     q_words.append('?')
  159.                 Q = 'Q: ' + ''.join([' '+w if not w.startswith("'") \
  160.                     and w not in string.punctuation \
  161.                     else w for w in q_words]).strip()
  162.                 A = 'A: ' + answer
  163.  
  164.                 image = plt.imread(image_file)
  165.                 plt.imshow(image)
  166.                 plt.axis('off')
  167.                 plt.title(Q+'\n'+A)
  168.                 plt.savefig(os.path.join(config.test_result_dir, \
  169.                     image_name + '_' + str(question_ids[idx]) \
  170.                     + '_result.jpg'))
  171.  
  172.                 idx += 1
  173.  
  174.         # Save the answers to a file
  175.         test_info = pd.read_csv(config.temp_test_info_file)
  176.         results = pd.DataFrame({'question_id': question_ids,
  177.                                 'answer': answers})
  178.         results = pd.merge(test_info, results)
  179.         results.to_csv(config.test_result_file)
  180.         print("Testing complete.")
  181.  
  182.     def save(self):
  183.         """ Save the model. """
  184.         config = self.config
  185.         data = {v.name: v.eval() for v in tf.global_variables()}
  186.         save_path = os.path.join(config.save_dir, str(self.global_step.eval()))
  187.  
  188.         print((" Saving the model to %s..." % (save_path+".npy")))
  189.         np.save(save_path, data)
  190.         info_file = open(os.path.join(config.save_dir, "config.pickle"), "wb")
  191.         config_ = copy.copy(config)
  192.         config_.global_step = self.global_step.eval()
  193.         pickle.dump(config_, info_file)
  194.         info_file.close()
  195.         print("Model saved.")
  196.  
  197.     def load(self, sess, model_file=None):
  198.         """ Load the model. """
  199.         config = self.config
  200.         if model_file is not None:
  201.             save_path = model_file
  202.         else:
  203.             info_path = os.path.join(config.save_dir, "config.pickle")
  204.             info_file = open(info_path, "rb")
  205.             config = pickle.load(info_file)
  206.             global_step = config.global_step
  207.             info_file.close()
  208.             save_path = os.path.join(config.save_dir,
  209.                                      str(global_step)+".npy")
  210.  
  211.         print("Loading the model from %s..." %save_path)
  212.         data_dict = np.load(save_path).item()
  213.         count = 0
  214.         for v in tqdm(tf.global_variables()):
  215.             if v.name in data_dict.keys():
  216.                 sess.run(v.assign(data_dict[v.name]))
  217.                 count += 1
  218.         print("%d tensors loaded." %count)
  219.  
  220.     def load_cnn(self, session, data_path, ignore_missing=True):
  221.         """ Load a pretrained CNN model. """
  222.         print("Loading the CNN from %s..." %data_path)
  223.         data_dict = np.load(data_path).item()
  224.         count = 0
  225.         for op_name in tqdm(data_dict):
  226.             with tf.variable_scope(op_name, reuse=True):
  227.                 for param_name, data in data_dict[op_name].iteritems():
  228.                     try:
  229.                         var = tf.get_variable(param_name)
  230.                         session.run(var.assign(data))
  231.                         count += 1
  232.                     except ValueError:
  233.                         pass
  234.         print("%d tensors loaded." %count)

（5）情景记忆模块episodic_memory.py

该文件是情景记忆模块文件：

  1. import tensorflow as tf
  2. from utils.nn import NN
  3.  
  4. class AttnGRU(object):
  5.     """ Attention-based GRU (used by the Episodic Memory Module). """
  6.     def __init__(self, config):
  7.         self.nn = NN(config)
  8.         self.num_units = config.num_gru_units
  9.  
  10.     def __call__(self, inputs, state, attention):
  11.         with tf.variable_scope('attn_gru'):
  12.             r_input = tf.concat([inputs, state], axis = 1)
  13.             r_input = self.nn.dropout(r_input)
  14.             r = self.nn.dense(r_input,
  15.                               units = self.num_units,
  16.                               activation = None,
  17.                               use_bias = False,
  18.                               name = 'fc1')
  19.             b = tf.get_variable('fc1/bias',
  20.                                 shape = [self.num_units],
  21.                                 initializer = tf.constant_initializer(1.0))
  22.             r = tf.nn.bias_add(r, b)
  23.             r = tf.sigmoid(r)
  24.  
  25.             c_input = tf.concat([inputs, r*state], axis = 1)
  26.             c_input = self.nn.dropout(c_input)
  27.             c = self.nn.dense(c_input,
  28.                               units = self.num_units,
  29.                               activation = tf.tanh,
  30.                               name = 'fc2')
  31.  
  32.             new_state = attention * c + (1 - attention) * state
  33.         return new_state
  34.  
  35. class EpisodicMemory(object):
  36.     """ Episodic Memory Module. """
  37.     def __init__(self, config, num_facts, question, facts):
  38.         self.nn = NN(config)
  39.         self.num_units = config.num_gru_units
  40.         self.num_facts = num_facts
  41.         self.question = question
  42.         self.facts = facts
  43.         self.attention = config.attention
  44.         if self.attention == 'gru':
  45.             self.attn_gru = AttnGRU(config)
  46.  
  47.     def new_fact(self, memory):
  48.         """ Get the context vector by using either soft attention or
  49.             attention-based GRU. """
  50.         fact_list = tf.unstack(self.facts, axis = 1)
  51.         mixed_fact = tf.zeros_like(fact_list[0])
  52.  
  53.         with tf.variable_scope('attend'):
  54.             attentions = self.attend(memory)
  55.  
  56.         if self.attention == 'gru':
  57.             with tf.variable_scope('attn_gate') as scope:
  58.                 attentions = tf.unstack(attentions, axis = 1)
  59.                 for ctx, att in zip(fact_list, attentions):
  60.                     mixed_fact = self.attn_gru(ctx,
  61.                                                mixed_fact,
  62.                                                tf.expand_dims(att, 1))
  63.                     scope.reuse_variables()
  64.         else:
  65.             mixed_fact = tf.reduce_sum(self.facts*tf.expand_dims(attentions, 2),
  66.                                        axis = 1)
  67.  
  68.         return mixed_fact
  69.  
  70.     def attend(self, memory):
  71.         """ Get the attention weights. """
  72.         c = self.facts
  73.         q = tf.tile(tf.expand_dims(self.question, 1), [1, self.num_facts, 1])
  74.         m = tf.tile(tf.expand_dims(memory, 1), [1, self.num_facts, 1])
  75.  
  76.         z = tf.concat([c*q, c*m, tf.abs(c-q), tf.abs(c-m)], 2)
  77.         z = tf.reshape(z, [-1, 4*self.num_units])
  78.  
  79.         z = self.nn.dropout(z)
  80.         z1 = self.nn.dense(z,
  81.                            units = self.num_units,
  82.                            activation = tf.tanh,
  83.                            name = 'fc1')
  84.         z1 = self.nn.dropout(z1)
  85.         z2 = self.nn.dense(z1,
  86.                            units = 1,
  87.                            activation = None,
  88.                            use_bias = False,
  89.                            name = 'fc2')
  90.         z2 = tf.reshape(z2, [-1, self.num_facts])
  91.  
  92.         attentions = tf.nn.softmax(z2)
  93.         return attentions

（6）模型文件model.py

该文件是模型文件，是根据前面的模块来集成问答模型：

  1. import tensorflow as tf
  2. import numpy as np
  3.  
  4. from base_model import BaseModel
  5. from episodic_memory import EpisodicMemory
  6.  
  7. class QuestionAnswerer(BaseModel):
  8.     def build(self):
  9.         """ Build the model. """
  10.         self.build_cnn()
  11.         self.build_rnn()
  12.         if self.is_train:
  13.             self.build_optimizer()
  14.             self.build_summary()
  15.  
  16.     def build_cnn(self):
  17.         """ Build the CNN. """
  18.         print("Building the CNN...")
  19.         if self.config.cnn =='vgg16':
  20.             self.build_vgg16()
  21.         else:
  22.             self.build_resnet50()
  23.         print("CNN built.")
  24.  
  25.     def build_vgg16(self):
  26.         """ Build the VGG16 net. """
  27.         config = self.config
  28.  
  29.         images = tf.placeholder(
  30.             dtype = tf.float32,
  31.             shape = [config.batch_size] + self.image_shape)
  32.  
  33.         conv1_1_feats = self.nn.conv2d(images, 64, name = 'conv1_1')
  34.         conv1_2_feats = self.nn.conv2d(conv1_1_feats, 64, name = 'conv1_2')
  35.         pool1_feats = self.nn.max_pool2d(conv1_2_feats, name = 'pool1')
  36.  
  37.         conv2_1_feats = self.nn.conv2d(pool1_feats, 128, name = 'conv2_1')
  38.         conv2_2_feats = self.nn.conv2d(conv2_1_feats, 128, name = 'conv2_2')
  39.         pool2_feats = self.nn.max_pool2d(conv2_2_feats, name = 'pool2')
  40.  
  41.         conv3_1_feats = self.nn.conv2d(pool2_feats, 256, name = 'conv3_1')
  42.         conv3_2_feats = self.nn.conv2d(conv3_1_feats, 256, name = 'conv3_2')
  43.         conv3_3_feats = self.nn.conv2d(conv3_2_feats, 256, name = 'conv3_3')
  44.         pool3_feats = self.nn.max_pool2d(conv3_3_feats, name = 'pool3')
  45.  
  46.         conv4_1_feats = self.nn.conv2d(pool3_feats, 512, name = 'conv4_1')
  47.         conv4_2_feats = self.nn.conv2d(conv4_1_feats, 512, name = 'conv4_2')
  48.         conv4_3_feats = self.nn.conv2d(conv4_2_feats, 512, name = 'conv4_3')
  49.         pool4_feats = self.nn.max_pool2d(conv4_3_feats, name = 'pool4')
  50.  
  51.         conv5_1_feats = self.nn.conv2d(pool4_feats, 512, name = 'conv5_1')
  52.         conv5_2_feats = self.nn.conv2d(conv5_1_feats, 512, name = 'conv5_2')
  53.         conv5_3_feats = self.nn.conv2d(conv5_2_feats, 512, name = 'conv5_3')
  54.  
  55.         self.permutation = self.get_permutation(14, 14)
  56.         conv5_3_feats_flat = self.flatten_feats(conv5_3_feats, 512)
  57.         self.conv_feats = conv5_3_feats_flat
  58.         self.conv_feat_shape = [196, 512]
  59.         self.images = images
  60.  
  61.     def build_resnet50(self):
  62.         """ Build the ResNet50. """
  63.         config = self.config
  64.  
  65.         images = tf.placeholder(
  66.             dtype = tf.float32,
  67.             shape = [config.batch_size] + self.image_shape)
  68.  
  69.         conv1_feats = self.nn.conv2d(images,
  70.                                   filters = 64,
  71.                                   kernel_size = (7, 7),
  72.                                   strides = (2, 2),
  73.                                   activation = None,
  74.                                   name = 'conv1')
  75.         conv1_feats = self.nn.batch_norm(conv1_feats, 'bn_conv1')
  76.         conv1_feats = tf.nn.relu(conv1_feats)
  77.         pool1_feats = self.nn.max_pool2d(conv1_feats,
  78.                                       pool_size = (3, 3),
  79.                                       strides = (2, 2),
  80.                                       name = 'pool1')
  81.  
  82.         res2a_feats = self.resnet_block(pool1_feats, 'res2a', 'bn2a', 64, 1)
  83.         res2b_feats = self.resnet_block2(res2a_feats, 'res2b', 'bn2b', 64)
  84.         res2c_feats = self.resnet_block2(res2b_feats, 'res2c', 'bn2c', 64)
  85.  
  86.         res3a_feats = self.resnet_block(res2c_feats, 'res3a', 'bn3a', 128)
  87.         res3b_feats = self.resnet_block2(res3a_feats, 'res3b', 'bn3b', 128)
  88.         res3c_feats = self.resnet_block2(res3b_feats, 'res3c', 'bn3c', 128)
  89.         res3d_feats = self.resnet_block2(res3c_feats, 'res3d', 'bn3d', 128)
  90.  
  91.         res4a_feats = self.resnet_block(res3d_feats, 'res4a', 'bn4a', 256)
  92.         res4b_feats = self.resnet_block2(res4a_feats, 'res4b', 'bn4b', 256)
  93.         res4c_feats = self.resnet_block2(res4b_feats, 'res4c', 'bn4c', 256)
  94.         res4d_feats = self.resnet_block2(res4c_feats, 'res4d', 'bn4d', 256)
  95.         res4e_feats = self.resnet_block2(res4d_feats, 'res4e', 'bn4e', 256)
  96.         res4f_feats = self.resnet_block2(res4e_feats, 'res4f', 'bn4f', 256)
  97.  
  98.         res5a_feats = self.resnet_block(res4f_feats, 'res5a', 'bn5a', 512)
  99.         res5b_feats = self.resnet_block2(res5a_feats, 'res5b', 'bn5b', 512)
  100.         res5c_feats = self.resnet_block2(res5b_feats, 'res5c', 'bn5c', 512)
  101.  
  102.         self.permutation = self.get_permutation(7, 7)
  103.         res5c_feats_flat = self.flatten_feats(res5c_feats, 2048)
  104.         self.conv_feats = res5c_feats_flat
  105.         self.conv_feat_shape = [49, 2048]
  106.         self.images = images
  107.  
  108.     def resnet_block(self, inputs, name1, name2, c, s=2):
  109.         """ A basic block of ResNet. """
  110.         branch1_feats = self.nn.conv2d(inputs,
  111.                                     filters = 4*c,
  112.                                     kernel_size = (1, 1),
  113.                                     strides = (s, s),
  114.                                     activation = None,
  115.                                     use_bias = False,
  116.                                     name = name1+'_branch1')
  117.         branch1_feats = self.nn.batch_norm(branch1_feats, name2+'_branch1')
  118.  
  119.         branch2a_feats = self.nn.conv2d(inputs,
  120.                                      filters = c,
  121.                                      kernel_size = (1, 1),
  122.                                      strides = (s, s),
  123.                                      activation = None,
  124.                                      use_bias = False,
  125.                                      name = name1+'_branch2a')
  126.         branch2a_feats = self.nn.batch_norm(branch2a_feats, name2+'_branch2a')
  127.         branch2a_feats = tf.nn.relu(branch2a_feats)
  128.  
  129.         branch2b_feats = self.nn.conv2d(branch2a_feats,
  130.                                      filters = c,
  131.                                      kernel_size = (3, 3),
  132.                                      strides = (1, 1),
  133.                                      activation = None,
  134.                                      use_bias = False,
  135.                                      name = name1+'_branch2b')
  136.         branch2b_feats = self.nn.batch_norm(branch2b_feats, name2+'_branch2b')
  137.         branch2b_feats = tf.nn.relu(branch2b_feats)
  138.  
  139.         branch2c_feats = self.nn.conv2d(branch2b_feats,
  140.                                      filters = 4*c,
  141.                                      kernel_size = (1, 1),
  142.                                      strides = (1, 1),
  143.                                      activation = None,
  144.                                      use_bias = False,
  145.                                      name = name1+'_branch2c')
  146.         branch2c_feats = self.nn.batch_norm(branch2c_feats, name2+'_branch2c')
  147.  
  148.         outputs = branch1_feats + branch2c_feats
  149.         outputs = tf.nn.relu(outputs)
  150.         return outputs
  151.  
  152.     def resnet_block2(self, inputs, name1, name2, c):
  153.         """ Another basic block of ResNet. """
  154.         branch2a_feats = self.nn.conv2d(inputs,
  155.                                      filters = c,
  156.                                      kernel_size = (1, 1),
  157.                                      strides = (1, 1),
  158.                                      activation = None,
  159.                                      use_bias = False,
  160.                                      name = name1+'_branch2a')
  161.         branch2a_feats = self.nn.batch_norm(branch2a_feats, name2+'_branch2a',)
  162.         branch2a_feats = tf.nn.relu(branch2a_feats)
  163.  
  164.         branch2b_feats = self.nn.conv2d(branch2a_feats,
  165.                                      filters = c,
  166.                                      kernel_size = (3, 3),
  167.                                      strides = (1, 1),
  168.                                      activation = None,
  169.                                      use_bias = False,
  170.                                      name = name1+'_branch2b')
  171.         branch2b_feats = self.nn.batch_norm(branch2b_feats, name2+'_branch2b')
  172.         branch2b_feats = tf.nn.relu(branch2b_feats)
  173.  
  174.         branch2c_feats = self.nn.conv2d(branch2b_feats,
  175.                                      filters = 4*c,
  176.                                      kernel_size = (1, 1),
  177.                                      strides = (1, 1),
  178.                                      activation = None,
  179.                                      use_bias = False,
  180.                                      name = name1+'_branch2c')
  181.         branch2c_feats = self.nn.batch_norm(branch2c_feats, name2+'_branch2c')
  182.  
  183.         outputs = inputs + branch2c_feats
  184.         outputs = tf.nn.relu(outputs)
  185.         return outputs
  186.  
  187.     def get_permutation(self, height, width):
  188.         """ Get the permutation corresponding to the snake-like walk decribed \
  189.            in the paper. Used to flatten the convolutional feats. """
  190.         permutation = np.zeros(height*width, np.int32)
  191.         for i in range(height):
  192.             for j in range(width):
  193.                 permutation[i*width+j] = i*width+j if i%2==0  \
  194.                                          else (i+1)*width-j-1
  195.         return permutation
  196.  
  197.     def flatten_feats(self, feats, channels):
  198.         """ Flatten the feats. """
  199.         temp1 = tf.reshape(feats, [self.config.batch_size, -1, channels])
  200.         temp1 = tf.transpose(temp1, [1, 0, 2])
  201.         temp2 = tf.gather(temp1, self.permutation)
  202.         temp2 = tf.transpose(temp2, [1, 0, 2])
  203.         return temp2
  204.  
  205.     def build_rnn(self):
  206.         """ Build the RNN. """
  207.         print("Building the RNN...")
  208.         config = self.config
  209.  
  210.         facts = self.conv_feats
  211.         num_facts, dim_fact = self.conv_feat_shape
  212.  
  213.         # Setup the placeholders
  214.         question_word_idxs = tf.placeholder(
  215.             dtype = tf.int32,
  216.             shape = [config.batch_size, config.max_question_length])
  217.         question_lens = tf.placeholder(
  218.             dtype = tf.int32,
  219.             shape = [config.batch_size])
  220.         if self.is_train:
  221.             answer_idxs = tf.placeholder(
  222.                 dtype = tf.int32,
  223.                 shape = [config.batch_size])
  224.             if config.question_encoding == 'positional':
  225.                 position_weights = tf.placeholder(
  226.                     dtype = tf.float32,
  227.                     shape = [config.batch_size, \
  228.                              config.max_question_length, \
  229.                              config.dim_embedding])
  230.  
  231.         # Setup the word embedding
  232.         with tf.variable_scope("word_embedding"):
  233.             embedding_matrix = tf.get_variable(
  234.                 name = 'weights',
  235.                 shape = [config.vocabulary_size, config.dim_embedding],
  236.                 initializer = self.nn.fc_kernel_initializer,
  237.                 regularizer = self.nn.fc_kernel_regularizer,
  238.                 trainable = self.is_train)
  239.  
  240.         # Encode the questions
  241.         with tf.variable_scope('question_encoding'):
  242.             question_embeddings = tf.nn.embedding_lookup(
  243.                 embedding_matrix,
  244.                 question_word_idxs)
  245.  
  246.             if config.question_encoding == 'positional':
  247.                 # use positional encoding
  248.                 self.build_position_weights()
  249.                 question_encodings = question_embeddings * position_weights
  250.                 question_encodings = tf.reduce_sum(question_encodings,
  251.                                                     axis = 1)
  252.             else:
  253.                 # use GRU encoding
  254.                 outputs, _ = tf.nn.dynamic_rnn(
  255.                     self.nn.gru(),
  256.                     inputs = question_embeddings,
  257.                     dtype = tf.float32)
  258.  
  259.                 question_encodings = []
  260.                 for k in range(config.batch_size):
  261.                     question_encoding = tf.slice(outputs,
  262.                                                  [k, question_lens[k]-1, 0],
  263.                                                  [1, 1, config.num_gru_units])
  264.                     question_encodings.append(tf.squeeze(question_encoding))
  265.                 question_encodings = tf.stack(question_encodings, axis = 0)
  266.  
  267.         # Encode the facts
  268.         with tf.variable_scope('input_fusion'):
  269.             if config.embed_fact:
  270.                 facts = tf.reshape(facts, [-1, dim_fact])
  271.                 facts = self.nn.dropout(facts)
  272.                 facts = self.nn.dense(
  273.                     facts,
  274.                     units = config.dim_embedding,
  275.                     activation = tf.tanh,
  276.                     name = 'fc')
  277.                 facts = tf.reshape(facts, [-1, num_facts, config.dim_embedding])
  278.  
  279.             outputs, _ = tf.nn.bidirectional_dynamic_rnn(
  280.                 self.nn.gru(),
  281.                 self.nn.gru(),
  282.                 inputs = facts,
  283.                 dtype = tf.float32)
  284.             outputs_fw, outputs_bw = outputs
  285.             fact_encodings = outputs_fw + outputs_bw
  286.  
  287.         # Episodic Memory Update
  288.         with tf.variable_scope('episodic_memory'):
  289.             episode = EpisodicMemory(config,
  290.                                      num_facts,
  291.                                      question_encodings,
  292.                                      fact_encodings)
  293.             memory = tf.identity(question_encodings)
  294.  
  295.             if config.tie_memory_weight:
  296.                 scope_list = ['layer'] * config.memory_step
  297.             else:
  298.                 scope_list = ['layer'+str(t) for t in range(config.memory_step)]
  299.  
  300.             for t in range(config.memory_step):
  301.                 with tf.variable_scope(scope_list[t], reuse = tf.AUTO_REUSE):
  302.                     fact = episode.new_fact(memory)
  303.                     if config.memory_update == 'gru':
  304.                         gru = self.nn.gru()
  305.                         memory = gru(fact, memory)[0]
  306.                     else:
  307.                         expanded_memory = tf.concat(
  308.                             [memory, fact, question_encodings],
  309.                             axis = 1)
  310.                         expanded_memory = self.nn.dropout(expanded_memory)
  311.                         memory = self.nn.dense(
  312.                             expanded_memory,
  313.                             units = config.num_gru_units,
  314.                             activation = tf.nn.relu,
  315.                             name = 'fc')
  316.  
  317.         # Compute the result
  318.         with tf.variable_scope('result'):
  319.             expanded_memory = tf.concat([memory, question_encodings],
  320.                                         axis = 1)
  321.             expanded_memory = self.nn.dropout(expanded_memory)
  322.             logits = self.nn.dense(expanded_memory,
  323.                                    units = config.vocabulary_size,
  324.                                    activation = None,
  325.                                    name = 'logits')
  326.             prediction = tf.argmax(logits, axis = 1)
  327.  
  328.         # Compute the loss and accuracy if necessary
  329.         if self.is_train:
  330.             cross_entropy_loss = tf.nn.sparse_softmax_cross_entropy_with_logits(
  331.                 labels = answer_idxs,
  332.                 logits = logits)
  333.             cross_entropy_loss = tf.reduce_mean(cross_entropy_loss)
  334.             reg_loss = tf.losses.get_regularization_loss()
  335.             total_loss = cross_entropy_loss + reg_loss
  336.  
  337.             ground_truth = tf.cast(answer_idxs, tf.int64)
  338.             prediction_correct = tf.where(
  339.                 tf.equal(prediction, ground_truth),
  340.                 tf.cast(tf.ones_like(prediction), tf.float32),
  341.                 tf.cast(tf.zeros_like(prediction), tf.float32))
  342.             accuracy = tf.reduce_mean(prediction_correct)
  343.  
  344.         self.question_word_idxs = question_word_idxs
  345.         self.question_lens = question_lens
  346.         self.prediction = prediction
  347.  
  348.         if self.is_train:
  349.             self.answer_idxs = answer_idxs
  350.             if config.question_encoding == 'positional':
  351.                 self.position_weights = position_weights
  352.             self.total_loss = total_loss
  353.             self.cross_entropy_loss = cross_entropy_loss
  354.             self.reg_loss = reg_loss
  355.             self.accuracy = accuracy
  356.  
  357.         print("RNN built.")
  358.  
  359.     def build_position_weights(self):
  360.         """ Setup the weights for the positional encoding of questions. """
  361.         config = self.config
  362.         D = config.dim_embedding
  363.         pos_weights = []
  364.         for M in range(config.max_question_length):
  365.             cur_pos_weights = []
  366.             for j in range(config.max_question_length):
  367.                 if j <= M:
  368.                     temp = [1.0-(j+1.0)/(M+1.0) \
  369.                             -((d+1.0)/D)*(1-2.0*(j+1.0)/(M+1.0)) \
  370.                             for d in range(D)]
  371.                 else:
  372.                     temp = [0.0] * D
  373.                 cur_pos_weights.append(temp)
  374.             pos_weights.append(cur_pos_weights)
  375.         self.pos_weights = np.array(pos_weights, np.float32)
  376.  
  377.     def build_optimizer(self):
  378.         """ Setup the training operation. """
  379.         config = self.config
  380.  
  381.         learning_rate = tf.constant(config.initial_learning_rate)
  382.         if config.learning_rate_decay_factor < 1.0:
  383.             def _learning_rate_decay_fn(learning_rate, global_step):
  384.                 return tf.train.exponential_decay(
  385.                     learning_rate,
  386.                     global_step,
  387.                     decay_steps = config.num_steps_per_decay,
  388.                     decay_rate = config.learning_rate_decay_factor,
  389.                     staircase = True)
  390.             learning_rate_decay_fn = _learning_rate_decay_fn
  391.         else:
  392.             learning_rate_decay_fn = None
  393.  
  394.         with tf.variable_scope('optimizer', reuse = tf.AUTO_REUSE):
  395.             if config.optimizer == 'Adam':
  396.                 optimizer = tf.train.AdamOptimizer(
  397.                     learning_rate = config.initial_learning_rate,
  398.                     beta1 = config.beta1,
  399.                     beta2 = config.beta2,
  400.                     epsilon = config.epsilon
  401.                     )
  402.             elif config.optimizer == 'RMSProp':
  403.                 optimizer = tf.train.RMSPropOptimizer(
  404.                     learning_rate = config.initial_learning_rate,
  405.                     decay = config.decay,
  406.                     momentum = config.momentum,
  407.                     centered = config.centered,
  408.                     epsilon = config.epsilon
  409.                 )
  410.             elif config.optimizer == 'Momentum':
  411.                 optimizer = tf.train.MomentumOptimizer(
  412.                     learning_rate = config.initial_learning_rate,
  413.                     momentum = config.momentum,
  414.                     use_nesterov = config.use_nesterov
  415.                 )
  416.             else:
  417.                 optimizer = tf.train.GradientDescentOptimizer(
  418.                     learning_rate = config.initial_learning_rate
  419.                 )
  420.  
  421.             opt_op = tf.contrib.layers.optimize_loss(
  422.                 loss = self.total_loss,
  423.                 global_step = self.global_step,
  424.                 learning_rate = learning_rate,
  425.                 optimizer = optimizer,
  426.                 clip_gradients = config.clip_gradients,
  427.                 learning_rate_decay_fn = learning_rate_decay_fn)
  428.  
  429.         self.opt_op = opt_op
  430.  
  431.     def build_summary(self):
  432.         """ Build the summary (for TensorBoard visualization). """
  433.         with tf.name_scope("variables"):
  434.             for var in tf.trainable_variables():
  435.                 with tf.name_scope(var.name[:var.name.find(":")]):
  436.                     self.variable_summary(var)
  437.  
  438.         with tf.name_scope("metrics"):
  439.             tf.summary.scalar("cross_entropy_loss", self.cross_entropy_loss)
  440.             tf.summary.scalar("reg_loss", self.reg_loss)
  441.             tf.summary.scalar("total_loss", self.total_loss)
  442.             tf.summary.scalar("accuracy", self.accuracy)
  443.  
  444.         self.summary = tf.summary.merge_all()
  445.  
  446.     def variable_summary(self, var):
  447.         """ Build the summary for a variable. """
  448.         mean = tf.reduce_mean(var)
  449.         tf.summary.scalar('mean', mean)
  450.         stddev = tf.sqrt(tf.reduce_mean(tf.square(var - mean)))
  451.         tf.summary.scalar('stddev', stddev)
  452.         tf.summary.scalar('max', tf.reduce_max(var))
  453.         tf.summary.scalar('min', tf.reduce_min(var))
  454.         tf.summary.histogram('histogram', var)
  455.  
  456.     def get_feed_dict(self, batch):
  457.         """ Get the feed dictionary for the current batch. """
  458.         config = self.config
  459.         if self.is_train:
  460.             # training phase
  461.             image_files, question_word_idxs, question_lens, answer_idxs = batch
  462.             images = self.image_loader.load_images(image_files)
  463.             if config.question_encoding == 'positional':
  464.                 position_weights = [self.pos_weights[question_lens[i]-1, :, :]
  465.                                     for i in range(config.batch_size)]
  466.                 position_weights = np.array(position_weights, np.float32)
  467.                 return {self.images: images,
  468.                         self.question_word_idxs: question_word_idxs,
  469.                         self.question_lens: question_lens,
  470.                         self.answer_idxs: answer_idxs,
  471.                         self.position_weights: position_weights}
  472.             else:
  473.                 return {self.images: images,
  474.                         self.question_word_idxs: question_word_idxs,
  475.                         self.question_lens: question_lens,
  476.                         self.answer_idxs: answer_idxs}
  477.         else:
  478.             # evaluation or testing phase
  479.             image_files, question_word_idxs, question_lens = batch
  480.             images = self.image_loader.load_images(image_files)
  481.             return {self.images: images,
  482.                     self.question_word_idxs: question_word_idxs,
  483.                     self.question_lens: question_lens}

（7）主文件main.py

主文件中确定运行的阶段，是训练还是测试还是评估？

  1. #!/usr/bin/python
  2. import tensorflow as tf
  3.  
  4. from config import Config
  5. from model import QuestionAnswerer
  6. from dataset import prepare_train_data, prepare_eval_data, prepare_test_data
  7.  
  8. FLAGS = tf.app.flags.FLAGS
  9.  
  10. tf.flags.DEFINE_string('phase', 'train',
  11.                        'The phase can be train, eval or test')
  12.  
  13. tf.flags.DEFINE_boolean('load', False,
  14.                         'Turn on to load a pretrained model from either \
  15.                         the latest checkpoint or a specified file')
  16.  
  17. tf.flags.DEFINE_string('model_file', None,
  18.                        'If sepcified, load a pretrained model from this file')
  19.  
  20. tf.flags.DEFINE_boolean('load_cnn', False,
  21.                         'Turn on to load a pretrained CNN model')
  22.  
  23. tf.flags.DEFINE_string('cnn_model_file', './vgg16_no_fc.npy',
  24.                        'File containing a pretrained CNN model')
  25.  
  26. tf.flags.DEFINE_boolean('train_cnn', False,
  27.                         'Turn on to train both CNN and RNN. \
  28.                          Otherwise, only RNN is trained')
  29.  
  30. def main(argv):
  31.     config = Config()
  32.     config.phase = FLAGS.phase
  33.     config.train_cnn = FLAGS.train_cnn
  34.  
  35.     with tf.Session() as sess:
  36.         if FLAGS.phase == 'train':
  37.             # training phase
  38.             data, config = prepare_train_data(config)
  39.             model = QuestionAnswerer(config)
  40.             sess.run(tf.global_variables_initializer())
  41.             if FLAGS.load:
  42.                 model.load(sess, FLAGS.model_file)
  43.             if FLAGS.load_cnn:
  44.                 model.load_cnn(sess, FLAGS.cnn_model_file)
  45.             tf.get_default_graph().finalize()
  46.             model.train(sess, data)
  47.  
  48.         elif FLAGS.phase == 'eval':
  49.             # evaluation phase
  50.             vqa, data, vocabulary, config = prepare_eval_data(config)
  51.             model = QuestionAnswerer(config)
  52.             model.load(sess, FLAGS.model_file)
  53.             tf.get_default_graph().finalize()
  54.             model.eval(sess, vqa, data, vocabulary)
  55.  
  56.         else:
  57.             # testing phase
  58.             data, vocabulary, config = prepare_test_data(config)
  59.             model = QuestionAnswerer(config)
  60.             model.load(sess, FLAGS.model_file)
  61.             tf.get_default_graph().finalize()
  62.             model.test(sess, data, vocabulary)
  63.  
  64. if __name__ == '__main__':
  65.     tf.app.run()

4. 实验训练

首先是训练：

第一次训练时候的参数设置，phase设置为train即可。

由于我的GPU（1050TI）比较小，直接运行会内存溢出，于是batch size改成了4，数据量比较大，训练时间很长，大概训练了6天吧（还没训练完，6天只训练了一个半epoch），下面这张图是训练了一天的景象：

下面是第四天第一个epoch快训练完的景象：

如果是训练了一半，想要接着之前的结果继续训练，将phase设置为train，然后将model_file设置为'./models/xxxxxx.npy'。

如果想要监测训练过程，可以在cmd中运行：tensorboard --logdir='./summary/'

如果想一起训练CNN和RNN，则打开--train_cnn即可，关闭则只训练RNN。

接下来是模型评价：

需要设置两个参数：--phase=eval --model_file='./models/xxxxxx.npy'

结果将会保存在'val/results.json'中。

最后就是测试：

可以用训练模型来回答JPEG图像的任何问题，将图像放在路径‘test/images’下，然后创建一个CSV（内容包含图像，问题，问题id），放在路径‘test’下，最后按照下面的参数设置运行：--phase=test --model_file='./models/xxxxxx.npy'

结果最终保存在‘test/results’里面。

5. 实验结果

训练时每1000个batch保存一次，我大概训练了74000+个batch，最后评估和测试用的是72999.npy：

模型的测试时间比较久，大概一共有40000多张图像，测试时间大约需要6个小时：

模型测试做到后面发现json的写入报错了，后面我考虑将它改写为csv保存结果。

先来看一下测试过程吧，我选了3张图，一张是航空影像拍摄的棒球场，一张是卡通图象皮卡丘，一张是足球运动图：

先要做一个csv文件，写入图像的路径和你想问的问题，放在路径“./test/question.csv”下：

来看一下回答结果吧，预想的应该是只有足球能回答正确：

第一张图问图像有多少个棒球场，回答两个，正确，第二张图问这个黄色的可爱小动物的名字是啥，回答长颈鹿，错误。

最后一个问题是他们在做什么运动？答案是飞碟，也回答错了。

四、小结

1. VQA还挺难的，不管是模型编写还是数据读取，都要非常细心。

本文内容由网友自发贡献，转载请注明出处：【wpsshop博客】
推荐阅读
相关标签

Copyright © 2003-2013 www.wpsshop.cn 版权所有，并保留所有权利。

  

闽ICP备14008679号