GQA 的简单介绍

编写不易如果觉得不错，麻烦关注一下~

GQA 的简单介绍

官网：GQA: Visual Reasoning in the Real World

问题减少 强的语言偏置，很多都是根据场景语义图进行构建

且有多个评价准则：

consistency, validity, plausibility, grounding and distribution scores

之外该数据集还有一个类似指引步骤的标注，将问题分解为图中的路径，去寻找答案

还有长句来回答问题 

 

语言偏置笼统的说，是指给定一个问题，可以不根据图片就直接作答，问题模态的权重很大。尤其当有10个关于香蕉的问题答案都是黄色的。那么给一个绿香蕉，多数情况下答案还是黄色。

Our starting point in creating the GQA dataset is the Visual Genome Scene Graph annotations [ 20 ] that cover 113k images from COCO [ 23 ] and Flickr [ 36 ]. 2 The scene graph serves as a formalized representation of the image: each node denotes an object , a visual entity within the image, like a person, an apple, grass or clouds. It is linked to a bounding box specifying its position and size, and is marked up with about 1–3 attributes , properties of the object: e.g., its color, shape, material or activity. The objects are connected by relation edges, representing actions (verbs), spatial relations (prepositions), and comparatives.

The GQA dataset consists of 22,669,678 questions over 113,018 images, which cover wide range of reasoning skills and vary in length and number of required inference-steps (fifigure 6 ). The dataset has a vocabulary size of 3097 words and 1878 possible answers. While smaller than natural language datasets, further investigation reveals that it covers 88.8% and 70.6% of VQA questions and answers respectively, corroborating its wide diversity. A wide selection of dataset visualizations is provided in the supplementary.

We associate each question with two types: structural and semantic. The structural type is derived from the fifinal operation in the question’s functional program. It can be (1) verify for yes/no questions, (2) query for all open questions, (3) choose for questions that present two alternatives to choose from, e.g . “Is it red or blue?”; (4) logical which involve logical inference, and (5) compare for comparison questions between two or more objects. The semantic type refers to the main subject of the question: (1) object : for existence questions, (2) attribute : consider the properties or position of an object, (3) category : related to object identification within some class, (4) relation : for questions asking about the subject or object of a described relation (e.g . “what is the girl wearing?” ), and (5) global : about overall properties of the scene such as weather or place. As shown in fifigure 6 , the questions’ types vary at both the semantic and structural levels.

新的标准准则 

Consistency

下面尝试获取截图的信息：

 下面进行各个图片的object num 个数的统计，以设计一个合适的图片框个数

即 统计objects 数组的长度即可

抽取图片的序号以及关系名

当然官网的trainscenegraph 的文件共有这些图片生成了

竟然还有126个目标的时候

将objnumtrain 中的数据筛选成和lxmert 一致的train 图片集

重新整理图片中的目标个数

平均为16张 ，大于50个的总共180张图，大于40个的766个。

选择50个看来比较合理

下面将各个object 的目标的坐标改为x1,y1,x2,y2，以及object 坐标，为其关系的提取做准备

 整理成json 格式，这里只展示最终的状态：

这是下面需要保存的dict形式

上面图的坐标已勾勒好！

---

最终整理成如下的dict。 

如果生成关系矩阵很有可能你需要对角为1的初始矩阵

import numpy as np
 
a=[1,2,3]
 
np.diag(a)

valscenegraph 的关系类型总共有295种 

trainscenegraph 总共有296 种类型,但是我看了大部分还是重复。有效的一般在150种左右

 

而vg fasterrcnn 总共提供了20个类型 

其中下面这些不再vg的20 类中

看一下总共有多少种类型的问题

 {'categoryThis', 'existAttrNotC', 'activityWho', 'verifyAttrAnd', 'weatherChoose', 'materialVerify', 'materialChoose', 'companyVerifyC', 'existAttrOr', 'existAnd', 'categoryThisChoose', 'directOf', 'typeVerifyC', 'typeVerify', 'weather', 'sameRelate', 'activityChoose', 'positionQuery', 'companyVerify', 'weatherVerifyC', 'existRelSC', 'objThisChoose', 'typeChoose', 'company', 'weatherVerify', 'sameGender', 'verifyAttrKC', 'categoryAttr', 'sameAnimalsC', 'chooseAttr', 'categoryThatChoose', 'categoryRelO', 'existThatOr', 'relS', 'categoryThat', 'existAttrOrC', 'categoryRelS', 'diffAnimalsC', 'diffAnimals', 'twoSameMaterial', 'verifyMaterialAnd', 'placeChoose', 'sameAnimals', 'materialVerifyC', 'existRelS', 'existThatNotC', 'stateChoose', 'dir', 'existOrC', 'relVerifyCop', 'relVerify', 'positionVerifyC', 'comparativeChoose', 'twoSameC', 'twoDifferent', 'existMaterialC', 'existAndC', 'twoCommon', 'diffGender', 'locationVerifyC', 'sameGenderC', 'positionVerify', 'material', 'locationChoose', 'sameMaterialRelate', 'place', 'twoSameMaterialC', 'existMaterialNot', 'existAttr', 'relChooser', 'relVerifyCo', 'verifyAttr', 'how', 'existOr', 'verifyAttrs', 'verifyAttrsC', 'verifyAttrC', 'placeVerifyC', 'companyChoose', 'existC', 'existAttrNot', 'existMaterialNotC', 'categoryRelOChoose', 'twoDifferentC', 'existThatOrC', 'category', 'existThat', 'verifyAttrThis', 'twoSame', 'existThatNot', 'activity', 'relVerifyCr', 'verifyAttrCThis', 'state', 'existMaterial', 'exist', 'existAttrC', 'positionChoose', 'relO', 'directWhich', 'existRelSRC', 'existThatC', 'placeVerify', 'locationVerify', 'verifyAttrK'}

问题json:

{"02930152": {"semantic": [{"operation": "select", "dependencies": [], "argument": "sky (2486325)"}, {"operation": "verify color", "dependencies": [0], "argument": "dark"}], "entailed": ["02930160", "02930158", "02930159", "02930154", "02930155", "02930156", "02930153"], "equivalent": ["02930152"], "question": "Is the sky dark?", "imageId": "2354786", "isBalanced": true, "groups": {"global": null, "local": "06-sky_dark"}, "answer": "yes", "semanticStr": "select: sky (2486325)->verify color: dark [0]", "annotations": {"answer": {}, "question": {"2": "2486325"}, "fullAnswer": {"2": "2486325"}}, "types": {"detailed": "verifyAttr", "semantic": "attr", "structural": "verify"}, "fullAnswer": "Yes, the sky is dark."}, "07333408": {"semantic": [{"operation": "select", "dependencies": [], "argument": "wall (722332)"}, {"operation": "filter color", "dependencies": [0], "argument": "white"}, {"operation": "relate", "dependencies": [1], "argument": "_,on,s (722335)"}, {"operation": "query", "dependencies": [2], "argument": "name"}], "entailed": [], "equivalent": ["07333408"], "question": "What is on the white wall?", "imageId": "2375429", "isBalanced": true, "groups": {"global": "", "local": "14-wall_on,s"}, "answer": "pipe", "semanticStr": "select: wall (722332)->filter color: white [0]->relate: _,on,s (722335) [1]->query: name [2]", "annotations": {"answer": {"0": "722335"}, "question": {"4:6": "722332"}, "fullAnswer": {"1": "722335", "5": "722332"}}, "types": {"detailed": "relS", "semantic": "rel", "structural": "query"}, "fullAnswer": "The pipe is on the wall."}, "07333405": {"semantic": [{"operation": "select", "dependencies": [], "argument": "pipe (722335)"}, {"operation": "verify color", "dependencies": [0], "argument": "red"}], "entailed": ["07333406"], "equivalent": ["07333405"], "question": "Is that pipe red?", "imageId": "2375429", "isBalanced": true, "groups": {"global": null, "local": "06-pipe_red"}, "answer": "no", "semanticStr": "select: pipe (722335)->verify color: red [0]", "annotations": {"answer": {}, "question": {"2": "722335"}, "fullAnswer": {"2": "722335"}}, "types": {"detailed": "verifyAttrC", "semantic": "attr", "structural": "verify"}, "fullAnswer": "No, the pipe is white."}, "15736264": {"semantic": [{"operation": "select", "dependencies": [], "argument": "clock (746851)"}, {"operation": "filter height", "dependencies": [0], "argument": "tall"}, {"operation": "choose size", "dependencies": [1], "argument": "large|small"}], "entailed": ["15736259", "15736258", "15736267", "15736253", "15736252", "15736251", "15736257", "15736256", "15736255", "15736254", "15736291", "15736249"], "equivalent": ["15736264"], "question": "Is the tall clock small or large?", "imageId": "2368326", "isBalanced": true, "groups": {"global": "size", "local": "10c-clock_size"}, "answer": "large", "semanticStr": "select: clock (746851)->filter height: tall [0]->choose size: large|small [1]", "annotations": {"answer": {}, "question": {"2:4": "746851"}, "fullAnswer": {"1": "746851"}}, "types": {"detailed": "chooseAttr", "semantic": "attr", "structural": "choose"}, "fullAnswer": "The clock is large."}

---

【题外话】

能否合成整张大图？ 

关于官网提交 leaderboard可以选择代码形式提交，比按钮体验好多了！按钮基本灰面提交状态不变！！！