Paper小计：Language Models as Knowledge Bases?

propose:

LAMA: 由一组知识源组成，每个知识源都由一组事实组成。

我们定义，一个预训练的语言模型知道一个事实（主语、关系、宾语），如（但丁出生在佛罗伦萨），如果它能成功预测MASK的对象，如 "但丁出生在 "这样的句子来表达这一事实。我们测试了各种类型的知识：存储在Wikidata中的实体之间的关系、常识性的概念网中的概念之间的关系，以及回答自然语言问题所需的知识SQuAD中的问题。在后一种情况下，我们手动将SQuAD问题的一个子集映射到cloze句子。

结论：

1.BERT模型(BERT-large)捕获了（准确的）关系型的知识，与用现成的关系提取器和基于甲骨文的实体提取器提取的知识库相当。                                                                                                                                            

2.事实知识可以从预训练的语言模型中恢复得很好，然而，对于某些关系（特别是
N-to-M关系）性能非常差。

3.BERT-large在恢复事实知识和语言模型方面一直优于其他语言模型在恢复事实和常识性知识方面一直优于其他语言模型，同时对查询的措辞更加稳健。

4.BERT-large在开放域QA方面取得了显著的结果，在@10时精度达到57.1%，而使用特定任务的监督关系提取系统构建的知识库为63.5%。

2.1 Unidirectional Language Models

2.2 Bidirectional “Language Models” 2

3 Related Work

我们的调查试图回答预训练的语言模型在多大程度上储存了事实性和常识性知识的程度。与传统关系提取方法所填充的符号知识库进行比较。

4 The LAMA Probe

We introduce the LAMA (LAnguage Model Analysis) probe to test the factual and commonsense
knowledge in language models：

It provides a setof knowledge sources which are composed of a corpus of facts. Facts are either subject-relationobject triples or question-answer pairs.

We evaluate each model based on how highly it ranks the ground truth token against every other word in a fifixed candidate vocabulary.

assumption： models which rank ground truth tokens high for these cloze state ments have more factual knowledge.

4.1 Knowledge Sources

we cover a variety of sources of factual and commonsense knowledge. For each source, we describe the origin of fact triples (or question answer pairs), how we transform them into cloze

templates, and to what extent aligned texts exist in Wikipedia that are known to express a partic ular fact. We use the latter information in super vised baselines that extract knowledge representa tions directly from the aligned text.

4.2 Models

4.3 Baselines

freq ：For a subject and relation pair......

re：For the relation-based knowledge source......

drqa：for open-domain question answering......

4.4 Metrics

We consider rank-based metrics and compute results per relation along with mean values across all relations. To account for multiple valid objects for a subject-relation pair ( i.e. , for N-Mrelations), we follow Bordes et al. ( 2013 ) and remove from the candidates when ranking at test time all other valid objects in the training data other than the one we test. We use the mean precision at k ( P@k ). For a given fact, this value is 1 if the object is ranked among the top k results, and 0 otherwise.

4.5 Considerations

Manually Defifined Templates

Single Token

Object Slots

Intersection of Vocabularies

5 Results                 

6 Discussion and Conclusion