今日arXiv精选 | 12篇EMNLP 2021最新论文

 关于 #今日arXiv精选 

这是「AI 学术前沿」旗下的一档栏目，编辑将每日从arXiv中精选高质量论文，推送给读者。

You should evaluate your language model on marginal likelihood overtokenisations

Comment: accepted at EMNLP 2021

Link: http://arxiv.org/abs/2109.02550

Abstract

Neural language models typically tokenise input text into sub-word units toachieve an open vocabulary. The standard approach is to use a single canonicaltokenisation at both train and test time. We suggest that this approach isunsatisfactory and may bottleneck our evaluation of language model performance.Using only the one-best tokenisation ignores tokeniser uncertainty overalternative tokenisations, which may hurt model out-of-domain performance.  In this paper, we argue that instead, language models should be evaluated ontheir marginal likelihood over tokenisations. We compare different estimatorsfor the marginal likelihood based on sampling, and show that it is feasible toestimate the marginal likelihood with a manageable number of samples. We thenevaluate pretrained English and German language models on both theone-best-tokenisation and marginal perplexities, and show that the marginalperplexity can be significantly better than the one best, especially onout-of-domain data. We link this difference in perplexity to the tokeniseruncertainty as measured by tokeniser entropy. We discuss some implications ofour results for language model training and evaluation, particularly withregard to tokenisation robustness.

Efficient Mind-Map Generation via Sequence-to-Graph and Reinforced Graph Refinement

Comment: Accepted by EMNLP 2021 main conference

Link: http://arxiv.org/abs/2109.02457

Abstract

A mind-map is a diagram that represents the central concept and key ideas ina hierarchical way. Converting plain text into a mind-map will reveal its keysemantic structure and be easier to understand. Given a document, the existingautomatic mind-map generation method extracts the relationships of everysentence pair to generate the directed semantic graph for this document. Thecomputation complexity increases exponentially with the length of the document.Moreover, it is difficult to capture the overall semantics. To deal with theabove challenges, we propose an efficient mind-map generation network thatconverts a document into a graph via sequence-to-graph. To guarantee ameaningful mind-map, we design a graph refinement module to adjust the relationgraph in a reinforcement learning manner. Extensive experimental resultsdemonstrate that the proposed approach is more effective and efficient than theexisting methods. The inference time is reduced by thousands of times comparedwith the existing methods. The case studies verify that the generated mind-mapsbetter reveal the underlying semantic structures of the document.

Eliminating Sentiment Bias for Aspect-Level Sentiment Classification with Unsupervised Opinion Extraction

Comment: 11 pages, Findings of EMNLP'2021, 7th-11th November 2021

Link: http://arxiv.org/abs/2109.02403

Abstract

Aspect-level sentiment classification (ALSC) aims at identifying thesentiment polarity of a specified aspect in a sentence. ALSC is a practicalsetting in aspect-based sentiment analysis due to no opinion term labelingneeded, but it fails to interpret why a sentiment polarity is derived for theaspect. To address this problem, recent works fine-tune pre-trained Transformerencoders for ALSC to extract an aspect-centric dependency tree that can locatethe opinion words. However, the induced opinion words only provide an intuitivecue far below human-level interpretability. Besides, the pre-trained encodertends to internalize an aspect's intrinsic sentiment, causing sentiment biasand thus affecting model performance. In this paper, we propose a span-basedanti-bias aspect representation learning framework. It first eliminates thesentiment bias in the aspect embedding by adversarial learning against aspects'prior sentiment. Then, it aligns the distilled opinion candidates with theaspect by span-based dependency modeling to highlight the interpretable opinionterms. Our method achieves new state-of-the-art performance on five benchmarks,with the capability of unsupervised opinion extraction.

Vision Guided Generative Pre-trained Language Models for Multimodal Abstractive Summarization

Comment: Long Paper Accepted in EMNLP 2021

Link: http://arxiv.org/abs/2109.02401

Abstract

Multimodal abstractive summarization (MAS) models that summarize videos(vision modality) and their corresponding transcripts (text modality) are ableto extract the essential information from massive multimodal data on theInternet. Recently, large-scale generative pre-trained language models (GPLMs)have been shown to be effective in text generation tasks. However, existing MASmodels cannot leverage GPLMs' powerful generation ability. To fill thisresearch gap, we aim to study two research questions: 1) how to inject visualinformation into GPLMs without hurting their generation ability; and 2) whereis the optimal place in GPLMs to inject the visual information? In this paper,we present a simple yet effective method to construct vision guided (VG) GPLMsfor the MAS task using attention-based add-on layers to incorporate visualinformation while maintaining their original text generation ability. Resultsshow that our best model significantly surpasses the prior state-of-the-artmodel by 5.7 ROUGE-1, 5.3 ROUGE-2, and 5.1 ROUGE-L scores on the How2 dataset,and our visual guidance method contributes 83.6% of the overall improvement.Furthermore, we conduct thorough ablation studies to analyze the effectivenessof various modality fusion methods and fusion locations.

PermuteFormer: Efficient Relative Position Encoding for Long Sequences

Comment: Accepted by EMNLP 2021

Link: http://arxiv.org/abs/2109.02377

Abstract

A recent variation of Transformer, Performer, scales Transformer to longersequences with a linear attention mechanism. However, it is not compatible withrelative position encoding, which has advantages over absolute positionencoding. In this paper, we discuss possible ways to add relative positionencoding to Performer. Based on the analysis, we propose PermuteFormer, aPerformer-based model with relative position encoding that scales linearly onlong sequences. PermuteFormer applies position-dependent transformation onqueries and keys to encode positional information into the attention module.This transformation is carefully crafted so that the final output ofself-attention is not affected by absolute positions of tokens. PermuteFormerintroduces negligible computational overhead by design that it runs as fast asPerformer. We evaluate PermuteFormer on Long-Range Arena, a dataset for longsequences, as well as WikiText-103, a language modeling dataset. Theexperiments show that PermuteFormer uniformly improves the performance ofPerformer with almost no computational overhead and outperforms vanillaTransformer on most of the tasks.

From Alignment to Assignment: Frustratingly Simple Unsupervised Entity Alignment

Comment: 11 pages; Accepted by EMNLP2021 (Full)

Link: http://arxiv.org/abs/2109.02363

Abstract

Cross-lingual entity alignment (EA) aims to find the equivalent entitiesbetween crosslingual KGs, which is a crucial step for integrating KGs.Recently, many GNN-based EA methods are proposed and show decent performanceimprovements on several public datasets. Meanwhile, existing GNN-based EAmethods inevitably inherit poor interpretability and low efficiency from neuralnetworks. Motivated by the isomorphic assumption of GNNbased methods, wesuccessfully transform the cross-lingual EA problem into the assignmentproblem. Based on this finding, we propose a frustratingly Simple but EffectiveUnsupervised entity alignment method (SEU) without neural networks. Extensiveexperiments show that our proposed unsupervised method even beats advancedsupervised methods across all public datasets and has high efficiency,interpretability, and stability.

Enhancing Visual Dialog Questioner with Entity-based Strategy Learning and Augmented Guesser

Comment: Accepted by Findings of EMNLP 2021. Code is available at:  https://github.com/zd11024/Entity_Questioner

Link: http://arxiv.org/abs/2109.02297

Abstract

Considering the importance of building a good Visual Dialog (VD) Questioner,many researchers study the topic under a Q-Bot-A-Bot image-guessing gamesetting, where the Questioner needs to raise a series of questions to collectinformation of an undisclosed image. Despite progress has been made inSupervised Learning (SL) and Reinforcement Learning (RL), issues still exist.Firstly, previous methods do not provide explicit and effective guidance forQuestioner to generate visually related and informative questions. Secondly,the effect of RL is hampered by an incompetent component, i.e., the Guesser,who makes image predictions based on the generated dialogs and assigns rewardsaccordingly. To enhance VD Questioner: 1) we propose a Related entity enhancedQuestioner (ReeQ) that generates questions under the guidance of relatedentities and learns entity-based questioning strategy from human dialogs; 2) wepropose an Augmented Guesser (AugG) that is strong and is optimized for the VDsetting especially. Experimental results on the VisDial v1.0 dataset show thatour approach achieves state-of-theart performance on both image-guessing taskand question diversity. Human study further proves that our model generatesmore visually related, informative and coherent questions.

Uncertainty-Aware Balancing for Multilingual and Multi-Domain Neural Machine Translation Training

Comment: 15 pages, 4 figures, to appear at EMNLP 2021 main conference

Link: http://arxiv.org/abs/2109.02284

Abstract

Learning multilingual and multi-domain translation model is challenging asthe heterogeneous and imbalanced data make the model converge inconsistentlyover different corpora in real world. One common practice is to adjust theshare of each corpus in the training, so that the learning process is balancedand low-resource cases can benefit from the high resource ones. However,automatic balancing methods usually depend on the intra- and inter-datasetcharacteristics, which is usually agnostic or requires human priors. In thiswork, we propose an approach, MultiUAT, that dynamically adjusts the trainingdata usage based on the model's uncertainty on a small set of trusted cleandata for multi-corpus machine translation. We experiments with two classes ofuncertainty measures on multilingual (16 languages with 4 settings) andmulti-domain settings (4 for in-domain and 2 for out-of-domain onEnglish-German translation) and demonstrate our approach MultiUAT substantiallyoutperforms its baselines, including both static and dynamic strategies. Weanalyze the cross-domain transfer and show the deficiency of static andsimilarity based methods.

Sent2Span: Span Detection for PICO Extraction in the Biomedical Text without Span Annotations

Comment: Accepted at EMNLP 2021 Findings

Link: http://arxiv.org/abs/2109.02254

Abstract

The rapid growth in published clinical trials makes it difficult to maintainup-to-date systematic reviews, which requires finding all relevant trials. Thisleads to policy and practice decisions based on out-of-date, incomplete, andbiased subsets of available clinical evidence. Extracting and then normalisingPopulation, Intervention, Comparator, and Outcome (PICO) information fromclinical trial articles may be an effective way to automatically assign trialsto systematic reviews and avoid searching and screening - the two mosttime-consuming systematic review processes. We propose and test a novelapproach to PICO span detection. The major difference between our proposedmethod and previous approaches comes from detecting spans without needingannotated span data and using only crowdsourced sentence-level annotations.Experiments on two datasets show that PICO span detection results achieve muchhigher results for recall when compared to fully supervised methods with PICOsentence detection at least as good as human annotations. By removing thereliance on expert annotations for span detection, this work could be used inhuman-machine pipeline for turning low-quality crowdsourced, and sentence-levelPICO annotations into structured information that can be used to quickly assigntrials to relevant systematic reviews.

STaCK: Sentence Ordering with Temporal Commonsense Knowledge

Comment: Accepted as a full paper at EMNLP 2021

Link: http://arxiv.org/abs/2109.02247

Abstract

Sentence order prediction is the task of finding the correct order ofsentences in a randomly ordered document. Correctly ordering the sentencesrequires an understanding of coherence with respect to the chronologicalsequence of events described in the text. Document-level contextualunderstanding and commonsense knowledge centered around these events are oftenessential in uncovering this coherence and predicting the exact chronologicalorder. In this paper, we introduce STaCK -- a framework based on graph neuralnetworks and temporal commonsense knowledge to model global information andpredict the relative order of sentences. Our graph network accumulates temporalevidence using knowledge of `past' and `future' and formulates sentenceordering as a constrained edge classification problem. We report results onfive different datasets, and empirically show that the proposed method isnaturally suitable for order prediction. The implementation of this work ispublicly available at: https://github.com/declare-lab/sentence-ordering.

BERT might be Overkill: A Tiny but Effective Biomedical Entity Linker based on Residual Convolutional Neural Networks

Comment: Accepted to EMNLP 2021 (Findings)

Link: http://arxiv.org/abs/2109.02237

Abstract

Biomedical entity linking is the task of linking entity mentions in abiomedical document to referent entities in a knowledge base. Recently, manyBERT-based models have been introduced for the task. While these models haveachieved competitive results on many datasets, they are computationallyexpensive and contain about 110M parameters. Little is known about the factorscontributing to their impressive performance and whether theover-parameterization is needed. In this work, we shed some light on the innerworking mechanisms of these large BERT-based models. Through a set of probingexperiments, we have found that the entity linking performance only changesslightly when the input word order is shuffled or when the attention scope islimited to a fixed window size. From these observations, we propose anefficient convolutional neural network with residual connections for biomedicalentity linking. Because of the sparse connectivity and weight sharingproperties, our model has a small number of parameters and is highly efficient.On five public datasets, our model achieves comparable or even better linkingaccuracy than the state-of-the-art BERT-based models while having about 60times fewer parameters.

The Devil is in the Detail: Simple Tricks Improve Systematic Generalization of Transformers

Comment: Accepted to EMNLP 2021

Link: http://arxiv.org/abs/2108.12284

Abstract

Recently, many datasets have been proposed to test the systematicgeneralization ability of neural networks. The companion baseline Transformers,typically trained with default hyper-parameters from standard tasks, are shownto fail dramatically. Here we demonstrate that by revisiting modelconfigurations as basic as scaling of embeddings, early stopping, relativepositional embedding, and Universal Transformer variants, we can drasticallyimprove the performance of Transformers on systematic generalization. We reportimprovements on five popular datasets: SCAN, CFQ, PCFG, COGS, and Mathematicsdataset. Our models improve accuracy from 50% to 85% on the PCFG productivitysplit, and from 35% to 81% on COGS. On SCAN, relative positional embeddinglargely mitigates the EOS decision problem (Newman et al., 2020), yielding 100%accuracy on the length split with a cutoff at 26. Importantly, performancedifferences between these models are typically invisible on the IID data split.This calls for proper generalization validation sets for developing neuralnetworks that generalize systematically. We publicly release the code toreproduce our results.

·