Pre-training Transformer from large-scale raw texts and fine-tuning on the desired task have achieved state-of-the-art results on diverse NLP tasks . However , it is unclear what the learned attention captures . The attention computed by attention heads seems not to match human intuitions about hierarchical structures . This paper proposes Tree Transformer , which adds an extra constraint to attention heads of the bidirectional Transformer encoder in order to encourage the attention heads to follow tree structures . The tree structures can be automatically induced from raw texts by our proposed " Constituent Attention " module , which is simply implemented by self-attention between two adjacent words . With the same training procedure identical to BERT , the experiments demonstrate the effectiveness of Tree Transformer in terms of inducing tree structures , better language modeling , and further learning more explainable attention scores 1 . Human languages exhibit a rich hierarchical structure which is currently not exploited nor mirrored by the self-attention mechanism that is the core of the now popular Transformer architecture . Prior work that integrated hierarchical structure into neural networks either used recursive neural networks ( Tree-RNNs ) ( C. Goller and A.Kuchler , 1996 ; Socher et al . , 2011 ; Tai et al . , 2015 ) or simultaneously generated a syntax tree and language in RNN ( Dyer et al . , 2016 ) , which have shown beneficial for many downstream tasks ( Aharoni and Goldberg , 2017 ; Eriguchi et al . , 2017 ; Strubell et al . , 2018 ; Zaremoodi and Haffari , 2018 ) . Considering the requirement of the annotated parse trees and the 1 The source code is publicly available at https:// github.com / yaushian / Tree-Transformer . costly annotation effort , most prior work relied on the supervised syntactic parser . However , a supervised parser may be unavailable when the language is low-resourced or the target data has different distribution from the source domain . Therefore , the task of learning latent tree structures without human-annotated data , called grammar induction ( Carroll and Charniak , 1992 ; Klein and Manning , 2002 ; Smith and Eisner , 2005 ) , has become an important problem and attractd more attention from researchers recently . Prior work mainly focused on inducing tree structures from recurrent neural networks ( Shen et al . , 2018a , b ) or recursive neural networks ( Yogatama et al . , 2017 ; Drozdov et al . , 2019 ) , while integrating tree structures into Transformer remains an unexplored direction . Pre-training Transformer from large-scale raw texts successfully learns high-quality language representations . By further fine-tuning pre-trained Transformer on desired tasks , wide range of NLP tasks obtain the state-of-the-art results ( Radford et al . , 2019 ; Devlin et al . , 2018 ; Dong et al . , 2019 ) . However , what pre-trained Transformer self-attention heads capture remains unknown . Although an attention can be easily explained by observing how words attend to each other , only some distinct patterns such as attending previous words or named entities can be found informative ( Vig , 2019 ) . The attention matrices do not match our intuitions about hierarchical structures . In order to make the attention learned by Transformer more interpretable and allow Transformer to comprehend language hierarchically , we propose Tree Transformer , which integrates tree structures into bidirectional Transformer encoder . At each layer , words are constrained to attend to other words in the same constituents . This constraint has been proven to be effective in prior work ( Wu et al . , 2018 ) . Different from the prior work that required a supervised parser , in Tree Transformer , the constituency tree structures is automatically induced from raw texts by our proposed " Constituent Attention " module , which is simply implemented by self-attention . Motivated by Tree-RNNs , which compose each phrase and the sentence representation from its constituent sub-phrases , Tree Transformer gradually attaches several smaller constituents into larger ones from lower layers to higher layers . The contributions of this paper are 3-fold : • Our proposed Tree Transformer is easy to implement , which simply inserts an additional " Constituent Attention " module implemented by self-attention to the original Transformer encoder , and achieves good performance on the unsupervised parsing task . • As the induced tree structures guide words to compose the meaning of longer phrases hierarchically , Tree Transformer improves the perplexity on masked language modeling compared to the original Transformer . • The behavior of attention heads learned by Tree Transformer expresses better interpretability , because they are constrained to follow the induced tree structures . By visualizing the self-attention matrices , our model provides the information that better matchs the human intuition about hierarchical structures than the original Transformer . This paper proposes Tree Transformer , a first attempt of integrating tree structures into Transformer by constraining the attention heads to attend within constituents . The tree structures are automatically induced from the raw texts by our proposed Constituent Attention module , which attaches the constituents to each other by selfattention . The performance on unsupervised parsing demonstrates the effectiveness of our model in terms of inducing tree structures coherent to human expert annotations . We believe that incorporating tree structures into Transformer is an important and worth exploring direction , because it allows Transformer to learn more interpretable attention heads and achieve better language modeling . The interpretable attention can better explain how the model processes the natural language and guide the future improvement .
This is unclear what attention heads in pre-training transformers models capture and it seems not to match human intuitions about hierarchical structures.
They propose to add an extra constraint to attention heads of the bidirectional Transformer encoder and a module induces tree structures from raw texts.
The proposed model achieves better unsupervised tree structure induction, language modelling, and more explainable attention scores which are coherent to human expert annotations.