Sequential models like the Long-Short Term Memory (LSTM) [1]} and Transformer [2]} have achieved state-of-the-art (SOTA) performance on a myriad of NLP tasks, such as language modeling [3]}, machine translation [4]}, summarization [5]} and document classification [6]}. Language is hierarchical in nature, where documents are composed of sentences, and in turn composed of words. These hierarchical structures are intricately cross-referencing, with words referencing others in different sentences or even paragraphs. For example, if we use the term 'aforementioned SOTA models' in this sentence, the reader needs to relate it to the terms 'LSTM' and 'Transformer' in the first sentence of this paragraph. In our paper, we ask this central question: Do these SOTA models exploit the ubiquitous relational hierarchies in NLP? Our experiments show they do not, and surprisingly fail to reason with multiple hierarchies when tested on systematic generalization of our proposed task. Transformer does no better than random chance with larger hierarchies.
