Recent work has shown that LSTMs trained on a generic language
modeling objective capture syntax-sensitive generalizations such as
long-distance number agreement. We have however no mechanistic
understanding of how they accomplish this remarkable feat. Some have
conjectured it depends on heuristics that do not truly take
hierarchical structure into account. We present here a detailed
study of the inner mechanics of number tracking in LSTMs at the
single neuron level. We discover that long-distance number
information is largely managed by two “number units”.
Importantly, the behaviour of these units is partially controlled by
other units independently shown to track syntactic structure. We
conclude that LSTMs are, to some extent, implementing genuinely
syntactic processing mechanisms, paving the way to a more general
understanding of grammatical encoding in LSTMs.