Given the fast development of analysis techniques for NLP and speech
 processing systems, few systematic studies have been conducted to
 compare the strengths and weaknesses of each method.  As a step in
 this direction we study the case of representations of phonology in
 neural network models of spoken language. We use two commonly applied
 analytical techniques, diagnostic classifiers and representational
 similarity analysis, to quantify to what extent neural activation
 patterns encode phonemes and phoneme sequences. We manipulate two
 factors that can affect the outcome of analysis. First, we investigate
 the role of learning by comparing neural activations extracted from
 trained versus randomly-initialized models. Second, we examine the
 temporal scope of the activations by probing both local activations
 corresponding to a few milliseconds of the speech signal, and global
 activations pooled over the whole utterance. We conclude that
 reporting analysis results with randomly initialized models is
 crucial, and that global-scope methods tend to yield more consistent
 and interpretable results and we recommend their use as a complement
 to local-scope diagnostic methods. As end-to-end architectures based on neural networks became the tool of choice for processing speech and language, there has been increased interest in techniques for analyzing and interpreting the representations emerging in these models. A large array of analytical techniques have been proposed and applied to diverse tasks and architectures . Given the fast development of analysis techniques for NLP and speech processing systems, relatively few systematic studies have been conducted to compare the strengths and weaknesses of each methodology and to assess the reliability and explanatory power of their outcomes in controlled settings. This paper reports a step in this direction: as a case study, we examine the representation of phonology in neural network models of spoken language. We choose three different models that process speech signal as input, and analyze their learned neural representations. We use two commonly applied analytical techniques: (i) diagnostic models and (ii) representational similarity analysis to quantify to what extent neural activation patterns encode phonemes and phoneme sequences. In our experiments, we manipulate two important factors that can affect the outcome of analysis. One pitfall not always successfully avoided in work on neural representation analysis is the role of learning. Previous work has shown that sometimes non-trivial representations can be found in the activation patterns of randomly initialized, untrained neural networks (Zhang and Bowman, 2018; ). Here we investigate the representations of phonology in neural models of spoken language in light of this fact, as extant studies have not properly controlled for role of learning in these representations. The second manipulated factor in our experiments is the scope of the extracted neural activations. We control for the temporal scope, probing both local activations corresponding to a few milliseconds of the speech signal, as well as global activations pooled over the whole utterance. When applied to global-scope representations, both the methods detect a robust difference between the trained and randomly initialized target models. However we find that in our setting, RSA applied to local representations shows low correlations between phonemes and neural activation patterns for both trained and randomly initialized target models, and for one of the target models the local diagnostic classifier only shows a minor difference in the decodability of phonemes from randomly initialized versus trained network. This highlights the importance of reporting analy-sis results with randomly initialized models as a baseline. We carried out a systematic study of analysis methods for neural models of spoken language and offered some suggestions on best practices in this endeavor. Nevertheless our work is only a first step, and several limitations remain. The main challenge is that it is often difficult to completely control for the many factors of variation present in the target models, due to the fact that a particular objective function, or even a dataset, may require relatively important architectural modifications. In future we plan to sample target models with a larger number of plausible combinations of factors. Likewise, a choice of an analytical method may often entail changes in other aspects of the analysis: for example unlike a global diagnostic classifier, global RSA captures the sequential order of phonemes. In future we hope to further disentangle these differences.
We study representations of phonology in neural network models of spoken language with several variants of analytical techniques.
{"src": "scitldr-AIC_train", "paper_id": "HZEg3S-Gppr"}