Various methods of measuring unit selectivity have been developed with the aim of better understanding how neural networks work.   But the different measures provide divergent estimates of selectivity, and this has led to different conclusions regarding the conditions in which selective object representations are learned and the functional relevance of these representations. In an attempt to better characterize object selectivity, we undertake a comparison of various selectivity measures on a large set of units in AlexNet, including localist selectivity, precision, class-conditional mean activity selectivity (CCMAS), network dissection, the human interpretation of activation maximization (AM) images, and standard signal-detection measures.   We find that the different measures provide different estimates of object selectivity, with precision and CCMAS measures providing misleadingly high estimates. Indeed, the most selective units had a poor hit-rate or a high false-alarm rate (or both) in object classification, making them poor object detectors.   We fail to find any units that are even remotely as selective as the 'grandmother cell' units reported in recurrent neural networks. In order to generalize these results, we compared selectivity measures on a few units in VGG-16 and GoogLeNet trained on the ImageNet or Places-365 datasets that have been described as 'object detectors'. Again, we find poor hit-rates and high false-alarm rates for object classification. There have been recent attempts to understand how neural networks (NNs) work by analyzing hidden units one-at-a-time using various measures such as localist selectivity (Bowers et al., 2014) , class-conditional mean activity selectivity (CCMAS) (Morcos et al., 2018) , precision (Zhou et al., 2015) , network dissection (Zhou et al., 2018a) , and activation maximization (AM) (Erhan et al., 2009) . These measures are all taken to provide evidence that some units respond highly selectively to categories of objects under some conditions. Not only are these findings surprising given the widespread assumption that NNs only learn highly distributed and entangled representations, they raise a host of questions, including the functional importance of these selective representations (Zhou et al., 2018b) , the conditions in which they are learned (e.g., Morcos et al., 2018) , and the relation between these representations and the selective neurons observed in cortex (Bowers, 2009 ). To answer these question, it is necessary to have a better understanding of what these metrics actually measure, and how they relate to one another. Accordingly, we directly compare these measures of selectivity on the same set of units as well as adopt standard signal-detection measures in an attempt to provide better measures of single-unit selectivity to object category. In addition, to provide a more intuitive assessment of selectivity, we report jitterplots for a few of the most selective units that visually display how the unit responds to the different image categories. We focus on AlexNet (Krizhevsky et al., 2012 ) trained on ImageNet (Deng et al., 2009 ) because many authors have studied the selectivity of single hidden units in this model using a range of quantitative (Zhou et al., 2018a; and qualitative (Nguyen et al., 2017; Yosinski et al., 2015; Simonyan et al., 2013) methods. But we also compare different selectivity measures on specific units in VGG-16 (Simonyan and Zisserman, 2014) and GoogLeNet (Szegedy et al., 2015) trained on the the ImageNet and Places-365 datasets that were characterized by Zhou et al. (2018a) as "object detectors" based on their Network Dissection method (Zhou et al., 2018a) . Our main findings are: 1. The precision and CCMAS measures are misleading with near-maximum selectivity scores associated with units that strongly respond to many different image categories. By contrast, the signal-detection measures more closely capture the level of selectivity displayed in the jitterplots (Sec. 3.1). 2. Units with interpretable AM images do not correspond to highly selective representations (Sec. 3.2). 3. The Network Dissection method also provides a misleading measure for "object detectors" (Sec. 3.3). In one line of research, Bowers et al. (2014; assessed the selectivity of single hidden units in recurrent neural networks (RNNs) designed to model human short-term memory. They reported many 'localist' or 'grandmother cell' units that were 100% selective for specific letters or words, where all members of the selective category were more active than and disjoint from all non-members, as can be shown in jitterplots (Berkeley et al., 1995 ) (see Fig. 1 for a unit selective to the letter 'j'). The authors argued that the network learned these representations in order to co-activate multiple letters or words at the same time in short-term memory without producing ambiguous blends of overlapping distributed patterns (the so-called 'superposition catastrophe'). Consistent with this hypothesis, localist units did not emerge when the model was trained on letters or words one-at-a-time (Bowers et al., 2014 ) (see Fig. 1 for an example of a non-selective unit). In parallel, researchers have reported selective units in the hidden layers of various CNNs trained to classify images into one of multiple categories (Zhou et al., 2015; Morcos et al., 2018; Zeiler and Fergus, 2014; Erhan et al., 2009) , for a review see Bowers (2017) . For example, Zhou et al. (2015) assessed the selectivity of units in the pool5 layer of two CNNs trained to classify images into 1000 objects and 205 scene categories, respectively. They reported many highly selective units that they characterized as 'object detectors' in both networks. Similarly, Morcos et al. (2018) reported that CNNs trained on CIFAR-10 and ImageNet learned many highly selective hidden units, with CCMAS scores approaching the maximum of 1.0. These later findings appear to be inconsistent with Bowers et al. (2016) who failed to observe selective representations in fully connected NNs trained on stimuli one-at-a-time (see Fig. 1 ), but the measures of selectivity that have been applied across studies are different, and accordingly, it is difficult to directly compare results. A better understanding of the relation between selectivity measures is vital given that different measures are frequently used to address similar issues. For example, both the human interpretability of generated images (Le, 2013) and localist selectivity (Bowers et al., 2014) have been used to make claims about 'grandmother cells', but it is not clear whether they provide similar insights into unit selectivity. Similarly, based on their precision metric, Zhou et al. (2015) claim that the object detectors learned in CNNs play an important role in identifying specific objects, whereas Morcos et al. (2018) challenge this conclusion based on their finding that units with high CCMAS measures were not especially important in the performance of their CNNs and concluded: "...it implies that methods for understanding neural networks based on analyzing highly selective single units, or finding optimal inputs for single units, such as activation maximization (Erhan et al., 2009 ) may be misleading". This makes a direct comparison between selectivity measures all the more important. In order to directly compare and have a better understanding of the different selectivity measures we assessed (1) localist, (2) precision, and (3) CCMAS selectivity of the conv5, fc6, and fc7 of AlexNet trained on ImageNet, and in addition, we employed a range of signal detection methods on these units, namely, (4) recall with 100% and 95% precision, (5) maximum informedness, (6) specificity at maximum informedness , and (7) recall (also called sensitivity) at maximum informedness, and false alarm rates at maximum informedness (described in Sec. 2). We also assessed the selectivity of a few units in VGG-16 and GoogLeNet models trained on the ImageNet and Places-365 dataset that were highly selective according to the Network Dissection method (Zhou et al., 2018a) . We show that the precision and CCMAS measures often provide misleadingly high estimates of object selectivity compared to other measures, and we do not find any units that can be reasonably described as 'object detectors' given that the most selective units show a low hit-rate or a high false-alarm rate (or both) when classifying images. At best, the most selective units in CNNs are sensitive to some unknown feature that is weakly associated with the class in question. (Bowers et al., 2016) . Top middle: jitterplot of a non-selective unit 160 found in an RNN trained on words one-at-a-time from (Bowers et al., 2016) . Top right: Activation maximization image of unit conv5 9 AlexNet that resembles a lighthouse (Nguyen et al., 2016) . Bottom: highest-activation images for a 'lamp' detector with 84% precision in the layer conv5 of AlexNet; from (Zhou et al., 2015) . In addition to these quantitative measures and jitterplots we assessed selectivity with a common qualitative measure, namely, human interpretation of images generated by a state-of-the-art activation maximization (AM) method (Nguyen et al., 2017) . AM images are generated to strongly activate individual units, and some of them are interpretable by humans (e.g., a generated image that looks like a lighthouse, see Fig. 1 ). For the first time, we systematically evaluated the interpretability of the AM images and compare these ratings with the selectivity measures for corresponding units. We show that the few hidden units with interpretable AM images are not highly selective. Our central finding is that different measures of single-unit selectivity for objects support very different conclusions when applied to the same units in AlexNet. In contrast with the precision (Zhou et al., 2015) and CCMAS (Morcos et al., 2018) measures that suggest some highly selective units for objects in layers conv5, fc6, and fc7, the recall with perfect precision and false alarm rates at maximum informedness show low levels of selectivity. Indeed, the most selective units have a poor hit-rate or a high false-alarm rate (or both) for identifying an object class. The same outcome was observed with units in VGG-16 and GoogLeNet trained on either ImageNet or the Places-365 dataset. Not only do the different measures provide very different assessments of selectivity, the precision, CCMAS, and Network Dissection measures provide highly misleading estimates of selectivity that have led to mistaken conclusions. For example, unit fc6 1199 in AlexNet trained on ImageNet is considered an Monarch Butterfly detector according to Zhou et al. (2015) with a precision score of 98% (and a CCMAS score of .93). But the jitterplot in Fig. 3 and signal detection scores (e.g., high false alarm rate at maximum informedness) show this is a mischaracterisation of this unit. In the same way, the Network Dissection method identified many object detectors in VGG-16 and GoogLeNet CNNs, but the jitterplots in Fig. 5 (and precision scores) show that this conclusion is unjustified. For additional problems with the CCMAS score see Figure 5 in Appendix C. Similarly, the images generated by Activation Maximization also provided a misleading estimate of selectivity given that interpretable images were associated with very low selectivity scores. This has led to confusions that have delayed theoretical progress. For example, describing single units in CNNs as "object detectors" in response to high precision measures (Zhou et al.) suggests similar types of representations are learned in CNNs and RNNs. Indeed, we are not aware of anyone in the machine learning community who has even considered the hypothesis that selectivity is reduced in CNNs compared RNNs. Our findings highlight the contrasting results. What should be made of the finding that localist representations are sometimes learned in RNNs (units with perfect specificity and recall), but not in AlexNet and related CNNs? The failure to observe localist units in the hidden layers of these CNNs is consistent with Bowers et al. (2014) 's claim that these units emerge in order to support the co-activation of multiple items at the same time in short-term memory. That is, localist representations may be the solution to the superposition catastrophe, and these CNNs only have to identify one image at a time. The pressure to learn highly selective representations in response to the superposition constraint may help explain the reports of highly selective neurons in cortex given that the cortex needs to co-activate multiple items at the same time in order to support short-term memory (Bowers et al., 2016) . Note, the RNNs that learned localist units were very small in scale compared to CNNs we have studied here, and accordingly, it is possible that the contrasting results reflect the size of the networks rather than the superposition catastrophe per se. Relevant to this issue a number of authors have reported the existence of selective units in larger RNNs with long-short term memory (LSTM) units (Karpathy et al., 2016; Radford et al., 2017; Lakretz et al., 2019; Na et al., 2019) . Indeed, Lakretz et al. (2019) use the term 'grandmother cell' to describe the units they observed. It will be interesting to apply our measures of selectivity to these larger RNNs and see whether these units are indeed 'grandmother units'. It should also be noted that there are recent reports of impressively selective representations in Generative Adversarial Networks (Bau et al., 2019) and Variational Autoencoders (Burgess et al., 2018) where the superposition catastrophe is not an issue. Again, it will be interesting to assess the selectivity of these units according to signal detection measures in order to see whether there are additional computational pressures to learn highly selective or even grandmother cells. We will be exploring these issues in future work.
Looking for object detectors using many different selectivity measures; CNNs are slightly selective , but not enough to be termed object detectors.
{"src": "scitldr-AIC_train", "paper_id": "Skltqh4KvB"}