How do we know what is captured by specific features in the deep layers of a CNN? While it is trivial to visualize the linear filters in the first layer, its less obvious how we can do so for features in later layers. One method is to pass a large number of images through a CNN, and keep track of the patches which trigger high activation values for a given filter, then inspect them and qualitatively identify their shared attributes [1]}, [2]}. Another method, which has had a much greater impact on the field of style transfer, is to produce the 'stimuli' which maximizes the activation of a certain feature by using gradient descent to directly optimize the pixels of an input image [3]}, [4]}, [1]}, [6]}. This same technique can be used to 'invert' the feature representation extracted by a CNN from a particular image, allowing some qualitative intuition about which image details are maintained or discarded at different depths of the CNN [7]}.
