<jupyter_start><jupyter_text>Activation Maximization on VGGNet  Dense Layer Visualizations To visualize activation over final dense layer outputs, we need to switch the `softmax` activation out for `linear` since gradient of output node will depend on all the other node activations. Doing this in keras is tricky, so we provide `utils.apply_modifications` to modify network parameters and rebuild the graph.If this swapping is not done, the results might be suboptimal. We will start by swapping out 'softmax' for 'linear' and compare what happens if we dont do this at the end.<jupyter_code>from keras.applications import VGG19 as VGG16
from vis.utils import utils
from keras import activations

# Build the VGG16 network with ImageNet weights
model = VGG16(weights='imagenet', include_top=True)

# Utility to search for layer index by name. 
# Alternatively we can specify this as -1 since it corresponds to the last layer.
layer_idx = utils.find_layer_idx(model, 'predictions')

# Swap softmax with linear
model.layers[layer_idx].activation = activations.linear
model = utils.apply_modifications(model)<jupyter_output>Using TensorFlow backend.
/home/yimengzh/miniconda2/envs/tf15/lib/python3.6/site-packages/keras/models.py:255: UserWarning: No training configuration found in save file: the model was *not* compiled. Compile it manually.
  warnings.warn('No training configuration found in save file: '<jupyter_text>Visualizing a specific output category Lets try visualizing a specific output category. We will pick `ouzel` which corresponds to imagenet category `20`<jupyter_code>from vis.visualization import visualize_activation

from matplotlib import pyplot as plt
%matplotlib inline
plt.rcParams['figure.figsize'] = (18, 6)<jupyter_output><empty_output><jupyter_text>Hmm, that sort of looks like a bird. Lets see if we can get better results with more iterations. This time, lets see the verbose output during the optimization process. We can see that the loss appears to be converging. So more iterations definitely seem to give better output. One way to get crisper results is to use `Jitter` input_modifier. As the name suggests, `Jitter` moves pixels around in the image. Lets try this out.<jupyter_code>from vis.input_modifiers import Jitter<jupyter_output><empty_output><jupyter_text>Look at that! Not only has the conv net captured what it means to be an ouzel, but it also seems to encode for different orientations and scales, a further proof of rotational and scale invariance.  Lets try this for a bunch of other random categories. This will take a while. Go grab a nice cup of coffee and prepare to be amused :)<jupyter_code>import numpy as np
plt.rcParams['figure.figsize'] = (50, 50)<jupyter_output><empty_output><jupyter_text>Some of them make sense if you stare at ot for a while. There are ways of improving this. We will cover some ideas for this in the next section. You can come back here and try those out as an exercise.  Visualizing Conv filters In a CNN, each Conv layer has several learned *template matching* filters that maximize their output when a similar template pattern is found in the input image. First Conv layer is easy to interpret; simply visualize the weights as an image. To see what the Conv layer is doing, a simple option is to apply the filter over raw input pixels. Subsequent Conv filters operate over the outputs of previous Conv filters (which indicate the presence or absence of some templates), making them hard to interpret.One way of interpreting them is to generate an input image that maximizes the filter output. This allows us to generate an input that activates the filter.Lets start by visualizing the second conv layer of vggnet (named as 'block1_conv2'). Here is the VGG16 model for reference.<jupyter_code>model.summary()
from vis.visualization import get_num_filters<jupyter_output><empty_output><jupyter_text>They mostly seem to match for specific color and directional patterns. Lets try a bunch of other layers.We will randomly visualize 10 filters within various layers.<jupyter_code>from tang_jcompneuro import dir_dictionary
import os.path
from skimage.io import imsave
selected_indices = []
np.random.seed(42)  # try some good seed

cherrypick_dict = {
    'block2_conv1': (0,4,5,7,9),
    'block3_conv1': range(10),
    'block4_conv1': (1,3,5,6,7)
}

for layer_name in ['block2_conv1', 'block3_conv1',
                   'block4_conv1',]:
    layer_idx = utils.find_layer_idx(model, layer_name)
    cherry_this = cherrypick_dict[layer_name]

    # Visualize all filters in this layer.
    filters = np.random.permutation(get_num_filters(model.layers[layer_idx]))[:10]
    selected_indices.append(filters)

    # Generate input image for each filter.
    vis_images = []
    for img_idx, idx in enumerate(filters):
        img = visualize_activation(model, layer_idx, filter_indices=idx,
                                   tv_weight=1,lp_norm_weight=1)
        assert img.shape == (224,224,3)
        img = img[112-25:112+25,112-25:112+25]

        # Utility to overlay text on image.
        # img = utils.draw_text(img, 'Filter {}'.format(idx))
        if img_idx in cherry_this:
            vis_images.append(img)

    # Generate stitched image palette with 5 cols so we get 2 rows.
    stitched = utils.stitch_images(vis_images, cols=10,margin=0)
    save_dir = os.path.join(dir_dictionary['plots'], 'main', 'vgg_vis')
    os.makedirs(save_dir, exist_ok=True)
    # save.
    imsave(os.path.join(save_dir, layer_name + '.png'), stitched)
    
    plt.close('all')
    plt.figure()
    plt.axis('off')
    plt.imshow(stitched)
    plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07126436781609195, 'fraction_numerical': 0.016622458001768346, 'mean_word_length': 4.534246575342466, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 24, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '106203', 'license': 'Apache-2.0', 'n_tokens_mistral': 1690, 'n_tokens_neox': 1600, 'n_words': 665, 'path': 'results_ipynb/plots/main/vgg_vis.ipynb', 'repo_name': 'leelabcnbc/tang_jcompneuro_revision', 'size': '5655'}