<jupyter_start><jupyter_text>Convolutional Layer---In this notebook, we visualize four filtered outputs (a.k.a. feature maps) of a convolutional layer.  Import the image<jupyter_code>import cv2
import matplotlib.pyplot as plt
%matplotlib inline

# TODO: Feel free to try out your own images here by changing img_path
# to a file path to another image on your computer!
img_path = 'images/udacity_sdc.png'

# load color image 
bgr_img = cv2.imread(img_path)
# convert to grayscale
gray_img = cv2.cvtColor(bgr_img, cv2.COLOR_BGR2GRAY)

# normalize, rescale entries to lie in [0,1]
gray_img = gray_img.astype("float32")/255 # <= same result if the operation is gray_img/255 in python 3
#gray_img = gray_img/255

# plot image
plt.imshow(gray_img, cmap='gray')
plt.show()<jupyter_output><empty_output><jupyter_text>Define and visualize the filters<jupyter_code>import numpy as np

## TODO: Feel free to modify the numbers here, to try out another filter!
filter_vals = np.array([[-1, -1, 1, 1], [-1, -1, 1, 1], [-1, -1, 1, 1], [-1, -1, 1, 1]])

print('Filter shape: ', filter_vals.shape)
print(filter_vals)
# Defining four different filters, 
# all of which are linear combinations of the `filter_vals` defined above

# define four filters
filter_1 = filter_vals
filter_2 = -filter_1
filter_3 = filter_1.T
filter_4 = -filter_3
filters = np.array([filter_1, filter_2, filter_3, filter_4])

# For an example, print out the values of filter 1
print('Filter 1: \n', filter_1)
print('Filter 2: \n', filter_2)
print('Filter 3: \n', filter_3)
print('Filter 4: \n', filter_4)
### do not modify the code below this line ###

# visualize all four filters
fig = plt.figure(figsize=(10, 5))
for i in range(4):
    ax = fig.add_subplot(1, 4, i+1, xticks=[], yticks=[])
    ax.imshow(filters[i], cmap='gray')
    ax.set_title('Filter %s' % str(i+1))
    width, height = filters[i].shape
    for x in range(width):
        for y in range(height):
            ax.annotate(str(filters[i][x][y]), xy=(y,x),
                        horizontalalignment='center',
                        verticalalignment='center',
                        color='white' if filters[i][x][y]<0 else 'black')<jupyter_output><empty_output><jupyter_text>Define a convolutional layer Initialize a single convolutional layer so that it contains all your created filters. Note that you are not training this network; you are initializing the weights in a convolutional layer so that you can visualize what happens after a forward pass through this network!<jupyter_code>import torch
import torch.nn as nn
import torch.nn.functional as F

    
# define a neural network with a single convolutional layer with four filters
class Net(nn.Module):
    
    def __init__(self, weight):
        super(Net, self).__init__()
        # initializes the weights of the convolutional layer to be the weights of the 4 defined filters
        k_height, k_width = weight.shape[2:] # 4 dimensions => Number_filters  x  1  x  (height x width)
        #                                                          shape[0]    shape[1] shape[2]  shape[3]
        # 1 input channel, i.e, it uses as input ONLY 1 image, with dim MxN (whatever M and N are)
        # 4 output channels, so 4 outputs. Each ouput is the result of one filter, so 4 filters. Each filter
        # of dim (k_height x k_width) => Remember rows x columns always, so rows => height, columns =>  width
        #                                rows x columns => height x width
        # So each of the 4 outputs is of dim (M - k_height + 1) x (N - k_width + 1) if stride = 1 
        self.conv = nn.Conv2d(1, 4, kernel_size=(k_height, k_width), bias=False)
        self.conv.weight = torch.nn.Parameter(weight)

    def forward(self, gray_img_tensor):
        # calculates the output of a convolutional layer
        # pre- and post-activation
        conv_result = self.conv(gray_img_tensor)
        activated_result = F.relu(conv_result)
        
        # returns both layers
        return conv_result, activated_result
    
# instantiate the model and set the weights
# fil_torch = torch.from_numpy(filters) <= 4x(4x4), [filter1, filter2, filter3, filter4] in tensor format for pytorch
# each filter fil_torch[i], with i = 0 to 3, has the form:
# [[a0,a1,a2,a3],[b0,b1,b2,b3],[c0,c1,c2,c3],[d0,d1,d2,d3]]
# fil_torch = [ [[....],[....],[....],[....]], [[....],[....],[....],[....]], [[....],[....],[....],[....]],[[....],[....],[....],[....]] ] 
# unsqueeze insert one dimension in dimension 1. dimension 1 means columns.
# so, from fil_torch = 4x4x4 to 4x1x4x4
# 4 x 4 x 4 => 4 x (_ x _) => list of 4 elements, each one of dimensions 4x4
# [ [[],[],[],[]], [[],[],[],[]], [[],[],[],[]], [[],[],[],[]] ] 
# 4 x 1 x 4 x 4 => 4 x 1 x (_ x _) => matrix of dim 4 x 1, each element is a matrix of dim 4 x 4
# [ [ [[....],[....],[....],[....]]  ],[],[],[] ]

fil_torch = torch.from_numpy(filters)
print("fil_torch:", fil_torch.shape)
print("fil_torch:\n", fil_torch)

weight = fil_torch.unsqueeze(1)
print("weight:",weight.shape)
print("weight:\n",weight)

weight = weight.type(torch.FloatTensor)
print("weight:",weight.shape)
print("weight:\n",weight)

model = Net(weight)

# print out the layer in the network
print(model)<jupyter_output>fil_torch: torch.Size([4, 4, 4])
fil_torch:
 tensor([[[-1, -1,  1,  1],
         [-1, -1,  1,  1],
         [-1, -1,  1,  1],
         [-1, -1,  1,  1]],

        [[ 1,  1, -1, -1],
         [ 1,  1, -1, -1],
         [ 1,  1, -1, -1],
         [ 1,  1, -1, -1]],

        [[-1, -1, -1, -1],
         [-1, -1, -1, -1],
         [ 1,  1,  1,  1],
         [ 1,  1,  1,  1]],

        [[ 1,  1,  1,  1],
         [ 1,  1,  1,  1],
         [-1, -1, -1, -1],
         [-1, -1, -1, -1]]])
weight: torch.Size([4, 1, 4, 4])
weight:
 tensor([[[[-1, -1,  1,  1],
          [-1, -1,  1,  1],
          [-1, -1,  1,  1],
          [-1, -1,  1,  1]]],


        [[[ 1,  1, -1, -1],
          [ 1,  1, -1, -1],
          [ 1,  1, -1, -1],
          [ 1,  1, -1, -1]]],


        [[[-1, -1, -1, -1],
          [-1, -1, -1, -1],
          [ 1,  1,  1,  1],
          [ 1,  1,  1,  1]]],


        [[[ 1,  1,  1,  1],
          [ 1,  1,  1,  1],
          [-1, -1, -1, -1],
          [-1, -1, -1, -1]]]])
weight: to[...]<jupyter_text>Visualize the output of each filterFirst, we'll define a helper function, `viz_layer` that takes in a specific layer and number of filters (optional argument), and displays the output of that layer once an image has been passed through.<jupyter_code># helper function for visualizing the output of a given layer
# default number of filters is 4
def viz_layer(layer, n_filters= 4):
    fig = plt.figure(figsize=(20, 20))
    
    for i in range(n_filters):
        ax = fig.add_subplot(1, n_filters, i+1, xticks=[], yticks=[])
        # grab layer outputs
        # layer is 1 x 4 x (row x col)
        print("layer[0,{}]                          : shape  is {} - type: {}". format(i, layer[0,i].shape, type(layer[0,i])))
        print("layer[0,{}].data                     : shape  is {} - type: {}". format(i, layer[0,i].data.shape, type(layer[0,i].data)))
        print("layer[0,{}].data.numpy()             : shape  is {} - type: {}". format(i, layer[0,i].data.numpy().shape, type(layer[0,i].data.numpy())))
        print("np.squeeze(layer[0,{}].data.numpy()) : shape  is {} - type: {}". format(i, np.squeeze(layer[0,i].data.numpy()).shape, type(np.squeeze(layer[0,i].data.numpy()))))
        
        ax.imshow(np.squeeze(layer[0,i].data.numpy()), cmap='gray')
        ax.set_title('Output %s' % str(i+1))<jupyter_output><empty_output><jupyter_text>Let's look at the output of a convolutional layer, before and after a ReLu activation function is applied.<jupyter_code># plot original image
plt.imshow(gray_img, cmap='gray')

# visualize all filters
fig = plt.figure(figsize=(12, 6))
fig.subplots_adjust(left=0, right=1.5, bottom=0.8, top=1, hspace=0.05, wspace=0.05)
for i in range(4):
    ax = fig.add_subplot(1, 4, i+1, xticks=[], yticks=[])
    ax.imshow(filters[i], cmap='gray')
    ax.set_title('Filter %s' % str(i+1))

    
# convert the image into an input Tensor
print("Gray Image dim:", gray_img.shape)
torch_img = torch.from_numpy(gray_img)
print("from numpy to torch:", torch_img.shape)
gray_img_tensor = torch_img.unsqueeze(0).unsqueeze(1)
print("Gray Image Tensor dim:", gray_img_tensor.shape)

# get the convolutional layer (pre and post activation)
conv_layer, activated_layer = model(gray_img_tensor)

# visualize the output of a conv layer
viz_layer(conv_layer)
# visualize the output of an activated conv layer
viz_layer(activated_layer)<jupyter_output>layer[0,0]                          : shape  is torch.Size([210, 317]) - type: <class 'torch.Tensor'>
layer[0,0].data                     : shape  is torch.Size([210, 317]) - type: <class 'torch.Tensor'>
layer[0,0].data.numpy()             : shape  is (210, 317) - type: <class 'numpy.ndarray'>
np.squeeze(layer[0,0].data.numpy()) : shape  is (210, 317) - type: <class 'numpy.ndarray'>
layer[0,1]                          : shape  is torch.Size([210, 317]) - type: <class 'torch.Tensor'>
layer[0,1].data                     : shape  is torch.Size([210, 317]) - type: <class 'torch.Tensor'>
layer[0,1].data.numpy()             : shape  is (210, 317) - type: <class 'numpy.ndarray'>
np.squeeze(layer[0,1].data.numpy()) : shape  is (210, 317) - type: <class 'numpy.ndarray'>
layer[0,2]                          : shape  is torch.Size([210, 317]) - type: <class 'torch.Tensor'>
layer[0,2].data                     : shape  is torch.Size([210, 317]) - type: <class 'torch.Tensor'>
layer[0,2].data.numpy() [...]
{'fraction_non_alphanumeric': 0.16936488169364883, 'fraction_numerical': 0.04140722291407223, 'mean_word_length': 2.725164282953228, 'pattern_counts': {'":': 0, '<': 32, '<?xml version=': 0, '>': 38, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '383189', 'license': 'MIT', 'n_tokens_mistral': 3582, 'n_tokens_neox': 3385, 'n_words': 1007, 'path': 'L06-07-Notebook-Visualizing_a_convolutional_layer.ipynb', 'repo_name': 'jfrascon/UDACITY-NANODEGREE-CVISION_AND_DLEARNING', 'size': '9636'}