<jupyter_start><jupyter_text>When matrix multiplications in feed-forward neural networks are replaced with convolutions, we can save a lot of connections and therefore computation. These are applicable whenever you have a grid like structure in your input that you would like to exploit.  Convolution Convolution is an important operation in deep learning. You can think of convolution as an operation which takes a signal and a kernel as input and produces an output. The signal is convolved with the kernel (meaning slided over and multiplied element-wise) and an output is produced. This can be done on 1D signal or 2D images. One can think of convolution as a density function that captures the local wiggle around a signal. Let's say you have position of a spaceship, but it has some local noise attached to it. Ideally, you would want to some invariant property that is global in nature that can capture this local wiggleness. The learned kernel (sometimes also called weight function) can be convolved with every point in time to get an aggragated version of the signal.<jupyter_code>import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
plt.style.use('ggplot')
function = [10,50,60,10,20,40,30]
kernel = [0.33,0.33,0.33]
np.convolve(function, kernel)
plt.plot(function)
plt.ylabel('Function')
plt.xlabel('Time')
plt.plot(np.convolve(function, kernel))
plt.ylabel('Convoled with a kernel (0.3,0.3,0.3)')
plt.xlabel('Time')<jupyter_output><empty_output><jupyter_text>Convolution on ImagesImages have properties that are local in nature, but globally they show a lot of variance. We can learn certain filters, for example, here is a mean filter, that blurs the image. This operation saves a lot of computational resources because it is same operation we are doing at each part of the input image. The book actually digs a little more deeper into the computational efficiency of convolutions.<jupyter_code>from skimage.io import imread
import urllib
from skimage import img_as_float
i = imread(urllib.urlopen('https://upload.wikimedia.org/wikipedia/en/2/24/Lenna.png?1486842310500'), 
           flatten= True, mode= 'F')
i.shape
i = img_as_float(i)
plt.imshow(i, cmap='gray')
from scipy.signal import convolve2d
mean_filter = np.array([[0.03,0.03,0.03],[0.03,0.03,0.03],[0.03,0.03,0.03]])
plt.imshow(convolve2d(i,mean_filter), cmap='gray')<jupyter_output><empty_output><jupyter_text>Why convolutional networks?At a higher level, convolutional networks offer three specific advantages over fully-connected neural networks.* Sparse interactions* Parameter Sharing* Equivalent representations  Sparse interactionsInstead of connecting every input neuron to a hidden neuron in the hidden layer, a hidden layer in ConvNet is only connected to a local region in the previous layer. The logical reason behind is surprisingly simple, images have properties that are local in nature and it is rather a waste of computational space trying to connect everything with everything else. As you dive deep into the networks, the receptive field of a neuron increases. Parameter SharingAs you slide the kernel along the image, you make a naive assumption that the kernel you learned at one part of image is applicable everywhere. So, you end up sharing the learned filter. You want your operations to be translation invariant, and sharing parameters across the image would be able to do that for you. In addition to that, it will save a lot of computation by reducing the weights.**Note**: Sometimes you don't want to share the weights (think of some images where they have properties that are very local and not moving anywhere). In that case, this assumption of weight sharing does not hold.For example, look at the parameters of the small network below.<jupyter_code>import keras
from keras.models import Sequential
from keras.layers.convolutional import Convolution2D, MaxPooling2D, ZeroPadding2D
from keras.layers.core import Activation
model = Sequential()
model.add(Convolution2D(6, 5, 5, border_mode='valid', input_shape = (32, 32, 1)))
model.add(MaxPooling2D(pool_size=(2, 2)))
model.add(Activation("sigmoid"))
model.add(Convolution2D(6, 5, 5, border_mode='valid', input_shape = (32, 32, 1)))
model.summary()<jupyter_output>____________________________________________________________________________________________________
Layer (type)                     Output Shape          Param #     Connected to                     
====================================================================================================
convolution2d_1 (Convolution2D)  (None, 28, 28, 6)     156         convolution2d_input_1[0][0]      
____________________________________________________________________________________________________
maxpooling2d_1 (MaxPooling2D)    (None, 14, 14, 6)     0           convolution2d_1[0][0]            
____________________________________________________________________________________________________
activation_1 (Activation)        (None, 14, 14, 6)     0           maxpooling2d_1[0][0]             
____________________________________________________________________________________________________
convolution2d_2 (Convolution2D)  (None, 10, 10, 6)     906         activation_1[0][0]      [...]<jupyter_text>There is a little caveat here. I am asking Keras not to pad with any zeros but this configuration is not possible because of the this painful formula. Filter output width = (W-F+2P)/(S+1)<jupyter_code>32 - 5 / 2.0<jupyter_output><empty_output><jupyter_text>I am guessing that Keras pads the input with a couple of zeros to make this convolution possible.<jupyter_code>32 - 5 + 2 / 2.0<jupyter_output><empty_output><jupyter_text>But the main goal of defining the small model here is to show sparse connectivity. In this model, the first layer has 156 learnable parameters.<jupyter_code>6 * 5 * 5 + 6<jupyter_output><empty_output><jupyter_text>What if instead of sparse connections, everything is connected to everything else?<jupyter_code>(6*6*6)*(32*32*1)<jupyter_output><empty_output><jupyter_text>We have to learn that many parameters if we were to not use sparse connections and parameter sharing, that is clearly a large number and therefore computationally very expensive.  Equivalence This weight sharing allows ConvNets to be equivariant to translation. You can switch the order of convolution operation, and translation operation but you end up with the same result.  Operations in ConvNetsTypically, we have three kinds of operations that are common among most popular networks.* Convolution* Activation* Pooling function  Pooling:* Pooling is an operation that replaces the output of a network at a certain location with summary statistic of the nearby inputs. The intuition behind pooling is to make the network become approximately invariant with small translations in the input.* Maxpooling is done over a rectangular grid of numbers, the maximum of the grid is selected as pooled output. This reduces the width and height of a particular filter, but does not change the depth.* One can also do pooling over learned filters. * Pooling improves the computational efficiency of the network.* One certain type of pooling called MaxOut pooling tends to make the network rotation invariant.!<jupyter_code>from scipy.ndimage.filters import maximum_filter
fig = plt.figure()
a=fig.add_subplot(1,2,1)
plt.imshow(i, cmap='gray')
a.set_title('Before MaxPooling')
b=fig.add_subplot(1,2,2)
plt.imshow(maximum_filter(i, size=(10,10)), cmap='gray')
b.set_title('After (interpolated)')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07900647377460696, 'fraction_numerical': 0.025234509182190515, 'mean_word_length': 5.334728033472803, 'pattern_counts': {'":': 0, '<': 32, '<?xml version=': 0, '>': 32, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '434293', 'license': 'MIT', 'n_tokens_mistral': 2092, 'n_tokens_neox': 1923, 'n_words': 883, 'path': 'output/downloads/notebooks/convnets.ipynb', 'repo_name': 'Sandeep42/Sandeep42.github.io-source', 'size': '7569'}