<jupyter_start><jupyter_text>CNN  Assignment  理论  复习上课内容以及复现课程代码 参考 `tensorflow_and_keras.ipynb`  回答以下理论题目?   Compared to FNN, what is the biggest advantage of CNN? 参数的数量大大减少  Suppose your input is a 100 by 100 gray image, and you use a convolutional layer with 50 filters that are each 5x5. How many parameters does this hidden layer have (including the bias parameters)?<jupyter_code># number of params

5 * 5 * 50 + 50<jupyter_output><empty_output><jupyter_text>What are "local invariant" and "parameter sharing" ? * local invariant, 卷积核特征提取时与位置无关* parameter sharing, CNN中卷积核的参数用于所有的输入，而FCN中权重参数只能影响一个输入   Why we use batch normalization ? 正则化防止数据进入饱和区，解决了梯度消失的问题。  What problem does dropout try to solve ? 防止过拟合   Is the following statement correct and why ? "Because pooling layers do not have parameters, they do not affect  the backpropagation(derivatives) calculation" 错误，不同的pooling，反向传播误差的方式不一样。  实践  Image Classification **In the first session of the practical part, you will implement an image classification model using any deep learning libraries that you are familiar with,  which means, except for tensorflow and keras, you can also use pytorch/caffe/... .  The dataset used in this session is the cifar10 which contains 50000 color (RGB) images, each with size 32x32x3.  All 50000 images are classified into ten categories.**<jupyter_code>import numpy as np
import pandas as pd
import tensorflow as tf
from tensorflow import keras
import matplotlib.pyplot as plt<jupyter_output>/home/yy/.local/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:523: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint8 = np.dtype([("qint8", np.int8, 1)])
/home/yy/.local/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:524: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_quint8 = np.dtype([("quint8", np.uint8, 1)])
/home/yy/.local/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:525: FutureWarning: Passing (type, 1) or '1type' as a synonym of type is deprecated; in a future version of numpy, it will be understood as (type, (1,)) / '(1,)type'.
  _np_qint16 = np.dtype([("qint16", np.int16, 1)])
/home/yy/.local/lib/python3.7/site-packages/tensorflow/python/framework/dtypes.py:526: FutureWarning: P[...]<jupyter_text>Load the dataset.<jupyter_code>import ssl
ssl._create_default_https_context = ssl._create_unverified_context
(x_train,y_train),(x_test,y_test) = tf.keras.datasets.cifar10.load_data()
for i in range(1,11):
    plt.subplot(2,5,i)
    plt.imshow(x_train[i-1])
    plt.text(3,10,str(y_train[i-1]))
    plt.xticks([])
    plt.yticks([])
plt.show()<jupyter_output><empty_output><jupyter_text>**Datasets**<jupyter_code>from tensorflow.keras.utils import to_categorical
(x_train,y_train),(x_test,y_test) = tf.keras.datasets.cifar10.load_data()
(x_train.shape, y_train.shape), (x_test.shape, y_test.shape)
y_train = to_categorical(y_train, 10)
y_test = to_categorical(y_test, 10)
(x_train.shape, y_train.shape), (x_test.shape, y_test.shape)<jupyter_output><empty_output><jupyter_text>**Model**<jupyter_code>from tensorflow.keras.layers import (Conv2D, MaxPooling2D, Dense,
                                     Flatten, Dropout)
# Hyperparameters

BATCH_SIZE = 128
EPOCHS = 2000
LEARNING_RATE = 1e-3<jupyter_output><empty_output><jupyter_text>LeNet5<jupyter_code>def lenet5():
    """
    LeNet5
    """
    inputs = keras.Input(shape=(32, 32, 3))

    # layer_1 convolution, pooling
    conv_1 = Conv2D(filters=6, kernel_size=(5, 5), activation='relu')(inputs)
    pooling_1 = MaxPooling2D(pool_size=(2, 2))(conv_1)

    # layer_2 convolution, pooling
    conv_2 = Conv2D(filters=16, kernel_size=(5, 5), activation='relu')(pooling_1)
    pooling_2 = MaxPooling2D(pool_size=(2, 2))(conv_2)

    # layer_3 full connection
    flatten = Flatten()(pooling_2)
    dense_3 = Dense(units=120, activation='relu')(flatten)

    # layer_4 full connection
    dense_4 = Dense(units=84, activation='relu')(dense_3)

    # layer_5 output
    outputs = Dense(units=10, activation='softmax')(dense_4)

    return keras.Model(inputs=inputs, outputs=outputs)
img_clf_lenet5 = lenet5()
img_clf_lenet5.summary()
from tensorflow.keras.layers import BatchNormalization
def lenet5_with_bn():
    """
    LeNet5
    """
    inputs = keras.Input(shape=(32, 32, 3))

    # layer_1 convolution, pooling
    conv_1 = Conv2D(filters=6, kernel_size=(5, 5), activation='relu')(inputs)
    pooling_1 = MaxPooling2D(pool_size=(2, 2))(BatchNormalization()(conv_1))

    # layer_2 convolution, pooling
    conv_2 = Conv2D(filters=16, kernel_size=(5, 5), activation='relu')(pooling_1)
    pooling_2 = MaxPooling2D(pool_size=(2, 2))(BatchNormalization()(conv_2))

    # layer_3 full connection
    flatten = Flatten()(pooling_2)
    dense_3 = Dense(units=120, activation='relu')(flatten)

    # layer_4 full connection
    dense_4 = Dense(units=84, activation='relu')(BatchNormalization()(dense_3))

    # layer_5 output
    outputs = Dense(units=10, activation='softmax')(BatchNormalization()(dense_4))

    return keras.Model(inputs=inputs, outputs=outputs)
lenet5_clf_with_bn_model = lenet5_with_bn()
lenet5_clf_with_bn_model.summary()<jupyter_output>Model: "model_1"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
input_2 (InputLayer)         [(None, 32, 32, 3)]       0         
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 28, 28, 6)         456       
_________________________________________________________________
batch_normalization_v2 (Batc (None, 28, 28, 6)         24        
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 14, 14, 6)         0         
_________________________________________________________________
conv2d_3 (Conv2D)            (None, 10, 10, 16)        2416      
_________________________________________________________________
batch_normalization_v2_1 (Ba (None, 10, 10, 16)        64        
___________________________________________________________[...]<jupyter_text>模型可视化<jupyter_code># For Linux 
#sudo apt-get install graphviz
# !pip3 install pydot==1.2.3
# !pip3 install graphviz==0.8.1
from tensorflow.keras.utils import plot_model

plot_model(lenet5_clf_with_bn_model, to_file='lenet5_bn_model.png', show_shapes=True)<jupyter_output><empty_output><jupyter_text>**Train**<jupyter_code>import pylab as pl
from tensorflow.keras import optimizers
from tensorflow.keras import losses
from tensorflow.keras.callbacks import Callback
img_clf_lenet5_bn = lenet5_with_bn()

img_clf_lenet5_bn.compile(optimizer=optimizers.SGD(learning_rate=LEARNING_RATE),
                          loss=losses.categorical_crossentropy,
                          metrics=['accuracy'])

# train_history = img_clf_lenet5_bn.fit(x_train, y_train, batch_size=BATCH_SIZE, 
#                                       epochs=EPOCHS, workers=4,
#                                       validation_split=0.25)
# img_clf_lenet5_bn.save('lenet5_bn.model')<jupyter_output><empty_output><jupyter_text>可视化<jupyter_code>import json
class FloatEncoder(json.JSONEncoder):
    def default(self, o):
        if isinstance(o, (np.float, np.float32, np.float64)):
            return float(o)
        return super(FloatEncoder, self).default(o)
def serialize(obj, filepath):
    with open(filepath, 'w+') as f:
        json.dump(obj, f, cls=FloatEncoder)
def deserialize(filepath):
    with open(filepath) as f:
        return json.load(f)
# record train history data

LENET5_BN_TRAIN_HISTORY_FILEPAHT = 'lenet5_bn_train_history.json'

# serialize(train_history.history, LENET5_BN_TRAIN_HISTORY_FILEPAHT)
# assert isinstance(deserialize(LENET5_BN_TRAIN_HISTORY_FILEPAHT), dict)
plt.legend?
def train_virtualize(history, name='Model'):
    """
    Train virtualization
    """
    # Accuracy
    _, ax = plt.subplots(1, 2, figsize=(12, 4))
    acc = 'acc' if 'acc' in history else 'accuracy'
    val_acc = 'val_acc' if 'val_acc' in history else 'val_accuracy'
    ax[0].plot(history[acc])
    ax[0].plot(history[val_acc])
    ax[0].set_title('{} Accuracy'.format(name))
    ax[0].set_ylabel('Accuracy')
    ax[0].set_xlabel('Epoch')
    ax[0].legend(['Train', 'Test'], loc='upper right')

    # Loss
    ax[1].plot(history['loss'])
    ax[1].plot(history['val_loss'])
    ax[1].set_title('{} Loss'.format(name))
    ax[1].set_ylabel('Loss')
    ax[1].set_xlabel('Epoch')
    ax[1].legend(['Train', 'Test'], loc='upper right')
history = deserialize(LENET5_BN_TRAIN_HISTORY_FILEPAHT)
train_virtualize(history, name='LeNet5_BN')<jupyter_output><empty_output><jupyter_text>模型过拟合，降低模型复杂度<jupyter_code>def img_clf_model():
    inputs = keras.Input(shape=(32, 32, 3))

    # layer_1 convolution, pooling
    conv_1 = Conv2D(filters=6, kernel_size=(3, 3), activation='relu')(inputs)
    pooling_1 = MaxPooling2D(pool_size=(2, 2))(BatchNormalization()(conv_1))

    # layer_2 convolution, pooling
    conv_2 = Conv2D(filters=5, kernel_size=(3, 3), activation='relu')(pooling_1)
    pooling_2 = MaxPooling2D(pool_size=(2, 2))(BatchNormalization()(conv_2))

    # layer_3 full connection
    flatten = Flatten()(pooling_2)
    dense_3 = Dense(units=64, activation='relu')(flatten)

    # layer_4 full connection
    dense_4 = Dense(units=32, activation='relu')(BatchNormalization()(dense_3))

    # layer_5 output
    outputs = Dense(units=10, activation='softmax')(BatchNormalization()(dense_4))

    return keras.Model(inputs=inputs, outputs=outputs)

img_clf_model = img_clf_model()
img_clf_model.summary()
plot_model(img_clf_model, to_file='img_clf_model.png', show_shapes=True)<jupyter_output><empty_output><jupyter_text>Train<jupyter_code>img_clf_model.compile(optimizer=optimizers.Adagrad(learning_rate=0.01), loss=losses.categorical_crossentropy,
                     metrics=['accuracy'])
# img_clf_train_history = img_clf_model.fit(x_train, y_train, batch_size=512, epochs=400,
#                                           validation_split=0.25, workers=8, use_multiprocessing=True)

# img_clf_model.save('img_cls.model')
IMG_CLF_TRAIN_HISOTRY_FILEPATH = 'img_cls_train_history.json'
# serialize(img_clf_train_history.history, IMG_CLF_TRAIN_HISOTRY_FILEPATH)
deserialize(IMG_CLF_TRAIN_HISOTRY_FILEPATH).keys()
train_virtualize(deserialize(IMG_CLF_TRAIN_HISOTRY_FILEPATH))<jupyter_output><empty_output><jupyter_text>继续优化 增加模型复杂度，添加L2范数，Dropout TODO **Evaluation**<jupyter_code>from tensorflow.keras.models import load_model

model = load_model('img_cls.model')

model.evaluate(x_test, y_test)<jupyter_output>10000/10000 [==============================] - 1s 127us/sample - loss: 1.3998 - accuracy: 0.5669<jupyter_text>Optional  TODO  Do you want to build a CNN from scratch ?  In this optianl part, you will build a cnn network step by step.<jupyter_code>import numpy as np
np.random.seed(1)<jupyter_output><empty_output><jupyter_text>4.1 Implement a convolution step Hints: Numpy.sum function might be useful.<jupyter_code>def conv_single_step(a_prev_slice,W,b):
    '''
    Apply one filter defined by parameters W on a single slice (a_slice_prev) of the output activation 
    of the previous layer.
    Arguments:
    a_prev_slice: slice of input data (shape=(f,f,n_C_prev))
    W: Weight parameters contained in a window. (shape = (f,f,n_C_prev))
    b: Bias parameters contained in a window. (shape=(1,1,1))
    
    Reutrns:
    
    Z: a scalar value, the result of convolving the sliding window (W, b) on a slice x of the input data 
    '''
    # Element-wise product
    s = None
    
    # Sum over s  
    Z = None
    
    # Add bias b to z.
    Z = None
    
    return Z
np.random.seed(1)
a_slice_prev = np.random.randn(4, 4, 3)
W = np.random.randn(4, 4, 3)
b = np.random.randn(1, 1, 1)

Z = conv_single_step(a_slice_prev, W, b)
print("Z =", Z)<jupyter_output><empty_output><jupyter_text>Use the code above to test your code. If your code is correct, you should be able to get the output shown as below.                         Z                            -6.99908945068              4.2 Padding Hints: Numpy.pad function might be useful.<jupyter_code>def zero_pad(X, pad):
    """
    Pad with zeros all images of the dataset X. The padding is applied to the height and width of an image, 
    as illustrated in Figure 1.
    
    Argument:
    X: python numpy array of shape (m, n_H, n_W, n_C) representing a batch of m images
    pad: integer, amount of padding around each image on vertical and horizontal dimensions
    
    Returns:
    X_pad: padded image of shape (m, n_H + 2*pad, n_W + 2*pad, n_C)
    """

    X_pad = None
    
    return X_pad
np.random.seed(1)
x = np.random.randn(4, 3, 3, 2)
x_pad = zero_pad(x, 2)
print ("x.shape =\n", x.shape)
print ("x_pad.shape =\n", x_pad.shape)
print ("x[1,1] =\n", x[1,1])
print ("x_pad[1,1] =\n", x_pad[1,1])<jupyter_output><empty_output><jupyter_text>Use the code above to test your code. If your code is correct, you should be able to get the output shown as below. ```x.shape = (4, 3, 3, 2)x_pad.shape = (4, 7, 7, 2)x[1,1] = [[ 0.90085595 -0.68372786] [-0.12289023 -0.93576943] [-0.26788808  0.53035547]]x_pad[1,1] = [[ 0.  0.] [ 0.  0.] [ 0.  0.] [ 0.  0.] [ 0.  0.] [ 0.  0.] [ 0.  0.]]```  4.3 Implement a forward propagation in CNN. Hints: The formulas to calculate the output shapes are : $$ n_H = \lfloor \frac{n_{H_{prev}} - f + 2 \times pad}{stride} \rfloor +1 $$$$ n_W = \lfloor \frac{n_{W_{prev}} - f + 2 \times pad}{stride} \rfloor +1 $$$$ n_C = \text{number of filters used in the convolution}$$<jupyter_code>def conv_forward(A_prev, W, b, hparameters):
    """
    Implements the forward propagation for a convolution function
    
    Arguments:
    A_prev: output activations of the previous layer, 
        numpy array of shape (m, n_H_prev, n_W_prev, n_C_prev)
    W: Weights, numpy array of shape (f, f, n_C_prev, n_C)
    b: Biases, numpy array of shape (1, 1, 1, n_C)
    hparameters: python dictionary containing "stride" and "pad"
        
    Returns:
    Z: conv output, numpy array of shape (m, n_H, n_W, n_C)
    cache: cache of values needed for the conv_backward() function
    """
    
    # Get dimensions from A_prev's shape  
    (m, n_H_prev, n_W_prev, n_C_prev) = None
    
    # Get dimensions from W's shape 
    (f, f, n_C_prev, n_C) = None
    
    # Get information from "hparameters" 
    stride = None
    pad = None
    
    # Compute the dimensions of the CONV output volume using the formula given above. 
    # Hint: use int() to apply the 'floor' operation. 
    n_H = None
    n_W = None
    
    # Initialize the output volume Z with zeros. 
    Z = None
    
    # Create A_prev_pad by padding A_prev
    A_prev_pad = None
    
    for i in range(None):               # loop over the batch of training examples
        a_prev_pad = None               # Select ith training example's padded activation
        for h in range(None):           # loop over vertical axis of the output volume
            # Find the vertical start and end of the current "slice" 
            vert_start = None
            vert_end = None
            
            for w in range(None):       # loop over horizontal axis of the output volume
                # Find the horizontal start and end of the current "slice"
                horiz_start = None
                horiz_end = None
                
                for c in range(None):   # loop over channels (= #filters) of the output volume
                                        
                    # Use the corners to define the (3D) slice of a_prev_pad (See Hint above the cell). 
                    a_slice_prev = None
                    
                    # Convolve the (3D) slice with the correct filter W and bias b, to get back one output neuron. 
                    weights = None
                    biases = None
                    Z[i, h, w, c] = None
                                        
    
    # Making sure your output shape is correct
    assert(Z.shape == (m, n_H, n_W, n_C))
    
    # Save information in "cache" for the backprop
    cache = (A_prev, W, b, hparameters)
    
    return Z, cache
np.random.seed(1)
A_prev = np.random.randn(10,5,7,4)
W = np.random.randn(3,3,4,8)
b = np.random.randn(1,1,1,8)
hparameters = {"pad" : 1,
               "stride": 2}

Z, cache_conv = conv_forward(A_prev, W, b, hparameters)
print("Z's mean =\n", np.mean(Z))
print("Z[3,2,1] =\n", Z[3,2,1])
print("cache_conv[0][1][2][3] =\n", cache_conv[0][1][2][3])<jupyter_output><empty_output><jupyter_text>Use the code above to test your code. If your code is correct, you should be able to get the output shown as below. ```Z's mean = 0.692360880758Z[3,2,1] = [ -1.28912231   2.27650251   6.61941931   0.95527176   8.25132576   2.31329639  13.00689405   2.34576051]cache_conv[0][1][2][3] = [-1.1191154   1.9560789  -0.3264995  -1.34267579]```  4.4 Pooling layers Hints: As there\'s no padding, the formulas binding the output shape of the pooling to the input shape is:$$ n_H = \lfloor \frac{n_{H_{prev}} - f}{stride} \rfloor +1 $$$$ n_W = \lfloor \frac{n_{W_{prev}} - f}{stride} \rfloor +1 $$$$ n_C = n_{C_{prev}}$$<jupyter_code>def pool_forward(A_prev, hparameters, mode = "max"):
    """
    Implements the forward pass of the pooling layer
    
    Arguments:
    A_prev: Input data, numpy array of shape (m, n_H_prev, n_W_prev, n_C_prev)
    hparameters: python dictionary containing "f" and "stride"
    mode: the pooling mode you would like to use, defined as a string ("max" or "average")
    
    Returns:
    A: output of the pool layer, a numpy array of shape (m, n_H, n_W, n_C)
    cache: cache used in the backward pass of the pooling layer, contains the input and hparameters 
    """
    
    # Get dimensions from the input shape
    (m, n_H_prev, n_W_prev, n_C_prev) = A_prev.shape
    
    # Get hyperparameters from "hparameters"
    f = hparameters["f"]
    stride = hparameters["stride"]
    
    # Define the dimensions of the output
    n_H = int(1 + (n_H_prev - f) / stride)
    n_W = int(1 + (n_W_prev - f) / stride)
    n_C = n_C_prev
    
    # Initialize output matrix A
    A = np.zeros((m, n_H, n_W, n_C))              
    
    for i in range(None):                         # loop over the training examples
        for h in range(None):                     # loop on the vertical axis of the output volume
            # Find the vertical start and end of the current "slice" (≈2 lines)
            vert_start = None
            vert_end = None
            
            for w in range(None):                 # loop on the horizontal axis of the output volume
                # Find the vertical start and end of the current "slice" (≈2 lines)
                horiz_start = None
                horiz_end = None
                
                for c in range (None):            # loop over the channels of the output volume
                    
                    # Use the corners to define the current slice on the ith training example of A_prev, channel c. (≈1 line)
                    a_prev_slice = None
                    
                    # Compute the pooling operation on the slice. 
                    # Use an if statement to differentiate the modes. 
                    # Use np.max and np.mean.
                    if mode == "max":
                        A[i, h, w, c] = None
                    elif mode == "average":
                        A[i, h, w, c] = None
    
    # Store the input and hparameters in "cache" for pool_backward()
    cache = (A_prev, hparameters)
    
    # Making sure your output shape is correct
    assert(A.shape == (m, n_H, n_W, n_C))
    
    return A, cache
np.random.seed(1)
A_prev = np.random.randn(2, 5, 5, 3)
hparameters = {"stride" : 1, "f": 3}

A, cache = pool_forward(A_prev, hparameters)
print("mode = max")
print("A.shape = " + str(A.shape))
print("A =\n", A)
print()
A, cache = pool_forward(A_prev, hparameters, mode = "average")
print("mode = average")
print("A.shape = " + str(A.shape))
print("A =\n", A)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11689837477971411, 'fraction_numerical': 0.03446250244762091, 'mean_word_length': 3.2765333891563744, 'pattern_counts': {'":': 4, '<': 66, '<?xml version=': 0, '>': 66, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 10, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '17', 'dir': 'jupyter-structured-clean-dedup', 'id': '220065', 'license': 'MIT', 'n_tokens_mistral': 7410, 'n_tokens_neox': 6995, 'n_words': 1828, 'path': 'nlp-2019-autumn/lecture06/Assignment-06.ipynb', 'repo_name': 'hotbaby/kkb-nlp', 'size': '20428'}