<jupyter_start><jupyter_text>DNN The text in the document by <NAME>, <NAME>, <NAME> is licensed under CC BY 3.0 https://creativecommons.org/licenses/by/3.0/us/ The code in the document by <NAME>, <NAME>, Zhiwei Zhang is licensed under the MIT License https://opensource.org/licenses/MIThttps://github.com/shujiefan/7390/blob/master/LICENSE DataSet Descriptiontrain.csv (42000 rows and 785 columns, columns are composed of one digit label and 784 pixels values of digit images)test.csv (28000 rows and 784 columns, columns are composed of 784 pixels values of digit images)Document DescriptionDigit Recognization with Deep learning using MNIST data and TensorflowResultAccuracy:0.98Model Architecture[[conv]-->[max_pool_2*2]]*2-->[Flatten]-->[Dropout]cite:https://www.kaggle.com/kakauandme/tensorflow-deep-nn<jupyter_code>import numpy as np
import pandas as pd
%matplotlib inline
import matplotlib.pyplot as plt
import matplotlib.cm as cm
import tensorflow as tf<jupyter_output><empty_output><jupyter_text>set the default parametersset the dropout to prevent overfitting set the validation_size to split the dataset into training set and validation set, and make the crossing validationset the beginning learning_rateset the specific image to display<jupyter_code># settings
LEARNING_RATE = 1e-4
TRAINING_ITERATIONS = 2500        
DROPOUT = 0.5
BATCH_SIZE = 50
VALIDATION_SIZE = 2000
IMAGE_TO_DISPLAY = 5<jupyter_output><empty_output><jupyter_text>load the data By <NAME>, is licensed under the MIT License https://opensource.org/licenses/MIT To start, we read provided data. The train.csv file contains 42000 rows and 785 columns. Each row represents an image of a handwritten digit and a label with the value of this digit.<jupyter_code># read training data from CSV file 
data = pd.read_csv('train.csv')
print('data({0[0]},{0[1]})'.format(data.shape))
print (data.head())<jupyter_output>data(42000,785)
   label  pixel0  pixel1  pixel2  pixel3  pixel4  pixel5  pixel6  pixel7  \
0      1       0       0       0       0       0       0       0       0   
1      0       0       0       0       0       0       0       0       0   
2      1       0       0       0       0       0       0       0       0   
3      4       0       0       0       0       0       0       0       0   
4      0       0       0       0       0       0       0       0       0   

   pixel8    ...     pixel774  pixel775  pixel776  pixel777  pixel778  \
0       0    ...            0         0         0         0         0   
1       0    ...            0         0         0         0         0   
2       0    ...            0         0         0         0         0   
3       0    ...            0         0         0         0         0   
4       0    ...            0         0         0         0         0   

   pixel779  pixel780  pixel781  pixel782  pixel783  
0         0         0         0   [...]<jupyter_text>normalize the data  normailze the data to speed up the traning processWe perform a grayscale normalization to reduce the effect of illumination's differences.Moreover the CNN converg faster on [0..1] data than on [0..255].<jupyter_code>images = data.iloc[:,1:].values
images = images.astype(np.float)
# convert from [0:255] => [0.0:1.0]
images = np.multiply(images, 1.0 / 255.0)
print('images({0[0]},{0[1]})'.format(images.shape))<jupyter_output>images(42000,784)<jupyter_text>the traning set shape is (42000,784), which means we have 42000 rows, each represents an image. and we have 784 columns, each column represents one pixel.and each image has 784 pixels. images reshape we reshpe each image(contains 784 pixels) into a image whose width and height is 28.Train and test images (28px x 28px) has been stock into pandas.Dataframe as 1D vectors of 784 values. We reshape all data to 28x28x1 3D matrices.MNIST images are gray scaled so it use only one channel. For RGB images, there is 3 channels, we would have reshaped 784px vectors to 28x28x3 3D matrices.<jupyter_code>image_size = images.shape[1]
print ('image_size => {0}'.format(image_size))
# in this case all images are square
image_width = image_height = np.ceil(np.sqrt(image_size)).astype(np.uint8)
print ('image_width => {0}\nimage_height => {1}'.format(image_width,image_height))<jupyter_output>image_size => 784
image_width => 28
image_height => 28<jupyter_text>display one image we choose one specific image to display, to validate our reshape process<jupyter_code># display image
def display(img):
    one_image = img.reshape(image_width,image_height)
    plt.axis('off')
    plt.imshow(one_image, cmap=cm.binary)
# output image     
display(images[IMAGE_TO_DISPLAY])<jupyter_output><empty_output><jupyter_text>output the displayed image's lable split the trainning into labels and pixels, show the labels‘ size.<jupyter_code>labels_flat = data.iloc[:,0].values.ravel()
print('labels_flat({0})'.format(len(labels_flat)))
print ('labels_flat[{0}] => {1}'.format(IMAGE_TO_DISPLAY,labels_flat[IMAGE_TO_DISPLAY]))<jupyter_output>labels_flat(42000)
labels_flat[5] => 0<jupyter_text>count the label count the unique labels‘ number, we have 10 labels, from 0 to 9, and we need to recognize these labels.<jupyter_code>labels_count = np.unique(labels_flat).shape[0]
print('labels_count => {0}'.format(labels_count))<jupyter_output>labels_count => 10<jupyter_text>encode the label to one-hot vectors convert class labels from scalars to one-hot vectors0 => [1 0 0 0 0 0 0 0 0 0]1 => [0 1 0 0 0 0 0 0 0 0] ...9 => [0 0 0 0 0 0 0 0 0 1]<jupyter_code>def dense_to_one_hot(labels_dense, num_classes):
    num_labels = labels_dense.shape[0]
    index_offset = np.arange(num_labels) * num_classes
    labels_one_hot = np.zeros((num_labels, num_classes))
    labels_one_hot.flat[index_offset + labels_dense.ravel()] = 1
    return labels_one_hot

labels = dense_to_one_hot(labels_flat, labels_count)
labels = labels.astype(np.uint8)

print('labels({0[0]},{0[1]})'.format(labels.shape))
print ('labels[{0}] => {1}'.format(IMAGE_TO_DISPLAY,labels[IMAGE_TO_DISPLAY]))<jupyter_output>labels(42000,10)
labels[5] => [1 0 0 0 0 0 0 0 0 0]<jupyter_text>split the data into training and validation we split the data into training data and validation data to make the crossing validationour training data‘s size is (40000,784)our validation data’s size is (2000,784)<jupyter_code># split data into training & validation
validation_images = images[:VALIDATION_SIZE]
validation_labels = labels[:VALIDATION_SIZE]
train_images = images[VALIDATION_SIZE:]
train_labels = labels[VALIDATION_SIZE:]
print('train_images({0[0]},{0[1]})'.format(train_images.shape))
print('validation_images({0[0]},{0[1]})'.format(validation_images.shape))<jupyter_output>train_images(40000,784)
validation_images(2000,784)<jupyter_text>initialize the weight and the biasHelper functionsFor this NN model, a lot of weights and biases are created. Generally, weights should be initialised with a small amount of noise for symmetry breaking, and to prevent 0 gradients.Since we are using ReLU neurones (ones that contain rectifier function f(x)=max(0,x)), it is also good practice to initialise them with a slightly positive initial bias to avoid "dead neurones".<jupyter_code># weight initialization
def weight_variable(shape):
    initial = tf.truncated_normal(shape, stddev=0.1)
    return tf.Variable(initial)
def bias_variable(shape):
    initial = tf.constant(0.1, shape=shape)
    return tf.Variable(initial)<jupyter_output><empty_output><jupyter_text>initialize the convolution the pooling layer the convolutional (Conv2D) layer. It is like a set of learnable filters. I choosed to set 32 filters for the first conv2D layers and 64 filters for the last one. In general, convolution layer is used to get the features of the data. In the case of digit recognition - a shape of each digit. It uses learnable kernels/filters each of which corresponds to one particular shape pattern. The number of the filter can differ for other problems.<jupyter_code># convolution
def conv2d(x, W):
    return tf.nn.conv2d(x, W, strides=[1, 1, 1, 1], padding='SAME')<jupyter_output><empty_output><jupyter_text>The pooling (MaxPool2D) layer. This layer simply acts as a downsampling filter. It looks at the 2 neighboring pixels and picks the maximal value. These are used to reduce computational cost, and to some extent also reduce overfitting. We have to choose the pooling size (i.e the area size pooled each time) more the pooling dimension is high, more the downsampling is important.<jupyter_code>def max_pool_2x2(x):
    return tf.nn.max_pool(x, ksize=[1, 2, 2, 1], strides=[1, 2, 2, 1], padding='SAME')
# input & output of NN
# images
x = tf.placeholder('float', shape=[None, image_size])
# labels
y_ = tf.placeholder('float', shape=[None, labels_count])<jupyter_output><empty_output><jupyter_text>first layer The first layer I use one convolutional layer and one max_pool(2D) layer. The first step is to reshape the image from(40000,784) into (40000,28,28,1). Secondly, use the initialized weight parameter for the next reshape process. After this step, the shape has been changed to (40000,28,28,32). The last step using MaxPool2D with the initialized parameter bias, and the shape changed into (40000,14,14,32).<jupyter_code># [[conv-->relu]-->[max_pool_2*2]]*2-->[Flatten]-->[Dropout]
# first convolutional layer
W_conv1 = weight_variable([5, 5, 1, 32])
b_conv1 = bias_variable([32])
# (40000,784) => (40000,28,28,1)
image = tf.reshape(x, [-1,image_width , image_height,1])
#print (image.get_shape()) # =>(40000,28,28,1)
h_conv1 = tf.nn.relu(conv2d(image, W_conv1) + b_conv1)
#print (h_conv1.get_shape()) # => (40000, 28, 28, 32)
h_pool1 = max_pool_2x2(h_conv1)
#print (h_pool1.get_shape()) # => (40000, 14, 14, 32)<jupyter_output><empty_output><jupyter_text>second layer the architecture of the second layer is the same as the first layer.we set the weight variable to be [5,5,32,64] and the bias variable to be [64], Using one convolution layer and one MaxPool2D.And for the second layer, we change the parameter of weight and bias,after the the convolution layer, the shape changed into (40000, 14, 14 64), and after the MaxPool the shape changed into (40000, 7,7, 64).<jupyter_code># second convolutional layer
W_conv2 = weight_variable([5, 5, 32, 64])
b_conv2 = bias_variable([64])
h_conv2 = tf.nn.relu(conv2d(h_pool1, W_conv2) + b_conv2)
#print (h_conv2.get_shape()) # => (40000, 14,14, 64)
h_pool2 = max_pool_2x2(h_conv2)
#print (h_pool2.get_shape()) # => (40000, 7, 7, 64)<jupyter_output><empty_output><jupyter_text>densely connected layer Now that the image size is reduced to 7x7, we add a fully-connected layer with 1024 neurones to allow processing on the entire image (each of the neurons of the fully connected layer is connected to all the activations/outpus of the previous layer)<jupyter_code># densely connected layer
W_fc1 = weight_variable([7 * 7 * 64, 1024])
b_fc1 = bias_variable([1024])
# (40000, 7, 7, 64) => (40000, 3136)
h_pool2_flat = tf.reshape(h_pool2, [-1, 7*7*64])
h_fc1 = tf.nn.relu(tf.matmul(h_pool2_flat, W_fc1) + b_fc1)
#print (h_fc1.get_shape()) # => (40000, 1024)<jupyter_output><empty_output><jupyter_text>dropout Dropout is a regularization method, where a proportion of nodes in the layer are randomly ignored (setting their wieghts to zero) for each training sample. This drops randomly a propotion of the network and forces the network to learn features in a distributed way. This technique also improves generalization and reduces the overfitting.<jupyter_code># dropout
keep_prob = tf.placeholder('float')
h_fc1_drop = tf.nn.dropout(h_fc1, keep_prob)<jupyter_output><empty_output><jupyter_text>readout layer In the last layer(labels_count=10,activation="softmax") the net outputs distribution of probability of each class.<jupyter_code># readout layer for deep net
W_fc2 = weight_variable([1024, labels_count])
b_fc2 = bias_variable([labels_count])
y = tf.nn.softmax(tf.matmul(h_fc1_drop, W_fc2) + b_fc2)
#print (y.get_shape()) # => (40000, 10)<jupyter_output><empty_output><jupyter_text>To evaluate network performance we use cross-entropy and to minimise it ADAM optimiser is used.ADAM optimiser is a gradient based optimization algorithm, based on adaptive estimates, it's more sophisticated than steepest gradient descent and is well suited for problems with large data or many parameters.<jupyter_code># cost function
cross_entropy = -tf.reduce_sum(y_*tf.log(y))
# optimisation function
train_step = tf.train.AdamOptimizer(LEARNING_RATE).minimize(cross_entropy)
# evaluation
correct_prediction = tf.equal(tf.argmax(y,1), tf.argmax(y_,1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, 'float'))<jupyter_output><empty_output><jupyter_text>defined the prediction function To predict values from test data, highest probability is picked from "one-hot vector" indicating that chances of an image being one of the digits are highest.<jupyter_code># prediction function
#[0.1, 0.9, 0.2, 0.1, 0.1 0.3, 0.5, 0.1, 0.2, 0.3] => 1
predict = tf.argmax(y,1)<jupyter_output><empty_output><jupyter_text>evaluate the model Ideally, we should use all data for every step of the training, but that's expensive. So, instead, we use small "batches" of random data.This method is called stochastic training. It is cheaper, faster and gives much of the same result.<jupyter_code>epochs_completed = 0
index_in_epoch = 0
num_examples = train_images.shape[0]

# serve data by batches
def next_batch(batch_size):
    
    global train_images
    global train_labels
    global index_in_epoch
    global epochs_completed
    
    start = index_in_epoch
    index_in_epoch += batch_size
    
    # when all trainig data have been already used, it is reorder randomly    
    if index_in_epoch > num_examples:
        # finished epoch
        epochs_completed += 1
        # shuffle the data
        perm = np.arange(num_examples)
        np.random.shuffle(perm)
        train_images = train_images[perm]
        train_labels = train_labels[perm]
        # start next epoch
        start = 0
        index_in_epoch = batch_size
        assert batch_size <= num_examples
    end = index_in_epoch
    return train_images[start:end], train_labels[start:end]
# start TensorFlow session
init = tf.global_variables_initializer()
sess = tf.InteractiveSession()
sess.run(init)<jupyter_output><empty_output><jupyter_text>data evaluation display Each step of the loop, we get a "batch" of data points from the training set and feed it to the graph to replace the placeholders. In this case, it's: x, y and dropout.Also, once in a while, we check training accuracy on an upcoming "batch".For result, we display the training accuracy and validation accuracy.<jupyter_code># visualisation variables
train_accuracies = []
validation_accuracies = []
x_range = []

display_step=1

for i in range(TRAINING_ITERATIONS):

    #get new batch
    batch_xs, batch_ys = next_batch(BATCH_SIZE)        

    # check progress on every 1st,2nd,...,10th,20th,...,100th... step
    if i%display_step == 0 or (i+1) == TRAINING_ITERATIONS:
        
        train_accuracy = accuracy.eval(feed_dict={x:batch_xs, 
                                                  y_: batch_ys, 
                                                  keep_prob: 1.0})       
        if(VALIDATION_SIZE):
            validation_accuracy = accuracy.eval(feed_dict={ x: validation_images[0:BATCH_SIZE], 
                                                            y_: validation_labels[0:BATCH_SIZE], 
                                                            keep_prob: 1.0})                                  
            print('training_accuracy / validation_accuracy => %.2f / %.2f for step %d'%(train_accuracy, validation_accuracy, i))
            
            validation_accuracies.append(validation_accuracy)
            
        else:
             print('training_accuracy => %.4f for step %d'%(train_accuracy, i))
        train_accuracies.append(train_accuracy)
        x_range.append(i)
        
        # increase display_step
        if i%(display_step*10) == 0 and i:
            display_step *= 10
    # train on batch
    sess.run(train_step, feed_dict={x: batch_xs, y_: batch_ys, keep_prob: DROPOUT})<jupyter_output>training_accuracy / validation_accuracy => 0.22 / 0.24 for step 0
training_accuracy / validation_accuracy => 0.30 / 0.28 for step 1
training_accuracy / validation_accuracy => 0.18 / 0.20 for step 2
training_accuracy / validation_accuracy => 0.10 / 0.14 for step 3
training_accuracy / validation_accuracy => 0.26 / 0.16 for step 4
training_accuracy / validation_accuracy => 0.24 / 0.24 for step 5
training_accuracy / validation_accuracy => 0.24 / 0.34 for step 6
training_accuracy / validation_accuracy => 0.32 / 0.40 for step 7
training_accuracy / validation_accuracy => 0.24 / 0.38 for step 8
training_accuracy / validation_accuracy => 0.24 / 0.40 for step 9
training_accuracy / validation_accuracy => 0.36 / 0.42 for step 10
training_accuracy / validation_accuracy => 0.48 / 0.52 for step 20
training_accuracy / validation_accuracy => 0.52 / 0.68 for step 30
training_accuracy / validation_accuracy => 0.70 / 0.80 for step 40
training_accuracy / validation_accuracy => 0.70 / 0.84 for step 50
train[...]<jupyter_text>final accuracy of the validation set the final accuracy of the validation is 0.9815, certainly, it is not the best one. and then we change the parameter varialbe  to strength the accuracy.<jupyter_code># check final accuracy on validation set  
if(VALIDATION_SIZE):
    validation_accuracy = accuracy.eval(feed_dict={x: validation_images, 
                                                   y_: validation_labels, 
                                                   keep_prob: 1.0})
    print('validation_accuracy => %.4f'%validation_accuracy)
    plt.plot(x_range, train_accuracies,'-b', label='Training')
    plt.plot(x_range, validation_accuracies,'-g', label='Validation')
    plt.legend(loc='lower right', frameon=False)
    plt.ylim(ymax = 1.1, ymin = 0.7)
    plt.ylabel('accuracy')
    plt.xlabel('step')
    plt.show()
sess.close()<jupyter_output><empty_output><jupyter_text>change the activation function from relu to elu To strength the accuracy, we change the activation function to be elu, to see the accuracy of this model. However, the result does not have obvious improvment. Therefore, we try to change another method. we use CNN to get the more higher accuracy.<jupyter_code># first convolutional layer
W_conv1 = weight_variable([5, 5, 1, 32])
b_conv1 = bias_variable([32])
# (40000,784) => (40000,28,28,1)
image = tf.reshape(x, [-1,image_width , image_height,1])
#print (image.get_shape()) # =>(40000,28,28,1)
h_conv1 = tf.nn.elu(conv2d(image, W_conv1) + b_conv1)
#print (h_conv1.get_shape()) # => (40000, 28, 28, 32)
h_pool1 = max_pool_2x2(h_conv1)
#print (h_pool1.get_shape()) # => (40000, 14, 14, 32)


# second convolutional layer
W_conv2 = weight_variable([5, 5, 32, 64])
b_conv2 = bias_variable([64])
h_conv2 = tf.nn.elu(conv2d(h_pool1, W_conv2) + b_conv2)
#print (h_conv2.get_shape()) # => (40000, 14,14, 64)
h_pool2 = max_pool_2x2(h_conv2)
#print (h_pool2.get_shape()) # => (40000, 7, 7, 64)


# densely connected layer
W_fc1 = weight_variable([7 * 7 * 64, 1024])
b_fc1 = bias_variable([1024])
# (40000, 7, 7, 64) => (40000, 3136)
h_pool2_flat = tf.reshape(h_pool2, [-1, 7*7*64])
h_fc1 = tf.nn.elu(tf.matmul(h_pool2_flat, W_fc1) + b_fc1)
#print (h_fc1.get_shape()) # => (40000, 1024)

# dropout
keep_prob = tf.placeholder('float')
h_fc1_drop = tf.nn.dropout(h_fc1, keep_prob)
# readout layer for deep net
W_fc2 = weight_variable([1024, labels_count])
b_fc2 = bias_variable([labels_count])

y = tf.nn.softmax(tf.matmul(h_fc1_drop, W_fc2) + b_fc2)


# cost function
cross_entropy = -tf.reduce_sum(y_*tf.log(y))
# optimisation function
train_step = tf.train.AdamOptimizer(LEARNING_RATE).minimize(cross_entropy)
# evaluation
correct_prediction = tf.equal(tf.argmax(y,1), tf.argmax(y_,1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, 'float'))
# prediction function
#[0.1, 0.9, 0.2, 0.1, 0.1 0.3, 0.5, 0.1, 0.2, 0.3] => 1
predict = tf.argmax(y,1)
epochs_completed = 0
index_in_epoch = 0
num_examples = train_images.shape[0]
# serve data by batches
def next_batch(batch_size):
    
    global train_images
    global train_labels
    global index_in_epoch
    global epochs_completed
    
    start = index_in_epoch
    index_in_epoch += batch_size
    
    # when all trainig data have been already used, it is reorder randomly    
    if index_in_epoch > num_examples:
        # finished epoch
        epochs_completed += 1
        # shuffle the data
        perm = np.arange(num_examples)
        np.random.shuffle(perm)
        train_images = train_images[perm]
        train_labels = train_labels[perm]
        # start next epoch
        start = 0
        index_in_epoch = batch_size
        assert batch_size <= num_examples
    end = index_in_epoch
    return train_images[start:end], train_labels[start:end]

# start TensorFlow session
init = tf.global_variables_initializer()
sess = tf.InteractiveSession()
sess.run(init)<jupyter_output><empty_output><jupyter_text>elu function<jupyter_code>import time 
t1=time.time()
# visualisation variables
train_accuracies = []
validation_accuracies = []
x_range = []

display_step=1

for i in range(TRAINING_ITERATIONS):

    #get new batch
    batch_xs, batch_ys = next_batch(BATCH_SIZE)        

    # check progress on every 1st,2nd,...,10th,20th,...,100th... step
    if i%display_step == 0 or (i+1) == TRAINING_ITERATIONS:
        
        train_accuracy = accuracy.eval(feed_dict={x:batch_xs, 
                                                  y_: batch_ys, 
                                                  keep_prob: 1.0})       
        if(VALIDATION_SIZE):
            validation_accuracy = accuracy.eval(feed_dict={ x: validation_images[0:BATCH_SIZE], 
                                                            y_: validation_labels[0:BATCH_SIZE], 
                                                            keep_prob: 1.0})                                  
            print('training_accuracy / validation_accuracy => %.2f / %.2f for step %d'%(train_accuracy, validation_accuracy, i))
            
            validation_accuracies.append(validation_accuracy)
            
        else:
             print('training_accuracy => %.4f for step %d'%(train_accuracy, i))
        train_accuracies.append(train_accuracy)
        x_range.append(i)
        
        # increase display_step
        if i%(display_step*10) == 0 and i:
            display_step *= 10
    # train on batch
    sess.run(train_step, feed_dict={x: batch_xs, y_: batch_ys, keep_prob: DROPOUT})
t = time.time()-t1 
print("running time is %g s" %(t)) 
# check final accuracy on validation set  
if(VALIDATION_SIZE):
    validation_accuracy = accuracy.eval(feed_dict={x: validation_images, 
                                                   y_: validation_labels, 
                                                   keep_prob: 1.0})
    print('validation_accuracy => %.4f'%validation_accuracy)
    plt.plot(x_range, train_accuracies,'-b', label='Training')
    plt.plot(x_range, validation_accuracies,'-g', label='Validation')
    plt.legend(loc='lower right', frameon=False)
    plt.ylim(ymax = 1.1, ymin = 0.7)
    plt.ylabel('accuracy')
    plt.xlabel('step')
    plt.show()<jupyter_output>validation_accuracy => 0.9805
{'fraction_non_alphanumeric': 0.10079405737704918, 'fraction_numerical': 0.04807035519125683, 'mean_word_length': 3.4970243808792474, 'pattern_counts': {'":': 0, '<': 100, '<?xml version=': 0, '>': 164, 'https://': 5, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 49, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '25', 'dir': 'jupyter-structured-clean-dedup', 'id': '153853', 'license': 'MIT', 'n_tokens_mistral': 8133, 'n_tokens_neox': 7399, 'n_words': 2223, 'path': 'input/DNN.ipynb', 'repo_name': 'shujiefan/7390', 'size': '23447'}