<jupyter_start><jupyter_text>0. Introduction<jupyter_code>%matplotlib inline<jupyter_output><empty_output><jupyter_text>Example based on: Deep Learning with Python by <NAME>:https://www.manning.com/books/deep-learning-with-python<jupyter_code>from tensorflow.examples.tutorials.mnist import input_data
import tensorflow as tf
import numpy as np<jupyter_output><empty_output><jupyter_text>1. Load data The MNIST database of handwritten digit has a training set of 60,000 examples, and a test set of 10,000 examples. It is a subset of a larger set available from NIST. The digits have been size-normalized and centered in a fixed-size image.It is a good database for people who want to try learning techniques and pattern recognition methods on real-world data while spending minimal efforts on preprocessing and formatting.<jupyter_code>!rm -rf myModel
#from tensorflow.models import mnist
import tensorflow as tf
import numpy as np

def weight_variable(shape):
    initial = tf.truncated_normal(shape, stddev=0.1)
    return tf.Variable(initial)


def bias_variable(shape):
    initial = tf.constant(0.1, shape=shape)
    return tf.Variable(initial)


def conv2d(x, W, name=None):
    return tf.nn.conv2d(x, W, strides=[1, 1, 1, 1], padding='SAME', name=name)


def max_pool_2x2(x):
    return tf.nn.max_pool(x, ksize=[1, 2, 2, 1],
                          strides=[1, 2, 2, 1], padding='SAME')


def conv_layer(input, shape, name=None):
    W = weight_variable(shape)
    b = bias_variable([shape[3]])
    return tf.nn.relu(conv2d(input, W, name=name) + b)


def full_layer(input, size, name=None):
    in_size = int(input.get_shape()[1])
    W = weight_variable([in_size, size])
    b = bias_variable([size])
    return tf.matmul(input, W, name=name) + b

DATA_DIR = '/tmp/data'
MINIBATCH_SIZE = 50
STEPS = 5000


mnist = input_data.read_data_sets(DATA_DIR, one_hot=True)

x = tf.placeholder(tf.float32, shape=[None, 784])
y_ = tf.placeholder(tf.float32, shape=[None, 10])

x_image = tf.reshape(x, [-1, 28, 28, 1])
conv1 = conv_layer(x_image, shape=[5, 5, 1, 32], name="inputLayer")
conv1_pool = max_pool_2x2(conv1)

conv2 = conv_layer(conv1_pool, shape=[5, 5, 32, 64])
conv2_pool = max_pool_2x2(conv2)

conv2_flat = tf.reshape(conv2_pool, [-1, 7*7*64])
full_1 = tf.nn.relu(full_layer(conv2_flat, 1024))

keep_prob = tf.placeholder(tf.float32)
full1_drop = tf.nn.dropout(full_1, keep_prob=keep_prob)

y_conv = full_layer(full1_drop, 10, name="inferenceLayer")

cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=y_conv, labels=y_))
train_step = tf.train.AdamOptimizer(1e-4).minimize(cross_entropy)
correct_prediction = tf.equal(tf.argmax(y_conv, 1), tf.argmax(y_, 1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

with tf.Session() as sess:
    sess.run(tf.global_variables_initializer())

    for i in range(STEPS):
        batch = mnist.train.next_batch(MINIBATCH_SIZE)

        if i % 100 == 0:
            train_accuracy = sess.run(accuracy, feed_dict={x: batch[0], y_: batch[1],
                                                           keep_prob: 1.0})
            print("step {}, training accuracy {}".format(i, train_accuracy))

        sess.run(train_step, feed_dict={x: batch[0], y_: batch[1], keep_prob: 0.5})

    X = mnist.test.images.reshape(10, 1000, 784)
    Y = mnist.test.labels.reshape(10, 1000, 10)
    test_accuracy = np.mean(
        [sess.run(accuracy, feed_dict={x: X[i], y_: Y[i], keep_prob: 1.0}) for i in range(10)])
    builder = tf.saved_model.builder.SavedModelBuilder("myModel")
    builder.add_meta_graph_and_variables(sess,
                                  ["myTag"]
                                  
                                  )
    builder.save()

print("test accuracy: {}".format(test_accuracy))
! ls -l myModel
!ls -l myModel/variables<jupyter_output>total 115128
-rw-r--r-- 1 root root 117886848 Jan 23 09:56 variables.data-00000-of-00001
-rw-r--r-- 1 root root      2559 Jan 23 09:56 variables.index<jupyter_text>2. Inspect data Try to get a feel for the data you are using to train and test your neural network.   Training data - Training data will be used to train our neural network to recognize hand-written digits.- MNIST provides 60000 labeled training images, each 28x28 pixels<jupyter_code>train_images.shape
train_labels.shape
def show_image(images, labels, index):
    img = images[index].reshape((28,28))
    label = labels[index]
    plt.imshow(img)
    plt.title(label)
    plt.show()
show_image(train_images, train_labels, 10)<jupyter_output><empty_output><jupyter_text>Test data - Test data will be used to validate how good our network performs on data it has never seen.- MNIST provides 10000 test images, each 28x28.- It's important to note that these should never be used in the training cycle. A 'test set' should never contain images the network has already seen during training. (read more: [Model Selection and Train/Validation/Test Sets](https://www.coursera.org/lecture/machine-learning/model-selection-and-train-validation-test-sets-QGKbr) and [How (and why) to create a good validation set](https://www.fast.ai/2017/11/13/validation-sets/))<jupyter_code>test_images.shape
test_labels.shape
show_image(test_images, test_labels, 168)<jupyter_output><empty_output><jupyter_text>3. Network architecture  3.1. ConvNet (Feature extraction) Define the network architecture that will be used for training- how many layers - which type of layer    - Convolution: channels, kernel size, activation    - MaxPool: matrix sizeImportant: ConvNets take as input tensors of shape: (image_height, image_width, channels)**Note:**Convolutional layers learn local patterns (features that can appear anywhere in the image)Dense layers learn global patterns.<jupyter_code>network = models.Sequential()
network.add(layers.Conv2D(32, (3,3), activation='relu', input_shape=(28, 28, 1)))
network.add(layers.MaxPool2D((2, 2)))
network.add(layers.Conv2D(64, (3, 3), activation='relu'))
network.add(layers.MaxPool2D((2, 2)))
network.add(layers.Conv2D(64, (3, 3), activation='relu'))
network.summary()<jupyter_output><empty_output><jupyter_text>3.2. Classifier layer<jupyter_code>network.add(layers.Flatten())
network.add(layers.Dense(64, activation='relu'))
network.add(layers.Dense(10, activation='softmax'))
network.summary()<jupyter_output><empty_output><jupyter_text>4. Compilation Step In the compilation step we define the:- the loss function- the optimizer- the evaluation metric<jupyter_code>network.compile(optimizer='rmsprop',
               loss='categorical_crossentropy',
               metrics=['accuracy'])<jupyter_output><empty_output><jupyter_text>5. Data Preparation Before feeding the data into the network for training, we make sure it is formatted properly.  Prepare the images<jupyter_code>train_images_reshaped = train_images.reshape((60000, 28, 28, 1))
test_images_reshaped = test_images.reshape((10000, 28, 28, 1))
train_images_reshaped.shape
train_images_transformed = train_images_reshaped.astype('float32') / 255
test_images_transformed = test_images_reshaped.astype('float32') / 255<jupyter_output><empty_output><jupyter_text>Prepare the labels<jupyter_code>train_labels_categorical = to_categorical(train_labels)
test_labels_categorical = to_categorical(test_labels)
train_labels_categorical[0]<jupyter_output><empty_output><jupyter_text>6. Network summary<jupyter_code>network.summary()<jupyter_output><empty_output><jupyter_text>7. Train the network Feed the training images and labels to the network.Two additional parameters need to be supplied:- epochs: how many times the network will look at the entire dataset. - batch_size: how many images will be put through the network at one time.<jupyter_code>network.fit(train_images_transformed, train_labels_categorical, epochs=5, batch_size=128)<jupyter_output><empty_output><jupyter_text>8. Test the network Use the test set (which the network has not seen yet) to test how well the network will perform on images it has not seen yet:<jupyter_code>test_loss, test_acc = network.evaluate(test_images_transformed, test_labels_categorical)
print('test_acc; ', test_acc)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11263299498593617, 'fraction_numerical': 0.03558762382291794, 'mean_word_length': 4.883453237410072, 'pattern_counts': {'":': 0, '<': 53, '<?xml version=': 0, '>': 53, 'https://': 3, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 3, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '112764', 'license': 'MIT', 'n_tokens_mistral': 2902, 'n_tokens_neox': 2672, 'n_words': 697, 'path': 'convolutional_mnist_tensorflow_go_inference.ipynb', 'repo_name': 'rvs-fluid-it/nn-py-training-go-inference', 'size': '8187'}