<jupyter_start><jupyter_text>Deep Learning - Hidden Layershttps://www.youtube.com/watch?v=UCG1FuKmIOs Why do we stack layers (adapted from http://stats.stackexchange.com/questions/63152/what-does-the-hidden-layer-in-a-neural-network-compute)Let's call the input vector $x$, the hidden layer activations $h$, and the output activation $y$.  You have some function $f$ that maps from $x$ to $h$ and another function $g$ that maps from $h$ to $y$.  So the hidden layer's activation is $f(x)$ and the output of the network is $g(f(x))$.**Why have two functions ($f$ and $g$) instead of just one?**If the level of complexity per function is limited, then $g(f(x))$ can compute things that $f$ and $g$ can't do individually.  ------**An example with logical functions:**For example, if we only allow $f$ and $g$ to be simple logical operators like "AND", "OR", and "NAND", then you can't compute other functions like "XOR" with just one of them.  On the other hand, we *could* compute "XOR" if we were willing to layer these functions on top of each other: First layer functions:* Make sure that at least one element is "TRUE" (using OR)* Make sure that they're not all "TRUE" (using NAND)Second layer function:* Make sure that both of the first-layer criteria are satisfied (using AND)The network's output is just the result of this second function.  The first layer *transforms the inputs* into something that the second layer can use so that the whole network can perform XOR.----**An example with images:**Slide 61 from [this talk](http://cs.nyu.edu/~fergus/tutorials/deep_learning_cvpr12/CVPR2012-Tutorial_lee.pdf) as a single image--shows (one way to visualize) what the different hidden layers in a particular neural network are looking for.![cnn](nn.png)The first layer looks for short pieces of edges in the image: these are very easy to find from raw pixel data, but they're not very useful by themselves for telling you if you're looking at a face or a bus or an elephant.The next layer composes the edges: if the edges from the bottom hidden layer fit together in a certain way, then one of the eye-detectors in the middle of left-most column might turn on.  It would be hard to make a single layer that was so good at finding something so specific from the raw pixels: eye detectors are much easier to build out of edge detectors than out of raw pixels.The next layer up composes the eye detectors and the nose detectors into faces.  In other words, these will light up when the eye detectors and nose detectors from the previous layer turn on with the right patterns.  These are very good at looking for particular kinds of faces: if one or more of them lights up, then your output layer should report that a face is present.This is useful because **face detectors are easy to build out of eye detectors and nose detectors, but really hard to build out of pixel intensities.**So each layer gets you farther and farther from the raw pixels and closer to your ultimate goal (e.g. face detection or bus detection).<jupyter_code>from IPython.display import YouTubeVideo
YouTubeVideo("UCG1FuKmIOs")
import numpy as np

y = 0.5
def loss(p,y):
    l = -(y*np.log(p)+(1-y)*np.log(1-p))
    return l

p = np.arange(1e-3,0.999,1e-3)

l = loss(p,y)
plt.plot(p,l)
plt.title('Cross Entropy Loss')
plt.show()<jupyter_output><empty_output><jupyter_text>Useful terms:1. Activation2. Softmax3. Cross Entropy4. One Hot Encoding5. Gradient Descent<jupyter_code>import tensorflow as tf
from tensorflow.examples.tutorials.mnist import input_data

import matplotlib.pyplot as plt
%matplotlib inline

mnist = input_data.read_data_sets('./', one_hot=True)
x, y = mnist.train.next_batch(20)
x.shape
28*28
y.shape
y
plt.imshow(x[1].reshape((28,28)),cmap='Greys')
plt.show()
x = tf.placeholder(tf.float32, [None, 784])
W = tf.Variable(tf.zeros([784, 10]))
b = tf.Variable(tf.zeros([10]))
y = tf.matmul(x,W) + b

y_true = tf.placeholder(tf.float32,[None, 10])

#   tf.reduce_mean(-tf.reduce_sum(y_true * tf.log(tf.nn.softmax(y)),
#                                 reduction_indices=[1]))
cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=y_true, logits=y))
train_step = tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)

sess = tf.InteractiveSession()
tf.initialize_all_variables().run()

# Training
for i in range(1000):
    batch_xs, batch_ys = mnist.train.next_batch(128)
    l,_ = sess.run([cross_entropy, train_step], feed_dict={x: batch_xs, y_true: batch_ys})
    if i%100 == 0:
        print('loss: '+str(l))<jupyter_output>WARNING:tensorflow:From <ipython-input-24-08f9e66bca55>:14: initialize_all_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2017-03-02.
Instructions for updating:
Use `tf.global_variables_initializer` instead.
loss: 2.30258
loss: 0.389749
loss: 0.485583
loss: 0.222426
loss: 0.302407
loss: 0.286258
loss: 0.24921
loss: 0.282081
loss: 0.29155
loss: 0.299976<jupyter_text>Softmax takes a $D$ dimensional vector and squeezes them through a function such that we have $D$ outputs whos values are positive and sums to one.$$\text{softmax}(\mathbf{y})_d = \frac{\exp(-y_d)}{\exp(-y_1)+...+\exp(-y_D)}$$<jupyter_code>[-2, 1]
exp(-2)/(exp(-2)+exp(1))
# test the model
correct_prediction = tf.equal(tf.argmax(y,1), tf.argmax(y_true,1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))
print(sess.run(accuracy, feed_dict={x: mnist.test.images, y_true: mnist.test.labels}))
print(sess.run(accuracy, feed_dict={x: mnist.train.images, y_true: mnist.train.labels}))
sess = tf.InteractiveSession()
tf.initialize_all_variables().run()

# Training
for i in range(1000):
    batch_xs, batch_ys = mnist.train.next_batch(128)
    l,_,a = sess.run([cross_entropy, train_step, accuracy], feed_dict={x: batch_xs, y_true: batch_ys})
    if i%100 == 0 or (i<100 and i%10==0):
        print(str(i)+': loss: '+str(l)+' accuracy: '+str(a))<jupyter_output>0: loss: 2.30258 accuracy: 0.0859375
10: loss: 0.924381 accuracy: 0.742188
20: loss: 0.679864 accuracy: 0.84375
30: loss: 0.598077 accuracy: 0.789062
40: loss: 0.398598 accuracy: 0.9375
50: loss: 0.473592 accuracy: 0.828125
60: loss: 0.571691 accuracy: 0.8125
70: loss: 0.475025 accuracy: 0.875
80: loss: 0.380094 accuracy: 0.929688
90: loss: 0.232757 accuracy: 0.96875
100: loss: 0.496833 accuracy: 0.851562
200: loss: 0.412172 accuracy: 0.867188
300: loss: 0.260597 accuracy: 0.9375
400: loss: 0.247081 accuracy: 0.929688
500: loss: 0.294148 accuracy: 0.929688
600: loss: 0.251191 accuracy: 0.914062
700: loss: 0.172335 accuracy: 0.945312
800: loss: 0.333997 accuracy: 0.914062
900: loss: 0.207521 accuracy: 0.953125<jupyter_text>Hidden Layered Network<jupyter_code>def fully_conntected_activation(input,size_in,size_out):
    # Initialise variables
    W = tf.Variable(tf.truncated_normal([size_in, size_out],stddev=0.1))
    b = tf.Variable(tf.truncated_normal([size_out], stddev=0.1))
    # NN part
    activation = tf.nn.relu(tf.matmul(input,W)+b)
    
    return activation

def fully_conntected(input,size_in,size_out):
    # Initialise variables
    W = tf.Variable(tf.truncated_normal([size_in, size_out],stddev=0.1))
    b = tf.Variable(tf.truncated_normal([size_out], stddev=0.1))
    # NN part
    activation = tf.matmul(input,W)+b
    
    return activation
x = tf.placeholder(tf.float32, [None, 784])
y_true = tf.placeholder(tf.float32,[None, 10])

hidden = fully_conntected_activation(x,784,100)
output = fully_conntected(hidden,100,10)
# output = fully_conntected(x,784,10)

cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(labels=y_true, logits=output))
train_step = tf.train.GradientDescentOptimizer(0.5).minimize(cross_entropy)

# test the model
correct_prediction = tf.equal(tf.argmax(output,1), tf.argmax(y_true,1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

sess = tf.InteractiveSession()
tf.initialize_all_variables().run()

# Training
for i in range(1000):
    batch_xs, batch_ys = mnist.train.next_batch(128)
    l,_,a = sess.run([cross_entropy, train_step, accuracy], 
                     feed_dict={x: batch_xs, y_true: batch_ys})
    if i%100 == 0 or (i<100 and i%10==0):
        print(str(i)+': loss: '+str(l)+' accuracy: '+str(a))
print(sess.run(accuracy, feed_dict={x: mnist.test.images, y_true: mnist.test.labels}))<jupyter_output>0.9665
{'fraction_non_alphanumeric': 0.12244165170556554, 'fraction_numerical': 0.06558946738479952, 'mean_word_length': 4.876230661040788, 'pattern_counts': {'":': 0, '<': 17, '<?xml version=': 0, '>': 15, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '57482', 'license': 'Apache-2.0', 'n_tokens_mistral': 3255, 'n_tokens_neox': 2875, 'n_words': 888, 'path': 'DL-Keras_Tensorflow/Lesson 04 - NeuralNets.ipynb', 'repo_name': 'eagleanurag/deepschool.io', 'size': '8355'}