<jupyter_start><jupyter_text>Data Analysis and Machine Learning: Elements of machine learning  **<NAME>**, Department of Physics, University of Oslo and Department of Physics and Astronomy and National Superconducting Cyclotron Laboratory, Michigan State UniversityDate: **Sep 28, 2018**Copyright 1999-2018, <NAME>. Released under CC Attribution-NonCommercial 4.0 license Neural networksArtificial neural networks are computational systems that can learn toperform tasks by considering examples, generally without beingprogrammed with any task-specific rules. It is supposed to mimic abiological system, wherein neurons interact by sending signals in theform of mathematical functions between layers. All layers can containan arbitrary number of neurons, and each connection is represented bya weight variable. Artificial neuronsThe field of artificial neural networks has a long history ofdevelopment, and is closely connected with the advancement of computerscience and computers in general. A model of artificial neurons wasfirst developed by McCulloch and Pitts in 1943 to study signalprocessing in the brain and has later been refined by others. Thegeneral idea is to mimic neural networks in the human brain, which iscomposed of billions of neurons that communicate with each other bysending electrical signals.  Each neuron accumulates its incomingsignals, which must exceed an activation threshold to yield anoutput. If the threshold is not overcome, the neuron remains inactive,i.e. has zero output.This behaviour has inspired a simple mathematical model for an artificial neuron. $$\begin{equation} y = f\left(\sum_{i=1}^n w_ix_i\right) = f(u)\label{artificialNeuron} \tag{1}\end{equation}$$ Here, the output $y$ of the neuron is the value of its activation function, which have as inputa weighted sum of signals $x_i, \dots ,x_n$ received by $n$ other neurons.Conceptually, it is helpful to divide neural networks into fourcategories:1. general purpose neural networks for supervised learning,2. neural networks designed specifically for image processing, the most prominent example of this class being Convolutional Neural Networks (CNNs),3. neural networks for sequential data such as Recurrent Neural Networks (RNNs), and4. neural networks for unsupervised learning such as Deep Boltzmann Machines.In natural science, DNNs and CNNs have already found numerous applications. Instatistical physics, they have been applied to detect phasetransitions in 2D Ising and Potts models, lattice gauge theories, anddifferent phases of polymers, or solving the Navier-Stokes equation in weather forecasting.Deep learning has also found interesting applications in quantumphysics. Various quantum phase transitions can be detected and studiedusing DNNs and CNNs,topological phases, and even non-equilibrium many-bodylocalization. Representing quantum states as DNNs quantum statetomography are among some of the impressiveachievements to reveal the potential of DNNs to facilitate the studyof quantum systems.In quantum information theory, it has been shown that one can performgate decompositions with the help of neural. In lattice quantum chromodynamics,DNNs have been used to learn action parameters in regions of parameterspace where PCA fails. The applications are not limited to the natural sciences. There is a plethora of applications in essentially all disciplines, from the humanities to life science and medicine. Neural network typesAn artificial neural network (NN), is a computational model thatconsists of layers of connected neurons, or *nodes*.  It is supposedto mimic a biological nervous system by letting each neuron interactwith other neurons by sending signals in the form of mathematicalfunctions between layers.  A wide variety of different NNs have beendeveloped, but most of them consist of an input layer, an output layerand eventual layers in-between, called *hidden layers*. All layers cancontain an arbitrary number of nodes, and each connection between twonodes is associated with a weight variable.Neural networks (also called neural nets) are neural-inspirednonlinear models for supervised learning.  As we will see, neural netscan be viewed as natural, more powerful extensions of supervisedlearning methods such as linear and logistic regression and soft-maxmethods. Feed-forward neural networksThe feed-forward neural network (FFNN) was the first and simplest type of NN devised. In this network, the information moves in only one direction: forward through the layers.Nodes are represented by circles, while the arrows display the connections between the nodes, including the direction of information flow. Additionally, each arrow corresponds to a weight variable, not displayed here. We observe that each node in a layer is connected to *all* nodes in the subsequent layer, making this a so-called *fully-connected* FFNN. A different variant of FFNNs are *convolutional neural networks* (CNNs), which have a connectivity patterninspired by the animal visual cortex. Individual neurons in the visual cortex only respond to stimuli fromsmall sub-regions of the visual field, called a receptive field. This makes the neurons well-suited to exploit the strongspatially local correlation present in natural images. The response of each neuron can be approximated mathematically as a convolution operation. CNNs emulate the behaviour of neurons in the visual cortex by enforcing a *local* connectivity patternbetween nodes of adjacent layers: Each nodein a convolutional layer is connected only to a subset of the nodes in the previous layer, in contrast to the fully-connected FFNN.Often, CNNs consist of several convolutional layers that learn local features of the input, with a fully-connected layer at the end, which gathers all the local data and produces the outputs. They have wide applications in image and video recognition Recurrent neural networksSo far we have only mentioned NNs where information flows in one direction: forward. *Recurrent neural networks* onthe other hand, have connections between nodes that form directed *cycles*. This creates a form of internal memory which are able to capture information on what has been calculated before; the output is dependent on the previous computations. Recurrent NNs make use of sequential information by performing the same task for every element in a sequence, where each element depends on previous elements. An example of such information is sentences, making recurrent NNs especially well-suited for handwriting and speech recognition. Other types of networksThere are many other kinds of NNs that have been developed. One type that is specifically designed for interpolationin multidimensional space is the radial basis function (RBF) network. RBFs are typically made up of three layers: an input layer, a hidden layer with non-linear radial symmetric activation functions and a linear output layer (''linear'' heremeans that each node in the output layer has a linear activation function). The layers are normally fully-connected and there are no cycles, thus RBFs can be viewed as a type of fully-connected FFNN. They are however usually treated asa separate type of NN due the unusual activation functions. Multilayer perceptronsOne uses often so-called fully-connected feed-forward neural networkswith three or more layers (an input layer, one or more hidden layersand an output layer) consisting of neurons that have non-linearactivation functions.Such networks are often called *multilayer perceptrons* (MLPs) Why multilayer perceptrons?According to the *Universal approximation theorem*, a feed-forward neural network with just a single hidden layer containing a finite number of neurons can approximate a continuous multidimensional function to arbitrary accuracy, assuming the activation function for the hidden layer is a **non-constant, bounded and monotonically-increasing continuous function**.Note that the requirements on the activation function only applies tothe hidden layer, the output nodes are always assumed to be linear, soas to not restrict the range of output values.We note that this theorem is only applicable to an NN with *one* hiddenlayer.  Therefore, we can easily construct an NN that employsactivation functions which do not satisfy the above requirements, aslong as we have at least one layer with activation functions that*do*. Furthermore, although the universal approximation theorem laysthe theoretical foundation for regression with neural networks, itdoes not say anything about how things work in practice: A neuralnetwork can still be able to approximate a given function reasonablywell without having the flexibility to fit *all other* functions. Mathematical model $$\begin{equation} y = f\left(\sum_{i=1}^n w_ix_i + b_i\right) = f(u)\label{artificialNeuron2} \tag{2}\end{equation}$$ In an FFNN of such neurons, the *inputs* $x_i$ are the *outputs* ofthe neurons in the preceding layer. Furthermore, an MLP isfully-connected, which means that each neuron receives a weighted sumof the outputs of *all* neurons in the previous layer. Mathematical modelFirst, for each node $i$ in the first hidden layer, we calculate a weighted sum $u_i^1$ of the input coordinates $x_j$, $$\begin{equation} u_i^1 = \sum_{j=1}^2 w_{ij}^1 x_j  + b_i^1 \label{_auto1} \tag{3}\end{equation}$$ This value is the argument to the activation function $f_1$ of each neuron $i$,producing the output $y_i^1$ of all neurons in layer 1, $$\begin{equation} y_i^1 = f_1(u_i^1) = f_1\left(\sum_{j=1}^2 w_{ij}^1 x_j  + b_i^1\right)\label{outputLayer1} \tag{4}\end{equation}$$ where we assume that all nodes in the same layer have identicalactivation functions, hence the notation $f_l$ $$\begin{equation} y_i^l = f_l(u_i^l) = f_l\left(\sum_{j=1}^{N_{l-1}} w_{ij}^l y_j^{l-1} + b_i^l\right)\label{generalLayer} \tag{5}\end{equation}$$ where $N_l$ is the number of nodes in layer $l$. When the output ofall the nodes in the first hidden layer are computed, the values ofthe subsequent layer can be calculated and so forth until the outputis obtained. Mathematical modelThe output of neuron $i$ in layer 2 is thus, $$\begin{equation} y_i^2 = f_2\left(\sum_{j=1}^3 w_{ij}^2 y_j^1 + b_i^2\right) \label{_auto2} \tag{6}\end{equation}$$ $$\begin{equation}  = f_2\left[\sum_{j=1}^3 w_{ij}^2f_1\left(\sum_{k=1}^2 w_{jk}^1 x_k + b_j^1\right) + b_i^2\right]\label{outputLayer2} \tag{7}\end{equation}$$ where we have substituted $y_m^1$ with. Finally, the NN output yields, $$\begin{equation} y_1^3 = f_3\left(\sum_{j=1}^3 w_{1m}^3 y_j^2 + b_1^3\right) \label{_auto3} \tag{8}\end{equation}$$ $$\begin{equation}  = f_3\left[\sum_{j=1}^3 w_{1j}^3 f_2\left(\sum_{k=1}^3 w_{jk}^2 f_1\left(\sum_{m=1}^2 w_{km}^1 x_m + b_k^1\right) + b_j^2\right)  + b_1^3\right]\label{_auto4} \tag{9}\end{equation}$$  Mathematical modelWe can generalize this expression to an MLP with $l$ hiddenlayers. The complete functional form is, $$\begin{equation}y^{l+1}_1\! = \!f_{l+1}\!\left[\!\sum_{j=1}^{N_l}\! w_{1j}^3 f_l\!\left(\!\sum_{k=1}^{N_{l-1}}\! w_{jk}^2 f_{l-1}\!\left(\! \dots \!f_1\!\left(\!\sum_{n=1}^{N_0} \!w_{mn}^1 x_n\! + \!b_m^1\!\right) \!\dots \!\right) \!+ \!b_k^2\!\right) \!+ \!b_1^3\!\right] \label{completeNN} \tag{10}\end{equation}$$ which illustrates a basic property of MLPs: The only independentvariables are the input values $x_n$. Mathematical modelThis confirms that an MLP, despite its quite convoluted mathematicalform, is nothing more than an analytic function, specifically amapping of real-valued vectors $\vec{x} \in \mathbb{R}^n \rightarrow\vec{y} \in \mathbb{R}^m$.  In our example, $n=2$ and$m=1$. Consequentially, the number of input and output values of thefunction we want to fit must be equal to the number of inputs andoutputs of our MLP.Furthermore, the flexibility and universality of a MLP can beillustrated by realizing that the expression is essentially a nestedsum of scaled activation functions of the form $$\begin{equation} h(x) = c_1 f(c_2 x + c_3) + c_4\label{_auto5} \tag{11}\end{equation}$$ where the parameters $c_i$ are weights and biases. By adjusting theseparameters, the activation functions can be shifted up and down orleft and right, change slope or be rescaled which is the key to theflexibility of a neural network. Matrix-vector notationWe can introduce a more convenient notation for the activations in a NN. Additionally, we can represent the biases and activationsas layer-wise column vectors $\vec{b}_l$ and $\vec{y}_l$, so that the $i$-th element of each vector is the bias $b_i^l$ and activation $y_i^l$ of node $i$ in layer $l$ respectively. We have that $\mathrm{W}_l$ is a $N_{l-1} \times N_l$ matrix, while $\vec{b}_l$ and $\vec{y}_l$ are $N_l \times 1$ column vectors. With this notation, the sum in  becomes a matrix-vector multiplication, and we can writethe equation for the activations of hidden layer 2 in $$\begin{equation} \vec{y}_2 = f_2(\mathrm{W}_2 \vec{y}_{1} + \vec{b}_{2}) =  f_2\left(\left[\begin{array}{ccc}    w^2_{11} &w^2_{12} &w^2_{13} \\    w^2_{21} &w^2_{22} &w^2_{23} \\    w^2_{31} &w^2_{32} &w^2_{33} \\    \end{array} \right] \cdot    \left[\begin{array}{c}           y^1_1 \\           y^1_2 \\           y^1_3 \\          \end{array}\right] +     \left[\begin{array}{c}           b^2_1 \\           b^2_2 \\           b^2_3 \\          \end{array}\right]\right).\label{_auto6} \tag{12}\end{equation}$$  Matrix-vector notation  and activationThe activation of node $i$ in layer 2 is $$\begin{equation} y^2_i = f_2\Bigr(w^2_{i1}y^1_1 + w^2_{i2}y^1_2 + w^2_{i3}y^1_3 + b^2_i\Bigr) =  f_2\left(\sum_{j=1}^3 w^2_{ij} y_j^1 + b^2_i\right).\label{_auto7} \tag{13}\end{equation}$$ This is not just a convenient and compact notation, but also a usefuland intuitive way to think about MLPs: The output is calculated by aseries of matrix-vector multiplications and vector additions that areused as input to the activation functions. For each operation$\mathrm{W}_l \vec{y}_{l-1}$ we move forward one layer. Activation functionsA property that characterizes a neural network, other than itsconnectivity, is the choice of activation function(s).  As describedin, the following restrictions are imposed on an activation functionfor a FFNN to fulfill the universal approximation theorem  * Non-constant  * Bounded  * Monotonically-increasing  * Continuous Activation functions, Logistic and Hyperbolic onesThe second requirement excludes all linear functions. Furthermore, ina MLP with only linear activation functions, each layer simplyperforms a linear transformation of its inputs.Regardless of the number of layers, the output of the NN will benothing but a linear function of the inputs. Thus we need to introducesome kind of non-linearity to the NN to be able to fit non-linearfunctions Typical examples are the logistic *Sigmoid* $$\begin{equation} f(x) = \frac{1}{1 + e^{-x}},\label{sigmoidActivationFunction} \tag{14}\end{equation}$$ and the *hyperbolic tangent* function $$\begin{equation} f(x) = \tanh(x)\label{tanhActivationFunction} \tag{15}\end{equation}$$  RelevanceThe *sigmoid* function are more biologically plausible because theoutput of inactive neurons are zero. Such activation function arecalled *one-sided*. However, it has been shown that the hyperbolictangent performs better than the sigmoid for training MLPs.  hasbecome the most popular for *deep neural networks*<jupyter_code>%matplotlib inline

"""The sigmoid function (or the logistic curve) is a 
function that takes any real number, z, and outputs a number (0,1).
It is useful in neural networks for assigning weights on a relative scale.
The value z is the weighted sum of parameters involved in the learning algorithm."""

import numpy
import matplotlib.pyplot as plt
import math as mt

z = numpy.arange(-5, 5, .1)
sigma_fn = numpy.vectorize(lambda z: 1/(1+numpy.exp(-z)))
sigma = sigma_fn(z)

fig = plt.figure()
ax = fig.add_subplot(111)
ax.plot(z, sigma)
ax.set_ylim([-0.1, 1.1])
ax.set_xlim([-5,5])
ax.grid(True)
ax.set_xlabel('z')
ax.set_title('sigmoid function')

plt.show()

"""Step Function"""
z = numpy.arange(-5, 5, .02)
step_fn = numpy.vectorize(lambda z: 1.0 if z >= 0.0 else 0.0)
step = step_fn(z)

fig = plt.figure()
ax = fig.add_subplot(111)
ax.plot(z, step)
ax.set_ylim([-0.5, 1.5])
ax.set_xlim([-5,5])
ax.grid(True)
ax.set_xlabel('z')
ax.set_title('step function')

plt.show()

"""Sine Function"""
z = numpy.arange(-2*mt.pi, 2*mt.pi, 0.1)
t = numpy.sin(z)

fig = plt.figure()
ax = fig.add_subplot(111)
ax.plot(z, t)
ax.set_ylim([-1.0, 1.0])
ax.set_xlim([-2*mt.pi,2*mt.pi])
ax.grid(True)
ax.set_xlabel('z')
ax.set_title('sine function')

plt.show()

"""Plots a graph of the squashing function used by a rectified linear
unit"""
z = numpy.arange(-2, 2, .1)
zero = numpy.zeros(len(z))
y = numpy.max([zero, z], axis=0)

fig = plt.figure()
ax = fig.add_subplot(111)
ax.plot(z, y)
ax.set_ylim([-2.0, 2.0])
ax.set_xlim([-2.0, 2.0])
ax.grid(True)
ax.set_xlabel('z')
ax.set_title('Rectified linear unit')

plt.show()<jupyter_output><empty_output><jupyter_text>Setting up a Multi-layer perceptron model<jupyter_code>from scipy import optimize

class Neural_Network(object):
    def __init__(self, Lambda=0):        
        #Define Hyperparameters
        self.inputLayerSize = 2
        self.outputLayerSize = 1
        self.hiddenLayerSize = 3
        
        #Weights (parameters)
        self.W1 = np.random.randn(self.inputLayerSize,self.hiddenLayerSize)
        self.W2 = np.random.randn(self.hiddenLayerSize,self.outputLayerSize)
        
        #Regularization Parameter:
        self.Lambda = Lambda
        
    def forward(self, X):
        #Propogate inputs though network
        self.z2 = np.dot(X, self.W1)
        self.a2 = self.sigmoid(self.z2)
        self.z3 = np.dot(self.a2, self.W2)
        yHat = self.sigmoid(self.z3) 
        return yHat
        
    def sigmoid(self, z):
        #Apply sigmoid activation function to scalar, vector, or matrix
        return 1/(1+np.exp(-z))
    
    def sigmoidPrime(self,z):
        #Gradient of sigmoid
        return np.exp(-z)/((1+np.exp(-z))**2)
    
    def costFunction(self, X, y):
        #Compute cost for given X,y, use weights already stored in class.
        self.yHat = self.forward(X)
        J = 0.5*sum((y-self.yHat)**2)/X.shape[0] + (self.Lambda/2)*(np.sum(self.W1**2)+np.sum(self.W2**2))
        return J
        
    def costFunctionPrime(self, X, y):
        #Compute derivative with respect to W and W2 for a given X and y:
        self.yHat = self.forward(X)
        
        delta3 = np.multiply(-(y-self.yHat), self.sigmoidPrime(self.z3))
        #Add gradient of regularization term:
        dJdW2 = np.dot(self.a2.T, delta3)/X.shape[0] + self.Lambda*self.W2
        
        delta2 = np.dot(delta3, self.W2.T)*self.sigmoidPrime(self.z2)
        #Add gradient of regularization term:
        dJdW1 = np.dot(X.T, delta2)/X.shape[0] + self.Lambda*self.W1
        
        return dJdW1, dJdW2
    
    #Helper functions for interacting with other methods/classes
    def getParams(self):
        #Get W1 and W2 Rolled into vector:
        params = np.concatenate((self.W1.ravel(), self.W2.ravel()))
        return params
    
    def setParams(self, params):
        #Set W1 and W2 using single parameter vector:
        W1_start = 0
        W1_end = self.hiddenLayerSize*self.inputLayerSize
        self.W1 = np.reshape(params[W1_start:W1_end], \
                             (self.inputLayerSize, self.hiddenLayerSize))
        W2_end = W1_end + self.hiddenLayerSize*self.outputLayerSize
        self.W2 = np.reshape(params[W1_end:W2_end], \
                             (self.hiddenLayerSize, self.outputLayerSize))
        
    def computeGradients(self, X, y):
        dJdW1, dJdW2 = self.costFunctionPrime(X, y)
        return np.concatenate((dJdW1.ravel(), dJdW2.ravel()))
    
        
class trainer(object):
    def __init__(self, N):
        #Make Local reference to network:
        self.N = N
        
    def callbackF(self, params):
        self.N.setParams(params)
        self.J.append(self.N.costFunction(self.X, self.y))
        self.testJ.append(self.N.costFunction(self.testX, self.testY))
        
    def costFunctionWrapper(self, params, X, y):
        self.N.setParams(params)
        cost = self.N.costFunction(X, y)
        grad = self.N.computeGradients(X,y)
        return cost, grad
        
    def train(self, trainX, trainY, testX, testY):
        #Make an internal variable for the callback function:
        self.X = trainX
        self.y = trainY
        
        self.testX = testX
        self.testY = testY

        #Make empty list to store training costs:
        self.J = []
        self.testJ = []
        
        params0 = self.N.getParams()

        options = {'maxiter': 200, 'disp' : True}
        _res = optimize.minimize(self.costFunctionWrapper, params0, jac=True, method='BFGS', \
                                 args=(trainX, trainY), options=options, callback=self.callbackF)

        self.N.setParams(_res.x)
        self.optimizationResults = _res<jupyter_output><empty_output><jupyter_text>Two-layer Neural Network<jupyter_code>import numpy as np

#sigmoid
def nonlin(x, deriv=False):
    if (deriv==True):
        return x*(1-x)
    return 1/(1+np.exp(-x))

#input data
x=np.array([[0,0,1],[0,1,1],[1,0,1],[1,1,1]])

#output data
y=np.array([0,1,1,0]).T

#seed random numbers to make calculation
np.random.seed(1)

#initialize weights with mean=0
syn0=2*np.random.random((3,4))-1

for iter in range(10000):
    #forward propogation
    l0=x
    l1=nonlin(np.dot(l0,syn0))
    l1_error=y-l1
    #multiply error by slope of sigmoid at values of l1
    l1_delta=l1_error*nonlin(l1,True)
    #update weights
    syn0+=np.dot(l0.T, l1_delta)
    
print("Output after training: ",l1 )
import numpy as np
import random
class Network(object):
    
    def _init_(self, sizes):
        self.num_layers=len(sizes)
        self.sizes=sizes
        self.biases=[np.random.randn(y,1) for y in sizes[1:]]
        self.weights=[np.random.randn(y,x) for x,y in zip(sizes[:-1], sizes[1:])]

#sizes is the number of neurons in each layer
#for example, say n_1st_layer=3, n_2nd_layer=3, n_3rd_layer=1, then net=Network([3,3,1])

#The biases and weights are initialized randomly, using Gaussian distributions of mean=0, stdev=1
#z is a vector (or a np.array)

    def feedforward(self,a):
        #returns output w/ 'a' as an input
        for b, w in zip(self.biases, self.weights):
            a=sigmoid(np.dot(w,b)+b)
        return a
    
#Apply a Stochastic Gradient Descent (SGD) method:
    def SGD(self, training_data, epochs, mini_batch_size, eta, test_data=None):
        """Trains network using batches incorporating SGD. The network will be evaluated against the
        test data after each epoch, with partial progress being printed out (this is useful for tracking,
        but slows the process.)"""
        if test_data: n_test=len(test_data)
        n=len(training_data)
        for j in xrange(epochs):
            random.shuffle(training_data)
            mini_batches=[training_data[k:k+mini_batch_size] for k in xrange(o,n,mini_batch_size)]
            for mini_batch in mini_batches:
                self.update_mini_batch(mini_batch, eta)
            if test_data:
                print ("Epoch {0}: {1}/{2}".format(j, self.evaluate(test_data), n_test))
            else:
                print ("Epoch {0} complete".format(j))
            
        
    def update_mini_batch(self, mini_batch, eta):
        #updates w and b using backpropagation to a single mini batch. eta is the learning rate."
        nabla_b=[np.zeros(b.shape) for b in self.biases]
        nabla_w=[np.zeros(w.shape) for w in self.weights]
        for x,y in mini_batch:
            delta_nabla_b, delta_nabla_w=self.backprop(x,y)
            nabla_b=[nb+dnb for nb, dnb in zip(nabla_b, delta_nabla_b)]
            nabla_w=[nw+dnw for nw, dnw in zip(nabla_w, delta_nabla_w)]
        self.weights=[w-(eta/len(mini_batch))*nw for w, nw in zip(self.weights, nabla_w)]
        self.biases=[b-(eta/len(mini_batch))*nb for b, nb in zip(self.biases, nabla_b)]
        
    def backprop(self, x, y):
        """Return a tuple ``(nabla_b, nabla_w)`` representing the
        gradient for the cost function C_x.  ``nabla_b`` and
        ``nabla_w`` are layer-by-layer lists of numpy arrays, similar
        to ``self.biases`` and ``self.weights``."""
        nabla_b = [np.zeros(b.shape) for b in self.biases]
        nabla_w = [np.zeros(w.shape) for w in self.weights]
        # feedforward
        activation = x
        activations = [x] # list to store all the activations, layer by layer
        zs = [] # list to store all the z vectors, layer by layer
        for b, w in zip(self.biases, self.weights):
            z = np.dot(w, activation)+b
            zs.append(z)
            activation = sigmoid(z)
            activations.append(activation)
        # backward pass
        delta = self.cost_derivative(activations[-1], y) * \
            sigmoid_prime(zs[-1])
        nabla_b[-1] = delta
        nabla_w[-1] = np.dot(delta, activations[-2].transpose())
        # Note that the variable l in the loop below is used a little
        # differently to the notation in Chapter 2 of the book.  Here,
        # l = 1 means the last layer of neurons, l = 2 is the
        # second-last layer, and so on.  It's a renumbering of the
        # scheme in the book, used here to take advantage of the fact
        # that Python can use negative indices in lists.
        for l in xrange(2, self.num_layers):
            z = zs[-l]
            sp = sigmoid_prime(z)
            delta = np.dot(self.weights[-l+1].transpose(), delta) * sp
            nabla_b[-l] = delta
            nabla_w[-l] = np.dot(delta, activations[-l-1].transpose())
        return (nabla_b, nabla_w)

    def evaluate(self, test_data):
        """Return the number of test inputs for which the neural
        network outputs the correct result. Note that the neural
        network's output is assumed to be the index of whichever
        neuron in the final layer has the highest activation."""
        test_results = [(np.argmax(self.feedforward(x)), y)
                        for (x, y) in test_data]
        return sum(int(x == y) for (x, y) in test_results)

    def cost_derivative(self, output_activations, y):
        """Return the vector of partial derivatives \partial C_x /
        \partial a for the output activations."""
        return (output_activations-y)
    
    
    
#Functions
def sigmoid(z):
        return 1.0/(1.0+np.exp(-z))

def sigmoid_prime(z):
    return sigmoid(z)*(1-sigmoid(z))

network=Network()
# %load neural-networks-and-deep-learning/src/mnist_loader.py
"""
mnist_loader
~~~~~~~~~~~~

A library to load the MNIST image data.  For details of the data
structures that are returned, see the doc strings for ``load_data``
and ``load_data_wrapper``.  In practice, ``load_data_wrapper`` is the
function usually called by our neural network code.
"""

#### Libraries
# Standard library
import pickle
import gzip

# Third-party libraries
import numpy as np

def load_data():
    """Return the MNIST data as a tuple containing the training data,
    the validation data, and the test data.

    The ``training_data`` is returned as a tuple with two entries.
    The first entry contains the actual training images.  This is a
    numpy ndarray with 50,000 entries.  Each entry is, in turn, a
    numpy ndarray with 784 values, representing the 28 * 28 = 784
    pixels in a single MNIST image.

    The second entry in the ``training_data`` tuple is a numpy ndarray
    containing 50,000 entries.  Those entries are just the digit
    values (0...9) for the corresponding images contained in the first
    entry of the tuple.

    The ``validation_data`` and ``test_data`` are similar, except
    each contains only 10,000 images.

    This is a nice data format, but for use in neural networks it's
    helpful to modify the format of the ``training_data`` a little.
    That's done in the wrapper function ``load_data_wrapper()``, see
    below.
    """
    f = gzip.open('../data/mnist.pkl.gz', 'rb')
    training_data, validation_data, test_data = cPickle.load(f)
    f.close()
    return (training_data, validation_data, test_data)

def load_data_wrapper():
    """Return a tuple containing ``(training_data, validation_data,
    test_data)``. Based on ``load_data``, but the format is more
    convenient for use in our implementation of neural networks.

    In particular, ``training_data`` is a list containing 50,000
    2-tuples ``(x, y)``.  ``x`` is a 784-dimensional numpy.ndarray
    containing the input image.  ``y`` is a 10-dimensional
    numpy.ndarray representing the unit vector corresponding to the
    correct digit for ``x``.

    ``validation_data`` and ``test_data`` are lists containing 10,000
    2-tuples ``(x, y)``.  In each case, ``x`` is a 784-dimensional
    numpy.ndarry containing the input image, and ``y`` is the
    corresponding classification, i.e., the digit values (integers)
    corresponding to ``x``.

    Obviously, this means we're using slightly different formats for
    the training data and the validation / test data.  These formats
    turn out to be the most convenient for use in our neural network
    code."""
    tr_d, va_d, te_d = load_data()
    training_inputs = [np.reshape(x, (784, 1)) for x in tr_d[0]]
    training_results = [vectorized_result(y) for y in tr_d[1]]
    training_data = zip(training_inputs, training_results)
    validation_inputs = [np.reshape(x, (784, 1)) for x in va_d[0]]
    validation_data = zip(validation_inputs, va_d[1])
    test_inputs = [np.reshape(x, (784, 1)) for x in te_d[0]]
    test_data = zip(test_inputs, te_d[1])
    return (training_data, validation_data, test_data)

def vectorized_result(j):
    """Return a 10-dimensional unit vector with a 1.0 in the jth
    position and zeroes elsewhere.  This is used to convert a digit
    (0...9) into a corresponding desired output from the neural
    network."""
    e = np.zeros((10, 1))
    e[j] = 1.0
    return e

net=network.Network([784,30,30])
net.SGD(training_data,30,10,3,test_data=test_data)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09853893957424302, 'fraction_numerical': 0.0172881131825995, 'mean_word_length': 3.6962123564110523, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 16, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '352758', 'license': 'CC0-1.0', 'n_tokens_mistral': 9373, 'n_tokens_neox': 8907, 'n_words': 3437, 'path': 'doc/pub/NeuralNet/ipynb/.ipynb_checkpoints/NeuralNet-checkpoint.ipynb', 'repo_name': 'anacost/MachineLearning', 'size': '30280'}