<jupyter_start><jupyter_text>Implementing ANN using Lasagne/Theano packageThis code just provides a general framework for my project. I simply do the same but switch out the sklearn functions with my own.  1. Get MNIST dataBefore we start with the NN, we need to import the libraries etc. Most importantly, we need to load the training and test data that we use for the NN.<jupyter_code># Import libraries
import idx2numpy
import numpy as np
import os
import pandas as pd
import matplotlib.pyplot as plt

# Import higher level libraries (these I should not use in the final project)
import lasagne # package to build NNs
import theano # package for md-array computation
import theano.tensor as T

#Set working directory
os.chdir('/Users/Max/Documents/Studium/Master/Semester 2/Programming/Data')

# Load datasets as np.arrays
train_images = idx2numpy.convert_from_file('train-images.idx3-ubyte')
train_labels = idx2numpy.convert_from_file('train-labels.idx1-ubyte')
test_images = idx2numpy.convert_from_file('t10k-images.idx3-ubyte')
test_labels = idx2numpy.convert_from_file('t10k-labels.idx1-ubyte')

# Divide by 256 to get values between 0 and 1
train_images = train_images/np.float32(256)
test_images = test_images/np.float32(256)
# Example
plt.show(plt.imshow(train_images[2]))
train_labels[2]
# Check data format
train_images.shape<jupyter_output><empty_output><jupyter_text>2. Set-up NNWith the high-level functions, we can simply define the attributes of the NN. We need to define the following:- Number of layers (2 hidden layers)- Number of nodes (800 in each hidden layer)- Output range (between 0 and 1)- Step size (0.01)- Momentum (0.9) - not necessary for stochastic gradient descent- Drop-out rates to avoid overfitting (20%/50%)<jupyter_code># First, we write a function that builds the neural network with the attributes above
# It takes one image ('input_var') as input, i.e. an array of size 28x28
def build_NN(input_var=None):
    # Set-up input layer:
    l_in = lasagne.layers.InputLayer(shape=(None,28,28), input_var=input_var)
    l_in_drop = lasagne.layers.DropoutLayer(l_in, p=0.2) # drop 20% to avoid overfitting
    
    # Set-up first hidden layer:
    l_hid1 = lasagne.layers.DenseLayer(l_in_drop, num_units=800, 
                                       nonlinearity=lasagne.nonlinearities.rectify, 
                                       W=lasagne.init.GlorotUniform()) # just default settings here
    l_hid1_drop = lasagne.layers.DropoutLayer(l_hid1, p=0.5) # drop 50%
    
    # Set-up second hidden layer (same procedure as first hidden layer):
    l_hid2 = lasagne.layers.DenseLayer(l_hid1_drop, num_units=800, 
                                       nonlinearity=lasagne.nonlinearities.rectify, 
                                       W=lasagne.init.GlorotUniform()) # just default settings here
    l_hid2_drop = lasagne.layers.DropoutLayer(l_hid2, p=0.5) # drop 50%
    
    # Set-up output layer:
    l_out = lasagne.layers.DenseLayer(l_hid2_drop, num_units=10, 
                                      nonlinearity=lasagne.nonlinearities.softmax) #'softmax' puts numbers between 0 and 1
    return l_out
# Second, we need to tell the NN we've just set up, how to train itself (how to update the weights)
# To do that we need some place-holder input:
input_var = T.tensor3('inputs') # empty 3d-array
target_var = T.ivector('targets') # empty integer vector

# Build NN with these place-holders:
network = build_NN(input_var)

# Set-up training algorithm:
# 1. Compute loss:
prediction = lasagne.layers.get_output(network)
loss = lasagne.objectives.categorical_crossentropy(prediction, target_var) # cross entropy is standard loss function
loss = loss.mean()

# 2. Update weights (based on loss):
weights = lasagne.layers.get_all_params(network, trainable=True) # current value of all weights
updates = lasagne.updates.nesterov_momentum(loss, weights, learning_rate = 0.01, momentum = 0.9) # step at each iteration
# nesterov-momentum is a fancy version of stochastic gradient descent

# 3. Put everything together into a single training step:
train_fn = theano.function([input_var, target_var], loss, updates=updates)<jupyter_output><empty_output><jupyter_text>3. Train NNWe have defined all the necessary attributes of the NN. Now, we need to feed it with the training data.<jupyter_code>n_iter = 25 # number of training steps, should be more! (just to check)
for step in range(n_iter):
    train_err = train_fn(train_images, train_labels)
    print('Step: ' + str(step))<jupyter_output>Step: 0
Step: 1
Step: 2
Step: 3
Step: 4
Step: 5
Step: 6
Step: 7
Step: 8
Step: 9
Step: 10
Step: 11
Step: 12
Step: 13
Step: 14
Step: 15
Step: 16
Step: 17
Step: 18
Step: 19
Step: 20
Step: 21
Step: 22
Step: 23
Step: 24<jupyter_text>4. Test NNNow, that we have a (hopefully) functioning neural network, we can test it on the training data. Before that, we can look at the output of it to see if it is roughly working.<jupyter_code># Look at the ouput for one test image and see if its correct:
test_prediction = lasagne.layers.get_output(network) # function to retreive output layer from network
val_fn = theano.function([input_var], test_prediction) # function to run one image through the network and return output

val_fn([test_images[0]]) # Output for the first image in the test set; from the probabilities you can tell 
# that the NN is not really confident (highest probability for number 7, but only about 26%)
# Check accuracy over whole test set:
test_prediction = lasagne.layers.get_output(network, deterministic = True)
test_acc = T.mean(T.eq(T.argmax(test_prediction, axis=1), target_var), dtype=theano.config.floatX)
# This function matches the index of the highest value with the label and gets the mean over the complete input

acc_fn = theano.function([input_var, target_var], test_acc) # function to feed dataset

# Test NN on test set, return accuracy
acc_fn(test_images, test_labels) # 80% after 25 iterations, not bad
# could be improved by using batches of data and more iterations (and improving attributes like step size etc.)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08262295081967214, 'fraction_numerical': 0.02442622950819672, 'mean_word_length': 4.201193520886616, 'pattern_counts': {'":': 0, '<': 16, '<?xml version=': 0, '>': 16, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 5, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '658947', 'license': 'MIT', 'n_tokens_mistral': 1998, 'n_tokens_neox': 1864, 'n_words': 684, 'path': '.ipynb_checkpoints/Sklearn implementation-checkpoint.ipynb', 'repo_name': 'Mittenbuhler/Neural_Network', 'size': '6100'}