<jupyter_start><jupyter_text>Initalize a single Hidden Layer Network<jupyter_code>def initialize_netowrk(n_inputs, n_hidden, n_outputs):
    network = list()
    hiddden_layer = [{'weights':[random() for i in range(n_inputs+1)]} for i in range(n_hidden)]
    network.append(hiddden_layer)
    output_layer = [{'weights':[random() for i in range(n_hidden+1)]} for i in range(n_outputs)]
    network.append(output_layer)
    return network<jupyter_output><empty_output><jupyter_text>Test Output<jupyter_code>from random import seed
from random import random
seed(1)
network = initialize_netowrk(2,1,3)
i=1
for layer in network:
    print()
    print('Layer {}: {}'.format(i, layer))
    i= i+1<jupyter_output>Layer 1: [{'weights': [0.13436424411240122, 0.8474337369372327, 0.763774618976614]}]

Layer 2: [{'weights': [0.2550690257394217, 0.49543508709194095]}, {'weights': [0.4494910647887381, 0.651592972722763]}, {'weights': [0.7887233511355132, 0.0938595867742349]}]<jupyter_text>Forward PropGenerating Predictions that suck on purpose  Neuron Activation Function<jupyter_code>#activation = sum(weights*inputs) + bias

def activation_function(weights, inputs):
    activation = weights[-1]
    for i in range(len(weights) - 1):
        activation += weights[i] * inputs[i]
    return activation<jupyter_output><empty_output><jupyter_text>Neuron Transfer FunctionWe're going to use softmax cuz, multiclass is more fun<jupyter_code>from math import exp
import numpy as np
'''e ^ (x - max(x)) / sum(e^(x - max(x))
By using the fact that a^(b - c) = (a^b)/(a^c) we have

= e ^ x / e ^ max(x) * sum(e ^ x / e ^ max(x))

= e ^ x / sum(e ^ x)'''

#output = e ^ activation / sum(e ^ activation)

#def transfer_function(activation_value):
#    return(exp(activation_value) / sum(exp(activation_value)))
def transfer_function(activation):
    return 1 / (1 + exp(-activation))

#Relu
#def transfer_function(activation_value):
#    (abs(activation_value) + activation_value) / 2
#    activation_value * (activation_value > 0)
#    activation_value[activation_value<0]=0
#    np.maximum(activation_value, 0, activation_value)<jupyter_output><empty_output><jupyter_text>Forward PropReturn the outputs from each Neuron / Layer<jupyter_code>def forward_propagate(network, row):
    inputs = row
    for layer in network:
        next_inputs = []
        #print(layer)
        for neuron in layer:
            #print(neuron)
            activation = activation_function(neuron['weights'], inputs)
            neuron['output'] = transfer_function(activation)
            next_inputs.append(neuron['output'])
        inputs = next_inputs
    return inputs<jupyter_output><empty_output><jupyter_text>Test it out!<jupyter_code>from math import exp
#network = [[{'weights': [0.13436424411240122, 0.8474337369372327, 0.763774618976614]}],
#           [{'weights': [0.2550690257394217, 0.49543508709194095]}, {'weights': [0.4494910647887381, 0.651592972722763]}]]
row = [1, 0, None]
output = forward_propagate(network, row)
print(output)<jupyter_output>[0.6629970129852887, 0.7253160725279748, 0.6579788665256563]
{'fraction_non_alphanumeric': 0.13329019734713685, 'fraction_numerical': 0.1112908443869298, 'mean_word_length': 5.050880626223092, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 24, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '649340', 'license': 'MIT', 'n_tokens_mistral': 1309, 'n_tokens_neox': 1085, 'n_words': 230, 'path': '1.0-whs2k-1986Hinton.ipynb', 'repo_name': 'whs2k/LearningInternalRepresentationsByErrorPropagation', 'size': '3091'}