<jupyter_start><jupyter_text>Artificial Neural Network - Multi-layered perceptronXOR function example<jupyter_code>#Artificial Neural Network Example
# XOR function
import math;
import numpy as np#for array/matrix calculations
import warnings
warnings.filterwarnings("ignore", category=np.VisibleDeprecationWarning);#ignore the warning<jupyter_output><empty_output><jupyter_text>Sigmoid Function$$f(x)=\frac{1}{1+e^{-\alpha x}}$$$$f'(x)=\alpha f(x)(1-f(x))$$<jupyter_code>aconst=2;
def sigmoid(x):
    return 1/(1+math.exp(-aconst*(x)));
def dsigmoid(x):
    sigx=sigmoid(x);    
    return aconst*sigx*(1-sigx);<jupyter_output><empty_output><jupyter_text>Error: $$e_{k}=d_{k}-y_{k}$$ where $d_k$ is the desired output, and $y_k$ is the output of the network$$total\space energy\space ξ=\frac{1}{2}\sum_{k}e_{k}^{2}(n)$$$$\sum_{k}e_{k}^{2}(n)=e_{k}\cdot e_{k}$$as $$a\cdot b=\sum_{i=1}^{n}=a_1b_1+a_2b_2+...+a_nb_n$$<jupyter_code>def sq_error(output,desired):#square error
    dif = output.ravel() - desired.ravel() #find the difference of each value in the array
    return np.dot( dif, dif )#dot product
def totalerr_energy(output,desired):
    return sq_error(output,desired)/2;<jupyter_output><empty_output><jupyter_text>Feed forward$v_k=\sum_{k=0}^{m}w_{km}x_k$$w_k0=b_k$ which is the bias$y_k=φ(v_k)$where $φ(v_k)=\frac{1}{1+exp^{-\alpha x}}$ the sigmoid function<jupyter_code>def feedforward(inp,weights):
    prev_layer=inp;
    prev_layer=np.append(prev_layer,1);#add the bias
    curlayers=[prev_layer];
    bcurlayers=[prev_layer];#current neuron states (before activation)
    nolayers=len(weights);
    for l in range(nolayers):
        no_neurons=weights[l].shape[0]
        thislayer=np.zeros(no_neurons);
        bthislayer=np.zeros(no_neurons);#before activation
        for i in range(no_neurons):
            bthislayer[i]=np.dot(prev_layer,weights[l][i]);#(before activation)
            thislayer[i]=sigmoid(bthislayer[i]);
        prev_layer=thislayer;
        curlayers.append(np.array(thislayer));
        bcurlayers.append(np.array(bthislayer));#add to the layers (before activation)
    return curlayers,bcurlayers;<jupyter_output><empty_output><jupyter_text>Back Propagation$Δw_{ji}=-\eta \frac{\partial ξ}{\partial w_{ji}}=-\eta \{-e_jφ'_j(v_j)y_i\}$$Δw_{ji}=\etaδ_jy_i$ where $\delta_j=e_jφ'_j(v_j)$$φ'_j(v_j)=\alpha y_j[1-y_j]$if neuron j is an output neuron $\delta_j=\alpha[d_j-y_j]y_j[1-y_j]$if neuron j is a hidden neuron $\delta_j=\alpha y_j[1-y_j]\sum_k{\delta_kw_{kj}}$<jupyter_code>def backpropagation(desired,curlayers,bcurlayers,weights,learning_rate):
    newweights=np.copy(weights);
    nolayers=len(weights);
    output=curlayers[nolayers];
    delta=aconst*(desired-output)*output*(1-output);#the output neuron
    for l in reversed(range(nolayers)):#propagate backwards
        pdelta=np.array([delta],dtype=object);#convert it to a 2D array for transpose    
        newweights[l]=weights[l]+ learning_rate*curlayers[l]*pdelta.T;
        delta=aconst*curlayers[l]*(1-curlayers[l])*(np.dot(delta,weights[l]));        
    return newweights;<jupyter_output><empty_output><jupyter_text>XOR Example| Input 1|Input 2      | Output || --- |--- | --- || 0 | 0 | 0|| 0 | 1 | 1|| 1 | 0 | 1|| 1 | 1 | 0|<jupyter_code>print('XOR example')
print('------------------------------------')
no_input=2;#  2 inputs 
no_output=1;# 1 output
training_data=np.array([[0,0],[0,1],[1,0],[1,1]]);#xor
desired_outputs=np.array([0,1,1,0]);#xor output
learning_rate=0.9;
Max_no_epochs=5000;#maximum no of epochs for training
target_err=0.0001;#targeted output average error
#-----------------------------------------------------
#only 1 hidden layer
neurons=[no_input+1,3,no_output];#i.e. no of neurons in each layer 1 input layer, 1 hidden layer + 1 output layer
weights=[];
nolayers=len(neurons)-1;
for i in range(nolayers):
    layer1=neurons[i];
    layer2=neurons[i+1];
    curweightlayer=np.random.rand(layer2,layer1);
    weights.append(curweightlayer);
#-----------------------------------------------------
no_trainingdata=training_data.shape[0];
#training the NN
average_err=1;#average error
epoch=0;#no epochs
while average_err>target_err and epoch<Max_no_epochs:
    for j in range(no_trainingdata):
        inp=training_data[j];
        desired=np.array(desired_outputs[j]);
        curlayers,bcurlayers=feedforward(inp,weights);#feed forward
        output=curlayers[nolayers];#output of the artificial neural network
        weights=backpropagation(desired,curlayers,bcurlayers,weights,learning_rate);
        average_err+=totalerr_energy(output,desired);#averaged error 
    average_err/=no_trainingdata;
    epoch+=1;    
print('no epochs:',epoch,'averaged err:',average_err);
#show inference results
print('------------------------------------')
print('Inference results')
for j in range(no_trainingdata):
    inp=training_data[j];
    desired=np.array(desired_outputs[j]);
    feedforwardresult,bcurlayers=feedforward(inp,weights);
    output=feedforwardresult[nolayers];
    print('input:',inp,'desired:',desired,'NN output:',output);<jupyter_output>XOR example
------------------------------------
no epochs: 2472 averaged err: 9.996571968328928e-05
------------------------------------
Inference results
input: [0 0] desired: 0 NN output: [0.00852342]
input: [0 1] desired: 1 NN output: [0.98589797]
input: [1 0] desired: 1 NN output: [0.98859425]
input: [1 1] desired: 0 NN output: [0.01404396]
{'fraction_non_alphanumeric': 0.19720382634289918, 'fraction_numerical': 0.02869757174392936, 'mean_word_length': 4.340864440078586, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '614556', 'license': 'MIT', 'n_tokens_mistral': 2041, 'n_tokens_neox': 1917, 'n_words': 337, 'path': 'ANN_for_XOR.ipynb', 'repo_name': 'bennyplo/python-ANN-XOR', 'size': '5436'}