<jupyter_start><jupyter_text>In this notebook, we will try to replicate what Tensorflow does. That is, we will try to create a simple Neural Network from scratch. We are going to create a model with 2 hidden layers.<jupyter_code>import numpy as np
import warnings
warnings.filterwarnings('ignore')
import pandas as pd
from sklearn.preprocessing import StandardScaler
from tqdm import tqdm
#load the data
data = pd.read_csv('D:/Datasets/Winsconsin/data.csv')
data.head()
data.drop(labels=['id',data.columns[-1]],axis=1,inplace=True)
data['diagnosis'].replace(to_replace=['M','B'],value=[1,0],inplace=True)
data.head()
#test,train splits
data_copy = data.copy()
data_copy = data_copy.sample(frac=1)

train_len = int(0.8*data.shape[0])

train,test = np.array(data_copy.iloc[0:train_len,1:]),np.array(data_copy.iloc[train_len:,1:])
c_train,c_test = np.array(data_copy.iloc[0:train_len,0]),np.array(data_copy.iloc[train_len:,0])
sc = StandardScaler()
train = sc.fit_transform(train)
test = sc.fit_transform(test)<jupyter_output><empty_output><jupyter_text>Just like in logistic regression, we will calculate score using a linear decision boundary to be fed into our Sigmoid Neuron$d = \sum_{i=1}^{N} \theta^T X + b$$\theta_{n,m}$ is the Weight matrix $X$ is the input matrix$b$ is the bias<jupyter_code>def decision_boundary(data,theta):
    score = data@theta
    return score.reshape(1,theta.shape[1])<jupyter_output><empty_output><jupyter_text>We will use Sigmoid function as the Activation Function$\large \frac{1}{1+e^{-z}}$<jupyter_code>def Sigmoid(x):
    '''Calculates the Sigmoid of the given input'''
    return (1/(1+np.exp(-x)))<jupyter_output><empty_output><jupyter_text>We will use cross-entropy as our loss function for Stochastic Gradient Descent$L(\theta) = -[y\, ln(\hat y) + (1-y) \,ln(1- \hat y)]$<jupyter_code>def loss(label,pred):
    ans = -(label*np.log(pred) + (1-label)*np.log(1-pred))
    return ans<jupyter_output><empty_output><jupyter_text>During forward pass we will calculate activations for each layer. Activation is nothing but the output of the layer$X->\,[Z=\theta^T X]->\,\sigma(Z)$ The trickiest part is updating the parameters or $\theta$ of our network. We do this with the help of Backpropagation. That is, we calculate the derivatives of our Loss function with respect to the parameters.$\large \frac{\delta L(\theta)}{\delta \theta} = \frac{\delta A}{\delta \theta}.\frac{\delta L(\theta)}{\delta A}$The derivative $\frac{\delta A}{\delta \theta}$ can be calculated during the forward passThe derivative $\frac{\delta L(\theta)}{\delta A}$ can only be calculated during the backward pass We will be using the Stochastic Gradient Descent algorithm for optimization of our Loss function.<jupyter_code>#initialise the values of theta for every layer
theta_1 = np.random.uniform(size=(train.shape[1],30))
theta_2 = np.random.uniform(size=(30,30))
theta_3 = np.random.uniform(size=(30,1))

#we are considering input to be the 0th layer

losses = [] #to track the final losses
errors = [] #to track the errors

alpha = 0.5 #learning rate for SGD
epochs = 200


for e in tqdm( range(epochs)):
    f_train = []
    f_test = []
    for i in range(train.shape[0]):
        #forward pass

        #layer 1
        x = train[i].reshape(1,train.shape[1])
        Z1 = decision_boundary(x,theta_1)
        A1 = Sigmoid(Z1)
        del_theta_1 = np.matmul(x.T,A1*(1-A1)) #del(A1)/del(theta_1)

        #layer 2
        Z2 = decision_boundary(A1,theta_2)
        A2 = Sigmoid(Z2)
        del_theta_2 = np.matmul(A1.T,A2*(1-A2)) #del(A2)/del(theta_2)

        #layer 3
        Z3 = decision_boundary(A2,theta_3)
        A3 = Sigmoid(Z3)
        del_theta_3 = np.matmul(A2.T,A3*(1-A3)) #del(A3)/del(theta_3)
        

        #we are done with the Forward prop
        #Now we will calculate the loss

        L = loss(c_train[i],A3) #Loss
        losses.append(L)

        E = A3 - c_train[i] #Error
        errors.append(E)
        
        f_train.append(1 if A3>0.5 else 0)
        
        

        #backward pass

        del_a3 = E/(A3*(1-A3)) #del(L)/del(A3)

        del_a2 = del_a3.T*np.matmul(theta_3,(A3*(1-A3)).T) #del(L)/del(A2)

        del_a1 = del_a2.T*np.matmul(theta_2,(A2*(1-A2)).T) #del(L)/del(A1)

        #update thetas

        theta_1 = theta_1 - alpha*del_a1.T*del_theta_1

        theta_2 = theta_2 - alpha*del_a2.T*del_theta_2

        theta_3 = theta_3 - alpha*del_a3*del_theta_3
        

for i in range(test.shape[0]):
    x = test[i].reshape(1,30)
    Z1 = decision_boundary(x,theta_1)
    A1 = Sigmoid(Z1)

    Z2 = decision_boundary(A1,theta_2)
    A2 = Sigmoid(Z2)

    Z3 = decision_boundary(A2,theta_3)
    A3 = Sigmoid(Z3)

    f_test.append(1 if A3>0.5 else 0)
print('training accuracy after epoch',e,'is',np.where(np.array(f_train)==c_train)[0].shape[0]/train.shape[0])
print('testing accuracy after epoch',e,'is',np.where(np.array(f_test)==c_test)[0].shape[0]/test.shape[0])<jupyter_output>100%|████████████████████████████████████████████████████████████████████████████████| 200/200 [00:07<00:00, 26.33it/s]
{'fraction_non_alphanumeric': 0.1573654111615066, 'fraction_numerical': 0.034115559061329126, 'mean_word_length': 4.011857707509882, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 24, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '66110', 'license': 'MIT', 'n_tokens_mistral': 1987, 'n_tokens_neox': 1864, 'n_words': 430, 'path': 'NN from Scratch.ipynb', 'repo_name': 'anantvindal/Deep-Learning', 'size': '5071'}