<jupyter_start><jupyter_text>**Artificial Neural Networks** input(x) * Weight(w) * Bias(b) = output(y) **Variants of Gradient Descent** 1. Batch Gradient Descent: Uses the entire dataset to compute the gradient of the cost function. This is slow as you have to compute the gradient of the entire dataset in order to perform a single update.2. Stochastic Gradient Descent: The gradient of the cost function is computed from a single training example in every iteration.Faster3. Mini-Batch Gradient Descent: The sample of the training data is used to compute the gradient of the cost function. **Deep Learning -<NAME> Notes** Link to Notes: https://docs.google.com/presentation/d/12oUP2g7gqpPBdZcmzuqH8_ttnzosOKA2cZKbOFJyPKU/editslide=id.g73ebe5debd_0_0 A tensor is an n-dimensional matrixWe can create a simple perceptron model replicating the core concepts behind a neuron.A single pereceptron is not enough to learn complicated systems. They can however be expanded to create a multi-layer perceptron model.Feedfoward- all the info goes from input layer to output layer.Hidden layers are the layers between the input and output layers. They are difficult to interpret due to their high interconnectivity and distance away from known input or output values.A Neural Network becomes a 'deep neural network' when it contains 2 or more hidden layers.**Activation functions** set boundaries to output values from the neuron.1. Step function2. sigmoid function3. Hyperbolic Tangent (Between -1 and 1)4. Rectified Linear Unit(ReLU): Below 0 outputs 0 but above ) outputs the actual value. They have been found to have very good performance, especially when dealing with the issue of vanishing gradient. one-hot-encoding or dummy variablesMutually Exclusive Classes- each data point can only have one class assigned to it.Non_Exclusive Class: A data point may have more than one class assigned to it.softmax function:- used for mutually exclusive classes  **Cost Functions and Gradient Descent**The cost functions are sometimes called loss functions or error functions. -Measured during every epoch of training.Adaptive gradient descent: Starting with larger learning rate then adjusting to smaller rates as the slope gets closer to zero.'Adam': A method for stochastic optimization. It is more efficient for searching for minimums.-used as an optimizer- uses adaptive step size.For classification problems, cross entropy loss function is often used. **Backpropagation**Moving backwards in a network to update the weights and biases. The main idea here is that we can use the gradient to go back through the network and adjust our weights and biases to minimize the output of the error vector on the last output layer.Hadamard Product - Element by element multiplication. **TensorFlow vs Keras**TensorFlow is an open-source deep learning library develped by Google. TF 2.0 released in late 2019. It has a large ecosystem of related components, including libraries like Tensorboard, Deployment and Production APIs, and support for various programming languages.Keras is a high-level python library that can use a variety of deep learning libraries underneath, such as: TensorFlow, CNTK or Theano. The official API for TF **Classification using Sckit_learn Neural Networks**<jupyter_code># importing necessary dependencies
import pandas as pd
#loading dataset
df = pd.read_csv('transfusion.data')
df.head()
#dropping the target variable
X = df.drop('whether he/she donated blood in March 2007',axis = 1)
y = df['whether he/she donated blood in March 2007']
from sklearn.model_selection import train_test_split
#splitting the dataset
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)
#Preprocessing or scaling the data
from sklearn.preprocessing import StandardScaler
#instance of the standard scaler
sc = StandardScaler()
#scalling - Fitting the dataset the dataset
#X_train = sc.fit(X_train)
#transforming- combines both fitting and transforming
X_train = sc.fit_transform(X_train)
#tranforming the test set
X_test = sc.transform(X_test)
#The Multi Layer Perceptron
from sklearn.neural_network import MLPClassifier
# creatinga a model
model = MLPClassifier(solver = 'adam',max_iter = 1000,verbose = True, 
                     hidden_layer_sizes = (5,5,5), random_state = 42)
#fitting the model
model.fit(X_train,y_train)
#Making predictions
predictions = model.predict(X_test)
#performance metrics
from sklearn.metrics import classification_report,accuracy_score
accuracy_score(predictions,y_test)
#classification report
print(classification_report(predictions,y_test))<jupyter_output>precision    recall  f1-score   support

           0       0.93      0.77      0.85       219
           1       0.24      0.57      0.34        28

    accuracy                           0.75       247
   macro avg       0.59      0.67      0.59       247
weighted avg       0.86      0.75      0.79       247<jupyter_text>**Regression in Scikit_Learn Neural Networks**<jupyter_code>#importing necessary dependencies
import pandas as pd
import numpy as np
df = pd.read_csv('sales.csv')
df.head()
df.dtypes
#There are no categorical features present in the dataset : Therefore there is no need for hot-encoding
#creating the features and target varibles dataframes
X = df.drop(['Price'],axis = 1)
y = df['Price']
# training and testing sets
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)
#scaling
from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
#fiting and transforming the training data
X_train = sc.fit_transform(X_train)
#tranforming the test set
X_test = sc.transform(X_test)
#Building model
from sklearn.neural_network import MLPRegressor
#picking hidden layers closer to the number of features present
model = MLPRegressor(max_iter = 1000, solver = 'lbfgs',hidden_layer_sizes = (5,5,5),random_state = 101)
#fitting the model
model.fit(X_train,y_train)
#predicting
predictions = model.predict(X_test)
#performance metrics
from sklearn.metrics import mean_absolute_error,mean_squared_error,RocCurveDisplay
print('The MAE is: ',mean_absolute_error(y_test,predictions))
print('The MSE is: ',mean_squared_error(y_test,predictions))
print('The RMSE is: ',np.sqrt(mean_squared_error(y_test,predictions)))<jupyter_output>The MAE is:  627.1930941547544
The MSE is:  628522.5931459988
The RMSE is:  792.794168208873<jupyter_text>**Tensorflow Introduction**<jupyter_code>#importing the necessary dependencies
import pandas as pd
import numpy as np
#loading the dataset
df = pd.read_csv('sales.csv')
df.head()
# training and testing sets
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)<jupyter_output><empty_output><jupyter_text>**Data Preprocessing**<jupyter_code>from sklearn.preprocessing import MinMaxScaler
#scales the dataset such that all the records are between 0 and 1
sc = MinMaxScaler()
sc.fit(X_train)
#tranform
X_train = sc.transform(X_train)
X_test = sc.transform(X_test)<jupyter_output><empty_output><jupyter_text>**Building the Model**<jupyter_code>#importing the necessary dependecies
from tensorflow import keras
from tensorflow.keras import Sequential #initializes the layers
from tensorflow.keras.layers import Dense #used to create the layers

df.head()
# Method1: of creating the model
#initialize sequential
my_model = Sequential()
#picking number in between number of features and the output
my_model.add(Dense(5,activation = 'relu',input_dim=5))
my_model.add(Dense(5,activation = 'relu'))
my_model.add(Dense(5,activation = 'relu'))

#including the output layer- output 1 because we are only predicting price
my_model.add(Dense(1, activation = keras.activations.linear))  #linear activation-regression problem
#compiling the model- application of the gradient descent
my_model.compile(optimizer='adam',loss='mean_squared_error',metrics=['mse'])

#Method 2
model = Sequential(
    [
        Dense(5,activation='relu',input_dim=5),
        Dense(5,activation = 'relu'),
        Dense(5,activation = 'relu'),
        Dense(1,activation = keras.activations.linear)
    ]

)
model.compile(optimizer='adam',loss = 'mean_squared_error',metrics=['mse'])
#can also update the batch size: Number of samples per gradient update, defualt set as 32
#fitting the model
#epochs : How many times the dataset is to be fitted to the network
#Validation_split: used to validate how the model is performing
history = model.fit(X_train,y_train,epochs = 100,validation_split=0.2)<jupyter_output>Epoch 1/100
168/168 [==============================] - 1s 3ms/step - loss: 238301520.0000 - mse: 238301520.0000 - val_loss: 244323632.0000 - val_mse: 244323632.0000
Epoch 2/100
168/168 [==============================] - 0s 2ms/step - loss: 236845024.0000 - mse: 236845024.0000 - val_loss: 241022144.0000 - val_mse: 241022144.0000
Epoch 3/100
168/168 [==============================] - 0s 2ms/step - loss: 230154272.0000 - mse: 230154272.0000 - val_loss: 229546048.0000 - val_mse: 229546048.0000
Epoch 4/100
168/168 [==============================] - 0s 2ms/step - loss: 212257120.0000 - mse: 212257120.0000 - val_loss: 203517504.0000 - val_mse: 203517504.0000
Epoch 5/100
168/168 [==============================] - 0s 2ms/step - loss: 178297120.0000 - mse: 178297120.0000 - val_loss: 160214352.0000 - val_mse: 160214352.0000
Epoch 6/100
168/168 [==============================] - 0s 2ms/step - loss: 129449944.0000 - mse: 129449944.0000 - val_loss: 105535432.0000 - val_mse: 105535432.0000
Epoch 7/10[...]<jupyter_text>**Evaluating the Model**<jupyter_code>#Making predictions
predictions = model.predict(X_test)
from sklearn.metrics import mean_absolute_error,mean_squared_error
print('The MAE is: ',mean_absolute_error(y_test,predictions))
print('The MSE is: ',mean_squared_error(y_test,predictions))
print('The RMSE is: ',np.sqrt(mean_squared_error(y_test,predictions)))
y_test.shape
predictions.shape
import matplotlib.pyplot as plt
%matplotlib inline
import seaborn as sns

plt.figure(figsize=(12,6))
sns.distplot((y_test-predictions.reshape(3300,)))<jupyter_output>In C:\Users\user\Anaconda3\lib\site-packages\matplotlib\mpl-data\stylelib\_classic_test.mplstyle: 
The text.latex.preview rcparam was deprecated in Matplotlib 3.3 and will be removed two minor releases later.
In C:\Users\user\Anaconda3\lib\site-packages\matplotlib\mpl-data\stylelib\_classic_test.mplstyle: 
The mathtext.fallback_to_cm rcparam was deprecated in Matplotlib 3.3 and will be removed two minor releases later.
In C:\Users\user\Anaconda3\lib\site-packages\matplotlib\mpl-data\stylelib\_classic_test.mplstyle: Support for setting the 'mathtext.fallback_to_cm' rcParam is deprecated since 3.3 and will be removed two minor releases later; use 'mathtext.fallback : 'cm' instead.
In C:\Users\user\Anaconda3\lib\site-packages\matplotlib\mpl-data\stylelib\_classic_test.mplstyle: 
The validate_bool_maybe_none function was deprecated in Matplotlib 3.3 and will be removed two minor releases later.
In C:\Users\user\Anaconda3\lib\site-packages\matplotlib\mpl-data\stylelib\_classic_test.mplstyle[...]<jupyter_text>**Plotting the Model Losses**<jupyter_code>#creating a dataframe
losses = pd.DataFrame(history.history)
losses.head()
#plotting
plt.figure(figsize=(8,6))
plt.xlabel('Epochs')
plt.ylabel('Loss')
losses['loss'].plot(label='Training Loss')
losses['val_loss'].plot(label = 'Validation Loss')
plt.legend(loc = 'center')<jupyter_output><empty_output><jupyter_text>The loss decreases as the model learns; increase in number of epochs. if the validation loss increases while the training loss decreases, it would mean the model is overfitting. **Overfitting - Classification Problem**<jupyter_code>#importing the neccessary dependencies
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
#importing the dataset
df = pd.read_csv('transfusion.data')
df.head()
df.describe()
#data preparation - creating features and target variable
X = df.drop(['whether he/she donated blood in March 2007'],axis = 1)
y = df['whether he/she donated blood in March 2007']
# training and testing sets
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=101)
#data preprocessing
from sklearn.preprocessing import MinMaxScaler
sc = MinMaxScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform(X_test)
#building the model
from tensorflow import keras
from tensorflow.keras.layers import Dense,Dropout
model = keras.Sequential([
    #passing layer- passing 20 neurons,input_dim: equal to number of features
    Dense(20,activation = 'relu',input_dim=4),
    Dense(15,activation = 'relu'),
    #means dropping 20% of the layers at this point to prevent overfitting
    Dropout(0.2),
    #creating output layer with actibvation using relu
    Dense(25,activation = 'relu'),
    #using sigmoid becaose we are predicting two classes
    Dense(1,activation=keras.activations.sigmoid)
    
])
#compiling the model
model.compile(optimizer = 'adam',loss='binary_crossentropy',metrics=['accuracy'])
#fitting the model
#validation data: passing in the test data
history = model.fit(X_train,y_train,epochs=1600,validation_data = (X_test,y_test))
#checking the model loss
losses = pd.DataFrame(history.history)
losses.head()
#plotting the loss
plt.figure(figsize =(12,6))
plt.xlabel('Epochs')
plt.ylabel('Loss')
losses['loss'].plot(label = 'Training Loss')
losses['val_loss'].plot(label = 'Validation Loss')
plt.legend(loc = 0)<jupyter_output><empty_output><jupyter_text>We can notice from the above plot that the model is overfitting as the validation loss is increasing as the training loss decreases.<jupyter_code>#evaluating the model
model.evaluate(X_test,y_test)
#Making predictions
predictions = model.predict(X_test)
import numpy as np
np.set_printoptions(suppress = True)
predictions_df = pd.DataFrame(predictions.round(3),columns = ['Prediction'])
predictions_df.head()
predictions_df.info()
#adding rows to the df
predictions_df['Row_ID'] = range(0,247)
predictions_df.tail()
#checking the maximum prediction
predictions_df['Prediction'].max()
#checking the minimum prediction
predictions_df['Prediction'].min()
#Making the predictions true or false
predictions = predictions>0.5
#checking the performance metric
from sklearn.metrics import classification_report
print(classification_report(predictions.reshape(247,),y_test))<jupyter_output>precision    recall  f1-score   support

       False       0.91      0.82      0.87       204
        True       0.43      0.63      0.51        43

    accuracy                           0.79       247
   macro avg       0.67      0.73      0.69       247
weighted avg       0.83      0.79      0.80       247<jupyter_text>**Keras Callbacks**<jupyter_code>#importing the dataset
df = pd.read_csv('transfusion.data')
df.head()
#data preparation - creating features and target variable
X = df.drop(['whether he/she donated blood in March 2007'],axis = 1)
y = df['whether he/she donated blood in March 2007']
# training and testing sets
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3, random_state=101)
#Datapreprocessing using StandardScaler
from sklearn.preprocessing import StandardScaler
#The StandardScaler tranforms the values to have a mean of 0 and a standard deviation of 1
sc = StandardScaler()
#transforming the data
X_train= sc.fit_transform(X_train)
X_test = sc.transform(X_test)
#building the model
from tensorflow import keras
from tensorflow.keras.layers import Dense,Dropout
model = keras.Sequential([
    #passing layer- passing 20 neurons,input_dim: equal to number of features
    Dense(20,activation = 'relu',input_dim=4),
    Dense(15,activation = 'relu'),
    #means dropping 20% of the layers at this point to prevent overfitting
    Dropout(0.2),
    #creating output layer with actibvation using relu
    Dense(25,activation = 'relu'),
    #using sigmoid becaose we are predicting two classes
    Dense(1,activation=keras.activations.sigmoid)
    
])
#compiling the model
model.compile(optimizer='adam',loss='binary_crossentropy',metrics=['accuracy'])
#How to stop the model when its no longer learning; improving with increasing epochs
from tensorflow.keras.callbacks import EarlyStopping
#list of callback, patience: checks that the model has stopped improving after 20 epochs
#mode: to maximize accuracy
callbacks = [
    EarlyStopping(patience = 20,mode = 'max',monitor='accuracy')    
]
#fitting the model
history = model.fit(X_train,y_train,epochs=600,validation_data=(X_test,y_test),
                   callbacks = callbacks)
#checking the model loss
losses = pd.DataFrame(history.history)
losses.head()
#plotting the loss
plt.figure(figsize =(12,6))
plt.xlabel('Epochs')
plt.ylabel('Loss')
losses['loss'].plot(label = 'Training Loss')
losses['val_loss'].plot(label = 'Validation Loss')
plt.legend(loc = 0)
#Making predictions
predictions = model.predict(X_test)
#Making the predictions true or false
predictions = predictions>0.5
predictions.shape
y_test.shape
#checking the performance metric
from sklearn.metrics import classification_report
print(classification_report(predictions.reshape(225,),y_test))<jupyter_output>precision    recall  f1-score   support

       False       0.98      0.78      0.87       211
        True       0.19      0.79      0.31        14

    accuracy                           0.78       225
   macro avg       0.59      0.78      0.59       225
weighted avg       0.93      0.78      0.84       225<jupyter_text>**Custom Keras Callbacks**<jupyter_code>class CustomCallBack(keras.callbacks.Callback):
    def on_train_begin(self, logs={}):
        self.metrics = []

    def on_batch_end(self, batch, logs={}):
        self.metrics.append(logs.get('mse'))
from tensorflow.keras.callbacks import EarlyStopping
custom_callback = CustomCallBack()
callbacks = [
            custom_callback,
            EarlyStopping(patience = 30,mode='min',monitor='val_loss')

            ]
history = model.fit(X_train,y_train,epochs=600,validation_split = 0.2,callbacks=callbacks)
pd.DataFrame(custom_callback.metrics).head()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09847244953627933, 'fraction_numerical': 0.04446262956901255, 'mean_word_length': 4.965180605271722, 'pattern_counts': {'":': 0, '<': 40, '<?xml version=': 0, '>': 42, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '11', 'dir': 'jupyter-structured-clean-dedup', 'id': '133912', 'license': 'MIT', 'n_tokens_mistral': 6272, 'n_tokens_neox': 5554, 'n_words': 1758, 'path': 'Practice/Introduction to Neural Networks & TensorFlow/Neural Networks_NoteBook.ipynb', 'repo_name': 'DennisChangach/Deep-Learning', 'size': '18337'}