<jupyter_start><jupyter_text>1. Download the data from here2. Code the model to classify data like below image3. Write your own callback function, that has to print the micro F1 score and AUC score after each epoch.4. Save your model at every epoch if your validation accuracy is improved from previous epoch. 5. you have to decay learning based on below conditions         Cond1. If your validation accuracy at that epoch is less than previous epoch accuracy, you have to decrese the               learning rate by 10%.         Cond2. For every 3rd epoch, decay your learning rate by 5%.        6. If you are getting any NaN values(either weigths or loss) while training, you have to terminate your training. 7. You have to stop the training if your validation accuracy is not increased in last 2 epochs.8. Use tensorboard for every model and analyse your gradients. (you need to upload the screenshots for each model for evaluation)9. use cross entropy as loss function10. Try the architecture params as given below.  Model-11. Use tanh as an activation for every layer except output layer.2. use SGD with momentum as optimizer.3. use RandomUniform(0,1) as initilizer.3. Analyze your output and training process. Model-21. Use relu as an activation for every layer except output layer.2. use SGD with momentum as optimizer.3. use RandomUniform(0,1) as initilizer.3. Analyze your output and training process. Model-31. Use relu as an activation for every layer except output layer.2. use SGD with momentum as optimizer.3. use he_uniform() as initilizer.3. Analyze your output and training process. Model-41. Try with any values to get better accuracy/f1 score.    1. Loading dataset and Visualize<jupyter_code>import numpy as np
import tensorflow as tf
import pandas as pd
data = pd.read_csv("data.csv")
data.head()
data.describe()
import matplotlib.pyplot as plt
import seaborn as sns
plt.figure(figsize=(10,6))
sns.scatterplot('f1', 'f2', hue='label', data=data)<jupyter_output><empty_output><jupyter_text>**Observations**1. data is in circular shape2. positive datapoints residing towards inside of the circle3. Negative datapoints residing towards outside of the circle. 4. There is a lot of overlap between the dataset classes.5. Linear Algorithms can't classify it better6. Even traditional ML Algorithms will survive to classify it better.7. DL Algorithms is the best option for this **reshaping the data**<jupyter_code>X, Y = data[['f1','f2']], data['label']
X.shape, Y.shape
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X,Y, test_size=0.2, random_state=42)
X_train.shape, y_train.shape, X_test.shape, y_test.shape<jupyter_output><empty_output><jupyter_text>2. Code the model to classify data like below image<jupyter_code>def create_model():
    return tf.keras.models.Sequential([
        tf.keras.layers.InputLayer(input_shape=(2,)),
        tf.keras.layers.Dense(512, activation='relu'),
        tf.keras.layers.Dense(256, activation='relu'),
        tf.keras.layers.Dense(128, activation='relu'),
        tf.keras.layers.Dense(84, activation='relu'),
        tf.keras.layers.Dense(32, activation='relu'),
        tf.keras.layers.Dense(1, activation='sigmoid')
    ])
model = create_model()
model.summary()<jupyter_output>Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense (Dense)                (None, 512)               1536      
_________________________________________________________________
dense_1 (Dense)              (None, 256)               131328    
_________________________________________________________________
dense_2 (Dense)              (None, 128)               32896     
_________________________________________________________________
dense_3 (Dense)              (None, 84)                10836     
_________________________________________________________________
dense_4 (Dense)              (None, 32)                2720      
_________________________________________________________________
dense_5 (Dense)              (None, 1)                 33        
========================================================[...]<jupyter_text>3. Write you own Callback function**that has to print the micro F1 score and AUC score after each epoch.**<jupyter_code>from sklearn.metrics import f1_score, roc_auc_score
class GetScore(tf.keras.callbacks.Callback):
    def __init__(self, validation_data, batch_size=64):
        super().__init__()
        self.validation_data = validation_data
        self.batch_size = batch_size
        
    def on_train_begin(self, logs= {}):
        self.mf1s = []
        self.auc = []
    
    def on_epoch_end(self, epoch, logs= {}):
#         print("self : ",dir(self))
#         print("self.model : ",dir(self.model))
#         print(self.validation_data)
        y_pred  = (np.asarray(self.model.predict(self.validation_data[0]))).round()
        
        y_true  = (np.asarray(self.validation_data[1]).reshape((-1,1))).round()
        
        y_score = (np.asarray(self.model.predict_proba(self.validation_data[0])).reshape((-1,1))).round()
        # https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html
        val_mf1s = f1_score(y_true, y_pred, average='micro')
        val_auc  = roc_auc_score(y_true, y_score)
        
        self.mf1s.append(val_mf1s)
        self.auc.append(val_auc)
        
        print(' - micro f1 score = {} - auc = {}'.format(val_mf1s,val_auc))
        return
score = GetScore((X_test,y_test))
# print(score.mf1s, score.auc)
optimizer = tf.keras.optimizers.Adam(0.1)
model.compile(optimizer=optimizer,
             loss='binary_crossentropy',
             metrics=['accuracy']
             )
X_test.shape, y_test.shape
model.fit(X_train, y_train, 
         validation_data=(X_test,y_test),
         epochs=10,
         batch_size=64,
         callbacks=[score]
         )
score.auc
score.mf1s<jupyter_output><empty_output><jupyter_text>4. Save your model at every epoch**If your validation accuracy is improved from previous epoch.**<jupyter_code>import os
os.getcwd()
# # filepath for windows(for my local system)
# filepathw="D:\\Applied AI Course\\Assignments\\20 Callbacks\\model_save\\weights-{epoch:02d}-{val_accuracy:.4f}.hdf5"#.format(epoch,self.val_accuracy) # val_acc ==> val_accuracy

class SaveModel(tf.keras.callbacks.Callback):
    
    def on_train_begin(self, logs={}):
#         print("train has began")
        self.accuracy = [] # list to store accuracy for each epoch
        
    def on_epoch_end(self, epoch, logs={}):
        
        self.accuracy.append(logs.get('val_accuracy'))
        # end of each epoch we will save our model if the accuracy has improved
        filepath = 'model_save/weights-{epoch:02d}-{val_accuracy:.4f}.hdf5'.format(epoch=epoch,val_accuracy=logs.get('val_accuracy'))
        if epoch==0:
            print('epoch is 0')
            self.model.save(filepath)
        if epoch>0:
            if self.accuracy[epoch]>self.accuracy[epoch-1]:
                print("accuracy is improved, prev_accuracy={}, current_accuracy={}".format(self.accuracy[epoch-1],self.accuracy[epoch]))
                self.model.save(filepath)
save_model = SaveModel()
optimizer = tf.keras.optimizers.Adam(0.1)
model.compile(optimizer=optimizer,
             loss='binary_crossentropy',
             metrics=['accuracy']
             )
model.fit(X_train, y_train, 
         validation_data=(X_test,y_test),
         epochs=10,
         batch_size=64,
         callbacks=[save_model]
         )<jupyter_output>Epoch 1/10
244/250 [============================>.] - ETA: 0s - loss: 0.6949 - accuracy: 0.4983epoch is 0<jupyter_text>**second method**<jupyter_code>from tensorflow.keras.callbacks import ModelCheckpoint
filepathw="D:\\Applied AI Course\\Assignments\\20 Callbacks\\model_save\\weights-{epoch:02d}-{val_accuracy:.4f}.hdf5" #.format(epoch,self.val_accuracy) # val_acc ==> val_accuracy
filepath="model_save/weights-{epoch:02d}-{val_accuracy:.4f}.hdf5"
checkpoint = ModelCheckpoint(
    filepath = filepath,
    monitor ='val_accuracy',
    verbose = 1,
    save_best_only=True,
    mode = 'max' # max in case of accuracy # min=> loss # auto=> detect automatic
)
optimizer = tf.keras.optimizers.Adam(0.1)
model.compile(optimizer=optimizer,
             loss='binary_crossentropy',
             metrics=['accuracy']
             )
model.fit(X_train, y_train, 
         validation_data=(X_test,y_test),
         epochs=10,
         batch_size=64,
         callbacks=[checkpoint]
         )<jupyter_output>Epoch 1/10
244/250 [============================>.] - ETA: 0s - loss: 0.6951 - accuracy: 0.4988
Epoch 00001: val_accuracy improved from -inf to 0.49925, saving model to D:\Applied AI Course\Assignments\20 Callbacks\model_save\weights-01-0.4992.hdf5
250/250 [==============================] - 2s 9ms/step - loss: 0.6950 - accuracy: 0.4993 - val_loss: 0.6984 - val_accuracy: 0.4992
Epoch 2/10
243/250 [============================>.] - ETA: 0s - loss: 0.6941 - accuracy: 0.5047
Epoch 00002: val_accuracy improved from 0.49925 to 0.50075, saving model to D:\Applied AI Course\Assignments\20 Callbacks\model_save\weights-02-0.5008.hdf5
250/250 [==============================] - 1s 5ms/step - loss: 0.6941 - accuracy: 0.5048 - val_loss: 0.6934 - val_accuracy: 0.5008
Epoch 3/10
241/250 [===========================>..] - ETA: 0s - loss: 0.6955 - accuracy: 0.4999
Epoch 00003: val_accuracy did not improve from 0.50075
250/250 [==============================] - 1s 6ms/step - loss: 0.6954 - accuracy: 0.50[...]<jupyter_text>5. Decay learning    you have to decay learning based on below conditions        cond1. If your validation accuracy at that epoch is less than previous epoch accuracy, you have to decrease the learning rate by 10%.    cond2. For every 3rd epoch, decay your learning rate by 5%.<jupyter_code>from tensorflow.keras.callbacks import LearningRateScheduler
class decayLR(tf.keras.callbacks.Callback):
    def on_train_begin(self, logs={}):
        self.valid_acc = {'accuracy':[]}
    def on_epoch_begin(self, epoch, logs={}):
        print(epoch)
        print("Learning rate is : {}".format(float(self.model.optimizer.learning_rate)))
        if epoch>1:
            if (epoch+1)%3==0:
                lr = self.model.optimizer.learning_rate
                self.model.optimizer.learning_rate = self.model.optimizer.learning_rate - (lr*0.05)
    def on_epoch_end(self, epoch, logs={}):
        self.valid_acc['accuracy'].append(logs.get('val_accuracy'))
        if epoch>0: # because epoch starts from 0
            if self.valid_acc['accuracy'][epoch] < self.valid_acc['accuracy'][epoch-1]:
                lr = self.model.optimizer.learning_rate
                self.model.optimizer.learning_rate = self.model.optimizer.learning_rate - (lr*0.1)
decaylr = decayLR()
optimizer = tf.keras.optimizers.Adam(0.1)
model.compile(optimizer=optimizer,
             loss='binary_crossentropy',
             metrics=['accuracy']
             )
model.fit(X_train, y_train, 
         validation_data=(X_test,y_test),
         epochs=20,
         batch_size=64,
         callbacks=[decaylr]
         )<jupyter_output>0
Learning rate is : 0.10000000149011612
Epoch 1/20
250/250 [==============================] - 2s 6ms/step - loss: 0.6953 - accuracy: 0.4937 - val_loss: 0.6939 - val_accuracy: 0.5008
1
Learning rate is : 0.10000000149011612
Epoch 2/20
250/250 [==============================] - 1s 6ms/step - loss: 0.6944 - accuracy: 0.4963 - val_loss: 0.6953 - val_accuracy: 0.4992
2
Learning rate is : 0.09000000357627869
Epoch 3/20
250/250 [==============================] - 1s 6ms/step - loss: 0.6941 - accuracy: 0.5067 - val_loss: 0.6956 - val_accuracy: 0.5008
3
Learning rate is : 0.08550000190734863
Epoch 4/20
250/250 [==============================] - 1s 6ms/step - loss: 0.6939 - accuracy: 0.5019 - val_loss: 0.6932 - val_accuracy: 0.5008
4
Learning rate is : 0.08550000190734863
Epoch 5/20
250/250 [==============================] - 1s 6ms/step - loss: 0.6943 - accuracy: 0.5044 - val_loss: 0.6942 - val_accuracy: 0.5008
5
Learning rate is : 0.08550000190734863
Epoch 6/20
250/250 [========================[...]<jupyter_text>6.Terminate training if getting NaN    6. If you are getting any NaN values(either weights or loss) while training, you have to terminate your training.<jupyter_code>from tensorflow.keras.callbacks import TerminateOnNaN
terminate = TerminateOnNaN()

class TerminateNaN(tf.keras.callbacks.Callback):
    def __init__(self):
        self.we = 0
    def on_epoch_end(self, epoch, logs={}):
        import numpy as np
        loss = logs.get('loss')
        w = self.model.get_weights()
        w = np.array(w)
        self.we = w
        if loss is not None:
            if np.isnan(loss) or np.isinf(loss):
                print("Invalid loss and terminated at epoch {}".foramt(epoch))
                self.model.stop_training = True
        
        for i in range(6):
            layer   = self.model.layers[i]
            weights = layer.get_weights()[0]
            biases  = layer.get_weights()[1]
        
            if np.isnan(weights).any() or np.isinf(weights).any():
                print("Invalid weights and terminated at epoch ={}".format(epoch))
                self.model.stop_training = True
terminate = TerminateNaN()
optimizer = tf.keras.optimizers.Adam(0.1)
model.compile(optimizer=optimizer,
             loss='binary_crossentropy',
             metrics=['accuracy']
             )
model.fit(X_train, y_train, 
         validation_data=(X_test,y_test),
         epochs=10,
         batch_size=64,
         callbacks=[terminate]
         )<jupyter_output>Epoch 1/10
250/250 [==============================] - 2s 6ms/step - loss: 0.6937 - accuracy: 0.5088 - val_loss: 0.6958 - val_accuracy: 0.5008
Epoch 2/10
250/250 [==============================] - 1s 6ms/step - loss: 0.6946 - accuracy: 0.4981 - val_loss: 0.7031 - val_accuracy: 0.4992
Epoch 3/10
250/250 [==============================] - 1s 6ms/step - loss: 0.6942 - accuracy: 0.5009 - val_loss: 0.6932 - val_accuracy: 0.4992
Epoch 4/10
250/250 [==============================] - 1s 6ms/step - loss: 0.6948 - accuracy: 0.4952 - val_loss: 0.6959 - val_accuracy: 0.5008
Epoch 5/10
250/250 [==============================] - 1s 6ms/step - loss: 0.6946 - accuracy: 0.4998 - val_loss: 0.6936 - val_accuracy: 0.4992
Epoch 6/10
250/250 [==============================] - 1s 6ms/step - loss: 0.6942 - accuracy: 0.4974 - val_loss: 0.6932 - val_accuracy: 0.5008
Epoch 7/10
250/250 [==============================] - 2s 6ms/step - loss: 0.6941 - accuracy: 0.5058 - val_loss: 0.6932 - val_accuracy: 0.4992
Epoch [...]<jupyter_text>7. Stop training if validation_accuracy not improved    You have to stop the training if your validation accuracy is not increased in last 2 epochs.<jupyter_code>from tensorflow.keras.callbacks import EarlyStopping
earlystop = EarlyStopping(
    monitor  = 'val_accuracy',
    mode     = 'max',
    patience = 2
)
optimizer = tf.keras.optimizers.Adam(0.1)
model.compile(optimizer=optimizer,
             loss='binary_crossentropy',
             metrics=['accuracy']
             )
model.fit(X_train, y_train, 
         validation_data=(X_test,y_test),
         epochs=10,
         batch_size=64,
         callbacks=[earlystop]
         )<jupyter_output>Epoch 1/10
250/250 [==============================] - 2s 7ms/step - loss: 0.6955 - accuracy: 0.4918 - val_loss: 0.6932 - val_accuracy: 0.5008
Epoch 2/10
250/250 [==============================] - 1s 6ms/step - loss: 0.6949 - accuracy: 0.4904 - val_loss: 0.6936 - val_accuracy: 0.4992
Epoch 3/10
250/250 [==============================] - 1s 6ms/step - loss: 0.6944 - accuracy: 0.5071 - val_loss: 0.6937 - val_accuracy: 0.4992<jupyter_text>Custom method<jupyter_code>class StopTraining(tf.keras.callbacks.Callback):
    def on_train_begin(self, logs={}):
        # to store validation accuracy
        self.val_acc = []
        self.val_inc = {} # dictionary to store that validation increased or not
        
    def on_epoch_end(self, epoch, logs={}):
        # getting validation accuracy from logs dictionary
        loss = logs.get('val_accuracy')
        # appending to val_acc list
        self.val_acc.append(loss)
        
        # check whether accuracy improved or not fro each epoch
        if epoch>0:
            if self.val_acc[epoch] > self.val_acc[epoch-1]:
                self.val_inc[epoch]=True
            else:
                self.val_inc[epoch]=False
        
        # check, whether to stop training or not
        if epoch>2:
            if self.val_inc[epoch-1]==False and self.val_inc[epoch-2]==False:
                self.model.stop_training = True
stop_train = StopTraining() # creating an object
optimizer = tf.keras.optimizers.Adam(0.1)
model.compile(optimizer=optimizer,
             loss='binary_crossentropy',
             metrics=['accuracy']
             )
model.fit(X_train, y_train, 
         validation_data=(X_test,y_test),
         epochs=10,
         batch_size=64,
         callbacks=[stop_train]
         )<jupyter_output>Epoch 1/10
250/250 [==============================] - 2s 6ms/step - loss: 0.6944 - accuracy: 0.4936 - val_loss: 0.6960 - val_accuracy: 0.4992
Epoch 2/10
250/250 [==============================] - 1s 6ms/step - loss: 0.6942 - accuracy: 0.5042 - val_loss: 0.6981 - val_accuracy: 0.4992
Epoch 3/10
250/250 [==============================] - 2s 6ms/step - loss: 0.6946 - accuracy: 0.4914 - val_loss: 0.6956 - val_accuracy: 0.4992
Epoch 4/10
250/250 [==============================] - 1s 6ms/step - loss: 0.6947 - accuracy: 0.4971 - val_loss: 0.6932 - val_accuracy: 0.4992<jupyter_text>8. Use tensorboard<jupyter_code>import datetime
import tensorflow as tf
# for windows = logs\\fit\\ , for colab = logs/fit
log_dir="logs/fit" + datetime.datetime.now().strftime("%Y%m%d-%H%M%S")
tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=log_dir,histogram_freq=1, write_graph=True)
# clear any logs from previous runs
!rm -rf ./logs/ # for colab
optimizer = tf.keras.optimizers.Adam(0.1)
model.compile(optimizer=optimizer,
             loss='binary_crossentropy',
             metrics=['accuracy']
             )
model.fit(X_train, y_train, 
         validation_data=(X_test,y_test),
         epochs=10,
         batch_size=64,
         callbacks=[tensorboard_callback]
         )
%load_ext tensorboard
%tensorboard --logdir logs/fit

# GetScore((X_test, y_test))=> Mf1s, auc score
# SaveModel() => 
# decayLR() => decaying learning rate
# TerminateNaN() => terminating on NaN
# StopTraining()<jupyter_output><empty_output><jupyter_text>Try different architectures Model-11. Use tanh as an activation for every layer except output layer.2. use SGD with momentum as optimizer.3. use RandomUniform(0,1) as initilizer.3. Analyze your output and training process.<jupyter_code># defining model 1
initializer = tf.keras.initializers.RandomUniform(0,1)
def create_model1():
    return tf.keras.models.Sequential([
        tf.keras.layers.Dense(512,
                              input_shape=(2,),
                              activation='tanh',
                              kernel_initializer=initializer
                             ),
        tf.keras.layers.Dense(256,
                              activation='tanh',
                              kernel_initializer=initializer
                             ),
        tf.keras.layers.Dense(128,
                              activation='tanh',
                              kernel_initializer=initializer
                             ),
        tf.keras.layers.Dense(84,
                              activation='tanh',
                              kernel_initializer=initializer
                             ),
        tf.keras.layers.Dense(32,
                              activation='tanh',
                              kernel_initializer=initializer
                             ),
        tf.keras.layers.Dense(1,
                              activation='sigmoid',
                              kernel_initializer=initializer
                             )
    ])
model1 = create_model1()
model1.summary()
optimizer = tf.keras.optimizers.SGD(learning_rate=0.1, momentum=0.9)
model1.compile(optimizer=optimizer,
             loss='binary_crossentropy',
             metrics=['accuracy']
             )<jupyter_output><empty_output><jupyter_text>**creating objects of custom callbacks**<jupyter_code># remove previous logs
!rd /s /q logs
score = GetScore((X_test, y_test))# => Mf1s, auc score
save_model = SaveModel() # to save model=> 
decay_lr = decayLR() # => decaying learning rate
terminate_nan = TerminateNaN()# => terminating on NaN
stop_training = StopTraining() # stop training on some conditions

log_dir="logs\\fit\\" + datetime.datetime.now().strftime("%Y%m%d-%H%M%S")
tensorboard_callback = tf.keras.callbacks.TensorBoard(log_dir=log_dir,histogram_freq=1, write_graph=True)
callback_list = [score, save_model, decay_lr, terminate_nan, stop_training, tensorboard_callback]
!rd /s /q model_save
model1.fit(X_train, y_train, 
         validation_data=(X_test,y_test),
         epochs=20,
         batch_size=64,
         callbacks=[callback_list]
         )
%load_ext tensorboard
%tensorboard --logdir logs/fit<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.15084319388800893, 'fraction_numerical': 0.0567874778719836, 'mean_word_length': 2.974634326976486, 'pattern_counts': {'":': 0, '<': 47, '<?xml version=': 0, '>': 69, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 2, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '392828', 'license': 'Apache-2.0', 'n_tokens_mistral': 7494, 'n_tokens_neox': 6662, 'n_words': 1582, 'path': 'Callbacks/Call_Backs_Assignment_mdiqbalbajmi00786@gmail.com.ipynb', 'repo_name': 'iqbal786786/Machine-Learning-Assignments', 'size': '21466'}