<jupyter_start><jupyter_text>Loading Data  Importing libraries and txt file We can seperate the txt file into different features, since we don't know the physical qualities each index represent we will keep them as strings that correspond to each number.<jupyter_code>import os
import tensorflow as tf
import datetime
import sys
from os import path
import numpy as np
import pandas as pd

# Import data to panda frames
headings_data =map(str, range(45 + 1))
data_input = pd.read_csv('input.txt', delim_whitespace=True, header=None, names =  [str(x) for x in range(49)])
data_output = pd.read_csv('output.txt', delim_whitespace=True, header=None, names = ['class'])

# Seperate the different classes (usefull in heavily unbalanced class problems like this one)
data_input_flares= data_input.loc[0:n_flares-1,:]
data_input_seps = data_input.loc[0+n_flares:n_SEPS+n_flares,:]
data_outputflares = data_output.loc[0:n_flares-1,:]
data_output_seps = data_output.loc[n_flares:n_SEPS+n_flares,:]<jupyter_output><empty_output><jupyter_text>Creating Train & Test setsNow we can generate all the different train and test sets for the project<jupyter_code>from sklearn.model_selection import train_test_split
from sklearn import preprocessing

x_train, x_test, y_train_init, y_test_init = train_test_split(data_input, data_output,  stratify=data_output, test_size=0.1, random_state=2)

# Add a min_max scaler which is fited in training data
standar_scaler_model= preprocessing.StandardScaler()
standar_scaler_model.fit(x_train)  

x_scaled_train = standar_scaler_model.transform(x_train) # Transform to normalized data 
x_scaled_test = standar_scaler_model.transform(x_test) # Transform to normalized data 

# Change back to dataframes
x_train_rescaled = pd.DataFrame(x_scaled_train, index=x_train.index, columns=x_train.columns)
x_test_rescaled = pd.DataFrame(x_scaled_test, index=x_test.index, columns=x_test.columns)<jupyter_output><empty_output><jupyter_text>Verification of Pre-process normzliationWe can check the results compared to the custom made function that we import ourselves<jupyter_code>from utility_functions_SR import normalizeRevisted

x_train_normalized = normalizeRevisted(x_train,x_train)
x_test_normalized = normalizeRevisted(x_test,x_train)<jupyter_output><empty_output><jupyter_text>Indeed, the normalization functions work equally well.  Split smaller dataset for flaresSince we have way too many flares we should split the training set to have fewer.<jupyter_code>number_train_flares = 1500

x_train_rescaled_SEPS = x_train_rescaled.loc[y_train_init['class'] == 1] 
x_train_rescaled_flares  = x_train_rescaled.loc[y_train_init['class'] == 0] 

#Alternative way to calculate remaining flares
#x_train_rescaled_flares2 =  pd.concat([x_train_rescaled_SEPS,x_train_rescaled]).drop_duplicates(keep=False)


#undersampled_flares
x_train_rescaled_undersampled_flares = x_train_rescaled_flares.sample(number_train_flares,random_state=2)

#remaining flares to be added to test set
remaining_flares_x = x_train_rescaled_flares.drop(x_train_rescaled_undersampled_flares.index.values)
remaining_flares__and_seps_y = y_train_init.drop(x_train_rescaled_undersampled_flares.index.values)
remaining_flares_y = remaining_flares__and_seps_y.drop(x_train_rescaled_SEPS.index.values)

x_test_rescaled = pd.concat([x_test_rescaled,remaining_flares_x])
y_test_init = pd.concat([y_test_init,remaining_flares_y])

#y_train_rescaled_undersampled_flares = y_train_init.loc[x_train_rescaled_undersampled_flares.index.values]

# Create the new training set
x_train_rescaled = pd.concat([x_train_rescaled_undersampled_flares,x_train_rescaled_SEPS])
y_train_init = y_train_init.loc[x_train_rescaled.index.values]

print(x_train_rescaled_flares.shape[0])
print(remaining_flares_x.shape[0]+x_train_rescaled_undersampled_flares.shape[0])
print(x_test_rescaled.shape[0] + x_train_rescaled.shape[0])
print(y_test_init.shape[0] + y_train_init.shape[0])
17875+226<jupyter_output>16087
16087
18101
18101<jupyter_text>Over-sampling / Under-sampling optionsWe can also try another approach, to unersample or oversample our data NOTE: this doesn't provide better results in the trials so far<jupyter_code>from imblearn.over_sampling import SMOTE,ADASYN,SVMSMOTE,KMeansSMOTE,BorderlineSMOTE,RandomOverSampler
from tensorflow.keras.utils import to_categorical

# change to categorical variable for the output

y_train = to_categorical(y_train_init)
y_test = to_categorical(y_test_init)


x_resampled, y_resampled = BorderlineSMOTE().fit_resample(x_train_rescaled, y_train)


x_train_rescaled = x_resampled
y_train = to_categorical(y_resampled)<jupyter_output>C:\Users\savvra\.conda\envs\ML_Tensor_2\lib\site-packages\sklearn\utils\validation.py:72: FutureWarning: Pass classes=[0 1] as keyword args. From version 1.0 (renaming of 0.25) passing these as positional arguments will result in an error
  "will result in an error", FutureWarning)<jupyter_text>Example of Model SetupLet's setup a model below<jupyter_code>import NN_models_SR 
import importlib
importlib.reload(NN_models_SR)
from NN_models_SR import simple_model
from NN_models_SR import deeper_model
from tensorflow.keras.utils import to_categorical
# change to categorical variable for the output

y_train = to_categorical(y_train_init)
y_test = to_categorical(y_test_init)

x_train_normalized_features = x_train_rescaled[columns_sigiava_uses]
x_test_normalized_features = x_test_rescaled[columns_sigiava_uses]

nin = len(x_train_normalized_features.columns)


SGD_2 = tf.keras.optimizers.SGD(
    learning_rate=0.005, momentum=0.5, nesterov=True
)

optimizer_choice_1 = 'Adam'
optimizer_choise_2 = SGD_2
scaling_factor_loss = 100

scaling_factor = len(y_train[y_train[:,1] ==0])/len(y_train[y_train[:,1] ==1])


model_used = deeper_model(nin,number_of_classes,optimizer_choise_2,scaling_factor_loss)

class_weight_choice = {0: 1,
                1: scaling_factor}
class_weight_choice<jupyter_output><empty_output><jupyter_text>Example of Training proccedureLet's now train our model<jupyter_code>from tensorflow.keras.callbacks import EarlyStopping

batch_size_choice = round(len(x_train_normalized_features))

early_stop = EarlyStopping(monitor='loss' , min_delta=1e-04, patience=50 , verbose=1,
                                   mode ='auto', restore_best_weights= True)

#history = model_used.fit(x_train_normalized_features, y_train, epochs=30, batch_size=batch_size_choice, 
#                         verbose=0)

history = model_used.fit(x_train_normalized_features, y_train, epochs=1000, batch_size=batch_size_choice, 
                         verbose=2, 
                         callbacks=[early_stop],
                             validation_data = (x_test_normalized_features, y_test))
#class_weight=class_weight_choice,
# callbacks=[early_stop],<jupyter_output>Train on 1703 samples, validate on 16398 samples
Epoch 1/1000
1703/1703 - 7s - loss: 3.8464 - acc: 0.5514 - f1_m: 0.5514 - precision_m: 0.5514 - recall_m: 0.5514 - val_loss: 0.9254 - val_acc: 0.2223 - val_f1_m: 0.2219 - val_precision_m: 0.2219 - val_recall_m: 0.2219
Epoch 2/1000
1703/1703 - 0s - loss: 1.7594 - acc: 0.6078 - f1_m: 0.6078 - precision_m: 0.6078 - recall_m: 0.6078 - val_loss: 0.9618 - val_acc: 0.2208 - val_f1_m: 0.2203 - val_precision_m: 0.2203 - val_recall_m: 0.2203
Epoch 3/1000
1703/1703 - 0s - loss: 1.6073 - acc: 0.6201 - f1_m: 0.6201 - precision_m: 0.6201 - recall_m: 0.6201 - val_loss: 0.9852 - val_acc: 0.2148 - val_f1_m: 0.2144 - val_precision_m: 0.2144 - val_recall_m: 0.2144
Epoch 4/1000
1703/1703 - 0s - loss: 1.5428 - acc: 0.6218 - f1_m: 0.6218 - precision_m: 0.6218 - recall_m: 0.6218 - val_loss: 1.0002 - val_acc: 0.2061 - val_f1_m: 0.2058 - val_precision_m: 0.2058 - val_recall_m: 0.2058
Epoch 5/1000
1703/1703 - 0s - loss: 1.5039 - acc: 0.6218 - f1_m: 0.6218 - preci[...]<jupyter_text>Visualziating results (simple metrics)Let's see the results<jupyter_code>from sklearn.metrics import classification_report
from sklearn.metrics import confusion_matrix


predictions = model_used.predict_classes(x_test_normalized_features)
y_test_binary = np.argmax(y_test, axis=1)
    
cr = classification_report(y_test_binary, predictions)
cm = confusion_matrix(y_test_binary, predictions)

print(cm)
print("Class 1 accuracy")
print(cm[0,0]/(cm[0,0]+cm[0,1]))

print("Class 2 accuracy")
print(cm[1,1]/(cm[1,0]+cm[1,1]))

print(model_used.summary())

from tensorflow.keras.utils import plot_model
plot_model(model_used, to_file='model.png', show_shapes=True, show_layer_names=True)<jupyter_output>[[15471   904]
 [    3    20]]
Class 1 accuracy
0.944793893129771
Class 2 accuracy
0.8695652173913043
Model: "sequential"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense (Dense)                (None, 50)                1600      
_________________________________________________________________
batch_normalization (BatchNo (None, 50)                200       
_________________________________________________________________
leaky_re_lu (LeakyReLU)      (None, 50)                0         
_________________________________________________________________
dense_1 (Dense)              (None, 80)                4080      
_________________________________________________________________
batch_normalization_1 (Batch (None, 80)                320       
_________________________________________________________________
leaky_re_lu_1 (Leaky[...]<jupyter_text>Visualziating results (Plots)Let's see the results<jupyter_code>import matplotlib.pyplot as plt

val_f1 = history.history['val_f1_m']
val_loss= history.history['val_loss']
val_acc= history.history['val_acc']
val_prec= history.history['val_precision_m']

train_f1 = history.history['f1_m']
train_loss = history.history['loss']
train_acc = history.history['acc']
train_prec = history.history['precision_m']


plt.figure(figsize=(11.69,8.27))
plt.subplot(2, 2, 1)
plt.plot(train_f1)
plt.plot(val_f1)
plt.ylabel('f1 score',fontsize=22)
plt.legend(['train', 'test'], loc='lower right',fontsize=16)

plt.subplot(2, 2, 2)
plt.plot(train_loss)
plt.plot(val_loss)
plt.ylabel('loss',fontsize=22)

plt.subplot(2, 2, 3)
plt.plot(train_acc)
plt.plot(val_acc)
plt.xlabel('epoches',fontsize=22)
plt.ylabel('acc',fontsize=22)

plt.subplot(2, 2, 4)
plt.plot(train_prec)
plt.plot(val_prec)
plt.xlabel('epoches',fontsize=22)
plt.ylabel('precision',fontsize=22)

plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1002184857984231, 'fraction_numerical': 0.05063170893891897, 'mean_word_length': 5.940013183915623, 'pattern_counts': {'":': 0, '<': 33, '<?xml version=': 0, '>': 33, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '382771', 'license': 'MIT', 'n_tokens_mistral': 4111, 'n_tokens_neox': 3704, 'n_words': 719, 'path': 'notebooks/.ipynb_checkpoints/example_1_SLAMS-checkpoint.ipynb', 'repo_name': 'SavvasRaptis/ISSI-International-Team-465', 'size': '10527'}