<jupyter_start><jupyter_text>Multilayer Perceptrons for Multiclass Classification (Exercise) The DataWe will be using the Glass Identification dataset from [UCI Machine Learning Repository](https://archive.ics.uci.edu/):https://archive.ics.uci.edu/ml/datasets/Glass+Identification**Information**From USA Forensic Science Service, 5 types of glass defined in terms of their oxide content (i.e. Na, Fe, K, etc).The study of classification of types of glass was motivated by criminological investigation. At the scene of the crime, the glass left can be used as evidence... if it is correctly identified!**Attributes**1. RI: refractive index2. Na: Sodium (unit measurement: weight percent in corresponding oxide, as are attributes 4-10)3. Mg: Magnesium4. Al: Aluminum5. Si: Silicon6. K: Potassium7. Ca: Calcium8. Ba: Barium9. Fe: Iron10. Type of glass: (class attribute)    - 1. building windows    - 2. vehicle windows    - 3. containers    - 4. tableware    - 5. headlamps Table of Contents-  [Python libraries](libraries)-  [Data Exploration and Feature Engineering](exploration)    -  [Read the data](read)    -  [Descriptive statistics](statistics)    -  [Class balance](balance)    -  [Correlations](correlation)    -  [Remove outliers](outliers)-  [Label and One-Hot encoding](one-hot)    -  [Create the `X` and `y` variables](variables)    -  [Encode target labels](encoders)-  [Split the Data](split)-  [Normalize the Data](normalize)-  [Create the Model](model)-  [Train the Model](training)    -  [Choosing too many epochs and overfit](overfit)    -  [Early Stopping](early_stop)-  [Evaluate the Model](evaluation)-  [Predictions](predictions)  Python libraries<jupyter_code>import numpy as np
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>Data Exploration and Feature Engineering  Read the data<jupyter_code>df = pd.read_csv('../../Data/glass.csv').drop(columns='Unnamed: 0')
df.head()
glass_types = {"building_windows": 1,
 "vehicle_windows": 2,
 "containers": 3,
 "tableware": 4,
 "headlamps": 5}

df['glass_type_id'] = df['glass_type'].apply(lambda x: glass_types[x]) 

df.head()<jupyter_output><empty_output><jupyter_text>Descriptive statistics<jupyter_code>df.describe().transpose()<jupyter_output><empty_output><jupyter_text>Class balance<jupyter_code>sns.countplot(x='glass_type_id',data=df)<jupyter_output><empty_output><jupyter_text>Correlations<jupyter_code>plt.figure(figsize=(10,8))
sns.heatmap(df.corr())
df.corr()['glass_type_id'].sort_values().plot(kind='bar')<jupyter_output><empty_output><jupyter_text>Remove outliersSeaborn uses [inter-quartile range](https://en.wikipedia.org/wiki/Interquartile_range) to detect the outliers. What we need to do is to reproduce the same function in the column you want to drop the outliers. We can do that by using the next function.<jupyter_code>def remove_outliers(df, column):
    Q1 = df[column].quantile(0.25)
    Q3 = df[column].quantile(0.75)
    IQR = Q3 - Q1    #IQR is interquartile range. 
    
    filter = (df[column] >= Q1 - 1.5 * IQR) & (df[column] <= Q3 + 1.5 *IQR)
    
    return df.loc[filter]<jupyter_output><empty_output><jupyter_text>Refractive index<jupyter_code>sns.boxplot(x='glass_type_id', y='RI', data=df)
df = remove_outliers(df, 'RI')
sns.boxplot(x='glass_type_id', y='RI', data=df)<jupyter_output><empty_output><jupyter_text>Potassium<jupyter_code>sns.boxplot(x='glass_type_id', y='K', data=df)
df = remove_outliers(df, 'K')
sns.boxplot(x='glass_type_id', y='K', data=df)<jupyter_output><empty_output><jupyter_text>Label and One-Hot encodingFor multiclass classification we have to represent categorical data in the form of binary vectors.  Create the `X` and `y` variables<jupyter_code>X = df.drop(columns=['glass_type', 'glass_type_id'],axis=1).values
y = df['glass_type'].values
X.shape
y.shape
y[:5]<jupyter_output><empty_output><jupyter_text>Encode target labels<jupyter_code>from sklearn.preprocessing import LabelEncoder
from sklearn.preprocessing import OneHotEncoder<jupyter_output><empty_output><jupyter_text>Label Encoder<jupyter_code>label_encoder = LabelEncoder()
y_integer = label_encoder.fit_transform(y)
y_integer.shape
y_integer[:5]<jupyter_output><empty_output><jupyter_text>One-Hot Encoder<jupyter_code>onehot_encoder = OneHotEncoder(sparse=False)
y_onehot = onehot_encoder.fit_transform(y_integer.reshape(len(y_integer), 1))
y_onehot.shape
y_onehot[:5,:]<jupyter_output><empty_output><jupyter_text>Split the Data<jupyter_code>from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y_onehot, test_size=0.25, random_state=42)
X_train.shape
y_train.shape<jupyter_output><empty_output><jupyter_text>Normalize the Data<jupyter_code>from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler()

X_train = scaler.fit_transform(X_train)
X_test = scaler.transform(X_test)<jupyter_output><empty_output><jupyter_text>Create the Model<jupyter_code>from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense
model = Sequential()

model.add(Dense(10,activation='relu'))
model.add(Dense(5,activation='softmax'))

# For a binary classification problem
model.compile(loss='categorical_crossentropy', 
              optimizer='adam',
              metrics=['accuracy']) # we can add in additional metrics https://keras.io/metrics/<jupyter_output><empty_output><jupyter_text>Train the Model  Choosing too many epochs and overfit<jupyter_code>model.fit(x=X_train, y=y_train, 
          validation_data=(X_test,y_test),
          epochs=2000)<jupyter_output>Train on 143 samples, validate on 48 samples
Epoch 1/2000
143/143 [==============================] - 0s 2ms/sample - loss: 1.4456 - accuracy: 0.6084 - val_loss: 1.4204 - val_accuracy: 0.6042
Epoch 2/2000
143/143 [==============================] - 0s 87us/sample - loss: 1.4082 - accuracy: 0.6224 - val_loss: 1.3909 - val_accuracy: 0.6458
Epoch 3/2000
143/143 [==============================] - 0s 97us/sample - loss: 1.3726 - accuracy: 0.6434 - val_loss: 1.3636 - val_accuracy: 0.6667
Epoch 4/2000
143/143 [==============================] - 0s 131us/sample - loss: 1.3396 - accuracy: 0.6573 - val_loss: 1.3376 - val_accuracy: 0.6875
Epoch 5/2000
143/143 [==============================] - 0s 87us/sample - loss: 1.3081 - accuracy: 0.7063 - val_loss: 1.3132 - val_accuracy: 0.6875
Epoch 6/2000
143/143 [==============================] - 0s 92us/sample - loss: 1.2769 - accuracy: 0.7273 - val_loss: 1.2895 - val_accuracy: 0.6875
Epoch 7/2000
143/143 [==============================] - 0s 98us/sample - [...]<jupyter_text>Check if the model overfits:- **Plot loss**<jupyter_code>model_loss = pd.DataFrame(model.history.history)
model_loss[['loss', 'val_loss']].plot()<jupyter_output><empty_output><jupyter_text>- **Plot accuracy**<jupyter_code>model_loss[['accuracy','val_accuracy']].plot()<jupyter_output><empty_output><jupyter_text>Early StoppingLet's use early stopping to track the val_loss and stop training once it begins increasing too much!<jupyter_code>from tensorflow.keras.callbacks import EarlyStopping
early_stop = EarlyStopping(monitor='val_loss', mode='min', verbose=1, patience=25)
model = Sequential()

model.add(Dense(10,activation='relu'))
model.add(Dense(5,activation='softmax'))

# For a binary classification problem
model.compile(loss='categorical_crossentropy', 
              optimizer='adam',
             metrics=['accuracy']) 
model.fit(x=X_train, y=y_train, 
          validation_data=(X_test, y_test),
          epochs=2000,
          callbacks=[early_stop]
          )<jupyter_output>Train on 143 samples, validate on 48 samples
Epoch 1/2000
143/143 [==============================] - 0s 2ms/sample - loss: 2.2883 - accuracy: 0.0350 - val_loss: 2.1999 - val_accuracy: 0.0625
Epoch 2/2000
143/143 [==============================] - 0s 85us/sample - loss: 2.2025 - accuracy: 0.0629 - val_loss: 2.1274 - val_accuracy: 0.0625
Epoch 3/2000
143/143 [==============================] - 0s 84us/sample - loss: 2.1205 - accuracy: 0.0909 - val_loss: 2.0555 - val_accuracy: 0.0625
Epoch 4/2000
143/143 [==============================] - 0s 84us/sample - loss: 2.0398 - accuracy: 0.1189 - val_loss: 1.9849 - val_accuracy: 0.0833
Epoch 5/2000
143/143 [==============================] - 0s 81us/sample - loss: 1.9598 - accuracy: 0.1329 - val_loss: 1.9164 - val_accuracy: 0.0625
Epoch 6/2000
143/143 [==============================] - 0s 122us/sample - loss: 1.8835 - accuracy: 0.1329 - val_loss: 1.8505 - val_accuracy: 0.0417
Epoch 7/2000
143/143 [==============================] - 0s 85us/sample - [...]<jupyter_text>- **Plot loss**<jupyter_code>model_loss = pd.DataFrame(model.history.history)
model_loss[['loss', 'val_loss']].plot()<jupyter_output><empty_output><jupyter_text>- **Plot accuracy**<jupyter_code>model_loss[['accuracy','val_accuracy']].plot()<jupyter_output><empty_output><jupyter_text>**Save the model**<jupyter_code>#model.save('multiclass_classification_model.h5')<jupyter_output><empty_output><jupyter_text>**Load the model**<jupyter_code>#model = load_model('multiclass_classification_model.h5')<jupyter_output><empty_output><jupyter_text>Evaluate the ModelCheck common classification metrics [here](https://scikit-learn.org/stable/modules/model_evaluation.htmlclassification-metrics).<jupyter_code>from sklearn.metrics import classification_report, confusion_matrix
predictions = model.predict_classes(X_test)<jupyter_output><empty_output><jupyter_text>**[Precision and recall](https://en.wikipedia.org/wiki/Precision_and_recall)**<jupyter_code>print(classification_report(onehot_encoder.inverse_transform(y_test),predictions))<jupyter_output>precision    recall  f1-score   support

           0       0.80      0.97      0.88        33
           1       0.75      0.75      0.75         4
           2       0.67      1.00      0.80         2
           3       1.00      0.33      0.50         3
           4       0.00      0.00      0.00         6

    accuracy                           0.79        48
   macro avg       0.64      0.61      0.59        48
weighted avg       0.70      0.79      0.73        48<jupyter_text>**[Confusion matrix](https://en.wikipedia.org/wiki/Confusion_matrix)**<jupyter_code>print(confusion_matrix(onehot_encoder.inverse_transform(y_test),predictions))<jupyter_output>[[32  1  0  0  0]
 [ 1  3  0  0  0]
 [ 0  0  2  0  0]
 [ 1  0  1  1  0]
 [ 6  0  0  0  0]]<jupyter_text>Predictions<jupyter_code>single_glass = df.drop(columns=['glass_type', 'glass_type_id'],axis=1).iloc[-1]
print("True glass type:", df['glass_type'].iloc[-1])
single_glass = scaler.transform(single_glass.values.reshape(-1, 9))
predicted_glass = model.predict_classes(single_glass)
print("Predicted glass type:", label_encoder.inverse_transform(predicted_glass)[0])<jupyter_output>Predicted glass type: headlamps
{'fraction_non_alphanumeric': 0.16275646333609348, 'fraction_numerical': 0.05851504278222468, 'mean_word_length': 5.005524861878453, 'pattern_counts': {'":': 5, '<': 105, '<?xml version=': 0, '>': 105, 'https://': 7, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 7, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '27', 'dir': 'jupyter-structured-clean-dedup', 'id': '622759', 'license': 'MIT', 'n_tokens_mistral': 4209, 'n_tokens_neox': 3697, 'n_words': 766, 'path': 'Notebooks/1-MLPs/4-Solutions_1.ipynb', 'repo_name': 'ikerey/Deep-Learning-with-TensorFlow2-Keras', 'size': '10869'}