<jupyter_start><jupyter_text>Technology Explorers Course 4, Session 3: FFNNs with Keras/TensorFlow**Instructor**: <NAME>**Contact**: <EMAIL>---In this session we will continue with our discussion on neural networks. Specifically, we will revisit the concept of learning curves, regularization and classification tasks, this time as they pertain to neural networks!_images in this notebook borrowed from [Ryan Holbrook](https://mathformachines.com/)_--- Contents* 2.0 [Preparing Environment and Importing Data](x.0)  * 2.0.1 [Import Packages](x.0.1)  * 2.0.2 [Load Dataset](x.0.2)* 2.1 [Feature Engineering](x.1)* 2.2 [Building the Model](x.2)  * 2.2.1 [Layers, Nodes, and Activation Functions](x.2.1)* 2.3 [Compiling the Model](x.3)  * 2.3.1 [Loss, Optimizer, and Metrics](x.3.1)* 2.4 [Fitting the Model](x.4)  * 2.4.1 [Epoch, Batch](x.4.1)* 2.5 [Evaluating the Model](x.5)* 2.6 [Convolutional NNs](x.6)  * 2.6.1 [Convolutional Layers](x.6.1)  * 2.6.2 [Pooling Layers](x.6.2)  * 2.6.3 [Flattening](x.6.3)  * 2.6.4 [Dropout](x.6.4)    ---  2.0 Preparing Environment and Importing Data[back to top](top)<jupyter_code>!pip uninstall scikit-learn -y

!pip install -U scikit-learn<jupyter_output>Found existing installation: scikit-learn 0.24.2
Uninstalling scikit-learn-0.24.2:
  Successfully uninstalled scikit-learn-0.24.2
Collecting scikit-learn
  Using cached scikit_learn-0.24.2-cp37-cp37m-manylinux2010_x86_64.whl (22.3 MB)
Requirement already satisfied: numpy>=1.13.3 in /usr/local/lib/python3.7/dist-packages (from scikit-learn) (1.19.5)
Requirement already satisfied: joblib>=0.11 in /usr/local/lib/python3.7/dist-packages (from scikit-learn) (1.0.1)
Requirement already satisfied: threadpoolctl>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from scikit-learn) (2.2.0)
Requirement already satisfied: scipy>=0.19.1 in /usr/local/lib/python3.7/dist-packages (from scikit-learn) (1.4.1)
Installing collected packages: scikit-learn
Successfully installed scikit-learn-0.24.2<jupyter_text>2.0.1 Import Packages[back to top](top)<jupyter_code>from tensorflow import keras
from tensorflow.keras import layers
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import tensorflow as tf
from copy import copy
import numpy as np
sns.set()

from sklearn.preprocessing import StandardScaler, OneHotEncoder, LabelEncoder
from sklearn.compose import make_column_transformer, make_column_selector
from sklearn.metrics import classification_report,confusion_matrix
from sklearn.model_selection import train_test_split
from sklearn import set_config
from sklearn.pipeline import make_pipeline
from sklearn.impute import SimpleImputer
set_config(display='diagram')<jupyter_output><empty_output><jupyter_text>2.0.2 Load Dataset[back to top](top)<jupyter_code>wine = pd.read_csv("https://raw.githubusercontent.com/wesleybeckner/"\
      "ds_for_engineers/main/data/wine_quality/winequalityN.csv")

# create X and y
X = wine.copy()
y = X.pop('quality')

# split into train/test
X_train, X_test, y_train, y_test = train_test_split(X, y)
X_train, X_val, y_train, y_val = train_test_split(X_train, y_train)

# the numerical values pipe
num_proc = make_pipeline(SimpleImputer(strategy='median'), StandardScaler())

# the categorical values pipe
cat_proc = make_pipeline(
    SimpleImputer(strategy='constant', fill_value='missing'),
    OneHotEncoder(handle_unknown='ignore'))

# parallelize the two pipes
preprocessor = make_column_transformer((num_proc,
                                make_column_selector(dtype_include=np.number)),
                                       (cat_proc,
                                make_column_selector(dtype_include=object)))

X_train_std = preprocessor.fit_transform(X_train) # fit_transform on train
X_test_std = preprocessor.transform(X_test) # transform test
X_val_std = preprocessor.transform(X_val)

y_train_std = np.log(y_train) # log output y
y_val_std = np.log(y_val) # log output y
y_test_std = np.log(y_test) # log output y

preprocessor<jupyter_output><empty_output><jupyter_text>2.1 Evaluating the Model[back to top](top)In the last session, we randomly chose 10 and then 30 epochs (10 rounds of the training data) to train our neural network. Now we'll garner the language to be more systematic in our approach.  2.1.1 Learning CurvesRecall learning curves from last week, where we plot model score (accuracy/mse/r2/etc.) over model complexity (trees in a forest, degrees in a polynomial, etc.):[img src](https://jakevdp.github.io/PythonDataScienceHandbook/05.03-hyperparameters-and-model-validation.html)We have a similar situation with neural networks, accept here, the model complexity is defined by both the number of epochs, and the capacity of the model.   2.1.1.1 Training Data (Epochs)A greater number of epochs allows the model to further tune itself to the training data. Remember that our model solves its weights by a solver function, most often a form of stochastic gradient descent. Because this solver is iterative, the longer we allow it run the closer it will get to finding its true weights. The caveat here is that, like we have seen before, the model is only learning the proper weights according to the training data, which we know includes noise, otherwise known as irreducible error. Our data science senses should be tingling: there's an optimum number of epochs here, and it will counter balance the trade-off between error due to bias (too few epochs) and error due to variance (too many epochs).  2.1.1.2 Complexity (Capacity)The capacity of the model is defined by the architecture. It is the total number of trainable weights available to the solver function. The more weights, the more capacity. Capacity determines the upper limit for which our model can learn relationships between the data. Again we should recall from our session on feature engineering: the more training data we have, the more capacity we should give our model to account for that abundance of training data. This is also influenced by the actual complexity between the input and output data, X and y, who's function we are attempting to approximate with the neural network. The more complicated the relationship, the more capacity we should give to our model.Capacity can be increased either by widening our model (increasing the neurons in a layer) or deepening our model(increasing the number of layers in our model).  2.1.2 Early Stopping[back to top](top)Without knowing the true relationship between X and y, or the degree to which there is irreducible error in our data, we return to our familiar learning curves to pragmaticaly determine how long we should train our model, that is, how many epochs should be ran and how many neurons we should give our model.When dealing with the number of epochs, we can program this into the training session automatically with early stopping. Early stopping allows us to discontinue training the model when either the validation score stops improving, or stops improving by some margin. This allows us to both save time during training and to avoid overfitting our model.To account for underfitting (not training the model long enough) we can simply set our number of training epochs to some large number and allow early stopping to take care of the rest.In TF/Keras, we can envoke early stopping by setting a **_callback_** a callback is simply a function that is called every so often.<jupyter_code>from tensorflow.keras.callbacks import EarlyStopping

early_stopping = EarlyStopping(
    min_delta=0.001, # minimium amount of change to count as an improvement
    patience=20, # how many epochs to wait before stopping
    restore_best_weights=True,
)<jupyter_output><empty_output><jupyter_text>In the above, we are saying detect when the validation loss stops improving by 0.001; check this over the past 20 epochs to avoid stopping early due to noise; and restore the best weights over that past 20 epoch period when early stopping is envoked. To envoke early stopping we would enlist it in our call to fit like so:```model.fit(     X, y, batch size etc, ...    callbacks=[early_stopping],)```  2.1.3 Exercise: Try Early StoppingTake your best model from the last exercise in session 1 and apply early stopping<jupyter_code>from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.callbacks import EarlyStopping

model = keras.Sequential([
    ### YOUR CODE HERE ###
    layers.Dense(512, activation='relu', input_shape=[X_train_std.shape[1]]),
    layers.Dense(512, activation='relu'),
    layers.Dense(512, activation='relu'),
    layers.Dense(1)
])

model.compile(
    optimizer='adam',
    loss='mse',
)

early_stopping = EarlyStopping(
    min_delta=0.001, # minimium amount of change to count as an improvement
    patience=20, # how many epochs to wait before stopping
    restore_best_weights=True,
)

print(model.summary())

history = model.fit(
    X_train_std, y_train_std,
    validation_data=(X_val_std, y_val_std),
    batch_size=256,
    epochs=1000,
    verbose=1,
    callbacks=[early_stopping],
)
# Show the learning curves
history_df = pd.DataFrame(history.history)
fig, ax = plt.subplots(1,1)
history_df.plot(ax=ax)
ax.set_ylim(0,.1)
y_test_pred = (np.exp(model.predict(X_test_std)))
plt.plot(y_test_pred, np.exp(y_test_std), ls='', marker='o')
plt.xlim(4, 10)<jupyter_output><empty_output><jupyter_text>2.2 Regularizing Layers: Dropout and Batch Normalization[back to top](top)There are dozens of different layer types for accomplishing different tasks (and more are being generated by researchers all the time). While we're on the topic of general model performance, there are two layer types we'll want to introduce here: **_dropout_** and **_batch normalization_** or just batchnorm. What we'll see with both of these types is that while they don't contain any neurons, they're useful to think of as layers, because they are an extra processing step between neural layers.  2.2.1 Dropout[back to top](top)Dropout is the Neural Network response to the wide success of ensemble learning. In a dropout layer, random neurons are dropped in each batch of training, i.e. their weighted updates are not sent to the next neural layer. Just as with random forests, the end result is that the neural network can be thought of as many _independent models_ that _vote_ on the final output. Put another way, when a network does not contain dropout layers, and has a capacity that exceeds that which would be suited for the true, underlying complexity level of the data, it can begin to fit to noise. This ability to fit to noise is based on very specific relationships between neurons, which fire uniquely given the particular training example. Adding dropout _breaks_ these specific neural connections, and so the neural network as a whole is forced to find weights that apply generally, as there is no guarantee they will be _turned on_ when their specific training example they would usually overfit for comes around again.  Network with 50% dropout A last thing to note, is that after adding dropout, we will typically need to add additional layers to our network to maintain the overall capacity of the network.<jupyter_code>keras.Sequential([
    # ...
    layers.Dropout(rate=0.3), # apply 30% dropout to the next layer
    layers.Dense(16),
    # ...
])<jupyter_output><empty_output><jupyter_text>When adding dropout to a model we will usually want to increase the depth:<jupyter_code># Dropout model

model = keras.Sequential([
    layers.Dense(512, activation='relu', input_shape=[X_train_std.shape[1]]),
    layers.Dropout(0.3),
    layers.Dense(512*2, activation='relu'),
    layers.Dropout(0.3),
    layers.Dense(512, activation='relu'),
    layers.Dense(1),
])

model.compile(
    optimizer='adam',
    loss='mse',
)

early_stopping = EarlyStopping(
    min_delta=0.001, # minimium amount of change to count as an improvement
    patience=20, # how many epochs to wait before stopping
    restore_best_weights=True,
)

print(model.summary())

history = model.fit(
    X_train_std, y_train_std,
    validation_data=(X_val_std, y_val_std),
    batch_size=256,
    epochs=1000,
    verbose=1,
    callbacks=[early_stopping],
)<jupyter_output>Model: "sequential_4"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_16 (Dense)             (None, 512)               7168      
_________________________________________________________________
dropout (Dropout)            (None, 512)               0         
_________________________________________________________________
dense_17 (Dense)             (None, 1024)              525312    
_________________________________________________________________
dropout_1 (Dropout)          (None, 1024)              0         
_________________________________________________________________
dense_18 (Dense)             (None, 512)               524800    
_________________________________________________________________
dense_19 (Dense)             (None, 1)                 513       
======================================================[...]<jupyter_text>2.2.2 Batch Normalization[back to top](top)Batch normalization ([batchnorm](https://arxiv.org/abs/1502.03167)) accomplishes a task very similar to sklearn's `StandardScaler`, and in fact, it can be added to the front of a neural network in place of sklearn's `StandardScaler` to scale and center the input data. But batchnorm is also used between layers, why would we want to rescale the output from one layer before feeding into the next? For the same reason we would want to scale and center our data in the first place: it levels the playing field for the following layer to find the important relationships. The optimizer algorithm, SGD, will shift weights in proportion to how large the activation is out of each neuron, and so large differences can lead to spurious behavior.In practice, batchnorm is often implemented to make the overall training time faster, rather than to improve the final loss (although in some cases it does do this as well). It is good to envoke batchnorm in particular, if you are dealing with neural networks that take a long time to train.Batchnorm can be added between layers, as well as between a layer and its activation function.<jupyter_code># BatchNorm model

model = keras.Sequential([
    layers.Dense(512, activation='relu', input_shape=[X_train_std.shape[1]]),
    layers.BatchNormalization(),
    layers.Dense(512*2, activation='relu'),
    layers.BatchNormalization(),
    layers.Dense(512, activation='relu'),
    layers.Dense(1),
])

model.compile(
    optimizer='adam',
    loss='mse',
)

early_stopping = EarlyStopping(
    min_delta=0.001, # minimium amount of change to count as an improvement
    patience=20, # how many epochs to wait before stopping
    restore_best_weights=True,
)

print(model.summary())

history = model.fit(
    X_train_std, y_train_std,
    validation_data=(X_val_std, y_val_std),
    batch_size=256,
    epochs=1000,
    verbose=1,
    callbacks=[early_stopping],
)
# Show the learning curves
history_df = pd.DataFrame(history.history)
fig, ax = plt.subplots(1,1)
history_df.plot(ax=ax)
ax.set_ylim(0,.2)
y_test_pred = np.exp(model.predict(X_test_std))
plt.plot(y_test_pred, np.exp(y_test_std), ls='', marker='o')<jupyter_output><empty_output><jupyter_text>Model with BatchNorm and Dropout:<jupyter_code>from tensorflow import keras
from tensorflow.keras import layers

# Dropout and BatchNorm Model
model = keras.Sequential([
    layers.Dense(512, input_shape=[X_train_std.shape[1]]),
    layers.BatchNormalization(),
    layers.Activation('relu'), # separate activation
    layers.Dropout(0.3),
    layers.Dense(1024, activation='relu'), # built-in activation
    layers.Dropout(0.3),
    layers.BatchNormalization(),
    layers.Dense(512, activation='relu'),
    layers.Dropout(0.3),
    layers.BatchNormalization(),
    layers.Dense(1),
])
print(model.summary())<jupyter_output>Model: "sequential_7"
_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
dense_28 (Dense)             (None, 512)               7168      
_________________________________________________________________
batch_normalization_5 (Batch (None, 512)               2048      
_________________________________________________________________
activation_1 (Activation)    (None, 512)               0         
_________________________________________________________________
dropout_5 (Dropout)          (None, 512)               0         
_________________________________________________________________
dense_29 (Dense)             (None, 1024)              525312    
_________________________________________________________________
dropout_6 (Dropout)          (None, 1024)              0         
______________________________________________________[...]<jupyter_text>2.2.3 Exercise: Try Batch Normalization and DropoutCreate 3 models and train for 100 epochs with early stopping:1. compile and train the model previously defined above2. recreate the architecture but remove the batchnorm  layers3. recreate the architecture but remove the dropout layers and one hidden, dense layerCompare the number of epochs required to converge, and the overall loss<jupyter_code># Dropout and BatchNorm Model
model = keras.Sequential([                      
    ### YOUR CODE HERE ###
    layers.Dense(512, input_shape=[X_train_std.shape[1]]),
    layers.BatchNormalization(),
    layers.Activation('relu'), # separate activation
    layers.Dropout(0.3),
    layers.Dense(1024, activation='relu'), # built-in activation
    layers.Dropout(0.3),
    layers.BatchNormalization(),
    layers.Dense(512, activation='relu'),
    layers.Dropout(0.3),
    layers.BatchNormalization(),
    layers.Dense(1),
])

model.compile(
    optimizer='adam',
    loss='mae',
)

early_stopping = EarlyStopping(
    min_delta=0.001, # minimium amount of change to count as an improvement
    patience=20, # how many epochs to wait before stopping
    restore_best_weights=True,
)

history = model.fit(
    X_train_std, y_train_std,
    validation_data=(X_val_std, y_val_std),
    batch_size=256,
    epochs=100,
    verbose=1,
    callbacks=[early_stopping],
)
# Dropout and BatchNorm Model
model = keras.Sequential([                      
    ### YOUR CODE HERE ###
    layers.Dense(512, input_shape=[X_train_std.shape[1]]),
    # layers.BatchNormalization(),
    layers.Activation('relu'), # separate activation
    layers.Dropout(0.3),
    layers.Dense(1024, activation='relu'), # built-in activation
    layers.Dropout(0.3),
    # layers.BatchNormalization(),
    layers.Dense(512, activation='relu'),
    layers.Dropout(0.3),
    # layers.BatchNormalization(),
    layers.Dense(1),
])

model.compile(
    optimizer='adam',
    loss='mae',
)

early_stopping = EarlyStopping(
    min_delta=0.001, # minimium amount of change to count as an improvement
    patience=20, # how many epochs to wait before stopping
    restore_best_weights=True,
)

history = model.fit(
    X_train_std, y_train_std,
    validation_data=(X_val_std, y_val_std),
    batch_size=256,
    epochs=100,
    verbose=1,
    callbacks=[early_stopping],
)
# Dropout and BatchNorm Model
model = keras.Sequential([                      
    ### YOUR CODE HERE ###
    layers.Dense(512, input_shape=[X_train_std.shape[1]]),
    layers.BatchNormalization(),
    layers.Activation('relu'), # separate activation
    # layers.Dropout(0.3),
    layers.Dense(1024, activation='relu'), # built-in activation
    # layers.Dropout(0.3),
    layers.BatchNormalization(),
    # layers.Dense(512, activation='relu'),
    # layers.Dropout(0.3),
    # layers.BatchNormalization(),
    layers.Dense(1),
])

model.compile(
    optimizer='adam',
    loss='mae',
)

early_stopping = EarlyStopping(
    min_delta=0.001, # minimium amount of change to count as an improvement
    patience=20, # how many epochs to wait before stopping
    restore_best_weights=True,
)

history = model.fit(
    X_train_std, y_train_std,
    validation_data=(X_val_std, y_val_std),
    batch_size=256,
    epochs=100,
    verbose=1,
    callbacks=[early_stopping],
)<jupyter_output>Epoch 1/100
15/15 [==============================] - 1s 42ms/step - loss: 1.9854 - val_loss: 1.1818
Epoch 2/100
15/15 [==============================] - 0s 31ms/step - loss: 1.0592 - val_loss: 0.5808
Epoch 3/100
15/15 [==============================] - 1s 35ms/step - loss: 0.5323 - val_loss: 0.1880
Epoch 4/100
15/15 [==============================] - 1s 34ms/step - loss: 0.4227 - val_loss: 0.3112
Epoch 5/100
15/15 [==============================] - 1s 35ms/step - loss: 0.3099 - val_loss: 0.1629
Epoch 6/100
15/15 [==============================] - 1s 34ms/step - loss: 0.2706 - val_loss: 0.1635
Epoch 7/100
15/15 [==============================] - 0s 33ms/step - loss: 0.2263 - val_loss: 0.1478
Epoch 8/100
15/15 [==============================] - 0s 33ms/step - loss: 0.2347 - val_loss: 0.1988
Epoch 9/100
15/15 [==============================] - 1s 34ms/step - loss: 0.2171 - val_loss: 0.1792
Epoch 10/100
15/15 [==============================] - 0s 33ms/step - loss: 0.2368 - val_loss: 0.1984[...]<jupyter_text>2.3 Binary Classification with Neural NetworksWhen dealing with classification tasks, everything we've covered up till now with regression still applies. The main difference is the format of the last layer's outputs and the choice of loss function.  2.3.1 Accuracy and Cross EntropyFormerly, we've encountered accuracy in classification tasks. It is the ratio of correct predictions over total predictions: `accuracy = number_corect / total` (and when classes were heavily imballanced we used a weighted accuracy). The problem with using accuracy as a loss function, is that it does not change smoothly (there are jumps in the numerator since it is just a count in correct predictions), which the SGD algorithm requires in order to work properly. We need another metric. the further a model is from predicting the correct class, the higher the loss. Instead we use **_cross-entropy_**, we won't go into detail here, other than that it is a distance measure between two probabilities. What two probabilities are we measuring? We want the probability for predicting the correct class to be 1 (100%). cross-entropy will measure the distance the current probability of the model is from 1. The main takeaway here is that we will want to use cross-entropy for classification tasks. We set cross-entropy as our loss when we compile the model. In this case we have two classes so we use `binary_crossentropy````model.compile(     optimizer...    loss='binary_crossentropy',    metrics=['binary_accuracy'],)```  2.3.2 0 or 1: The Sigmoid FunctionFinally, we need to introduce a special activation function that will map our last layer outputs from 0 to 1, to feed into our loss function. The traditional function we use for this is the **_sigmoid function_**. The sigmoid function maps values on the interval [0, 1]To get the final class prediction, we need a threshold probability, typically 0.5 where we will round up to the class label. Keras will set to 0.5 by default.We set the sigmoid activation function in the last layer:```model = keras.Sequential([     ... previous layers ...    layers.Dense(1, activation='sigmoid'),])```  2.3.3 Classification Example<jupyter_code>wine = pd.read_csv("https://raw.githubusercontent.com/wesleybeckner/"\
      "ds_for_engineers/main/data/wine_quality/winequalityN.csv")

# create X and y
X = wine.copy()
y = X.pop('type')

# split into train/test
X_train, X_test, y_train, y_test = train_test_split(X, y)
X_train, X_val, y_train, y_val = train_test_split(X_train, y_train)

# the numerical values pipe
num_proc = make_pipeline(SimpleImputer(strategy='median'), StandardScaler())

# the categorical values pipe
cat_proc = make_pipeline(
    SimpleImputer(strategy='constant', fill_value='missing'),
    OneHotEncoder(handle_unknown='ignore'))

# parallelize the two pipes
preprocessor = make_column_transformer((num_proc,
                                make_column_selector(dtype_include=np.number)),
                                       (cat_proc,
                                make_column_selector(dtype_include=object)))

X_train_std = preprocessor.fit_transform(X_train) # fit_transform on train
X_test_std = preprocessor.transform(X_test) # transform test
X_val_std = preprocessor.transform(X_val)

y_train_std = y_train.map({'white': 0, 'red': 1}) # convert to int
y_val_std = y_val.map({'white': 0, 'red': 1}) # convert to int
y_test_std = y_test.map({'white': 0, 'red': 1}) # convert to int

preprocessor
from tensorflow import keras
from tensorflow.keras import layers

model = keras.Sequential([
    layers.Dense(4, activation='relu', input_shape=[X_train_std.shape[1]]),
    layers.Dense(4, activation='relu'),    
    layers.Dense(1, activation='sigmoid'),
])

print(model.summary())
tf.keras.utils.plot_model(model, 
                          show_layer_names=True, 
                          # show_dtype=True, 
                          show_shapes=True)<jupyter_output><empty_output><jupyter_text>We set cross-entropy as our loss when we compile the model. In this case we have two classes so we use `binary_crossentropy`> Introducing `metrics`: we can track other forms of performance during training with the `metrics` parameter in `model.compile`. We will look at the results when we finish training.<jupyter_code>model.compile(
    optimizer='adam',
    loss='binary_crossentropy',
    metrics=['binary_accuracy'], # new metrics flag here
)
early_stopping = keras.callbacks.EarlyStopping(
    patience=10,
    min_delta=0.001,
    restore_best_weights=True,
)

history = model.fit(
    X_train_std, y_train_std,
    validation_data=(X_val_std, y_val_std),
    batch_size=512,
    epochs=1000,
    callbacks=[early_stopping],
    verbose=1, 
)
history_df = pd.DataFrame(history.history)

history_df.loc[:, ['loss', 'val_loss']].plot()
history_df.loc[:, ['binary_accuracy', 'val_binary_accuracy']].plot()

print(("Best Validation Loss: {:0.4f}" +\
      "\nBest Validation Accuracy: {:0.4f}")\
      .format(history_df['val_loss'].min(), 
              history_df['val_binary_accuracy'].max()))
# predict test set
pred_probability = model.predict(X_test_std)

# convert to bool
predictions = pred_probability > 0.5

# precision / recall / f1-score 
print(classification_report(y_test_std,predictions))
plt.figure(figsize=(10,6))
sns.heatmap(confusion_matrix(y_test_std,predictions),annot=True)<jupyter_output><empty_output><jupyter_text>2.3.4 Multi-Class Classification[back to top](top)It is good practice, and often necessary to one-hot encode the target values for a multi-class problem. We will need to do that with our wine data here.  2.3.4.1 Exercise: Multi-Class Classification1) Define ModelDefine a model with both batch normalization and dropout layers. 2) Add Optimizer, Loss, and Metric3) Train and Evaluate<jupyter_code>from keras.utils import np_utils
wine = pd.read_csv("https://raw.githubusercontent.com/wesleybeckner/"\
      "ds_for_engineers/main/data/wine_quality/winequalityN.csv")

# create X and y
X = wine.copy()
y = X.pop('quality')

# preprocess y for quality and/or type
encoder = LabelEncoder()
encoder.fit(y)
encoded_y = encoder.transform(y)
y = np_utils.to_categorical(encoded_y)

# split into train/test
X_train, X_test, y_train, y_test = train_test_split(X, y)
X_train, X_val, y_train, y_val = train_test_split(X_train, y_train)

# the numerical values pipe
num_proc = make_pipeline(SimpleImputer(strategy='median'), StandardScaler())

# the categorical values pipe
cat_proc = make_pipeline(
    SimpleImputer(strategy='constant', fill_value='missing'),
    OneHotEncoder(handle_unknown='ignore'))

# parallelize the two pipes
preprocessor = make_column_transformer((num_proc,
                                make_column_selector(dtype_include=np.number)),
                                       (cat_proc,
                                make_column_selector(dtype_include=object)))

X_train_std = preprocessor.fit_transform(X_train) # fit_transform on train
X_test_std = preprocessor.transform(X_test) # transform test
X_val_std = preprocessor.transform(X_val)

y_train_std = y_train
y_val_std = y_val
y_test_std = y_test

preprocessor
from tensorflow import keras
from tensorflow.keras import layers

model = keras.Sequential([
  ### YOUR MODEL HERE ###
])
model.compile(
    optimizer='adam',
    loss='categorical_crossentropy',
    metrics=['accuracy'],
)
early_stopping = keras.callbacks.EarlyStopping(
    patience=10,
    min_delta=0.001,
    restore_best_weights=True,
)

history = model.fit(
    X_train_std, y_train_std,
    validation_data=(X_val_std, y_val_std),
    batch_size=512,
    epochs=1000,
    callbacks=[early_stopping],
    verbose=1, # hide the output because we have so many epochs
)
history_df = pd.DataFrame(history.history)

history_df.loc[:, ['loss', 'val_loss']].plot()
history_df.loc[:, ['accuracy', 'val_accuracy']].plot()

print(("Best Validation Loss: {:0.4f}" +\
      "\nBest Validation Accuracy: {:0.4f}")\
      .format(history_df['val_loss'].min(), 
              history_df['val_accuracy'].max()))
# predict test set
pred_probability = model.predict(X_test_std)

# convert to bool
predictions = pred_probability > 0.5

# precision / recall / f1-score 
print(classification_report(y_test_std,predictions))<jupyter_output>precision    recall  f1-score   support

           0       0.50      0.11      0.18         9
           1       0.00      0.00      0.00        54
           2       0.62      0.63      0.63       516
           3       0.56      0.62      0.59       717
           4       0.62      0.16      0.26       286
           5       0.00      0.00      0.00        41
           6       0.00      0.00      0.00         2

   micro avg       0.59      0.51      0.54      1625
   macro avg       0.33      0.22      0.24      1625
weighted avg       0.56      0.51      0.51      1625
 samples avg       0.51      0.51      0.51      1625
{'fraction_non_alphanumeric': 0.1066552338236264, 'fraction_numerical': 0.03182507750148407, 'mean_word_length': 4.037880046519355, 'pattern_counts': {'":': 0, '<': 50, '<?xml version=': 0, '>': 57, 'https://': 6, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 3, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 14, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '35353', 'license': 'MIT', 'n_tokens_mistral': 9528, 'n_tokens_neox': 8828, 'n_words': 2924, 'path': 'Statistics and Models II/SOLUTIONS/SOLUTION_Tech_Exp_C4_S3_FFNNs_with_Keras_and_TensorFlow.ipynb', 'repo_name': 'wesleybeckner/technology_explorers', 'size': '30346'}