<jupyter_start><jupyter_text>Deep Learning - Beginners TrackInstructor: <NAME>  Multi Class ClassificationIn the previous notebeook we used logistic regression for Binary Classification, now we will see how to train a classifier model for Multi-Class Classification.**What is Multi-Class Classification?** If the target values have n discrete classification classes ie: y can take discrete value from 0 to n-1. If $y \in \{0, 1, 2, 3, ..., n-1\}$, then the classification task is n-Multi-Class.![](https://miro.medium.com/max/972/1*SwXHlCzh-d9UqHOglp3vcA.png)  Task - 1 Create a 3-Multi-Class dataset with sklearn.datasets and visualize it.It's very easy, use the same code form previous notebook and make changes for 3 classes.<jupyter_code>from sklearn.datasets import make_blobs

# make dataset with 2 input features and 3 output classes
X, y = 

X.shape, y.shape, set(y)<jupyter_output><empty_output><jupyter_text>If you made 3 centers, you can see ```set(y)``` will return ```{0, 1, 2}```. where 0 represent the first class, 1 represent second and 2 represents the third class.<jupyter_code>import numpy as np

# getting the index of each class
class_0 = 
class_1 = 
class_2 =

# getting data from index
X_0 = 
X_1 = 
X_2 = 

X_0.shape, X_1.shape, X_2.shape
import matplotlib.pyplot as plt

# make scatter plot of each class data with different colour and marker
plt.figure(figsize=(12,9))

plt.xlabel('$x1$', fontsize=20)
plt.ylabel('$x2$', fontsize=20)
plt.grid(True)
plt.legend(fontsize=20)
plt.show()<jupyter_output><empty_output><jupyter_text>How to get the probabilities? : SoftMaxNow the target is going to be $y \in \{0, 1, 2\}$, so sigmoid cannot be used here, as sigmoid will convert any number to range 0 to 1 , so it can only be used for binary classification.We need a function which converts the scores/logits of linear mapping into probabilities for all n classes.That function should have some properties:- all probabilities should be >0- probabilities should be in range $[0,1]$- some of all class probabilities = 1One possible function can be class_logit/sum of all class logits. Lets try itExample:$logits = [-100, 40, -10]$, right now don't bother how do we get 3 logits, we will discuss it below.$probabilities = [-100/(-100+40-10), 40/(-100+40-10), -10/(-100+40-10)]$$probabilities = [100/70, -40/70, 10/70]$ you can see this example satisfies only the third property(sum=1). So we need a function which gives positive numbers. Exponential function can help us.$Logits = [l_0, l_1, l_2, ...., l_{n-1}]$$Probabilities = [\dfrac{e^{l_0}}{e^{l_1} + e^{l_2}+ ... + e^{l_{n-1}}}, \dfrac{e^{l_1}}{e^{l_1} + e^{l_2}+ ... + e^{l_{n-1}}}, ..., \dfrac{e^{l_{n-1}}}{e^{l_1} + e^{l_2}+ ... + e^{l_{n-1}}}]$This function is called Softmax, and this gives the probability that a data belongs to class j, given the logits.$P(y=j|z) = \dfrac{e^{z_j}}{\sum_{i=0}^{n-1}e^{z_i}}$Let's code softmax function in Numpy.<jupyter_code>import numpy as np

def softmax(x):
  exp = np.exp(x)
  exp_sum = exp.sum(axis=1).reshape(-1,1)
  return exp/exp_sum
x = np.array([[22, 40, 10]])

softmax(x)<jupyter_output><empty_output><jupyter_text>Now we know, replacing sigmoid with with softmax will help in the case of multi class classification. This softmax model is also called **Softmax Regression**.  Loss FunctionAs we have already seen, for classification task we will use Cross Entropy loss.The prediction of softmax regression $\hat{y} = [0.129, 0.8, 0.071]$, whereas the true label will be one of $y \in \{0, 1, 2\}$. We cannot directly use Cross Entropy loss with $\hat{y}$ and $y$. So we convert the true label into One-Hot Encoding form. One hot encoding is a vector representation of the label which has '1' at the index corresponding to the label and '0' elsewhere.Example:Let $y \in \{0, 1, 2, 3, 4\}$, then - '4' is represented as $[0, 0, 0, 0, 1]$- '3' is represented as $[0, 0, 0, 1, 0]$- '2' is represented as $[0, 0, 1, 0, 0]$- '1' is represented as $[0, 1, 0, 0, 0]$- '0' is represented as $[1, 0, 0, 0, 0]$Let's code the label to one hot conversion<jupyter_code>import numpy as np

def to_one_hot(labels, num_classes):
    return np.eye(num_classes)[labels]

num_classes = 5
labels = np.array([0, 1, 2, 3, 4])

to_one_hot(labels, num_classes)<jupyter_output><empty_output><jupyter_text>You can also use sklearn.preprocessing.OneHotEncoder to convert labels to one hot vectors.<jupyter_code>from sklearn.preprocessing import OneHotEncoder

labels = np.array([[0], 
                   [1], 
                   [2], 
                   [3], 
                   [4]])

OneHotEncoder(categories='auto').fit_transform(labels).toarray()<jupyter_output><empty_output><jupyter_text>Keras also have some utils functions which can help in one-hot encoding<jupyter_code>from keras.utils.np_utils import to_categorical   

labels = np.array([0, 1, 2, 3, 4])
to_categorical(labels, num_classes=5)<jupyter_output><empty_output><jupyter_text>Softmax Regression ModelFrom what we discussed so far, if the number of classes = 3, then we expect model to give a prediction $\hat{y} = Softmax(z)$ and $z$ will be like $z = [-10, 20, 5]$(Example).$z = X.W + b$ will only give one number like $z=[4]$ in logistic regression. But now we are using softmax regression which expect a model which gives 3 output for a 3 class classifier. If the no of input features = 2 and no of out[ut classes = 3So we will use 3 linear classifier. $z_1 = X.W_1 + b_1$, $z_2 = X.W_2 + b_2$, $z_3 = X.W_3 + b_3$ which can be combined together withz = $\begin{bmatrix}z_1&z_2&z_3\\ \end{bmatrix}$ $ z = X . W + b $$W = \begin{bmatrix}W_1&W_2&W_3\\ \end{bmatrix} $$b = \begin{bmatrix}b_1&b_2&b_3\\ \end{bmatrix}$each $W_i = \begin{bmatrix}W_{i1}\\W_{i2}\\W_{}i3\\ \end{bmatrix} $so the final $W = \begin{bmatrix}W_{11}&W_{12}&W_{13}\\ W_{21}&W_{22}&W_{23}\\ \end{bmatrix}$Let $X = \begin{bmatrix}x_1&x_2\\ \end{bmatrix}$$z = \begin{bmatrix}x_1&x_2\\ \end{bmatrix} . \begin{bmatrix}W_{11}&W_{12}&W_{13}\\ W_{21}&W_{22}&W_{23}\\ \end{bmatrix} + \begin{bmatrix}b_1&b_2&b_3\\ \end{bmatrix}$- For 1 data, $(1,2).(2,3) + (1,3) = (1,3)$- For n data, $(n,2).(2,3) + (1,3) = (n,3)$, b will be added to all n data, this is called broadcasting.Frameworks like Tensorflow, PyTorch will take care of this matrix form of $W$ and $b$ for you.  Task-2Train a Softmax Regression with Tensorflow<jupyter_code># Data

from sklearn.datasets import make_blobs

X, y = make_blobs(n_samples=1000, n_features=2, centers=3, random_state=42)
X.shape, y.shape, set(y)<jupyter_output><empty_output><jupyter_text>We need to convert the labels into one hot vectors to train the model. Let's use keras to_categorical function<jupyter_code>print(y.shape)

y = to_categorical(y, num_classes=3)
print(y.shape)<jupyter_output><empty_output><jupyter_text>Train-Validation-Test SplitSo far we had a dataset and we used it for training and checked the accuracy/loss to see the performance. But its not the right way to check the preformance of a model. Usually any dataset is split into 3 parts namely Train-Validation-Test. train setThis dataset is used to train the model. If the training is good, metrics of this dataset will be always good. Almost 60-70% of dataset is given to training set. validation setAfter every epoch(generally) of training, metrics of this dataset is checked to ensure, the model is also performing good on unseen data as much as it performs on the training dataset. If the model performs well on training dataset but not good on validation set, it means the model has a problem called 'Overfitting' whcih we will look in more detail later. Some hyper parameters are adjusted to make the model perform well in validation set as well during training. 10-20% of data is given to validation set. testing setAfter the training is over for n epochs, when the model performs good in both training and validation sets, a final check is done to see the performance of the model on new unseen dataset. 20-30% of data is given to Test set.The percentage numbers depends on the total number of data we have access to, which you will understand as you work on more projects.We can split the dataset into train-test using ```sklearn.model_selection.train_test_split```<jupyter_code>from sklearn.model_selection import train_test_split

print(X.shape, y.shape)
# test_size is the percent of split 0.2 means 20% of data is for testset.
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

print(X_train.shape, X_test.shape, y_train.shape, y_test.shape)<jupyter_output><empty_output><jupyter_text>Tensorflow Model- Make a Dataset with 2 input features, 3 output classes- one hot encode y- Split Dataset into Train-Validation-Test- Train Model with Validation Dataset, check the [docs](https://keras.io/models/model/) on how to use validation data.- Do not forget to change the input shape and units in Dense layer and use "softmax" activation ```keras.layers.Activation('softmax')```- Use "categorical_crossentropy" loss- predict X_test with the trained model, refer the docs(model.predict function)- convert the prediction of X_test and y_test from one-hot to labels using np.argmax(pred, axis=1)- use sklearn.metrics.accuracy_score on prediction of X_test and y_test to find the accuracy on Test set.If you are stuck somewhere before checking the solution- check the keras docs- google search will help and you will learn many possible solutions in stackoverflow- then go to the solutions<jupyter_code>import tensorflow as tf
from tensorflow import keras
import matplotlib.pyplot as plt
import numpy as np
from sklearn.datasets import make_blobs

# Make the Dataset
num_classes = 3
num_input_features = 2
X, y = 

# to categorical
y = 

# train-test split
# 20% of dataset to testset
X_train, X_test, y_train, y_test = 

# train-validation split
# 20% of trainset to valset
X_train, X_val, y_train, y_val = 

# Model
model = 
model.compile()
tf_history = model.fit()

# Prediction for Test set with trained Model
from sklearn.metrics import accuracy_score
y_test_pred = 
test_accuracy = 

print('\nTest Accuracy = ', test_accuracy)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11265523358959195, 'fraction_numerical': 0.031243839936920953, 'mean_word_length': 4.618493909191583, 'pattern_counts': {'":': 0, '<': 42, '<?xml version=': 0, '>': 43, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '310228', 'license': 'MIT', 'n_tokens_mistral': 3496, 'n_tokens_neox': 3234, 'n_words': 1284, 'path': '1-Intro-to-Deep-Learning/.ipynb_checkpoints/1_4_1_MultiClass_Classification-checkpoint.ipynb', 'repo_name': 'shangeth/Google-Explore-ML-Deep-Learning', 'size': '10154'}