<jupyter_start><jupyter_text><jupyter_code>import seaborn as sns
import pandas as pd
import matplotlib.pyplot as plt
from pandas.plotting import parallel_coordinates

data = sns.load_dataset("iris")<jupyter_output><empty_output><jupyter_text>**Task 1.	Brief description on its value and possible applications.**The IRIS dataset has the following characteristics:* 150 examples of Iris flowers * The first four fields are features that are the characteristics of flower examples. All these fields hold float numbers representing flower measurements. * The last column is the label which represents the Iris species. * Balance class distribution meaning that each category has even amount of instances * Has no missing values One example of possible application is for botanists to find an automated way to categorize each Iris flower they find. For instance, to classify based on photographs, or in our case based on the length and width measurements of their sepals and petals.<jupyter_code>data
print(f'CLASS DISTRIBUTION:\n{data.groupby("species").size()}')
print(f'\nSHAPE: {data.shape}')
print(f'\nTOTAL MISSING VALUES:\n{data.isnull().sum()}\n')<jupyter_output>CLASS DISTRIBUTION:
species
setosa        50
versicolor    50
virginica     50
dtype: int64

SHAPE: (150, 5)

TOTAL MISSING VALUES:
sepal_length    0
sepal_width     0
petal_length    0
petal_width     0
species         0
dtype: int64<jupyter_text>_________  **Task 2. Summarize and visually report on the Size of this data set including labeling or non-labeled status** For all three species, the respective values of the mean and median of its features are found to be pretty close. This indicates that data is nearly symmetrically distributed with very less presence of outliers.<jupyter_code>data.groupby('species').agg(['mean', 'median'])<jupyter_output><empty_output><jupyter_text>Standard deviation (or variance) is an indication of how widely the data is spread about the mean.<jupyter_code>data.groupby('species').std()<jupyter_output><empty_output><jupyter_text>The isolated points for each feature that can be seen in the box-plots below are the outliers in the data. Since these are very few in number, it wouldn't have any significant impact on our analysis.<jupyter_code>sns.set(style="ticks") 
plt.figure(figsize=(12,10))
plt.subplot(2,2,1)
sns.boxplot(x='species',y='sepal_length',data=data)
plt.subplot(2,2,2)
sns.boxplot(x='species',y='sepal_width',data=data)
plt.subplot(2,2,3)
sns.boxplot(x='species',y='petal_length',data=data)
plt.subplot(2,2,4)
sns.boxplot(x='species',y='petal_width',data=data)
plt.show()<jupyter_output><empty_output><jupyter_text>Scatter plot helps to analyze the relationship between 2 features on the x and y<jupyter_code>sns.pairplot(data)<jupyter_output><empty_output><jupyter_text>Next, we can make a correlation matrix to see how these features are correlated to each other using a heatmap in the seaborn library. It can be observed that petal measurements are highly correlated, while the sepal one are uncorrelated. Also we can see that petal length is highly correlated with speal length, but not with sepal width.<jupyter_code>plt.figure(figsize=(10,11))
sns.heatmap(data.corr(),annot=True, square = True)
plt.plot()<jupyter_output><empty_output><jupyter_text>Another way to visualize the data is by parallel coordinate plot, which represents each row as a line. As we have seen below, petal measurements can separate species better than the sepal ones.<jupyter_code>parallel_coordinates(data, "species", color = ['blue', 'red', 'green']);<jupyter_output><empty_output><jupyter_text>Now, we can plot a scatter plot between the sepal length and the sepal width to visualise the iris dataset. We can observe that the blue dots(setosa) are quite clear separated from red(versicolor) and green dots(virginica), while separation between red dots and green dots might be a very difficult task given the two features available.<jupyter_code>labels_names = { 'setosa': 'blue',
                'versicolor': 'red',
                'virginica': 'green'}

for species, color in labels_names.items():
  x = data.loc[data['species'] == species]['sepal_length']
  y = data.loc[data['species'] == species]['sepal_width']
  plt.scatter(x, y, c=color)
plt.legend(labels_names.keys())
plt.xlabel('sepal_length')
plt.ylabel('sepal_width')
plt.show()<jupyter_output><empty_output><jupyter_text>We can also visualise the data on different features such as petal width and petal length. In this case, the decision boundary between blue, green and red dots can be easily determined, which indicates that using all features for training is a good choice.<jupyter_code>labels_names = { 'setosa': 'blue',
                'versicolor': 'red',
                'virginica': 'green'}

for species, color in labels_names.items():
  x = data.loc[data['species'] == species]['petal_length']
  y = data.loc[data['species'] == species]['petal_width']
  plt.scatter(x, y, c=color)
plt.legend(labels_names.keys())
plt.xlabel('petal_length')
plt.ylabel('petal_width')
plt.show()<jupyter_output><empty_output><jupyter_text>___________ **3.	Propose and perform Deep Learning using this data set.**Report on your implementation as follows:* Justify your selection of techniques and platform* Explain your results and their applicability &nbsp;&nbsp;&nbsp;&nbsp;In our project we are using python language. There are two well-known libraries for deep learning such as PyTorch and Tensorflow. Each library has its own API implementation for example, Keras is high level API for Tensorflow, while fastai is an API for PyTorch.&nbsp;&nbsp;&nbsp;&nbsp;The Iris classification problem is an example of supervised machine learning: the model is trained from examples that contain labels and for our model we are planning to use Keras wrapper for Tensor flow.&nbsp;&nbsp;&nbsp;&nbsp;The Deep Learning would be performed in the following steps:  * Data preprocessing  * Model Building  * Model Selection &nbsp;&nbsp;&nbsp;&nbsp;In the **Data preprocessing**, we need to create data frames for features and labels, normalize the feature data by converting all values in a range between 0 and 1, convert species labels to numerical representation and then to binary string. Then, the data needs to be split into train and test data sets.   **Phase 1: Data Preprocessing**  Step 1: Create Dataframes for features and lables<jupyter_code>import pandas as pd
from sklearn.preprocessing import LabelBinarizer, LabelEncoder


encoder = LabelBinarizer()
le=LabelEncoder() 
seed = 42

data = sns.load_dataset("iris")
# Create X variable with four features
X = data.drop(['species'],axis=1)
# Convert species to int
Y_int = le.fit_transform(data['species'])
# Convert species int to binary representation
Y_binary = encoder.fit_transform(Y_int)
target_names = data['species'].unique()
Y = pd.DataFrame(data=Y_binary, columns=target_names) 

print(f'\nNormalized X_test values:\n{X[:5]}')
print(f'\nEncoded Y_test:\n{Y[:5]}')<jupyter_output>Normalized X_test values:
   sepal_length  sepal_width  petal_length  petal_width
0           5.1          3.5           1.4          0.2
1           4.9          3.0           1.4          0.2
2           4.7          3.2           1.3          0.2
3           4.6          3.1           1.5          0.2
4           5.0          3.6           1.4          0.2

Encoded Y_test:
   setosa  versicolor  virginica
0       1           0          0
1       1           0          0
2       1           0          0
3       1           0          0
4       1           0          0<jupyter_text>Step 2: Create training and testing datasets<jupyter_code>from sklearn.model_selection import train_test_split

# Split data in train and test with percentage proportion 70%/30%
X_train,X_test,y_train,y_test = train_test_split(X, Y, test_size=0.30,random_state=seed)

print(f'X_train: {X_train.shape}, y_train: {y_train.shape}')
print(f'X_test : {X_test.shape}, y_test : {y_test.shape}')<jupyter_output>X_train: (105, 4), y_train: (105, 3)
X_test : (45, 4), y_test : (45, 3)<jupyter_text>Step 3: Normalize the feature data, all values should be in a range from 0 to 1<jupyter_code>import pandas as  pd
 from sklearn import preprocessing
 
# Normalize X features, make all values between 0 and 1
X_train = pd.DataFrame(preprocessing.normalize(X_train),
                               columns=X_train.columns,
                               index=X_train.index)
X_test = pd.DataFrame(preprocessing.normalize(X_test),
                           columns=X_test.columns,
                           index=X_test.index)

print(f'Train sample:\n{X_train.head(4)},\nShape: {X_train.shape}')
print(f'\nTest sample:\n{X_test.head(4)},\nShape: {X_test.shape}')<jupyter_output>Train sample:
     sepal_length  sepal_width  petal_length  petal_width
81       0.772429     0.337060      0.519634     0.140442
133      0.723660     0.321627      0.585820     0.172300
137      0.698048     0.338117      0.599885     0.196326
75       0.767857     0.349026      0.511905     0.162879,
Shape: (105, 4)

Test sample:
     sepal_length  sepal_width  petal_length  petal_width
73       0.736599     0.338111      0.567543     0.144905
18       0.806828     0.537885      0.240633     0.042465
118      0.706006     0.238392      0.632655     0.210885
78       0.733509     0.354530      0.550132     0.183377,
Shape: (45, 4)<jupyter_text>**Phase 2: Model Building** Step 1: Build model&nbsp;&nbsp;&nbsp;&nbsp;The IRIS is a classification problem, we need to classify if an Iris flower is setosa, versicolor or virginia. Softmax activation function is commonly used in multi classification problems in the output layer, that would return the label with the highest probability. &nbsp;&nbsp;&nbsp;&nbsp;The **tf.keras.Sequential** model is a linear stack of layers. Its constructor takes a list of layer instances, in this case, one tf.keras.layers.Dense layer with 8 nodes, two layers with 10 nodes, and an output layer with 3 nodes representing our label predictions. The first layer’s input_shape parameter corresponds to the number of features from the dataset which is equal 4. &nbsp;&nbsp;&nbsp;&nbsp;The **activation** function determines the output shape of each node in the layer. These non linearities are important, without them the model would be equivalent to a single layer. There are many tf.keras.activations such as tanh, like sigmoid or relu. In our two models we have decided to use "tahn" and "relu" and compare the performance.&nbsp;&nbsp;&nbsp;&nbsp;The ideal number of hidden layers and neurons depends on the problem and the dataset. Like many aspects of machine learning, picking the best shape of the neural network requires a mixture of knowledge and experimentation. As a rule of thumb, increasing the number of hidden layers and neurons typically creates a more powerful model, which requires more data to train effectively. For our illustration we have used two models with 3 and 4 layers. Our expectation that the model with many layers should give a better result.<jupyter_code>from keras.models import Sequential
from keras.layers import Dense

def model_with_3_layers():
  model = Sequential()
  model.add(Dense(27, input_dim=4, activation='relu', name='input_layer'))
  model.add(Dense(9, activation='relu', name='layer_1'))
  model.add(Dense(3, activation='softmax', name='output_layer'))

  model.compile(loss='categorical_crossentropy',
                optimizer='adam',
                metrics=['accuracy'])
  return model


def model_with_4_layers():
    """build the Keras model callback"""
    model = Sequential()
    model.add(Dense(8, input_dim=4, activation='tanh', name='layer_1'))
    model.add(Dense(10, activation='tanh', name='layer_2'))
    model.add(Dense(10, activation='tanh', name='layer_3'))
    model.add(Dense(3, activation='softmax', name='output_layer'))
     
    model.compile(loss="categorical_crossentropy",
                  optimizer="adam",
                  metrics=['accuracy'])
    return model<jupyter_output><empty_output><jupyter_text>Step 2: Create estimator&nbsp;&nbsp;&nbsp;&nbsp;We can also pass arguments in the construction of the KerasClassifier class that will be passed on to the fit() function internally used to train the neural network. Here, we pass the number of epochs as 200 and batch size as 20 to use when training the model.<jupyter_code>from keras.wrappers.scikit_learn import KerasClassifier

estimator = KerasClassifier(
        build_fn=model_with_4_layers,
        epochs=200, batch_size=20,
        verbose=0)<jupyter_output><empty_output><jupyter_text>Step 3: Evaluate The Model with k-Fold Cross Validation&nbsp;&nbsp;&nbsp;&nbsp;Now, the neural network model can be evaluated on a training dataset. The scikit-learn has excellent capability to evaluate models using a suite of techniques. The gold standard for evaluating machine learning models is k-fold cross validation.Since the dataset is quite small, we can pass 5 fold for cross validation.<jupyter_code>from sklearn.model_selection import KFold
from sklearn.model_selection import cross_val_score
import tensorflow as tf

# Suppress Tensorflow warning
tf.compat.v1.logging.set_verbosity(tf.compat.v1.logging.ERROR)

estimator = KerasClassifier(
        build_fn=model_with_3_layers,
        epochs=200, batch_size=20,
        verbose=0)

kfold = KFold(n_splits=5, shuffle=True, random_state=seed)
results = cross_val_score(estimator, X_train, y_train, cv=kfold)
print(f'Model Performance:\nmean: {results.mean()*100:.2f}\
        \nstd: {results.std()*100:.2f}') 
from sklearn.model_selection import KFold
from sklearn.model_selection import cross_val_score

estimator = KerasClassifier(
        build_fn=model_with_4_layers,
        epochs=200, batch_size=20,
        verbose=0)

kfold = KFold(n_splits=5, shuffle=True, random_state=seed)
results = cross_val_score(estimator, X_train, y_train, cv=kfold)
print(f'Model Performance:\nmean: {results.mean()*100:.2f}\
        \nstd: {results.std()*100:.2f}')<jupyter_output>Model Performance:
mean: 98.10        
std: 2.33<jupyter_text>Phase 3 : Model Selection&nbsp;&nbsp;&nbsp;&nbsp; For our illustration, two models have been used. One with 3 layers and another with 4 layers. We can observe that the accuracy are almost the same, but the loss value is much lower in the model with 4 layers. It can be concluded that by adding more layers, it improves the accuracy and lost and at the same time requires more computational power.<jupyter_code>md1 = model_with_3_layers()
md1.fit(X_train,
       y_train,
       epochs=200,
       shuffle=True, # shuffle data randomly.
       verbose=0 # this will tell keras to print more detailed info
       )

# Validate the model with test dateset
test_error_rate = md1.evaluate(X_test, y_test, verbose=0)
print(f'{md1.metrics_names[1]}: {test_error_rate[1]*100:.2f}')
print(f'{md1.metrics_names[0]}: {test_error_rate[0]*100:.2f}')
md2 = model_with_4_layers()
md2.fit(X_train,
       y_train,
       epochs=200,
       shuffle=True, # shuffle data randomly.
       verbose=0 # this will tell keras to print more detailed info
       )

# Validate the model with test dateset
test_error_rate = md2.evaluate(X_test, y_test, verbose=0)
print(f'{md2.metrics_names[1]}: {test_error_rate[1]*100:.2f}')
print(f'{md2.metrics_names[0]}: {test_error_rate[0]*100:.2f}')<jupyter_output>accuracy: 95.56
loss: 11.00<jupyter_text>STEP 4: Evaluate model performs on the test data<jupyter_code>from sklearn.metrics import confusion_matrix


def evaluate_performace(actual, expected):
    """
    Function accepts two lists with actual and expected lables
    """
    flowers = {0:'setosa',
               1:'versicolor',
               2:'virginica'}
    print(f'Flowers in test set: \nSetosa={y_test["setosa"].sum()}\
          \nVersicolor={y_test["versicolor"].sum()}\
          \nVirginica={y_test["virginica"].sum()}')
    
    for act,exp in zip(actual, expected):
        if act != exp:
            print(f'ERROR: {flowers[exp]} predicted as {flowers[act]}')


for i,model in enumerate((md1, md2), 1):
  print(f'\nEVALUATION OF MODEL {i}')
  predicted_targets = model.predict_classes(X_test)
  true_targets = encoder.inverse_transform(y_test.values)
  evaluate_performace(predicted_targets, true_targets)
   # Calculate the confusion matrix using sklearn.metrics
  fig, ax =plt.subplots(1,1)
  conf_matrix = confusion_matrix(true_targets, predicted_targets)
  sns.heatmap(conf_matrix, annot=True, cmap='Blues', xticklabels=target_names,yticklabels=target_names)
  print('\n')<jupyter_output>EVALUATION OF MODEL 1
Flowers in test set: 
Setosa=19          
Versicolor=13          
Virginica=13
ERROR: versicolor predicted as virginica
ERROR: virginica predicted as versicolor



EVALUATION OF MODEL 2
Flowers in test set: 
Setosa=19          
Versicolor=13          
Virginica=13
ERROR: versicolor predicted as virginica
ERROR: virginica predicted as versicolor<jupyter_text>&nbsp;&nbsp;&nbsp;&nbsp;From the confusion matrix above we can see that the second model with 4 layers outperformed the model with 3 layers and the prediction was wrong only once for versicolor species.  ___ **4.	Find a publication or report that uses this same data set and compare it’s methodology and results to what you did**&nbsp;&nbsp;&nbsp;&nbsp;In the last assignment, we would like to analyze the approach that has been suggested by TensorFlow in "Custom Training: walkthrough" report [1]. The same deep learning framework has been used. Feature and labels are stored in tf.Tensor structure, where in my model all data was stored in pandas.DataFrame. Label data is converted to numerical representation, where in our side we have decided to use binary string representation. The Author decided to not normalize the feature data, to be more specific to represent in the range from 0 to 1. It is a preferable approach, because it allows the model to learn faster.&nbsp;&nbsp;&nbsp;&nbsp;Suggested model is using a Sequential model which is a linear stack of layers. The stack is built with 4 layers, input and output layers and two Dense layers with 10 nodes each can be simply represented as 4/10/10/3. One of our models that showed better accuracy and loss contains 5 layers which can be represented as follows: 4/8/10/10/3. The relu activation function has been chosen for inner layers, that outputs 0 if input is negative or 0, and returns the value when it is positive. Both models are using **SparseCategoricalCrossentropy** function which calculates the loss value by taking the model's class probability predictions and the desired labels, and returns the average loss across all examples. To minimize the loss the Stochastic gradient algorithm has been chosen with learning rate 0.01, in contrast our model is built with Adam which is an extension to stochastic gradient descent algorithm.&nbsp;&nbsp;&nbsp;&nbsp;Both models are run with almost the same number of epochs. It can be observed that both models return almost the same accuracy and loss.&nbsp;&nbsp;&nbsp;&nbsp;To summarize, we can see that both models performed similarly, but in our approach to the same result can be achieved by adding the new inner layer that does help to improve the model but it might be very resource-intensive.**References:**[1] - "Custom training: walkthrough", https://colab.research.google.com/github/tensorflow/docs/blob/master/site/en/tutorials/customization/custom_training_walkthrough.ipynbscrollTo=rwxGnsA92emp, Accessed 2018, The TensorFlow Authors<jupyter_code># To convert colab notebook to pdf
!apt-get install texlive texlive-xetex texlive-latex-extra pandoc >/dev/null
!pip install pypandoc >/dev/null
from google.colab import drive
drive.mount('/content/drive')
!cp drive/My\ Drive/Colab\ Notebooks/HW2.ipynb ./
!jupyter nbconvert --to PDF "HW2.ipynb" 2>/dev/null
!cp ./HW2.pdf drive/My\ Drive/Colab\ Notebooks/<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.08867933992075036, 'fraction_numerical': 0.03019511461102473, 'mean_word_length': 4.288594164456233, 'pattern_counts': {'":': 0, '<': 70, '<?xml version=': 0, '>': 73, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '19', 'dir': 'jupyter-structured-clean-dedup', 'id': '747', 'license': 'MIT', 'n_tokens_mistral': 6319, 'n_tokens_neox': 5859, 'n_words': 2125, 'path': 'HW2/HW2.ipynb', 'repo_name': 'DSNortsev/CSE-694-Case-Studies-in-Deep-Learning', 'size': '19937'}