<jupyter_start><jupyter_text>Problem statement: To build a CNN based model which can accurately detect melanoma. Melanoma is a type of cancer that can be deadly if not detected early. It accounts for 75% of skin cancer deaths. A solution which can evaluate images and alert the dermatologists about the presence of melanoma has the potential to reduce a lot of manual effort needed in diagnosis.  Importing Skin Cancer Data To do: Take necessary actions to read the data  Importing all the important libraries<jupyter_code>import pathlib
import tensorflow as tf
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import os
import PIL
from tensorflow import keras
from tensorflow.keras import layers
from tensorflow.keras.models import Sequential
## If you are using the data by mounting the google drive, use the following :
## from google.colab import drive
## drive.mount('/content/gdrive')

##Ref:https://towardsdatascience.com/downloading-datasets-into-google-drive-via-google-colab-bcb1b30b0166<jupyter_output><empty_output><jupyter_text>This assignment uses a dataset of about 2357 images of skin cancer types. The dataset contains 9 sub-directories in each train and test subdirectories. The 9 sub-directories contains the images of 9 skin cancer types respectively.<jupyter_code># Defining the path for train and test images
## Todo: Update the paths of the train and test dataset
data_dir_train = pathlib.Path("path_to_train_directory")
data_dir_test = pathlib.Path('path_to_test_directory')
image_count_train = len(list(data_dir_train.glob('*/*.jpg')))
print(image_count_train)
image_count_test = len(list(data_dir_test.glob('*/*.jpg')))
print(image_count_test)<jupyter_output><empty_output><jupyter_text>Load using keras.preprocessingLet's load these images off disk using the helpful image_dataset_from_directory utility.  Create a datasetDefine some parameters for the loader:<jupyter_code>batch_size = 32
img_height = 180
img_width = 180<jupyter_output><empty_output><jupyter_text>Use 80% of the images for training, and 20% for validation.<jupyter_code>## Write your train dataset here
## Note use seed=123 while creating your dataset using tf.keras.preprocessing.image_dataset_from_directory
## Note, make sure your resize your images to the size img_height*img_width, while writting the dataset
train_ds = ##todo
## Write your validation dataset here
## Note use seed=123 while creating your dataset using tf.keras.preprocessing.image_dataset_from_directory
## Note, make sure your resize your images to the size img_height*img_width, while writting the dataset
val_ds = ##todo
# List out all the classes of skin cancer and store them in a list. 
# You can find the class names in the class_names attribute on these datasets. 
# These correspond to the directory names in alphabetical order.
class_names = train_ds.class_names
print(class_names)<jupyter_output><empty_output><jupyter_text>Visualize the data Todo, create a code to visualize one instance of all the nine classes present in the dataset<jupyter_code>import matplotlib.pyplot as plt

### your code goes here, you can use training or validation data to visualize<jupyter_output><empty_output><jupyter_text>The `image_batch` is a tensor of the shape `(32, 180, 180, 3)`. This is a batch of 32 images of shape `180x180x3` (the last dimension refers to color channels RGB). The `label_batch` is a tensor of the shape `(32,)`, these are corresponding labels to the 32 images. `Dataset.cache()` keeps the images in memory after they're loaded off disk during the first epoch.`Dataset.prefetch()` overlaps data preprocessing and model execution while training.<jupyter_code>AUTOTUNE = tf.data.experimental.AUTOTUNE
train_ds = train_ds.cache().shuffle(1000).prefetch(buffer_size=AUTOTUNE)
val_ds = val_ds.cache().prefetch(buffer_size=AUTOTUNE)<jupyter_output><empty_output><jupyter_text>Create the model Todo: Create a CNN model, which can accurately detect 9 classes present in the dataset. Use ```layers.experimental.preprocessing.Rescaling``` to normalize pixel values between (0,1). The RGB channel values are in the `[0, 255]` range. This is not ideal for a neural network. Here, it is good to standardize values to be in the `[0, 1]`<jupyter_code>### Your code goes here<jupyter_output><empty_output><jupyter_text>Compile the modelChoose an appropirate optimiser and loss function for model training<jupyter_code>### Todo, choose an appropirate optimiser and loss function
model.compile(optimizer='your_optimser',
              loss=your_loss_function_goes_here,
              metrics=['accuracy'])
# View the summary of all layers
model.summary()<jupyter_output><empty_output><jupyter_text>Train the model<jupyter_code>epochs = 20
history = model.fit(
  train_ds,
  validation_data=val_ds,
  epochs=epochs
)<jupyter_output><empty_output><jupyter_text>Visualizing training results<jupyter_code>acc = history.history['accuracy']
val_acc = history.history['val_accuracy']

loss = history.history['loss']
val_loss = history.history['val_loss']

epochs_range = range(epochs)

plt.figure(figsize=(8, 8))
plt.subplot(1, 2, 1)
plt.plot(epochs_range, acc, label='Training Accuracy')
plt.plot(epochs_range, val_acc, label='Validation Accuracy')
plt.legend(loc='lower right')
plt.title('Training and Validation Accuracy')

plt.subplot(1, 2, 2)
plt.plot(epochs_range, loss, label='Training Loss')
plt.plot(epochs_range, val_loss, label='Validation Loss')
plt.legend(loc='upper right')
plt.title('Training and Validation Loss')
plt.show()<jupyter_output><empty_output><jupyter_text>Todo: Write your findings after the model fit, see if there is an evidence of model overfit or underfit  Write your findings here<jupyter_code># Todo, after you have analysed the model fit history for presence of underfit or overfit, choose an appropriate data augumentation strategy. 
# Your code goes here
# Todo, visualize how your augmentation strategy works for one instance of training image.
# Your code goes here<jupyter_output><empty_output><jupyter_text>Todo: Create the model, compile and train the model<jupyter_code>## You can use Dropout layer if there is an evidence of overfitting in your findings

## Your code goes here<jupyter_output><empty_output><jupyter_text>Compiling the model<jupyter_code>## Your code goes here<jupyter_output><empty_output><jupyter_text>Training the model<jupyter_code>## Your code goes here, note: train your model for 20 epochs
history = ## your training code<jupyter_output><empty_output><jupyter_text>Visualizing the results<jupyter_code>acc = history.history['accuracy']
val_acc = history.history['val_accuracy']

loss = history.history['loss']
val_loss = history.history['val_loss']

epochs_range = range(epochs)

plt.figure(figsize=(8, 8))
plt.subplot(1, 2, 1)
plt.plot(epochs_range, acc, label='Training Accuracy')
plt.plot(epochs_range, val_acc, label='Validation Accuracy')
plt.legend(loc='lower right')
plt.title('Training and Validation Accuracy')

plt.subplot(1, 2, 2)
plt.plot(epochs_range, loss, label='Training Loss')
plt.plot(epochs_range, val_loss, label='Validation Loss')
plt.legend(loc='upper right')
plt.title('Training and Validation Loss')
plt.show()<jupyter_output><empty_output><jupyter_text>Todo: Write your findings after the model fit, see if there is an evidence of model overfit or underfit. Do you think there is some improvement now as compared to the previous model run?  **Todo:** Find the distribution of classes in the training dataset. **Context:** Many times real life datasets can have class imbalance, one class can have proportionately higher number of samples compared to the others. Class imbalance can have a detrimental effect on the final model quality. Hence as a sanity check it becomes important to check what is the distribution of classes in the data.<jupyter_code>## Your code goes here.<jupyter_output><empty_output><jupyter_text>**Todo:** Write your findings here:  - Which class has the least number of samples? - Which classes dominate the data in terms proportionate number of samples?  **Todo:** Rectify the class imbalance **Context:** You can use a python package known as `Augmentor` (https://augmentor.readthedocs.io/en/master/) to add more samples across all classes so that none of the classes have very few samples.<jupyter_code>!pip install Augmentor<jupyter_output><empty_output><jupyter_text>To use `Augmentor`, the following general procedure is followed:1. Instantiate a `Pipeline` object pointing to a directory containing your initial image data set.2. Define a number of operations to perform on this data set using your `Pipeline` object.3. Execute these operations by calling the `Pipeline’s` `sample()` method.<jupyter_code>path_to_training_dataset="To do"
import Augmentor
for i in class_names:
    p = Augmentor.Pipeline(path_to_training_dataset + i)
    p.rotate(probability=0.7, max_left_rotation=10, max_right_rotation=10)
    p.sample(500) ## We are adding 500 samples per class to make sure that none of the classes are sparse.<jupyter_output><empty_output><jupyter_text>Augmentor has stored the augmented images in the output sub-directory of each of the sub-directories of skin cancer types.. Lets take a look at total count of augmented images.<jupyter_code>image_count_train = len(list(data_dir_train.glob('*/output/*.jpg')))
print(image_count_train)<jupyter_output><empty_output><jupyter_text>Lets see the distribution of augmented data after adding new images to the original training data.<jupyter_code>path_list = [x for x in glob(os.path.join(data_dir_train, '*','output', '*.jpg'))]
path_list
lesion_list_new = [os.path.basename(os.path.dirname(os.path.dirname(y))) for y in glob(os.path.join(data_dir_train, '*','output', '*.jpg'))]
lesion_list_new
dataframe_dict_new = dict(zip(path_list_new, lesion_list_new))
df2 = pd.DataFrame(list(dataframe_dict_new.items()),columns = ['Path','Label'])
new_df = original_df.append(df2)
new_df['Label'].value_counts()<jupyter_output><empty_output><jupyter_text>So, now we have added 500 images to all the classes to maintain some class balance. We can add more images as we want to improve training process.  **Todo**: Train the model on the data created using Augmentor<jupyter_code>batch_size = 32
img_height = 180
img_width = 180<jupyter_output><empty_output><jupyter_text>**Todo:** Create a training dataset<jupyter_code>data_dir_train="path to directory with training data + data created using augmentor"
train_ds = tf.keras.preprocessing.image_dataset_from_directory(
  data_dir_train,
  seed=123,
  validation_split = 0.2,
  subset = ## Todo choose the correct parameter value, so that only training data is refered to,,
  image_size=(img_height, img_width),
  batch_size=batch_size)<jupyter_output><empty_output><jupyter_text>**Todo:** Create a validation dataset<jupyter_code>val_ds = tf.keras.preprocessing.image_dataset_from_directory(
  data_dir_train,
  seed=123,
  validation_split = 0.2,
  subset = ## Todo choose the correct parameter value, so that only validation data is refered to,
  image_size=(img_height, img_width),
  batch_size=batch_size)<jupyter_output><empty_output><jupyter_text>**Todo:** Create your model (make sure to include normalization)<jupyter_code>## your code goes here<jupyter_output><empty_output><jupyter_text>**Todo:** Compile your model (Choose optimizer and loss function appropriately)<jupyter_code>## your code goes here<jupyter_output><empty_output><jupyter_text>**Todo:**  Train your model<jupyter_code>epochs = 30
## Your code goes here, use 50 epochs.
history = # your model fit code<jupyter_output><empty_output><jupyter_text>**Todo:**  Visualize the model results<jupyter_code>acc = history.history['accuracy']
val_acc = history.history['val_accuracy']

loss = history.history['loss']
val_loss = history.history['val_loss']

epochs_range = range(epochs)

plt.figure(figsize=(8, 8))
plt.subplot(1, 2, 1)
plt.plot(epochs_range, acc, label='Training Accuracy')
plt.plot(epochs_range, val_acc, label='Validation Accuracy')
plt.legend(loc='lower right')
plt.title('Training and Validation Accuracy')

plt.subplot(1, 2, 2)
plt.plot(epochs_range, loss, label='Training Loss')
plt.plot(epochs_range, val_loss, label='Validation Loss')
plt.legend(loc='upper right')
plt.title('Training and Validation Loss')
plt.show()<jupyter_output><empty_output><jupyter_text>**Todo:**  Analyze your results here. Did you get rid of underfitting/overfitting? Did class rebalance help?<jupyter_code><jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09289311695579183, 'fraction_numerical': 0.011111999360460468, 'mean_word_length': 6.604863221884498, 'pattern_counts': {'":': 0, '<': 113, '<?xml version=': 0, '>': 113, 'https://': 2, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '28', 'dir': 'jupyter-structured-clean-dedup', 'id': '125583', 'license': 'Apache-2.0', 'n_tokens_mistral': 3996, 'n_tokens_neox': 3778, 'n_words': 1268, 'path': '12. CNN Assignment/.ipynb_checkpoints/Starter_code_Assignment_CNN_Skin_Cancer-checkpoint.ipynb', 'repo_name': 'vshubh24/machineLearning', 'size': '12509'}