<jupyter_start><jupyter_text>First Year Project Project 3 - Effectiveness of Skin Cancer PredictionThis notebook contains all of the code developed for project 3, completing tasks similar to data scientists working for a dermatologist to investigate whether some characteristics of skin lesions can be reliably measure with a smartphone app.The goal is to measure at least 2 of the following characteristics in a set of skin lesion images; asymmetry, border, and color.Then, we will try to assess how good the measurements are, by predicting the diagnosis of the skin lesions based on these features.We will focus on the **Melanoma** form of skin cancer.Group 3:<NAME> (<EMAIL>)<NAME> (<EMAIL>)<NAME> (<EMAIL>)<NAME> (<EMAIL>)<NAME> (<EMAIL>)Created: 07-04-2021Last Modified: 18-04-2021  Data Sources Data was provided by the ISIC challenge data sets. https://challenge.isic-archive.com/dataCodella N, <NAME>, <NAME>, <NAME>, <NAME>, <NAME>, <NAME>, <NAME>, <NAME>, <NAME>, <NAME>. "Skin Lesion Analysis Toward Melanoma Detection: A Challenge at the 2017 International Symposium on Biomedical Imaging (ISBI), Hosted by the International Skin Imaging Collaboration (ISIC)". arXiv: 1710.05006 [cs.CV]  Imports Libraries used in this notebook.<jupyter_code>import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import skimage
from skimage.transform import rotate
from skimage import morphology
from skimage import measure
import math

from sklearn.neighbors import KNeighborsClassifier
from sklearn.preprocessing import StandardScaler
from sklearn.metrics import classification_report,\
confusion_matrix, accuracy_score, plot_confusion_matrix, recall_score
from sklearn import tree
from sklearn.utils import resample
from sklearn.ensemble import RandomForestClassifier<jupyter_output><empty_output><jupyter_text>Constants Constants to access data on the directory structure of this repository.<jupyter_code>TRAIN = './data/training/' 
VALID = './data/validation/'
TEST = './data/test/'

IMG = 'example_image/'
SEG = 'example_segmentation/'
FEAT = 'features/'
TRUTH = 'ground_truth.csv'<jupyter_output><empty_output><jupyter_text>Functions Functions created for this notebook.<jupyter_code>"""FUNCTIONS FOR MASKED IMAGES"""

def get_boundaries(image):
    """Function to locate the boundaries of the lesion over the whole image.
    Takes a segmentation mask image as argument and returns the upper, lower, left and right boundaries."""

    mask = np.where(image == 1)
    left = min(mask[1])
    right = max(mask[1])
    upper = min(mask[0])
    lower = max(mask[0])
    return upper, lower, left, right

def get_center(image): # NOT NEEDED ANYMORE ?
    """Function that takes an image as input, and returns the centerpoint of the lesion."""
    up, dw, lt, rt = get_boundaries(image)
    center = ((up+dw)/2, (lt+rt)/2)
    return center
    
def zoom(image):
    """Function to zoom-in (crop) the lesion from blank space. Takes a segmentation mask image as input,
    and returns the rectangle where the lesion is found."""

    up, dw, lt, rt = get_boundaries(image)
    rectangle = image[up:dw+1, lt:rt+1]
    return rectangle

def cuts(image):
    """Function to perform a double cut (vertical and horizontal) of the lesion. Takes a segmentation mask image as input,
    and returns the vertical and horizontal cuts (2 for each dimension). It handles uneven shapes."""

    center_h = image.shape[0] // 2 # The image shape contains a tuple with height and width (in pixels)
    if image.shape[0] % 2 == 0: # If the height is an even number of pixels, the cut returns 2 equal sides
        upside = image[:center_h,:]
        downside = image[center_h:,:]
    else: # If the height is an uneven number of pixels, the cut has to "share" the center, to return 2 equal sides
        upside = image[:center_h,:]
        downside = image[center_h+1:,:]
        
    center_w = image.shape[1] // 2    
    if image.shape[1] % 2 == 0:
        leftside = image[:,:center_w]
        rightside = image[:,center_w:]
    else:
        leftside = image[:,:center_w]
        rightside = image[:,center_w+1:]
    return upside, downside, leftside, rightside


def test_symmetry(image, rot_deg=5):
    """Function to test the symmetry of an image. Takes a segmentation mask image and the rotation degree interval and
    returns a symmetry score between zero (non-symmetric) to one (completely symmetric)."""

    assert (rot_deg <= 90) and (rot_deg >= 0), "Rotation degree should be positive and at most 90 deg"
    optimal = 0
    
    for deg in range(0,91, rot_deg):
        rot_image = skimage.transform.rotate(image, deg, resize=True)
        z = zoom(rot_image)
        
        upside, downside, leftside, rightside = cuts(z)

        up_dw = np.sum(np.bitwise_and(upside.astype(int), np.flipud(downside).astype(int))) /\
        np.sum(np.bitwise_or(upside.astype(int), np.flipud(downside).astype(int)))

        lt_rt = np.sum(np.bitwise_and(leftside.astype(int), np.fliplr(rightside).astype(int))) /\
        np.sum(np.bitwise_or(leftside.astype(int), np.fliplr(rightside).astype(int)))
    
        symmetry = (up_dw+lt_rt)/2
        
        if symmetry > optimal: optimal = symmetry

    return symmetry
    
def rgb2gray(rgb):
    """Function to convert a RGB image to grayscale."""
    return np.dot(rgb[...,:3], [0.2989, 0.5870, 0.1140])

def crop(image, mask, resize=True, warning=True):
    if image.shape[:2] != mask.shape[:2]:
        if warning:
            print("Image and Mask must have the same size. OPERATION CANCELLED.")
        else: return
    else:
        img = image.copy()
        img[mask==0] = 0

        if resize:
            u,d,l,r = get_boundaries(mask)
            img = img[u:d,l:r,...]
        return img

def color_std(image):
    """A function that takes an image as input, computes and returns the average standard deviation of all the
    rgb color values."""
    R = image[np.where(image[:,:,0] != 0) and np.where(image[:,:,1] != 0) and np.where(image[:,:,2] != 0)][:,0]
    G = image[np.where(image[:,:,0] != 0) and np.where(image[:,:,1] != 0) and np.where(image[:,:,2] != 0)][:,1]
    B = image[np.where(image[:,:,0] != 0) and np.where(image[:,:,1] != 0) and np.where(image[:,:,2] != 0)][:,2]
    color_std = (np.std(R) + np.std(G) + np.std(B)) /3
    #except:
    #    color_std = 'NA'
    return color_std

def check_border(image, border=0.01, tolerance=0.2, warning=True):
    """Function to check if the lesion might be exceeding the image. Take the following arguments:
    - image: segmentation mask image to check.
    - border: the percentage of pixels to consider as a border. 10% by default.
    - tolerance: the percentage of tolerance for a lesion to be at the border of the image. 20% by default.
    - warning: boolean to indicate if a textual warning should be issue when checking the border. True by default."""
    h = int(image.shape[0] * border)
    w = int(image.shape[1] * border)
    up = (np.sum(image[h,:]) / image.shape[1]) > tolerance
    dw = (np.sum(image[-h,:]) / image.shape[1]) > tolerance
    lt = (np.sum(image[:,w]) / image.shape[0]) > tolerance
    rt = (np.sum(image[:,w]) / image.shape[0]) > tolerance
    if warning:
        if up or dw or lt or rt: return "This lesion might be overflowing the image"
        else: return "This lesion does not seem to be overflowing the image"
    else:
        return up or dw or lt or rt


def masker(image, sens):
    '''Takes image, converts to a grayscale image, and returns a masked 
    image that only shows values below the sensitivity given as input.'''
    
    gray = rgb2gray(image) # Create grayscale image
    img2 = gray < sens # **This level needs manually adjusting, also need to be able to automate**
    
    # use plt.imshow(masker(image,sens), cmap='gray') to see image
    
    return img2.astype(int)

def dimensions(mask1):
    '''calculates height(max) and width(90 deg to height)
        returns height, width, rotated mask image, degree of rotation'''
    pixels_in_col = np.max(np.sum(mask1, axis=0))

    rot = 0
    max_col = 0
    rot_max = 0
    for _ in range(9):
        rot_im = transform.rotate(mask1,rot)
        pixels_in_col = np.max(np.sum(rot_im, axis=0))
        if pixels_in_col > max_col:
            max_col = pixels_in_col
            rot_max = rot
            pixels_in_row = np.max(np.sum(rot_im, axis=1))
        rot += 10

    return max_col, pixels_in_row, rot_max

def measure_area_perimeter(mask):
    """A function that takes either a segmented image or perimeter 
    image as input, and calculates the length of the perimeter of a lesion."""
    
    # Measure area: the sum of all white pixels in the mask image
    area = np.sum(mask)

    # Measure perimeter: first find which pixels belong to the perimeter.
    perimeter = measure.perimeter(mask)
    
    return area, perimeter

def predict(bi_image): # Predict might be a little confusing ?
    
    area = np.sum(bi_image)
    _, peri = perimeter(bi_image)
    
    area_from_peri = pi*((peri/(2*pi))**2)
    peri_from_area = 2*pi*sqrt(area/pi)
    
    return area, area_from_peri, peri, peri_from_area<jupyter_output><empty_output><jupyter_text>Directories Creation (NOTE) We load and access the metadata included in our source data.<jupyter_code>df = {} # A main dictionary will hold our different labels for datasets

df['train'] = {'path': TRAIN, 'label': pd.read_csv(TRAIN + TRUTH, index_col='image_id')}
df['validation'] = {'path': VALID, 'label': pd.read_csv(VALID + TRUTH, index_col='image_id')}
df['test'] = {'path': TEST, 'label': pd.read_csv(TEST + TRUTH, index_col='image_id')}<jupyter_output><empty_output><jupyter_text>Exploratory Data Analysis (Task 0)  Dataset Cleaning and Manipulation  Figure compression In order to work more efficiently, we decide to crop the images to reduce their dimensions to the part of the image that contains the lesion. For this, we will use our segmented masks, as following: the color images will be cropped to the rectangle where the lesion is, and saving the new image with reduced dimensions. This process must be done only once.<jupyter_code>WARN = input("This operation may take several minutes. Do you wish to continue: (Yes/No) ")

print("\n----- PLEASE BE PATIENT -----\n")

if WARN.lower().startswith("y"):
    i = 1
    for k, v in df.items():
        for img_id in v['label'].index:
            imgpath = v['path'] + IMG + img_id + '.jpg'
            mskpath = v['path'] + SEG + img_id + '_segmentation.png'
            img = plt.imread(imgpath)
            msk = plt.imread(mskpath)
            new = crop(img, msk, warning=False)
            if new is None:
                pass
            else: plt.imsave(imgpath, new)
            print(f'\rResizing image # {i}', end='\r')
            i += 1
            
else: print("OPERATION CANCELLED")<jupyter_output><empty_output><jupyter_text>We check for lesions overflowing the image Here we go through the images and find those that are potentially too zoomed in and cut off areas of the border. Since this study focused on asymmetry and compactness of lesions, it is necessary to have images that show the entire border. The output is a list of images that are potentially cutting off the border. We then manually went through to verify the quality of the border in the images before excluding them from the model.<jupyter_code># We loop through all images applying our automatic border detection function

WARN = input("This operation may take several minutes. Do you wish to continue: (Yes/No) ")

print("\n----- PLEASE BE PATIENT -----\n")

if WARN.lower().startswith("y"):
    with open('./to_check.csv', 'w') as outfile:
        outfile.write('image_id'+','+'from Dataset'+'\n')
        for k, data in df.items():
                for img_id in data['label'].index:
                    img = plt.imread(data['path'] + SEG + img_id + '_segmentation.png') 
                    if check_border(img, warning=False) == True:
                        outfile.write(img_id+','+data['path']+'\n')
else:
    print("OPERATION CANCELLED")
# After careful manual inspection, we read the file and filter the images to be ignored.
# We then exclude them from our datasets.

visual_inspection = pd.read_csv('./to_check.csv')
to_be_ignored = visual_inspection[visual_inspection.loc[:,'Visual inspection'] == 'Ignore']
mela, kera, non = 0,0,0
for ix, row in to_be_ignored.iterrows():
    try:
        image = row.loc['image_id']
        from_dataset = row.loc['from Dataset']
        if df[from_dataset]['label'].loc[image,'melanoma'] == 1:
            mela += 1
        elif df[from_dataset]['label'].loc[image,'seborrheic_keratosis'] == 1:
            kera += 1
        else: non += 1
        df[from_dataset]['label'].drop(image, axis=0, inplace=True)
    except:
        image = row.loc['image_id']
        from_dataset = row.loc['from Dataset']
        if df[from_dataset]['label'].loc[image,'melanoma'] == 1:
            mela += 1
        elif df[from_dataset]['label'].loc[image,'seborrheic_keratosis'] == 1:
            kera += 1
        else: non += 1
            
print(f'{to_be_ignored.shape[0]} images excluded from the model.')
print(f'{mela} melanomas, {kera} keratosis and {non} benigns')<jupyter_output><empty_output><jupyter_text>Visualize the Data (NOTE) Here is shown how we visualize and mask the images to get familiar with the images and their attributes. The lesion images present in the dataset have been cropped and masked using the crop function.  Load Image and Segmentation Image Side-by-Side<jupyter_code>IM = 'ISIC_0000000'
image = plt.imread(df['train']['path']+IMG+IM+'.jpg')
seg = plt.imread(df['train']['path']+SEG+IM+'_segmentation.png')

fig, axes = plt.subplots(nrows=1, ncols=2, figsize=(7, 5))
axes[0].imshow(image)
axes[1].imshow(seg, cmap='gray')
fig.tight_layout()<jupyter_output><empty_output><jupyter_text>Showing how the zoom function works This shows how the image was cropped so that the border is cut to the edges of the lesion, resulting in the cropped image of the lesion seen above on the left.<jupyter_code>plt.imshow(zoom(seg), cmap='gray')
plt.show()<jupyter_output><empty_output><jupyter_text>Convert an image to grayscale This shows how the rgb2gray function works to convert an image to grayscale.<jupyter_code>plt.imshow(rgb2gray(image), cmap='gray')<jupyter_output><empty_output><jupyter_text>VIsualize the masker function This function was written to create our own segmentation images. When the image has been masked with a segmentation image already, the function outputs differently. Since the images in this repository have all already been masked, it is not as clear to see how this functino would work. In addition, since we already had access to segmentation images, we did not further explore optimization of this technique for large amounts of images.<jupyter_code>plt.imshow(masker(image,140), cmap='gray')<jupyter_output><empty_output><jupyter_text>Visualize output of function for Area and Perimeter This shows the output of the function that shows area and perimeter of a lesion, as well as a visual display of the perimeter of the lesion.<jupyter_code>area, perimeter = measure_area_perimeter(seg)
print(f'Area: {area}\nPerimeter: {perimeter}')

struct_el = morphology.disk(1)
mask_eroded = morphology.binary_erosion(seg, struct_el)
image_perimeter = seg - mask_eroded
plt.imshow(image_perimeter, cmap='gray')
plt.show()<jupyter_output><empty_output><jupyter_text>Output of the cuts function The output of this function is then used in the test_symmetry function.<jupyter_code>u,d,l,r = cuts(seg)
fig, ((ax1, ax2), (ax3, ax4)) = plt.subplots(2,2, figsize=(5,3), dpi=350)
ax1.imshow(u, cmap='gray')
ax2.imshow(d, cmap='gray')
ax3.imshow(l, cmap='gray')
ax4.imshow(r, cmap='gray')

plt.show()<jupyter_output><empty_output><jupyter_text>Result of the test_symmetry function<jupyter_code>test_symmetry(seg)<jupyter_output><empty_output><jupyter_text>This lesion shows a 76.65% symmetry, which lines up well with the visual symmetry of the image.  Feature Extraction (NOTE) We proceed to extract features of interest for our predictive model  Asymmetry To test for asymmetry we run a function to calculate a score based on how similar an image is when cut horizontally and vertically. We assign a score between 0 (non asymmetric) and 1 (totally asymmetric) for both cuts, and we take the average to convey a unique value.<jupyter_code># We perform a test on a black circle, where it's symmetry should be close to 1
test_circle = plt.imread('./data/test-black-circle.png') # Load the circle test
test_circle = test_circle[:,:,0]
print(f'Symmetry test for black circle: {test_symmetry(test_circle):.2%}')
DATASET = input("Which dataset to calculate? [train, validation, test] ")
if DATASET.lower() not in ['train', 'validation', 'test']:
    print('OPERATION CANCELLED')
else:
    data = df[DATASET]['label']

    DoBatch = int(input("How many batches? "))
    if DoBatch > data.shape[0]:
        DoBatch = data.shape[0]
    batch = int(input("Do batch # "))
    assert batch <= DoBatch, "Wrong Batch #"

    WARN = input("This operation may take several minutes. Do you wish to continue: (Yes/No) ")

    REWRITE = input("Do you wish to overwrite the /symmetry.csv file?: (Yes/No) ")
    print("\n----- PLEASE BE PATIENT -----\n")


    length = data.shape[0] // DoBatch
    start = length * (batch - 1)
    end = length * (batch)

    if WARN.lower().startswith("y"):
        symmetry = {}
        i = 1
        for ix, row in data[start:end].iterrows():
            file_path = df[DATASET]['path'] + SEG + str(ix) + "_segmentation.png"
            image = plt.imread(file_path)

            ptg = round(i / length,2)
            print(f'\rCalculating symmetry: {ptg:.2%}', end='\r')
            symmetry[ix] = test_symmetry(image)
            i += 1
            %xdel image
    else: print("OPERATION CANCELLED")

    if REWRITE.lower().startswith("y"):
        with open(df[DATASET]['path'] + FEAT + f'symmetry_{str(batch)}.csv', 'w') as outfile:
            outfile.write('image_id'+','+'symmetry'+'\n')
            for k, v in symmetry.items():
                line = k +','+str(v)
                outfile.write(line+'\n')<jupyter_output><empty_output><jupyter_text>Border (Compactness method) To test for border smoothness we use the compactness method. Compactness is defined as the ratio of thearea of an object to the area of a circle with the same perimeter.The measure takes a maximum value of 1 for a circle.<jupyter_code># We perform a test on a black circle, where it's compactness should be close to 1
circle_area, circle_perimeter = measure_area_perimeter(test_circle)
circle_compactness = (4* math.pi * circle_area) / (circle_perimeter**2)
print(f'Compactness test for black circle: {circle_compactness:.2%}')
DATASET = input("Which dataset to calculate? [train, validation, test] ")
if DATASET.lower() not in ['train', 'validation', 'test']:
    print('OPERATION CANCELLED')
else:
    data = df[DATASET]['label']

    DoBatch = int(input("How many batches? "))
    if DoBatch > data.shape[0]:
        DoBatch = data.shape[0]
    batch = int(input("Do batch # "))
    assert batch <= DoBatch, "Wrong Batch #"

    WARN = input("This operation may take several minutes. Do you wish to continue: (Yes/No) ")

    REWRITE = input("Do you wish to overwrite the /compactness.csv file?: (Yes/No) ")
    print("\n----- PLEASE BE PATIENT -----\n")


    length = data.shape[0] // DoBatch
    start = length * (batch - 1)
    end = length * (batch)

    if WARN.lower().startswith("y"):
        compactness = {}
        i = 1
        for ix, row in data[start:end].iterrows():
            file_path = df[DATASET]['path'] + SEG + str(ix) + "_segmentation.png"
            image = plt.imread(file_path)

            ptg = round(i / length,2)
            print(f'\rCalculating compactness: {ptg:.2%}', end='\r')
            area, per = measure_area_perimeter(image)
            compactness[ix] = (4* math.pi * area) / (per**2)
            i += 1
    else: print("OPERATION CANCELLED")

    if REWRITE.lower().startswith("y"):
        with open(df[DATASET]['path'] + FEAT + f'compactness_{str(batch)}.csv', 'w') as outfile:
            outfile.write('image_id'+','+'compactness'+'\n')
            for k, v in compactness.items():
                line = k +','+str(v)
                outfile.write(line+'\n')<jupyter_output><empty_output><jupyter_text>Color In order to evaluate the difference in lesion colors, we take the standard deviation for each of the 3 RGB channels of the cropped image (to reduce noise), and then we average the 3 deviations to obtain a unique value.<jupyter_code>DATASET = input("Which dataset to calculate? [train, validation, test] ")
if DATASET.lower() not in ['train', 'validation', 'test']:
    print('OPERATION CANCELLED')
else:
    data = df[DATASET]['label']

    DoBatch = int(input("How many batches? "))
    if DoBatch > data.shape[0]:
        DoBatch = data.shape[0]
    batch = int(input("Do batch # "))
    assert batch <= DoBatch, "Wrong Batch #"

    WARN = input("This operation may take several minutes. Do you wish to continue: (Yes/No) ")

    REWRITE = input("Do you wish to overwrite the /color_deviation.csv file?: (Yes/No) ")
    print("\n----- PLEASE BE PATIENT -----\n")


    length = data.shape[0] // DoBatch
    start = length * (batch - 1)
    end = length * (batch)

    if WARN.lower().startswith("y"):
        color_deviation = {}
        i = 1
        for ix, row in data[start:end].iterrows():
            file_path = df[DATASET]['path'] + IMG + str(ix) + ".jpg"
            image = plt.imread(file_path)

            ptg = round(i / length,2)
            print(f'\rCalculating color deviation: {ptg:.2%}', end='\r') 
            color_deviation[ix] = color_std(image)
            i += 1
    else: print("OPERATION CANCELLED")

    if REWRITE.lower().startswith("y"):
        with open(df[DATASET]['path'] + FEAT + f'color_deviation{str(batch)}.csv', 'w') as outfile:
            outfile.write('image_id'+','+'color_deviation'+'\n')
            for k, v in color_deviation.items():
                line = k +','+str(v)
                outfile.write(line+'\n')<jupyter_output><empty_output><jupyter_text>Aggregating features to datasets We will now add the recently extracted features to our main dataset<jupyter_code>feat_list = ['symmetry', 'compactness', 'color_deviation']
for dataset in df.keys():
    for feat in feat_list:
        symmetry = pd.read_csv(df[dataset]['path'] + FEAT + 'symmetry.csv', index_col='image_id') 
        compactness = pd.read_csv(df[dataset]['path'] + FEAT + 'compactness.csv', index_col='image_id')
        color_deviation = pd.read_csv(df[dataset]['path'] + FEAT + 'color_deviation.csv', index_col='image_id')
        df[dataset]['features'] = symmetry.merge(compactness, how = 'inner', on = 'image_id')\
        .merge(color_deviation, how = 'inner', on = 'image_id')

df['train']['features'].head()<jupyter_output><empty_output><jupyter_text>Analysis of Features  Feature Scaling As our features Symmetry and Compactness have a metric between zero and one, we will only scale the color deviation feature to match the same range as the others.<jupyter_code>for dataset in df.keys():
    scaled_color_deviation = []
    to_scale = df[dataset]['features'].color_deviation
    for i in to_scale.iteritems():
        new_x = (i[1] - np.min(to_scale)) /\
        (np.max(to_scale) - np.min(to_scale))
        scaled_color_deviation.append(new_x)
    df[dataset]['features'].color_deviation = scaled_color_deviation
    
# Example of Dataframe with scaling
df['train']['features'].head()
train_set = df['train']['label'].merge(df['train']['features'], how = 'inner', on = 'image_id')
train_set.drop('seborrheic_keratosis', axis= 1, inplace=True)

train_set.loc[train_set.melanoma == 1, 'melanoma'] = "Melanoma"
train_set.loc[train_set.melanoma == 0, 'melanoma'] = "Non-Melanoma"
train_set.columns = ['label'] + list(train_set.columns)[1:]<jupyter_output><empty_output><jupyter_text>Resampling<jupyter_code>melanoma = train_set.loc[train_set.label == "Melanoma"]
n = train_set.shape[0]//2 - melanoma.shape[0] # We want 1/3 to be melanoma
resample_melanoma = resample(melanoma, n_samples=n)
train_set_rs = train_set.append(resample_melanoma)
print(f'The new training set contains now {train_set_rs[train_set_rs.label == "Melanoma"].shape[0]}\
 Melanoma observations and {train_set_rs[train_set_rs.label != "Melanoma"].shape[0]}\
 non Melanoma observations.')
fig, ax = plt.subplots(3,2, figsize=(15,12), dpi=350)
fig.suptitle("Scatterplots of Lesion Features")
fig.tight_layout(h_pad=2)
i = 0
for f1 in feat_list:
    for f2 in feat_list:
        if f1 != f2:
            if i < 3:
                axis = (i, 0)
            else:
                axis = (i-3, 1)
            ax[0,0] = sns.scatterplot(x=f1, y=f2, data=train_set_rs, hue='label', ax=ax[axis])
            i += 1
#plt.show()
plt.savefig("./reports/figures/features_scatter.png")            
positive = train_set_rs.loc[train_set_rs.label == "Melanoma"]
negative = train_set_rs.loc[train_set_rs.label == "Non-Melanoma"]
fig, axs = plt.subplots(3,1, figsize=(7,5), dpi=350)
fig.suptitle("Melanoma Vs Non-Melanoma Feature Distributions")
fig.tight_layout()

sns.kdeplot(x = 'symmetry', data=positive, cumulative= False, shade=True, clip=(0,1), color='r',label="Melanoma",ax=axs[0])
sns.kdeplot(x = 'symmetry', data=negative, cumulative= False, shade=True, clip=(0,1), color='g',label="Non-Melanoma",ax=axs[0])
axs[0].axvline(np.mean(positive.symmetry), ymin= 0, ymax= 0.78, color='r', linestyle = 'dashed')
axs[0].axvline(np.mean(negative.symmetry), ymin= 0, ymax= 0.80, color='g', linestyle = 'dashed')
axs[0].legend(loc="upper left")

sns.kdeplot(x = 'compactness', data=positive, cumulative= False, shade=True, clip=(0,1), color='r',label="Melanoma",ax=axs[1])
sns.kdeplot(x = 'compactness', data=negative, cumulative= False, shade=True, clip=(0,1), color='g',label="Non-Melanoma",ax=axs[1])
axs[1].axvline(np.mean(positive.compactness), ymin= 0, ymax= 0.67, color='r', linestyle = 'dashed')
axs[1].axvline(np.mean(negative.compactness), ymin= 0, ymax= 0.80, color='g', linestyle = 'dashed')
#axs[1].legend(loc="upper left")

sns.kdeplot(x = 'color_deviation', data=positive, cumulative= False, shade=True, clip=(0,1), color='r',label="Melanoma",ax=axs[2])
sns.kdeplot(x = 'color_deviation', data=negative, cumulative= False, shade=True, clip=(0,1), color='g',label="Non-Melanoma",ax=axs[2])
axs[2].axvline(np.mean(positive.color_deviation), ymin= 0, ymax= 0.77, color='r', linestyle = 'dashed')
axs[2].axvline(np.mean(negative.color_deviation), ymin= 0, ymax= 0.88, color='g', linestyle = 'dashed')
#axs[2].legend(loc="upper right")

#plt.show()
plt.savefig("./reports/figures/densitySubplots.png")
fig, ax = plt.subplots(dpi=250)
fig.suptitle("Melanoma Vs Non-Melanoma Symmetry Density ")
fig.tight_layout()

sns.kdeplot(x = 'symmetry', data=positive, cumulative= False, shade=True, clip=(0,1), color='r',label="Melanoma")
sns.kdeplot(x = 'symmetry', data=negative, cumulative= False, shade=True, clip=(0,1), color='g',label="Non-Melanoma")
ax.axvline(np.mean(positive.symmetry), ymin= 0, ymax= 0.78, color='r', linestyle = 'dashed')
ax.axvline(np.mean(negative.symmetry), ymin= 0, ymax= 0.80, color='g', linestyle = 'dashed');

plt.legend(loc="upper left")

plt.savefig("./reports/figures/sym_density.png")
fig, ax = plt.subplots(dpi=250)
fig.suptitle("Melanoma Vs Non-Melanoma Compactness Density")
fig.tight_layout()

sns.kdeplot(x = 'compactness', data=positive, cumulative= False, shade=True, clip=(0,1), color='r',label="Melanoma")
sns.kdeplot(x = 'compactness', data=negative, cumulative= False, shade=True, clip=(0,1), color='g',label="Non-Melanoma")
ax.axvline(np.mean(positive.compactness), ymin= 0, ymax= 0.67, color='r', linestyle = 'dashed')
ax.axvline(np.mean(negative.compactness), ymin= 0, ymax= 0.80, color='g', linestyle = 'dashed');

plt.legend(loc="upper left")

plt.savefig("./reports/figures/compact_density.png")
fig, ax = plt.subplots(dpi=250)
fig.suptitle("Melanoma Vs Non-Melanoma Color Density")
fig.tight_layout()

sns.kdeplot(x = 'color_deviation', data=positive, cumulative= False, shade=True, clip=(0,1), color='r',label="Melanoma")
sns.kdeplot(x = 'color_deviation', data=negative, cumulative= False, shade=True, clip=(0,1), color='g',label="Non-Melanoma")
ax.axvline(np.mean(positive.color_deviation), ymin= 0, ymax= 0.77, color='r', linestyle = 'dashed')
ax.axvline(np.mean(negative.color_deviation), ymin= 0, ymax= 0.88, color='g', linestyle = 'dashed');

plt.legend(loc="upper right")

plt.savefig("./reports/figures/color_density.png")<jupyter_output><empty_output><jupyter_text>From the plots is difficult to see any correlation. We will take each feature by pairs and see if they are correlated.<jupyter_code>print('For Melanoma, the features have the following correlation coefficients:\n')
feat_iter = feat_list.copy()
for f1 in feat_iter:
    for f2 in feat_iter:
        if f1 != f2:
            corr = np.corrcoef(positive[f1], positive[f2])
            print(f'Corr. {f1} and {f2}: {corr[0,1]:.2%}')
    feat_iter.remove(f1)
    
print('\nFor Non-Melanoma, the features have the following correlation coefficients:\n')
feat_iter = feat_list.copy()
for f1 in feat_iter:
    for f2 in feat_iter:
        if f1 != f2:
            corr = np.corrcoef(negative[f1], negative[f2])
            print(f'Corr. {f1} and {f2}: {corr[0,1]:.2%}')
    feat_iter.remove(f1)<jupyter_output><empty_output><jupyter_text>We see a positive correlation between features for Melanoma lesions, but smaller correlations for Non-Melanoma, with even a negative correlation between Symmetry and Color Deviation. Should we take another approach in our splitting of Melanoma/Non-Melanoma?  Models We first create the set with features and labels for validation and test data.<jupyter_code># For Validation Data
validation_set = df['validation']['label'].merge(df['validation']['features'],\
                                                 how = 'inner', on = 'image_id')

# For Test Data
test_set = df['test']['label'].merge(df['test']['features'], how = 'inner', on = 'image_id')

labels = []
for ix, row in validation_set.iterrows():
    if row['melanoma'] == 1.0:
        labels.append("Melanoma")
    else:
        labels.append("Non-Melanoma")
        
validation_set['label'] = labels
validation_set.drop("melanoma", axis=1, inplace=True)
validation_set.drop("seborrheic_keratosis", axis=1, inplace=True)  

labels = []
for ix, row in test_set.iterrows():
    if row['melanoma'] == 1.0:
        labels.append("Melanoma")
    else:
        labels.append("Non-Melanoma")
        
test_set['label'] = labels
test_set.drop("melanoma", axis=1, inplace=True)
test_set.drop("seborrheic_keratosis", axis=1, inplace=True)
# We move the label column to the last place to match the other datasets
cols = list(train_set_rs.columns)
cols = cols[1:] + [cols[0]]
train_set_rs = train_set_rs[cols]
train_set_rs.head()<jupyter_output><empty_output><jupyter_text>Data splitting<jupyter_code># For Training Data
X_train = train_set_rs.iloc[:,:-1].reset_index(drop=True).values
y_train = train_set_rs.iloc[:,-1].reset_index(drop=True).values

# For Validation Data
X_valid = validation_set.iloc[:,:-1].reset_index(drop=True).values
y_valid = validation_set.iloc[:,-1].reset_index(drop=True).values

# For Test Data
X_test = test_set.iloc[:,:-1].reset_index(drop=True).values
y_test = test_set.iloc[:,-1].reset_index(drop=True).values<jupyter_output><empty_output><jupyter_text>K-Nearest Neighbors  Selecting best K We will try the model on a different range of Ks to find the optimal one.<jupyter_code>scores_KNN = []

# Calculating score for K values between 1 and 40
for i in range(1, 40):
    KNN = KNeighborsClassifier(n_neighbors=i)
    KNN.fit(X_train, y_train)
    pred_i = KNN.predict(X_valid)
    scores_KNN.append(recall_score(y_valid, pred_i, average='macro'))
plt.figure(figsize=(12, 6))
plt.plot(range(1, 40), scores_KNN, color='blue', linestyle='dashed', marker='o',
         markerfacecolor='green', markersize=8)
plt.title('Scores for K Values')
plt.xlabel('K Value')
plt.ylabel('Model Score');
best_k = [i for i, x in enumerate(scores_KNN) if x == max(scores_KNN)]
best_k[0]<jupyter_output><empty_output><jupyter_text>Model fitting<jupyter_code>KNN = KNeighborsClassifier(n_neighbors=best_k[0])
KNN.fit(X_train, y_train)<jupyter_output><empty_output><jupyter_text>Model prediction on Validation Data<jupyter_code>y_pred_KNN = KNN.predict(X_test)
print(classification_report(y_test, y_pred_KNN))
print(f'Overall Model Accuracy: {accuracy_score(y_test, y_pred_KNN):.2%}')
disp = plot_confusion_matrix(KNN, X_test, y_test,
                                 display_labels=['Melanoma', "Non-Melanoma"],
                                 cmap=plt.cm.Blues);
disp.ax_.set_title("Quality of KNN Classifier")
plt.savefig("./reports/figures/knnConfMat.png")<jupyter_output><empty_output><jupyter_text>PRECISION: is the ability of a classifier not to label an instance positive that is actually negative. For each class it is defined as the ratio of true positives to the sum of true and false positives.TP – True PositivesFP – False PositivesPrecision – Accuracy of positive predictions.Precision = TP/(TP + FP)Our model achieved a precision of 0.16 for Melanoma and 0.80 for Non-Melanoma, with a joint accuracy of 73.50 %.---------RECALL: is the ability of a classifier to find all positive instances. For each class it is defined as the ratio of true positives to the sum of true positives and false negatives.FN – False NegativesRecall: Fraction of positives that were correctly identified.Recall = TP/(TP+FN)Our model did not perform specially well on predicting Melanoma instances.---------The F1 score is a weighted harmonic mean of precision and recall such that the best score is 1.0 and the worst is 0.0. Generally speaking, F1 scores are lower than accuracy measures as they embed precision and recall into their computation. As a rule of thumb, the weighted average of F1 should be used to compare classifier models, not global accuracy.F1 Score = 2*(Recall * Precision) / (Recall + Precision)  Decision Tree<jupyter_code>DT = tree.DecisionTreeClassifier(random_state=0).fit(X_train, y_train)
y_pred_DT = DT.predict(X_test)
print(classification_report(y_test, y_pred_DT))
print(f'Overall Model Accuracy: {accuracy_score(y_test, y_pred_DT):.2%}')
disp = plot_confusion_matrix(DT, X_test, y_test,
                                 display_labels=['Melanoma', "Non-Melanoma"],
                                 cmap=plt.cm.Blues)
disp.ax_.set_title("Quality of Decision Tree Classifier")
plt.savefig("./reports/figures/deciTreeConfMat.png")<jupyter_output><empty_output><jupyter_text>Our Decision Tree model performs a little better than the KNN, with 19% of true melanomas detected and 81% of non-melanoma correctly detected.<jupyter_code>RF = RandomForestClassifier()
RF.fit(X_train, y_train)
preds = RF.predict(X_valid)
plot_confusion_matrix(RF, X_valid, y_valid,
                                 display_labels=['Melanoma', 'Non-Melanoma'],
                                 cmap=plt.cm.Blues);<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13165567413026863, 'fraction_numerical': 0.013021201913550542, 'mean_word_length': 3.947206273631144, 'pattern_counts': {'":': 0, '<': 135, '<?xml version=': 0, '>': 139, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '27', 'dir': 'jupyter-structured-clean-dedup', 'id': '67095', 'license': 'MIT', 'n_tokens_mistral': 12033, 'n_tokens_neox': 11393, 'n_words': 3343, 'path': 'notebooks/dee.ipynb', 'repo_name': 'dmdequin/fyp2021p3g03', 'size': '35437'}