<jupyter_start><jupyter_text>Project 3: Skin Lesion Detection in Medical Treatment using Machine Learning Detecting Skin Cancer Type 'Melanoma' through Machine Learning---**Group 9: <NAME>, <NAME>, <NAME>, <NAME>**Submission: *23.04.2021* / Last Modified: *22.04.2021*---This notebook contains the step-by-step data science process performed on the *ISIC 2017* public test data and official training data on medical image recognition. The goal of this project was to extract and automatically analyse features from medical images of skin lesions in order to predict whether or not the lesion is *melanoma* using image processing and machine learning classification algorithms.The initial data (containing the medical images, masked images and information on features and disease) was given for 150 medical images (equivalent to the public test data of the *ISIC 2017* challenge) by the project manager *Veronika *.To develop more accurate models, we extended the initially given data by the official training data that could be obtained from the official [ISIC 2017 Website](https://challenge.isic-archive.com/data). In a later part of the notebook, we will explain how to download the images for reproducing the results of this notebook.  Introduction---When diagnosing the skin cancer **melanoma**, there are some key features doctors look for. Melanoma is typically: Asymmetrical, has an uneven Border, non-uniformly Colored, larger in Diameter than a pencil eraser, and can over time causing bleeding and other effects. This common guide for spotting melanoma is known as the ABCDE model.A quick search on mobile phone app stores will show dozens of different skin diagnosis apps, where a user can take a photo of their affected area of skin and get an unofficial diagnosis of what the condition could be. These apps calculate some of the aforementioned features from images to estimate a diagnosis. Medical imaging is also widely used to monitor and diagnose several conditions by medical professionals (*[Source](https://www.postdicom.com/en/blog/medical-imaging-science-and-applications)*). Thus there is great demand for machine learning algorithms to aid in diagnosing large volumes of such image data, thereby reducing the workload of doctors.This notebook contains all the code to explore the digital diagnosis process, by analysing images of skin lesions and using image processing and machine learning to attempt to accurately diagnose whether or not a lesion is melanoma. This formed the research question:> **To what degree of certainty can we predict if a patient has melanoma from a given medical image using machine learning?**  Running this Notebook---This notebook contains all code to reproduce the findings of the project as can be seen on the [GitHub](https://github.com/jonas-mika/fyp2021p03g09) page of this project. If one downloads the images used within this project, it is important that the global paths are correctly set. This can either be achieved through locally changing the file structure (ie. naming folders as defined in the notebook) or by adjusting the global variables within the `Constants` section of this notebook.*Note that the rest of the file structure as can be seen on the [GitHub](https://github.com/jonas-mika/fyp2021p03g09) page of the project generates automatically*  Required Libraries ---Throughout the project, we will use a range of both built-in and external Python Libraries. This notebook will only run if all libraries and modules are correctly installed on your local machines. To install missing packages use `pip install ` (PIP (Python Package Index) is the central package management system, read more [here](https://pypi.org/project/pip/)). You can do this directly within the notebook by uncommenting the below cell.In case you desire further information about the used packages, click the following links to find detailed documentations:- [Pandas](https://pandas.pydata.org/)- [Numpy](https://numpy.org/)- [Matplotlib](https://matplotlib.org/stable/index.html)- [PIL](https://pillow.readthedocs.io/en/stable/)- [SciKit Learn](https://scikit-learn.org/stable/)- [SciKit Image](https://scikit-image.org/)- [Scipy](https://www.scipy.org/)- [ImbLearn](https://imbalanced-learn.org/stable/)<jupyter_code>%%capture 
# stop cell from displaying output

# uncomment lines with uninstalled packages on local machine
#!pip install scikit-image
#!pip install imblearn
#!pip install scikit-learn
#!pip install pillow
#!pip install itertools
# python standard libraries
import os                                              # automates saving of export files (figures, summaries, ...)
import re                                              # used for checking dateformat in data cleaning

# external libraries
import pandas as pd                                    # provides major datastructure pd.DataFrame() to store datasets
import numpy as np                                     # used for numerical calculations and fast array manipulations
import matplotlib.pyplot as plt                        # standard data visualisation
import seaborn as sns                                  # convenient data visualisation
from PIL import Image                                  # fork from PIL (python image library), image processing in python

# specific functions
import matplotlib.cm as cm
from skimage import morphology
#from scipy.stats.stats import mode

# preprocessing
from sklearn.preprocessing import StandardScaler # normalise features
from imblearn.under_sampling import RandomUnderSampler # balancing data
from imblearn.over_sampling import RandomOverSampler # balancing data
from imblearn.over_sampling import SMOTE # balancing data

# feature selection
from sklearn.feature_selection import mutual_info_classif, SelectKBest # univariate feature selection with mutual information for feature scoring
from sklearn.feature_selection import SequentialFeatureSelector # sequential selection of k-features given fixed classifier 

# classifiers used in this project
from sklearn.ensemble import RandomForestClassifier
from sklearn.neighbors import KNeighborsClassifier # k-nearest neighbour classifier

# model selection
from sklearn.model_selection import train_test_split
from sklearn.model_selection import KFold  # producing equal cross-validation sets (reproducable results)
from sklearn.model_selection import cross_val_score # perform cross-validation and report single metric
from sklearn.model_selection import cross_validate # perform cross-validation and report set of metrics
from sklearn.model_selection import GridSearchCV # perform cross-validation on given set of hyperparameters to find best hyperparamters optimising single metric
from imblearn.pipeline import Pipeline, make_pipeline # balancing on each k-th training set in cross validation

# evaluating model
from sklearn.metrics import classification_report
from sklearn.metrics import accuracy_score
from sklearn.metrics import recall_score<jupyter_output><empty_output><jupyter_text>Downloading Additional Data---The data analysis was performed on the concatenated data of the 150 initially given images and 2000 additional images obtained from the same data source - the [ISIC](https://challenge.isic-archive.com/data) (International Skin Imaging Collaboration) 2017 Competition. The links automatically start the download from the website - so be aware of clicking them, if no big downloads are intended.**REMARK: Not downloading the images will cause the image processing and feature extraction part of the notebook to not run properly. However, these sections can be skipped, by simply loading in the `feature.csv` dataframe that is provided in the submission**> Raw Data>> [Raw Data Images](https://isic-challenge-data.s3.amazonaws.com/2017/ISIC-2017_Test_v2_Data.zip)>> [Raw Data Masks](https://isic-challenge-data.s3.amazonaws.com/2017/ISIC-2017_Test_v2_Part1_GroundTruth.zip)>> [Raw Data Ground Truth](https://isic-challenge-data.s3.amazonaws.com/2017/ISIC-2017_Test_v2_Part3_GroundTruth.csv)> External Data>> [External Data Images](https://isic-challenge-data.s3.amazonaws.com/2017/ISIC-2017_Training_Data.zip)>> [External Data Masks](https://isic-challenge-data.s3.amazonaws.com/2017/ISIC-2017_Training_Part1_GroundTruth.zip)>> [External Ground Truth](https://isic-challenge-data.s3.amazonaws.com/2017/ISIC-2017_Training_Part3_GroundTruth.csv)  Constants---To enhance the readibilty, as well as to decrease the maintenance effort, it is useful for bigger projects to define contants that need to be accessed globally throughout the whole notebook in advance. The following cell contains all of those global constants. By convention, we write them in caps (https://www.python.org/dev/peps/pep-0008/constants)<jupyter_code># global parameters for running the notebooks (each computation )
PREPROCESS_IMAGES = False
COMPUTE_FEATURES = False
PATH = {}
PATH['data'] = {}
PATH['data']['raw'] = "../data/raw/"
PATH['data']['processed'] = "../data/processed/"
PATH['data']['external'] = "../data/external/"

PATH['images'] = 'images/'
PATH['masks'] = 'masks/'
PATH['filtered_images'] = 'filtered_images/'
PATH['dummy_images']= 'dummy_images/'

PATH['reports'] = "../reports/"

# filename lookup dictionary storing the most relevant filenames
FILENAME = {}
FILENAME['diagnosis'] = 'diagnosis.csv' # changed from original name
FILENAME['features'] = 'features.csv' # changed from original name

# store filenames of different files in the project for easy iteration
FILENAME['raw_images'] = sorted([image[:-4] for image in os.listdir(PATH['data']['raw'] + PATH['images']) if not re.match('.*super.*', image) and re.match('^ISIC', image)])
FILENAME['raw_masks'] = sorted([mask[:-4] for mask in os.listdir(PATH['data']['raw'] + PATH['masks'])])
FILENAME['external_images'] = sorted([image[:-4] for image in os.listdir(PATH['data']['external'] + PATH['images']) if not re.match('.*super.*', image)])[1:] # need to subscript because of weird dotfile
FILENAME['external_masks'] = sorted([mask[:-4] for mask in os.listdir(PATH['data']['external'] + PATH['masks'])])
FILENAME['all_images'] = sorted(FILENAME['raw_images'] + FILENAME['external_images'])
FILENAME['all_masks'] = sorted(FILENAME['raw_masks'] + FILENAME['external_masks'])

# defining main dictionaries to store data. each dictionary will reference different pd.DataFrames
DATA = {}<jupyter_output><empty_output><jupyter_text>**TASK 0** Image Preprocessing---In this section we preprocess our image to make it nicer to deal with them in the later part of the project.1. Crop Images and Masks to be bound by lesion2. Make Width and Length an even number to be able to crop evenly3. Maybe save filtered image with color<jupyter_code># helper function to make width and lengths even
def make_even(img):
    if img.size[0] % 2 != 0: #  making number of cols even
        img = np.array(img)
        mid = int(img.shape[1] / 2)
        img = np.delete(img, mid, axis=1)
        img = Image.fromarray(img)

    if img.size[1] % 2 != 0: # making number of rows even
        img = np.array(img)
        mid = int(img.shape[0] / 2)
        img = np.delete(img, mid, axis=0)
        img = Image.fromarray(img)
    
    return img
# preprocessing (170 seconds to run)
if PREPROCESS_IMAGES == True:
    try: 
        os.makedirs(PATH['data']['processed'] + PATH['images'])
        os.makedirs(PATH['data']['processed'] + PATH['masks'])
        os.makedirs(PATH['data']['processed'] + PATH['filtered_images'])
    except: print('Directories already exist.')

    print('Starting Preprocessing of Raw Images...')
    for i in range(150):
        # get image names
        img_name = FILENAME['raw_images'][i] + '.jpg'
        mask_name = FILENAME['raw_masks'][i] + '.png'

        # open temporarily
        img = Image.open(PATH['data']['raw'] + PATH['images'] + img_name)
        mask = Image.open(PATH['data']['raw'] + PATH['masks'] + mask_name)

        # crop to only store lesion
        cropped_img = img.crop(mask.getbbox())
        cropped_mask = mask.crop(mask.getbbox())

        # make width and length even (two cases)
        cropped_img = make_even(cropped_img)
        cropped_mask = make_even(cropped_mask)

        # create filtered with color
        dummy = Image.new("RGB", cropped_img.size, 0)
        filtered_img = Image.composite(cropped_img, dummy, cropped_mask)
        
        # save to '../data/processed' in correct subfolder
        cropped_img.save(PATH['data']['processed'] + PATH['images'] + img_name)
        cropped_mask.save(PATH['data']['processed'] + PATH['masks'] + mask_name)

        filtered_img.save(PATH['data']['processed'] + PATH['filtered_images'] + img_name)

    print('Processed Raw-Data\n')
    print('Starting Preprocessing of External Images...')

    for i in range(2000):
        # get image names
        img_name = FILENAME['external_images'][i] + '.jpg'
        mask_name = FILENAME['external_masks'][i] + '.png'

        # open temporarily
        img = Image.open(PATH['data']['external'] + PATH['images'] + img_name)
        mask = Image.open(PATH['data']['external'] + PATH['masks'] + mask_name)

        # crop to only store lesion
        cropped_img = img.crop(mask.getbbox())
        cropped_mask = mask.crop(mask.getbbox())

        # make width and length even (two cases)
        cropped_img = make_even(cropped_img)
        cropped_mask = make_even(cropped_mask)

        # create filtered with color
        dummy = Image.new("RGB", cropped_img.size, 0)
        filtered_img = Image.composite(cropped_img, dummy, cropped_mask)
        
        # save to '../data/processed' in correct subfolder
        cropped_img.save(PATH['data']['processed'] + PATH['images'] + img_name)
        cropped_mask.save(PATH['data']['processed'] + PATH['masks'] + mask_name)
        filtered_img.save(PATH['data']['processed'] + PATH['filtered_images'] + img_name)
        
        
    print('Preprocessing Done')<jupyter_output><empty_output><jupyter_text>*TASK 0.5* Data Exploration---  Loading in Data---The task involves different sources of data, namely:> **Images**: 150 initially given medical images of skin lesions + 2000 externally obtained images of the same format.> **Masks**: 150 initially given binary masks, specificying the region of the skin lesion + 2000 externally obtained masks of the same format.> **Diagnosis**: Dataset storing whether or not the lesion was either *melanoma*, *seborrheic_keratosis* or *neither* through binary values.We conveniently load in the raw and external *diagnosis* dataset into individual `pd.DataFrames` using the built-in pandas method `pd.read_csv()`. We store those in our `DATA` dictionary in the corresponding keys and concatenate them, to have one central diagnosis dataframe holding all 2150 diagnoses. For saving RAM, neither the *images* nor the *masks* are read in locally into the script, but instead read in one-by-one from the file system, whenever they are needed.<jupyter_code># load in raw and external dataset from different paths
DATA['raw_diagnosis'] = pd.read_csv(PATH['data']['raw'] + FILENAME['diagnosis']) # 150 x 3
DATA['external_diagnosis'] = pd.read_csv(PATH['data']['external'] + FILENAME['diagnosis']) # 2000 x 3

# concatenate into big dataframe
DATA['diagnosis'] = pd.concat([DATA['raw_diagnosis'], DATA['external_diagnosis']], ignore_index=True).sort_values(by=['image_id']) # 2150 x 3<jupyter_output><empty_output><jupyter_text>Inspection of Dataset (Target Variable)---The *diagnosis* dataset contains our target variable - the binary label *melanoma* or *not melanoma*. In this section we get some first understanding of the frequency of our target label in the data, which will play an important role when developing the machine learning model.We start by reporting the number of records and fields/ variables in the dataset by using the shape property of the `pd.DataFrame`. We then continue to have an actual look into the data. Similiar to the head command in terminal, we can use the method `head()` onto our DataFrames, which outputs an inline representation of the first five data records of the dataset.We then visualise the frequency distribution of the three possible labels within the data. **Shape**<jupyter_code>print(f"Diagnosis: {DATA['diagnosis'].shape}")<jupyter_output><empty_output><jupyter_text>**Diagnosis Dataset**<jupyter_code>DATA['diagnosis'].head() # confirm its sorted
%%capture
# extract cols
mel = DATA['diagnosis']['melanoma']
ker = DATA['diagnosis']['seborrheic_keratosis']

# mask all cases
mel_mask = (mel == 1) & (ker == 0)
ker_mask = (mel == 0) & (ker == 1)
neither_mask = (mel == 0) & (ker == 0)
both_mask = (mel == 1) & (ker == 1)

# append 3-class label into 'diagnosis' col
DATA['diagnosis']['diagnosis'] = 0
for i in range(DATA['diagnosis'].shape[0]):
    if mel_mask[i]: DATA['diagnosis']['diagnosis'].loc[i] = 2
    elif ker_mask[i]: DATA['diagnosis']['diagnosis'].loc[i] = 1
    else: DATA['diagnosis']['diagnosis'].loc[i] = 0
# distribution of diagnosis
diagnosis, counts = np.unique(DATA['diagnosis']['diagnosis'], return_counts=True)

for x in range(len(diagnosis)):
    print(f"{diagnosis[x]}: {counts[x]}")

# plot
fig,ax = plt.subplots()
ax.bar(diagnosis, counts, color='gray')
# maybe add text with numeric count
ax.set_title('Distribution of Diagnosis', fontweight='bold'); ax.set_xlabel('Diagnosis', fontstyle='italic'); ax.set_ylabel('Frequency', fontstyle='italic');
ax.set_xticks(diagnosis); ax.set_xticklabels(['Neither', 'Keratosis', 'Melanoma']);<jupyter_output><empty_output><jupyter_text>The majority (~67%) of the recorded skin lesions were neither *melanoma* nor *non-melanoma*. Only ~14% were *keratosis* and ~19% were *melanoma*.This significant imbalance is important to keep in mind, when building our model, as high class imbalance, makes machine learning models often biased towards rigoursly detecting the dominating the label - result in high accuracy scores, but 0 sensitivity.Furthermore, the visualisation reveals that he two diseases are - as expected - mutually exclusive, meaning that a single skin lesion cannot be diagnosed with multiple diseases.  Inspection of Images---The main part of the project is to analyse medical images for a set of features. To do so, we can analyse both the original image and a binary mask. In this section, we will look at examples of images and their corresponding mask to get a feel for the type of images we are dealing with and assess the quality of the masks.<jupyter_code># load test image using PIL
fig, ax = plt.subplots(nrows=3, ncols=2, figsize=(15,15))
fig.suptitle('Examples of Images and Corresponding Masks', fontsize=16, fontweight='bold')

for i in range(3):
    ex_img = Image.open('../data/processed/images/' + FILENAME['all_images'][i] + '.jpg')
    ex_img_mask = Image.open('../data/processed/masks/' + FILENAME['all_masks'][i] + '.png')

    ax[i][0].imshow(ex_img)
    ax[i][0].set_xlabel(f"{ex_img.format}, {ex_img.size}, {ex_img.mode}");
    ax[i][1].imshow(ex_img_mask, cmap='gray')
    ax[i][1].set_xlabel(f"{ex_img_mask.format}, {ex_img_mask.size}, {ex_img_mask.mode}");

ax[0][0].set_title('Medical Image');
ax[0][1].set_title('Corresponding Binary Mask');<jupyter_output><empty_output><jupyter_text>The initially given high-resolution images (often 3000X2000px) were sucessfully cut down in size within our preprocessing to only display the part of the image that is relevant for our analysis and decrease running times of feature computation and training of models. Furthermore, all images and masks were modified in a way that all images and masks have an even width and length, which is necessary to correctly compute the asymmetry score in a later section. From the three sampled pictures, it generally appears that the masks give a well-enough approximation of the actual shape of the lesion. However, they do seem to differ in the way they were computed, as some smoothen out borders more than others.  *TASK 1* Extracting Features ---We need to find quantitative measures of how to best classify *melanoma*. The following handcrafted features are assessed during this project:> **Compactness**. A quantitative measure of the shape of the lesion. The smaller the value, the more compact the lesion is. A perfect circle has a compactness score of roughly 1.> **Average Luminance**. A quantitative measure of the averaged brightness of the lesion. The higher the value, the lighter the lesion, and vice versa. Values range from 0 (meaning 100% black) to 255 (meaning 100% white).> **Luminance Variability**. A quantitative measure to determine the variation of luminance of the lesion. The higher the value, the more variation can be found on the lesion. > **Average Colour**. A quantitative measure of the averaged colour of the lesion. This metric is split up inthe the average color in the three RGB channels individually. Each ranges between 0 and 255.> **Colour Variability**. A quantitative measure to determine the variation of color of the lesion. The higher the value, the more variation can be found on the lesion. output in the format of (rvariation,gvariation,bvariation),average variation> **Asymmetry**. A quantitative measure to assess the symmetry of the lesion. Measured through relative number of non-overlapping pixels in different rotations. The higher the value, the less symmmetric the lesion is. A perfect circle, should score a 0 asymmetry score.  Functions for Feature Extraction<jupyter_code>def measure_area_perimeter(mask):
    # Measure area: the sum of all white pixels in the mask image
    mask = np.where(np.array(mask)==255, 1, 0)
    
    # compute area as number of white pixels 
    area = np.sum(mask)

    # compute perimeter by eroding 1 pixel from mask and then compute the difference between original and eroded mask
    struct_el = morphology.disk(1)
    mask_eroded = morphology.binary_erosion(mask, struct_el)
    image_perimeter = mask - mask_eroded
    perimeter = np.sum(image_perimeter)
    
    return area, perimeter
# compactness
def get_compactness(mask):
    area, perimeter = measure_area_perimeter(mask)
    # return compactness formula
    return ( ( perimeter ** 2) / (4 * np.pi * area ) )
# average luminance
def get_average_luminance(filtered_image): # image needs to be filtered for lesion
    gray = np.array(filtered_image.convert('L')) # converting to gray scale 
    return round(np.mean(gray[gray > 0]))
# luminance variability
def get_luminance_variability(filtered_image, measure='variance'):
    gray = np.array(filtered_image.convert('L')) # converting to gray scale 
    if measure == 'variance': return round(np.var(gray[gray > 0]))
    elif measure == 'standard_deviation': return round(np.std(gray[gray > 0]))
    else: print('Cannot compute this `measure`. Try `variance` or `standard_deviation`')
# average color
def get_average_color(filtered_image): # image needs to be filtered for lesion
    r, g, b = filtered_image.split() # converting to separate channels  
    r= np.array(r) 
    g= np.array(g)
    b= np.array(b) 
    return [round(np.mean(r[r > 0])),round(np.mean(g[g > 0])),round(np.mean(b[b > 0]))]
# color variability
def get_color_variability(filtered_image, measure='variance'): # image needs to be filteredd for lesion
    r, g, b = filtered_image.split() # converting to separate channels  
    r= np.array(r) 
    g= np.array(g)
    b= np.array(b) 
    if measure == 'variance': rgb=(np.var(r[r > 0]),np.var(g[g > 0]),np.var(b[b > 0]))
    elif measure == 'standard_deviation': rgb=(np.std(r[r > 0]),np.std(g[g > 0]),np.std(b[b > 0]))
    else: return 
    return np.mean(rgb)
def get_asymmetry(mask):
    return round(np.mean([asymmetry(mask), asymmetry(mask.rotate(90, expand=True))]),2)
# helper for asymmetry
def asymmetry(mask):
    # calculate basic properties of image
    width, length = mask.size  # requires even number of pixels in both dimensions
    size = width * length

    if width%2!=0:
        print("Uneven Number of Pixels. Can't calculate asymmetry.")

    # cut in half and fold
    left = mask.crop((0,0,(width/2), length)) 
    right = mask.crop((width/2,0,width,length))
    right = right.transpose(Image.FLIP_LEFT_RIGHT)

    # get binary array of unequal positions (the closer the sum to 0, the better the symmetry)
    diff = np.where(np.array(left) != np.array(right), 1, 0)

    return np.sum(diff) / (size / 2) # percentage of asymmetric pixels<jupyter_output><empty_output><jupyter_text>Evaluating Feature Extraction---Before accepting our functions to compute the featurs as universal truth, we need to test them on a set of dummy images, for which we know what metrics we are expecting. If the computed values of our functions match with the expectation, it is reasonable to believe that the functions indeed compute the desired feature.<jupyter_code>dummies = ['dummy' + str(i+1) for i in range(2)]

# show dummy image and mask
fig, ax = plt.subplots(nrows=len(dummies), ncols=2, figsize=(16, 6 * len(dummies)))
fig.suptitle("Feature Extration on Dummies", fontsize=16, fontweight='bold')

for i, img_name in enumerate(dummies):
    # compute features
    img = Image.open('../data/external/dummy_images/' + img_name + '.jpg')
    img_mask = Image.open('../data/external/dummy_images/' + img_name + '_mask.png').convert('L')

    area, perimeter = measure_area_perimeter(img_mask)
    compactness = get_compactness(img_mask)
    asymmetry_score = get_asymmetry(img_mask)
    average_luminance = get_average_luminance(img)
    luminance_variance = get_luminance_variability(img)
    average_colors = get_average_color(img)
    color_variance = get_color_variability(img)

    # print out computed features
    print('#'*15 + f"   Dummy {i+1}   " + '#'*15)
    print(f"Area / Perimeter      : {area}px / {perimeter}px")
    print(f"Compactness           : {compactness}")
    print(f"Asymmetry             : {asymmetry_score * 100}%")
    print(f"Average Luminance     : {average_luminance}")
    print(f"Luminance Variance    : {luminance_variance}")
    print(f"Average Colors (RGB)  : {average_colors}")
    print(f"Color Variability     : {color_variance}\n")

    ax[i][0].imshow(img);
    ax[i][1].imshow(img_mask, cmap='gray');
    ax[i][0].set_title(f'Dummy {i+1}', fontsize=10, fontstyle='italic')
    ax[i][1].set_title(f'Dummy {i+1} Mask', fontsize=10, fontstyle='italic')
    ax[i][0].set_xlabel(f'{img.format}, {img.size}, {img.mode}', fontstyle='italic');
    ax[i][1].set_xlabel(f'{img_mask.format}, {img_mask.size}, {img_mask.mode}', fontstyle='italic');<jupyter_output><empty_output><jupyter_text>**Evaluation**> *Compactness*: A perfect circle should have a lower compactnes score (and is thusly more compact), than the triangle. This is indeed computed by our `get_compactness()` function.> *Asymmetry*: A perfect circle shouldn't be asymmetric accross the two main axes, and thusly a asymmetry score of 0%, whereas the triangle should report a higher value, since the second fold is asymmetric. This is indeed computed by our `get_asymmetry()` function.> *Average Luminance*: We expect the red-gradient circle to have a higher average luminance than the dark grey triangle. This is indeed computed by our `get_average_luminance()` function.> *Luminance Variance*: We expect the gradient circle to have a higher variance in the luminance than the uniformly colored triangle. This is indeed computed by our `get_luminance_variability()` function.> *Average Color*:  We expect the red-gradient circle to have a high average color value in the red-channel and a lower in the remaining two, whereas the uniformly grey-colored triangle is expected to have equal values in all three channels. This is indeed computed by our `get_average_color()` function.> *Color Variability*: We expect the gradient circle to have a higher variance amongst all three color channels than the uniformly colored triangle. This is indeed computed by our `get_color_variability()` function.  Compute Features---We have developed our functions to extract features from the medical images of the lesions and have proven them to work on dummy images, where we were able to validate assumed scores in each feature. We can therefore now compute the features for each image and append it to our main dataframe to store the data, that is later used to train our model. At the end of the cell, a pd.Dataframe called `DATA['features']` is computed, which holds the Image ID, all eight computed features and the diagnosis for each image. *Note*: If the `COMPUTE_FEATURES` variable is set to `False` (to save roughly 8min of computation), the already computed dataframe is read in from the file structure.<jupyter_code>features = ['compactness', 'average_luminance', 'luminance_variability', 'average_red', 'average_green', 'average_blue', 'color_variability', 'asymmetry']

feature_functions = {
    'compactness': get_compactness,
    'average_luminance': get_average_luminance,
    'luminance_variability': get_luminance_variability,
    'average_red': get_average_color,
    'average_green': get_average_color,
    'average_blue': get_average_color,
    'color_variability': get_color_variability,
    'asymmetry': get_asymmetry
}
# compute all features (execution time: >8min)
if COMPUTE_FEATURES == True:
    feature_dict = {feature: [] for feature in features}

    for i in range(len(FILENAME['all_images'])):
        name = FILENAME['all_images'][i]
        img_name = FILENAME['all_images'][i] + '.jpg'
        mask_name = FILENAME['all_masks'][i] + '.png'

        # open temporarily
        filtered_img = Image.open(PATH['data']['processed'] + PATH['filtered_images'] + img_name)
        mask = Image.open(PATH['data']['processed'] + PATH['masks'] + mask_name)

        # measure features and append to feature_dict
        for feature in features:
            if feature in ['compactness', 'asymmetry']:
                feature_dict[feature].append(feature_functions[feature](mask))
            elif feature in ['average_luminance', 'luminance_variability', 'color_variability']:
                feature_dict[feature].append(feature_functions[feature](filtered_img))
            elif feature == 'average_red':
                feature_dict[feature].append(feature_functions[feature](filtered_img)[0])
            elif feature == 'average_green':
                feature_dict[feature].append(feature_functions[feature](filtered_img)[1])
            elif feature == 'average_blue':
                feature_dict[feature].append(feature_functions[feature](filtered_img)[2])
        
    # append extracted features and diagnosis to features.csv
    DATA['features'] = pd.DataFrame({'id': FILENAME['all_images']})
    DATA['features'] = pd.concat([DATA['features'], pd.DataFrame(feature_dict), DATA['diagnosis'][['melanoma', 'seborrheic_keratosis', 'diagnosis']]], axis=1) # concatenating handcrafted features and diagnosis

    # save all computed features
    DATA['features'].to_csv(PATH['data']['processed'] + 'features.csv')

else: DATA['features'] = pd.read_csv(PATH['data']['processed'] + 'features.csv').iloc[: , 1:]<jupyter_output><empty_output><jupyter_text>Inspect Main DataFrame---Check that the computation or reading in of the dataframe was successful, by displaying it inline in Jupyter.<jupyter_code>DATA['features']<jupyter_output><empty_output><jupyter_text>*TASK 1.5* Evaluate Features for Classification---In this section, we are plotting the handcrafted features in order to see, whether any correaltions appear. Since we are trying to classify a binary label, namely whether a lesion is diseased with *melanoma* or not, we plot the distribution of each feature in the two classes *melanoma* and *no melanoma* to get a visual intuition of how good the specific feature might be able to distinguish between our target label. The more distinct the two distributions are, the better the feature will be at prediciting the tartget value.  Visual Exploration<jupyter_code>fig, ax = plt.subplots(nrows=len(features), figsize=(8, 6 * len(features)))

for i, feature in enumerate(features):
    sns.violinplot(x='melanoma', 
                   y=feature, 
                   data=DATA['features'],  
                   ax=ax[i]).set_title(f"Distribution of {feature.title().replace('_', ' ')} in Different Classes");
sns.pairplot(DATA['features'], hue='melanoma', height=2, diag_kind="hist");<jupyter_output><empty_output><jupyter_text>*TASK 2* Predict Diagnosis---We have 'handcrafted' a number of features and now aim to find the model that is the most robust to predict our target variable - whether the lesion is our is not diseased with melanoma. In the process of doing so, a number of questions arise that we will step-by-step tackle in order to build the 'best' model. > **Performance Metrics**: Which metrics do we use to assess whether or not our model does good?> **Defining Features and Target Variable**: Which label are we trying to predict and what features do we have at hand to do so?> **Train-Test Split**: How can we get scores that resemble the model's performance on real out-of-sample data?> **Normalisation**: How do we make sure that each feature contributes equally to the model's prediction?> **Selecting Features**: Which (combination of) features are (/is) likely do perform best?> **Building Model**: Which model (and with which hyperparameters) does best in achieving the desired performance?  Performance Metrics---Choosing performance metrics is arguably a design choice for the model, and different approaches are surely possible. For the purpose of this ML model, which detects a minority class, we chose to go with the widely used metric of `Recall`, which measures the *sensitivity* of the model towards detecting positives (The percentage of positives that were detected by the model). That is, because in medical diagnosis detecting false positives is generally more acceptable than false negatives, since the latter could potentially result into serious diseases.  Define Initial Features and Target Variable---First, we need to define from which set of features our model should choose the ones that are likely to perform best on a specified target variable. In our case, each measured feature is potentially interesting for the model, and we want to predict the binary label, whether or not the classifier is diseased with *melanoma* or not.By, convention, we call the feature matrix `X` and the target column `y`.<jupyter_code>X = DATA['features'][features]
y = DATA['features'][['melanoma']]<jupyter_output><empty_output><jupyter_text>Train-Test Split---Before, we doing anything further with our data, we split our data into a `development` set and a `test` set. The test set will be used to assess the final performance of the model by mimicing a true *out-of-sample* set of datapoints. For this project, we chose a standard split size of `0.3`, meaning that 30% of the original data will be used for testing, and 70% for the development of the model.We split the data at this early stage to not bias our model towards the test dataset, ie. performing the feature selection, normalisation or balancing process on all of the data, might bias it towards the test data, and thus result in a sligthly overfitted model, that might perform good on the test, but not in real-life. *Note: Whenever there is randomness involved, we set `random_state=1` to produce reproducable results.*<jupyter_code># split into dev-test data (hold-out data to mimic true 'out-of-sample' data)
X_dev, X_test, y_dev, y_test = train_test_split(X, y, stratify=y, test_size=0.3, random_state=1)
print(X_dev.shape, y_dev.shape)
print(X_test.shape, y_test.shape)<jupyter_output><empty_output><jupyter_text>Normalisation---Our measured handcrafted features all perform on different scales, ie. the *average luminance* can only obtain values between 0 and 255, whereas the *asymmetry* is the average percentage of non-matching pixels in two-folds of the image.Since we want each of our features to equally contribute to the prediction of the model, we need to normalise them. We do this through the formula of the standard score $\frac{x-\mu}{\sigma}$ ([Wikipedia](https://en.wikipedia.org/wiki/Standard_score)). We computed the values for the normalisation solely on the development set (again, to not bias the model towards the test data) and then use this scaling to scale both the development and test features matrices. Within the whole process the target column (the binary label 0 and 1), is not scaled.We can easily use `sklearn`'s `StandardScaler`, which does the job for us.<jupyter_code># distribution of features after normalisation
plt.figure(figsize=(15,5));
sns.boxplot(data=X_dev, width=0.5);
scaler = StandardScaler().fit(X_dev) # (x - mu) / std
X_dev = pd.DataFrame(scaler.transform(X_dev), columns=features)
X_test = pd.DataFrame(scaler.transform(X_test), columns=features)
# distribution of features before normalisation
plt.figure(figsize=(15,5));
sns.boxplot(data=X_dev, width=0.5);<jupyter_output><empty_output><jupyter_text>Balancing Data---As we have seen in the initial exploration of the data, the labels are heavily imbalanced. Out of the 150 observed images, only 30 are diseased with *melanoma*, which results in precisely 20% of the data. If we would naively train our model disregarding the imbalance and only evaluate the model's performance by its *accuracy score*, we are likely to see a model performing equally or very similar to the so-called *null-hypotheses* (always predicting the dominating label), which would lead to a 80% accuracy in our case. To prevent this we balance our data, such that at the end of the process the data we train our model has equal amounts of *melanoma* and *not melanoma*.There are two ways of doing this:> **Random Undersampling**: Cutting down the frequency of the dominant label.> **Random Oversampling**: Duplicating the less dominant feature.Although, both option are obviously not ideal (since we loose information in the first, and create duplicate information in the second), and we would always prefer a orginally balanced dataset, we need to make a choice. For this project, we *upsampled* our data, since undersampling, would have limited our whole analyis to only 60 images, which is likely to not create an accurate model.We oversample using `imblearn`'s class `RandomOverSampler`.> **REMARK:** As we will later see, estimating the model's performance by cross-validating on an oversampled development set, produces highly overconfident scores. Correct upsampling, is therefore done by iteratively upsampling for each k-fold during the cross-validation, to create unbiased results. This is shown in detail in a later section.<jupyter_code># plot before oversampling
fig, axes = plt.subplots(ncols=3, figsize=(12, 3))
for ax, dataset in zip(axes, [y, y_dev, y_test]):
    unique, counts = np.unique(dataset, return_counts=True)
    ax.bar(unique, counts); ax.set_xticks([1,0])
axes[0].set_title('Target Imbalance on `y`');
axes[1].set_title('Target Imbalance on `y_dev`');
axes[2].set_title('Target Imbalance on `y_test`');
# balance data (using over-sampling)
oversampler = RandomOverSampler(random_state=1)
X_dev_sampled, y_dev_sampled = oversampler.fit_resample(X_dev, y_dev)
# plot after oversampling
fig, axes = plt.subplots(ncols=3, figsize=(12, 3))
for ax, dataset in zip(axes, [y, y_dev_sampled, y_test]):
    unique, counts = np.unique(dataset, return_counts=True)
    ax.bar(unique, counts); ax.set_xticks([1,0])
axes[0].set_title('Target Imbalance on `y`');
axes[1].set_title('Target Imbalance on `y_dev_sampled`');
axes[2].set_title('Target Imbalance on `y_test`');
print(f'#Images before Upsampling:   {X_dev.shape[0]}')
print(f'#Images after Upsampling:    {X_dev_sampled.shape[0]}')
print(f'Upsampled Images:            {X_dev_sampled.shape[0]-X_dev.shape[0]}')<jupyter_output><empty_output><jupyter_text>Feature Selection---*Feature Selection* is the process of identifying the most related features from the data and removing the irrelvant or less important features which do not contribute much to our target variable.There are a number of advantages that come along with feature selection, which make it an important step of every machine learning project.1. **Reduces Overfitting**: Less redundant data means less opportunity to make decisions based on noise.2. **Improves Accuracy**: Less misleading data means modeling accuracy improves.3. **Reduces Training Time**: fewer data points reduce algorithm complexity and algorithms train faster.Since brute-forcing the best features on a given model with 8 features already results in $\sum_{i=1}^{8} {8 \choose i}=255$ unique combination of features, the computational effort quickly becomes infeasible. We can therefore use statistics to try to pick the most relevant features in a computatinally less expensive way.  K-Best---Statistical tests can be used to select those features that have the strongest relationship with the output variable.The scikit-learn library provides the `SelectKBest` class that can be used with a suite of different statistical tests to select a specific number of features.In order to evaluate our features with respect to randomly generated data, we add some uniformly random columns to our development data, in order to select the features that are likely to perform best. It is however important to notice, that K-Best Selection is a way of selecting univariate features, meaning that it only checks the likely performance for each single feature. This method might ie. give us 5 individually good performing features, which are highly correlated. In that case, the 4 additionally selected features, don't add any value to our model and thus increase running time and potentially accuracy.<jupyter_code># generate some noise
noise_cols = 4
features_noise = features + ['Noise' for _ in range(noise_cols)] # list of features + noise columns
noise = np.random.RandomState(1).uniform(0,0.1, size=(noise_cols, X_dev_sampled.shape[0])).transpose()


# feature matrix and target variable
X_select = np.hstack((X_dev_sampled, noise))
y_select = y_dev_sampled

# plot scores for features and noise
fig, ax = plt.subplots(ncols=5, figsize=(4*5, 3))
fig.suptitle('Univariate Feature Scores for Features and Noise for different K-Best', fontsize=14)

# select k-best
for k in range(1,5+1):
    kbest = SelectKBest(mutual_info_classif, k=k)
    kbest.fit(X_select, np.ravel(y_select)) # ravel cause of sklearn warning 
    scores = kbest.scores_ # univariate features scores for each feature and noise

    print('-'*10 + f' k = {k} ' + '-'*10)
    scores_dict = {scores[i]: i for i in range(len(scores))}
    for val in sorted(scores_dict, reverse=True)[:k]:
        print(features_noise[scores_dict[val]])

    ax[k-1].bar(np.arange(0,len(scores)), scores)
    ax[k-1].tick_params(labelrotation=90) # readable labels
    ax[k-1].set_xticks(np.arange(0, len(features) + noise_cols));
    ax[k-1].set_xticklabels([f.replace('_', ' ').title() for f in features] + ['Noise' for _ in range(noise_cols)]);<jupyter_output><empty_output><jupyter_text>Building KNN-Model---For the scope of this project, we first limit ourselves to a basic classifier called `KNN` (*K-Nearest-Neighbors*) algorithm, classifies out-of-sample data by evaluating the label frequency of k-surrounding (mostly measured through *euclidean distance*) neighbors in a n-dimensional space (where n is the number of features).To build the model, we choose the features selected by *K-Best Features* and and train our model using cross-validation of 5 partitions. In that way we get a more robust estimate for the performance of the metric. In this section we will build three different models:> **Case 1 (Baseline)**: In the baseline model, we train our model on the unbalanced dataset. We expect the model to not perform on the recall score, given the high class imbalance. > **Case 2 (Wrong Upsampling)**: Here we do the cross-validation on the entire upsampled dataset. We expect highly overconfident metrics since on each fold, we expect a number of images being a copy from an image in the training in the validation set, which results in a highly biased model that overfits the training data. This overfitting through wrong upsampling, however, is expected to be exposed when testing the model on the real test data, which achieves a similar score as the baseline.> **Case 3 (Correct Upsampling)**: Instead, we need to upsample for each training set in each of the folds in the cross-validation to create an unbiased estimation of the model’s performance. With this technique, we expect to increase the model’s performance on recall and make it robust to out-of-sample data in a way, that we perform equally good on the test data.<jupyter_code># define splitting for cross-validation (shuffle=False for reproducable results) 
kf = KFold(n_splits=5,shuffle=False) 
# selected features from k-best
selected_features = ['compactness', 'color_variability', 'luminance_variability']

# condense feature matrix to only contain selected features (in non-upsampled and upsampled development set and test set)
X_dev_selected = X_dev[selected_features]
X_dev_sampled_selected = X_dev_sampled[selected_features]

X_test_selected = X_test[selected_features]
MODEL = {}
MODEL['baseline'] = {}
MODEL['upsample_before_cv'] = {}
MODEL['upsample_during_cv'] = {}
# initialise data structure to hold performance results in the three cases
DATA['results'] = pd.DataFrame({'Method': ['No Upsampling (Baseline)', 'Upsample Training Data before CV', 'Upsample within CV']})<jupyter_output><empty_output><jupyter_text>Base-Performance on Unbalanced Data---<jupyter_code>cross_val_reports = {'Classifier': [], 'Hyperparameter': [], 'Accuracy': [], 'Precision': [], 'Recall (Sensitivity)': []}

params = {'n_neighbors': range(1,6), 'weights': ['distance', 'uniform']}

knn = KNeighborsClassifier()
grid_no_upsampling = GridSearchCV(knn, param_grid=params, cv=kf, 
                          scoring=['accuracy', 'recall'], refit='recall')

grid_no_upsampling.fit(X_dev_selected, np.ravel(y_dev))

# cv results
MODEL['baseline']['cv_results'] = pd.DataFrame(grid_no_upsampling.cv_results_)
try: os.makedirs(PATH['reports'] + 'model_evaluation/cross_val/')
except: None
MODEL['baseline']['cv_results'].to_csv(PATH['reports'] + 'model_evaluation/cross_val/' + 'baseline.csv')

# report best score (and the hyperparameters used)
MODEL['baseline']['best_recall'] = grid_no_upsampling.best_score_
MODEL['baseline']['hyperparameters'] = grid_no_upsampling.best_estimator_

# display results
print(MODEL['baseline']['best_recall'])
print(MODEL['baseline']['hyperparameters'])
MODEL['baseline']['cv_results'] # display inline<jupyter_output><empty_output><jupyter_text>Performance on Wrongly Upsampled Data---<jupyter_code>cross_val_reports = {'Classifier': [], 'Hyperparameter': [], 'Accuracy': [], 'Precision': [], 'Recall (Sensitivity)': []}

params = {'n_neighbors': range(1,11), 'weights': ['distance', 'uniform']}

knn = KNeighborsClassifier()
grid_upsampling_before_cv = GridSearchCV(knn, param_grid=params, cv=kf, 
                          scoring=['accuracy', 'recall'], refit='recall')

grid_upsampling_before_cv.fit(X_dev_sampled_selected, np.ravel(y_dev_sampled))

# cv results
MODEL['upsample_before_cv']['cv_results'] = pd.DataFrame(grid_upsampling_before_cv.cv_results_)
try: os.makedirs(PATH['reports'] + 'model_evaluation/cross_val/')
except: None
MODEL['upsample_before_cv']['cv_results'].to_csv(PATH['reports'] + 'model_evaluation/cross_val/' + 'upsample_before_cv.csv')

# report best score (and the hyperparameters used)
MODEL['upsample_before_cv']['best_recall'] = grid_upsampling_before_cv.best_score_
MODEL['upsample_before_cv']['hyperparameters'] = grid_upsampling_before_cv.best_estimator_

# display results
print(MODEL['upsample_before_cv']['best_recall'])
print(MODEL['upsample_before_cv']['hyperparameters'])
MODEL['upsample_before_cv']['cv_results'] # display inline<jupyter_output><empty_output><jupyter_text>Performance on Correctly Upsampled Data---<jupyter_code>cross_val_reports = {'Classifier': [], 'Hyperparameter': [], 'Accuracy': [], 'Precision': [], 'Recall (Sensitivity)': []}

params = {
    'kneighborsclassifier__n_neighbors': range(1, 11), # technicalities
    'kneighborsclassifier__weights': ['distance', 'uniform']
}

pipeline = make_pipeline(SMOTE(random_state=1), 
                              KNeighborsClassifier())
grid_upsampling_during_cv = GridSearchCV(pipeline, param_grid=params, cv=kf, scoring=['accuracy', 'recall'], refit='recall')
grid_upsampling_during_cv.fit(X_dev_selected, np.ravel(y_dev));

# cv results
MODEL['upsample_during_cv']['cv_results'] = pd.DataFrame(grid_upsampling_during_cv.cv_results_)
try: os.makedirs(PATH['reports'] + 'model_evaluation/cross_val/')
except: None
MODEL['upsample_during_cv']['cv_results'].to_csv(PATH['reports'] + 'model_evaluation/cross_val/' + 'upsample_during_cv.csv')

# report best score (and the hyperparameters used)
MODEL['upsample_during_cv']['best_recall'] = grid_upsampling_during_cv.best_score_
MODEL['upsample_during_cv']['hyperparameters'] = grid_upsampling_during_cv.best_estimator_

# display results
print(MODEL['upsample_during_cv']['best_recall'])
print(MODEL['upsample_during_cv']['hyperparameters'])
MODEL['upsample_during_cv']['cv_results'] # display inline
DATA['results'] = pd.concat([DATA['results'], pd.DataFrame({'Recall Score (Validation)': [MODEL[key]['best_recall'] for key in MODEL.keys()]})], axis=1)
DATA['results']<jupyter_output><empty_output><jupyter_text>Evaluating Final Performance---<jupyter_code># fit best performing model on recall saved through grid search cv to whole dev set
grid_no_upsampling.fit(X_dev_selected, np.ravel(y_dev)) # knn(n_neighbors=1, weights='distance')
grid_upsampling_before_cv.fit(X_dev_sampled_selected, np.ravel(y_dev_sampled)) # KNeighborsClassifier(n_neighbors=8, weights='distance')
grid_upsampling_during_cv.fit(X_dev_sampled_selected, np.ravel(y_dev_sampled)) # KNeighborsClassifier(n_neighbors=7)

# predict test set
pred1 = grid_no_upsampling.predict(X_test_selected)
pred2 = grid_upsampling_before_cv.predict(X_test_selected)
pred3 = grid_upsampling_during_cv.predict(X_test_selected)

# make classification report and 
try: os.makedirs(PATH['reports'] + 'model_evaluation/test_data/')
except: None
pd.DataFrame(classification_report(y_test, pred1, output_dict=True)).transpose().to_csv(PATH['reports'] + 'model_evaluation/test_data/baseline.csv')
pd.DataFrame(classification_report(y_test, pred2, output_dict=True)).transpose().to_csv(PATH['reports'] + 'model_evaluation/test_data/upsampling_before_cv.csv')
pd.DataFrame(classification_report(y_test, pred3, output_dict=True)).transpose().to_csv(PATH['reports'] + 'model_evaluation/test_data/upsampling_during_cv.csv')

# recall
sens1 = recall_score(y_test, pred1)
sens2 = recall_score(y_test, pred2)
sens3 = recall_score(y_test, pred3)

DATA['results'] = pd.concat([DATA['results'], pd.DataFrame({'Recall Score (Test)': [sens1, sens2, sens3]})], axis=1)
DATA['results'].to_csv(PATH['reports'] + 'commparison_of_recall_scores.csv')
DATA['results']
# final model's performance
print(classification_report(y_test, pred3))<jupyter_output><empty_output><jupyter_text>*TASK 3* Open Question: How can we make our model more sensitive?---In the previous section we have developed a model, that is optimised for performing as well as possible on the `recall` metric on a specified classifer with specified features. However, it becomes obvious that the model is stil only able to predict roughly 40% of the *melanoma* lesions as diseased. In real-world, we would rather want to model to be more sensitive to positives, as a false negative (we do not detect cancer) has potentially fatal consequences, whereas a false positive only motivates people do go to a doctor. In this section we are therefore investigating, how we can tweak our model in a way that it gets more sensitive (increasing the 'recall' metric). In order to do so, we need a model, which outputs predictive labeling (ie. gives a probability on the binary label). With such a model, we can adjust the threshold of prediciting positive. This increases sensitivity, knowing that we will also have more false positives.  Model and Feature Selection---We now use the `Random Forest Classifier`, which is a widely used classifier, which allows us to adjust the performance of detecting positives through lowering the threshold for detecting them. We use the same balanced scaled development set to monitor the performance on cross-validated sets and later evaluate the final performance on the spared test-set - one time with the regular threshold of 0.5 and another time with the adjusted one.<jupyter_code>rf = RandomForestClassifier(n_estimators=100) # initialise random forest classifier

rf.fit(X_dev_sampled, y_dev_sampled) # fit on all features and select highest scoring features

rf_feature_importance = pd.DataFrame({'Feature': features, 'Importance': rf.feature_importances_})

# plot
x = sns.barplot(x='Feature', y='Importance', data=rf_feature_importance);
x.set_title("Feature Importance in Random Forest Classifier");
x.set_xticklabels(x.get_xticklabels(), rotation=90);
# select best performing features from above output
rf_selected_features = ['compactness', 'color_variability', 'average_blue']<jupyter_output><empty_output><jupyter_text>Train and Evaluate Models---<jupyter_code>rf = RandomForestClassifier(n_estimators=100) # initialise random forest classifier

X_train, y_train = X_dev_sampled[rf_selected_features], np.ravel(y_dev_sampled)

rf.fit(X_dev_sampled[rf_selected_features], np.ravel(y_dev_sampled))
y_pred = rf.predict(X_test[rf_selected_features])

fig, ax = plt.subplots()
labels, counts = np.unique(y_pred, return_counts=True)
ax.bar(labels, counts, color='gray'); ax.set_xticks(labels); ax.set_xticklabels(['Non-Melanoma', 'Melanoma']); ax.set_title('Predicted Labels')

print(classification_report(y_test, y_pred))
threshold = 0.1 # setting the threshold for positive predictions low 

rf = RandomForestClassifier(n_estimators=100)

rf.fit(X_dev_sampled[rf_selected_features], np.ravel(y_dev_sampled))
predicted_proba = rf.predict_proba(X_test[rf_selected_features])
y_pred = (predicted_proba [:,1] >= threshold).astype('int')

fig, ax = plt.subplots()
labels, counts = np.unique(y_pred, return_counts=True)
ax.bar(labels, counts, color='gray'); ax.set_xticks(labels); ax.set_xticklabels(['Non-Melanoma', 'Melanoma']); ax.set_title('Predicted Labels')

#y_pred=clf.predict(X_test)
print(classification_report(y_test, y_pred))<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10218857989139378, 'fraction_numerical': 0.00886767959336661, 'mean_word_length': 5.111744328975305, 'pattern_counts': {'":': 0, '<': 101, '<?xml version=': 0, '>': 157, 'https://': 22, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 17, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '24', 'dir': 'jupyter-structured-clean-dedup', 'id': '510624', 'license': 'MIT', 'n_tokens_mistral': 16322, 'n_tokens_neox': 15277, 'n_words': 5696, 'path': 'notebooks/project3.ipynb', 'repo_name': 'jonas-mika/skin-lesion-detection', 'size': '54725'}