<jupyter_start><jupyter_text>Table of Contents1&nbsp;&nbsp;Report Introduction2&nbsp;&nbsp;Data Background3&nbsp;&nbsp;Data Description4&nbsp;&nbsp;Data Exploration and Analysis4.1&nbsp;&nbsp;Graphical Summaries4.1.1&nbsp;&nbsp;Frequencies4.1.1.1&nbsp;&nbsp;Sex4.1.1.2&nbsp;&nbsp;Diagnosis4.1.1.3&nbsp;&nbsp;Diagnosis Method4.1.1.4&nbsp;&nbsp;Localisation4.1.1.5&nbsp;&nbsp;Sex by Skin Lesion Type4.1.2&nbsp;&nbsp;Location and Spread4.1.2.1&nbsp;&nbsp;Age4.1.2.2&nbsp;&nbsp;Age by Skin Lesion Type5&nbsp;&nbsp;Conclusions and Data Quality<jupyter_code>import pandas as pd
import matplotlib.pyplot as plt
import numpy as np

BASE_PROCESSED_DATA_DIR = '../data/processed'
"""
str: Base processed data directory
"""

PROCESSED_CSV_FILE = BASE_PROCESSED_DATA_DIR + '/processed.csv'
"""
str: HAM1000_metadata.csv metadata file location 
"""
        
# Read datasets in
    
skin_df = pd.read_csv(PROCESSED_CSV_FILE, index_col=0)
"""
pandas.core.frame.DataFrame:: final processed dataframe
"""

# plotting 

lesion_type_label = skin_df[
    ['lesion_type_idx', 'lesion_type']].sort_values(
    'lesion_type_idx').drop_duplicates()['lesion_type']
"""
pandas.core.series.Series: Lesion types (text) series sorted by idx for labels
"""<jupyter_output><empty_output><jupyter_text>Report Introduction This report documents the Data Understanding stage of the CRISP DM cycle. The Data Understandingstage involves the process of collecting insights about the data, which are used to help form hypothesis forlater analysis and/or to form results for deployment/modelling to stakeholder. The process involves data descriptions, explorations and quality verification. In this project this would involve analysing the frequencies, spread, location and patterns of skin lesions through population attributes like age. Data Background As stated in the Business Understanding and Data Preparation Stage reports, the data is provided from a kaggle competition in a dataset called the HAM10000. The dataset consists of a set of dermatoscopic images collected from various populations. The final dataset consists of 10015 images. Ground truths are provided by various confirmation techniques (follow-up examination, expert consensus or in-vivo confocal microscopy). The csv files provided consist of files for the metadata and pixel values (RGB and Luminance of 28 X 28 sizes). Data Description The full description of the data is covered in the data preparation stage, but to sum up the following fields from the final dataset are formed after preprocessing: * lesion_type (textual): The diagnosis (ground truth) as a textual description. Values:    - Actinic keratoses    - Basal cell carcinoma    - Benign keratosis-like lesions    - Dermatofibroma    - Melanocytic nevi    - Melanoma    - Vascular lesions * dx_type (textual): The method of diagnosis, textual. Values:     - histopathology follow-up examination (follow_up)    - expert consensus (consensus)    - in-vivo confocal microscopy (confocal). * lesion_type_idx: codes for diagnosis:    - 0: Actinic keratoses    - 1: Basal cell carcinoma    - 2: Benign keratosis-like lesions    - 3: Dermatofibroma    - 4: Melanocytic nevi    - 5: Melanoma    - 6: Vascular lesions* age (numeric): Natural numerical age of the individual the image is taken from.* sex (textual): Sex of the individual the image is taken from (male, female or unknown).* localization (textual): Location of skin lesion in individual.* pixelXXXX_l_28_28 (numeric): Luminance value of images in 28 by 28 pixel representation.* pixelXXXX_rgb_28_28 (numeric): RGB value of images in 28 by 28 pixel representation.<jupyter_code>skin_df.head()<jupyter_output><empty_output><jupyter_text>Data Exploration and AnalysisThis section concentrates on the process of exploring the data for patterns and interesting features usingExploratory Data Analysis, Graphical Summaries  Frequencies  Sex<jupyter_code># sex frequency (value counts) bar plot 

skin_df['sex'].value_counts().plot(kind='bar')

# Labels and titles

plt.title('Sex Frequencies')
plt.xlabel('Sex')
plt.ylabel('Frequency')

plt.rcParams['figure.figsize'] = [10, 6]<jupyter_output><empty_output><jupyter_text>As can be seen in the frequency bar plot - while not overwhelmingly higher than females - the majority of the sample population are male. Lastly, 'unknown' sexes account for a very small percentage of the population sex and hence it is going to be difficult to make statements about that category with high confidence. Diagnosis<jupyter_code># diagnosis frequency (value counts) bar plot 

skin_df['lesion_type'].value_counts().plot(kind='bar')

# Labels and titles

plt.xticks(rotation='vertical')
plt.title('Skin Lesion Types Frequencies')
plt.xlabel('Skin Lesion Types')
plt.ylabel('Frequency')

plt.rcParams['figure.figsize'] = [10, 6]<jupyter_output><empty_output><jupyter_text>The most common lesion and overwhelmingly so is nevi as it is more than 5 times larger than the second most common lesion (melanoma). Both Dematofibroma and Vascular lesions account for a very small minority of the lesions as they are less than half the next lesion type (Actinic Keratoses).   Diagnosis Method<jupyter_code># diagnosis method frequency (value counts) bar plot 

skin_df['dx_type'].value_counts().plot(kind='bar')

# Labels and titles

plt.xticks([0, 1, 2, 3], ['Histopathology','Follow Up Examination',
                          'Expert Consensus','in-vivo confocal microscopy'],
           rotation='vertical')
plt.title('Diagnosis Method Frequencies')
plt.xlabel('Diagnosis Method')
plt.ylabel('Frequency')

plt.rcParams['figure.figsize'] = [10, 6]<jupyter_output><empty_output><jupyter_text>When it comes to diagnosis methods, it seems various different methods were used to come up with the ground truths. This is mostly done by Histopathology and Follow Ups, but a small number of them rely on Expert Consensus and In-Vivo Vonfocal Microscopy.  Localisation<jupyter_code># localisation frequency (value counts) bar plot 

skin_df['localization'].value_counts().plot(kind='bar')

# Labels and titles

plt.title('Localization Frequencies')
plt.xlabel('Localization')
plt.ylabel('Frequency')

plt.rcParams['figure.figsize'] = [10, 6]<jupyter_output><empty_output><jupyter_text>The Localization of lesions seems to spread around various parts of the body, but are more common on extremities, the back and trunk.  Sex by Skin Lesion Type<jupyter_code>def group_by_lesion(df):
    """
    Used to group len of lesions for a df
    """
    return( ( len(df[df['lesion_type_idx'] == 0]),
              len(df[df['lesion_type_idx'] == 1]),
              len(df[df['lesion_type_idx'] == 2]),
              len(df[df['lesion_type_idx'] == 3]),
              len(df[df['lesion_type_idx'] == 4]),
              len(df[df['lesion_type_idx'] == 5]),
              len(df[df['lesion_type_idx'] == 6])
             ) )

bar_width = 0.35
n_lesion = len(skin_df['lesion_type'].drop_duplicates())
             
# find frequencies for each sex

male_freq = group_by_lesion(skin_df[skin_df['sex']  == 'male'])
female_freq = group_by_lesion(skin_df[skin_df['sex']  == 'female'])
unknown_freq = group_by_lesion(skin_df[skin_df['sex']  == 'unknown'])

# set up subplot

fig, ax = plt.subplots()
index = np.arange(n_lesion)

# Draw bars
 
ax.bar(index, male_freq, bar_width, color='b', label='Male')
ax.bar(index + bar_width, female_freq, bar_width, color='r', label='Female')

# set labels, titles and legends

ax.set_xlabel('Skin Lesion')
ax.set_ylabel('Frequency')
ax.set_title('Frequency by Skin Lesion and Sex')
# plot using labels from lesion type sorted by idx
plt.xticks([i for i, _ in enumerate(lesion_type_label)],
           lesion_type_label, rotation='vertical')
ax.set_xticks(index + bar_width / 2)
ax.legend()

# Draw plot

fig.tight_layout()
plt.rcParams['figure.figsize'] = [22, 8]
plt.show()<jupyter_output><empty_output><jupyter_text>As previously stated, unknown accounted a very small proportion of the sex labels and because it is by default "unknown", it is not plotted. Nonetheless, as previously stated the number of males in the pool is noticeably higher which is shown here in most lesion types. Hence, males seem to be more likely to encounter lesions according to this sample.  Location and Spread Age<jupyter_code># box plot with labels

plt.boxplot(skin_df['age'])
plt.title('Distribution of Age')
plt.ylabel('Age')

# drop x axis

plt.tick_params(axis='x', which='both',
                bottom=False, top=False, labelbottom=False) 

plt.rcParams['figure.figsize'] = [10, 6]
plt.show()<jupyter_output><empty_output><jupyter_text>This box plot shows that the most common ages are around 40 to 60 according to the quantiles, but there are not many outlier points which suggests that ages outside this range are not exceedingly rare and the age range is representative.  Age by Skin Lesion Type<jupyter_code># group age by lesion type 

plt.boxplot([skin_df[skin_df['lesion_type_idx'] == 0]['age'],
            skin_df[skin_df['lesion_type_idx'] == 1]['age'],
            skin_df[skin_df['lesion_type_idx'] == 2]['age'],
            skin_df[skin_df['lesion_type_idx'] == 3]['age'],
            skin_df[skin_df['lesion_type_idx'] == 4]['age'],
            skin_df[skin_df['lesion_type_idx'] == 5]['age'],
            skin_df[skin_df['lesion_type_idx'] == 6]['age']])
 
# setup labels and titles

plt.title('Distribution of Age by Skin Lesion')
plt.xlabel('Skin Lesion')
plt.ylabel('Age')
# plot using labels from lesion type sorted by idx
plt.xticks([i+1 for i, _ in enumerate(lesion_type_label)],
           lesion_type_label, rotation='vertical')

# draw plot

plt.rcParams['figure.figsize'] = [10, 6]
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10382794813747685, 'fraction_numerical': 0.01358304177814365, 'mean_word_length': 4.966236955187232, 'pattern_counts': {'":': 0, '<': 37, '<?xml version=': 0, '>': 37, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 7, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 4, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '271200', 'license': 'MIT', 'n_tokens_mistral': 3130, 'n_tokens_neox': 2876, 'n_words': 967, 'path': 'notebooks/data-understanding-ammar-150454388.ipynb', 'repo_name': 'Castdeath97/csc8635-project', 'size': '9718'}