<jupyter_start><jupyter_text>Data Processing - Exploration<jupyter_code>import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline
metadata = pd.read_csv('HAM10000_metadata')<jupyter_output><empty_output><jupyter_text>Understanding the data Our data come from 4 different distinct datasets.- <NAME>- RosenDahl- <NAME>- <NAME>If interested, we can always augment our dataset accordingly by adding in pictures from other dataset, but it is good to start at this baseline.<jupyter_code>metadata['dataset'].unique()<jupyter_output><empty_output><jupyter_text>The following diseases are being diagnoses:<jupyter_code>sorted(metadata['dx'].unique())<jupyter_output><empty_output><jupyter_text>**Legend**:- `akiec`: Actinic Keratosis         This disease is common and non-invasive. They can be treated without locally without surgery. Only small fraction becomes malignant.- `bcc`: Basal Cell Carcinoma        This disease is a common variant of ephitelial skin cancer that rarely metastasizes or grow destructively if untreated.- `bkl`: Benign Keratosis           Benign supergroup of different kinds of warts- `df`: Dermatofibroma        Benign skin lesion regardes as either a benign proliferation or an inflammatory reaction to minimal trauma.- `nv`: Melanocytic nevi        Benign neoplasms of melanocytes. If diagnosed properly, are biologically stable, completely benign lesions. However, melanocytic nevi can be found in association with melanoma- `mel`: Melanoma        **Malignant** neoplasms derived from melanocytes. In early stage it is treatable, but can become highly invasive in the future.- `vasc`: Vascular Skin Lesions                They can be both malignant or benign.  OpportunitiesWe can augment the HAM10000 dataset with the [DermNet](https://www.kaggle.com/shubhamgoel27/dermnet) dataset which contains 23 diseases. The only downside is that this dataset is way harder to find bounding boxes. So, I think a right way to approach is to train a model on HAM10000 first as a baseline and then we can always come back and feed in more data if we decide we want to.<jupyter_code>sns.countplot(data=metadata, x='dx')<jupyter_output><empty_output><jupyter_text>Metadata Information<jupyter_code>sns.countplot(data=metadata, y='localization')
plt.xticks(rotation=90);
sns.countplot(data=metadata, x='age')
plt.xticks(rotation=40);<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.0754639175257732, 'fraction_numerical': 0.009896907216494845, 'mean_word_length': 5.4866310160427805, 'pattern_counts': {'":': 0, '<': 24, '<?xml version=': 0, '>': 24, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '34404', 'license': 'MIT', 'n_tokens_mistral': 735, 'n_tokens_neox': 657, 'n_words': 267, 'path': 'Archive/dataset_test/explore_dataprocessing.ipynb', 'repo_name': 'Kevin-Miao/FSDL-FinalProject', 'size': '2443'}