<jupyter_start><jupyter_text>Exploratory Data Analysis with the SimulacrumHere we explore the data structure of the Simulacrum with a view to writing a synthetic data generator in Python according to the methodology described [here][1].The `bnlearn` package for R presents a possible alternative implementation. Find out more at http://www.bnlearn.com/, where [lecture][2] [slides][3] on the general theory of Bayesian Networks can also be found.[1]: https://simulacrum.healthdatainsight.org.uk/wp/wp-content/uploads/2018/11/Methodology-Overview-Nov18.pdf[2]: http://www.bnlearn.com/about/teaching/slides-bnshort.pdf[3]: http://www.bnlearn.com/about/slides/slides-useRconf13.pdf  Read the data from the .csv fileCode to extract the data from database using SQL will also be provided soon.<jupyter_code>from __future__ import print_function
import os
# Please set the path below as per your system data folder location
data_path = ['simulacrum_release_v1.1.0']
import pandas as pd
import numpy as np
from ipywidgets import interact, interact_manual

filepath = os.sep.join(data_path + ['sim_av_tumour.csv'])
data = pd.read_csv(filepath, sep=',', dtype=str).fillna('NaN')<jupyter_output><empty_output><jupyter_text>Peek at the data<jupyter_code>data.dtypes<jupyter_output><empty_output><jupyter_text>Fun with `ipywidgets` and the `interact` user interface. Read the docs [here](https://ipywidgets.readthedocs.io/en/stable/examples/Using%20Interact.html).Here we examine the unique values appearing in each column of the table and their frequencies, including missing values which are displayed as `'NaN'`.<jupyter_code>def view_value_counts(column='BEHAVIOUR_ICD10_O2'):
    return data[column].value_counts(dropna=False)

interact(view_value_counts, column=list(data.columns));<jupyter_output><empty_output><jupyter_text>Split the table records data up into strata defined by the tumour’s cancer siteSelect the breast cancer stratum from the tumour table using `SITE_ICD10_O2_3CHAR` == 'C50'<jupyter_code>data = data.loc[data['SITE_ICD10_O2_3CHAR'] == 'C50', :] # Select the breast cancer stratum from the tumour table

# View the first 5 rows in the table after selecting the breast cancer stratum
data.head()<jupyter_output><empty_output><jupyter_text>Data preprocessing: encode data categories with integer labelsWe encode the labels in each column as nonnegative integers using scikit-learn's `LabelEncoder` class.For example, each `SITE_ICD10_O2` cancer site code will be mapped uniquely to a nonnegative integer.This is necessary in order to compute $\chi^2$ values using scikit-learn's `chi2` function to assess the correlation between columns.<jupyter_code>from sklearn.preprocessing import LabelEncoder
# Select the columns to compare using the chi-squared statistic
# In particular we exclude patient ID, tumour ID, and link number
comparison_cols = list(data.columns)[2:]
comparison_cols.remove('LINKNUMBER')

# Keep track of the label encoders for each column in a dictionary
label_encoder_dict = {col_name: LabelEncoder().fit(data[col_name]) for col_name in comparison_cols}
# Swap out the string labels for their corresponding integer encodings in place within the dataframe
# cleaned = data.copy() can be used if we want to retain a copy of the raw data
for col_name, encoder in label_encoder_dict.items():
    data[col_name] = encoder.transform(data[col_name])<jupyter_output><empty_output><jupyter_text>Review label encodingsHere we can take a look at the categories of data which appear in each column. These should be the same as the categories which appeared in the value counts cell above, but instead of ordering by frequency, here the displayed order corresponds to the way the categories have been encoded as integers.For example, the first class which appears in the list will be encoded by the number 0, the second class in the list will be encoded as 1, and so on...<jupyter_code>def display_label_encodings(feature='SITE_ICD10_O2'):
    r'''Returns the list of classes that were encoded for a given feature column, in order'''
    return label_encoder_dict[feature].classes_

interact(display_label_encodings, feature=label_encoder_dict.keys());<jupyter_output><empty_output><jupyter_text>We create a dictionary for how missing values 'NaN' have been encoded in each column, if present.If a column does not have any missing values, then we match the column with the string 'NaN'.<jupyter_code>NaN_encoding_dict = dict()
for key, encoder in label_encoder_dict.items():
    try:
        NaN_encoding_dict[key] = encoder.transform(['NaN'])[0]
    except:
        NaN_encoding_dict[key] = 'NaN'
#print(pd.Series(NaN_encoding_dict))<jupyter_output><empty_output><jupyter_text>Compute the $\chi^2$ test statistics between pairs of characteristicsThe mathematical formula for computing $\chi^2$ test-statistics is described [here](https://en.wikipedia.org/wiki/Pearson%27s_chi-squared_testTesting_for_statistical_independence).  Discussion on handling missing valuesHow do you deal with missing values when computing chi-squared values between pairs of characteristics and why?Do you:1. Treat missing values as a separate category and compute the test statistic between the pair of characteristics over all records (with a given ICD10 site code);2. Treat missing values as a separate category, but ignore records containing missing values for both characteristics (outer join of columns with missing values removed);3. Ignore records containing missing values in either characteristic (inner join of columns with missing values removed);4. Fill missing values, etc.Retaining missing values might inflate test statistics between (relatively) independent characteristics whose missing values are highly correlated (e.g. due to data collection practices), whereas removing records with missing values may distort the results, especially when the remaining data is quite small. Option 2 is a combination of options 1 and 3 and inherits the disadvantages of both.In general, how one deals with missing data requires an understanding of its nature - how was the data collected, what do the missing values represent?- Was the data off the scale of usual measurements (too small or large to register on measuring equipment)?- Were measurements not taken due to faulty or unavailable equipment, or lost due to data corruption or mismanagement?- Etc.When considering the data at hand, we will choose option 1, remaining aware of the caveats mentioned above."Some organisations, for instance, have consistently low data quality between some fields and some have high. However missing data in fields (e.g. screening status or tumour staging) is informative (screening was not involved)." The function below computes $\chi^2$ test statistics between a given characteristic and all the others, taking missing values as a separate category. [Documentation][1] for `sklearn.feature_selection.chi2`, reveals that in the multiclass case where features are not Booleans, we must provide class frequencies to the `chi2` function, as discussed on [Stack Overflow][2] and [Data Science Stack Exchange][3]. [Documentation] for sparse matrices in coordinate format using scipy.[1]: https://scikit-learn.org/stable/modules/generated/sklearn.feature_selection.chi2.html[2]: https://stackoverflow.com/questions/21281328/scikit-learn-%CF%87%C2%B2-chi-squared-statistic-and-corresponding-contingency-table[3]: https://datascience.stackexchange.com/questions/11469/how-would-i-chi-squared-test-these-simple-results-from-a-b-experiment[4]: https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.coo_matrix.htmlTODO:- Compare implementation of $\chi^2$ in scikit-learn with `scipy` and R. Either the current input should be modified to use class frequencies rather than the raw data (e.g. with a method such a `value_counts()`, or the date-based characteristics `DIAGNOSISDATEBEST` and `DATE_FIRST_SURGERY` return unexpectedly high test statistics compared to the methodology document.<jupyter_code>from sklearn.feature_selection import chi2

def compute_chi2_values(column='SITE_ICD10_O2'):
    r'''Returns a dataframe containing chi2 test statistics between the input column and the rest sorted in descending order,
    along with the associated p-values for each test statistic in a second column'''
    results = chi2(data[comparison_cols], data[column])
    return pd.DataFrame({'chi2': results[0], 'p_val': results[1]}, index=comparison_cols).sort_values('chi2', ascending=False)

interact_manual(compute_chi2_values, column=comparison_cols);<jupyter_output><empty_output><jupyter_text>Alternative treatment of missing dataIf we choose option 3 we must first select the appropriate rows in the data before computing the test statistic. The code below performs the selection. We can also use SQL to select the relevant data from the database directly, and the corresponding SQL command for this can be found below (soon).<jupyter_code>def clean_col(column):
    r'''Returns a column from a dataframe after dropping encoded missing values'''
    return data.loc[~data[column].isin([NaN_encoding_dict[column]]), column]

def clean_inner_join(column1='CANCERCAREPLANINTENT', column2='ACE27', verbose=False):
    r'''Returns a pair of columns from a dataframe after dropping rows with any encoded missing values'''
    clean_col1 = clean_col(column1)
    clean_col2 = clean_col(column2)
    combined = pd.concat([clean_col1, clean_col2], axis=1, join='inner')
    if verbose:
        print('Column | preclean size | cleaned size\n{} | {} | {}\n{} | {} | {}'.format(
            column1, data[column1].size, clean_col1.size, column2, data[column2].size, clean_col2.size))
        print('The joined data has shape', combined.shape)
    return combined

interact(clean_inner_join, column1=comparison_cols, column2=comparison_cols, verbose=False);<jupyter_output><empty_output><jupyter_text>The function below computes a pair of a $\chi^2$ test-statistic and the associated p-value for a given pair of columns/characteristics after dropping any records with missing values for the given characteristics.<jupyter_code>from sklearn.feature_selection import chi2

def compute_chi2_value(column1='SITE_ICD10_O2', column2='CREG_CODE'):
    r'''Returns the chi2 test statistic and associated p-value between the given pair of columns from the dataset
    where we ignore any records in those columns containing missing values'''
    combined = clean_inner_join(column1, column2, verbose=True)
    return chi2(combined[column1].values.reshape(-1, 1), combined[column2])

interact(compute_chi2_value, column1=comparison_cols, column2=comparison_cols);<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07920142468834943, 'fraction_numerical': 0.013403318024182211, 'mean_word_length': 5.906148867313916, 'pattern_counts': {'":': 0, '<': 41, '<?xml version=': 0, '>': 41, 'https://': 7, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '10', 'dir': 'jupyter-structured-clean-dedup', 'id': '580451', 'license': 'MIT', 'n_tokens_mistral': 3010, 'n_tokens_neox': 2871, 'n_words': 1140, 'path': 'Exploratory Data Analysis.ipynb', 'repo_name': 'EdwardMPearce/simulacrum', 'size': '10669'}