<jupyter_start><jupyter_text>Course 3 Week 3 lecture notebook 01 Permutation Method In this lecture notebook, we'll demonstrate the permutation method on a dataset you worked on in Week 1 of Course 2. By going through this short exercise, you'll hopefully develop some intuition on how to do the related task in this week's programming assignment.  Setup First, let's import the packages we'll be using:<jupyter_code>from google.colab import drive

drive.mount('/content/drive')
import os

os.chdir('drive/My Drive/AI for Health/Treatment/Week 3')
import pandas as pd
import pickle
from util import cindex<jupyter_output><empty_output><jupyter_text>Next, we load the model we trained during the said assignment in Course 2. This accepts a patient's age, systolic blood pressure, diastolic blood pressure, and cholesterol to output a risk score (i.e. probability of an event).<jupyter_code>model_X = pickle.load(open('C2M1_model.model', 'rb'))<jupyter_output><empty_output><jupyter_text>We will also use the same test dataset from that programming assignment:<jupyter_code># patient data (test set in the previous assignment)
X_baseline = pd.read_csv('./lecture_nb_permutation_data/X_data_normalized.csv',index_col=0)

# corresponding patient outcome
y = pd.read_csv('./lecture_nb_permutation_data/y_data_test.csv',index_col=0)<jupyter_output><empty_output><jupyter_text>Let's print the first 10 cells to peek into the dataset. Take note that these are already standardized and normalized as you did previously.<jupyter_code>X_baseline.head(10)<jupyter_output><empty_output><jupyter_text>C-index and Baseline Performance Now let's get the c-index of the trained model using our test dataset. Just to refresh:* The c-index measures the discriminatory power of a risk score. * Intuitively, a higher c-index indicates that the model's prediction is in agreement with the actual outcomes of a pair of patients.* The formula for the c-index is$$ \mbox{cindex} = \frac{\mbox{concordant} + 0.5 \times \mbox{ties}}{\mbox{permissible}} $$* A permissible pair is a pair of patients who have different outcomes.* A concordant pair is a permissible pair in which the patient with the higher risk score also has the worse outcome.* A tie is a permissible pair where the patients have the same risk score.Computing this for our test dataset will give us our baseline performance. For efficiency, we will be using the `concordance_index()` method from the [lifelines](https://lifelines.readthedocs.io/en/latest/lifelines.utils.html) package instead of the homemade c-index function we developed in the Course 2 assignment. This is wrapped in a `cindex()` function in the `util` library we imported earlier.<jupyter_code># get the patient's risk scores by feeding the baseline data to the trained model
scores = model_X.predict_proba(X_baseline)[:, 1]

# measure the model's cindex given the patient outcome and computed risk scores above
c_index = cindex(y.values, scores)

print(f"baseline c-index is {c_index:.4f}")<jupyter_output>baseline c-index is 0.8182<jupyter_text>This should look familiar as this is also the result you got in the Course 2 assignment.   Feature Importance and the Permutation Method As discussed in the lecture videos, one way we can compute the importance of a feature in a given model is by shuffling the values of a particular column in our test set. Evaluating the performance of the model after this shuffling will allow us to quantify the feature's importance. For this exercise, we'll choose to shuffle the `Age` column. You will do this programmatically in this week's assignment but for now, you'll use a dataset that is shuffled beforehand.<jupyter_code># patient data permuted at the Age column
X_permuted_1 = pd.read_csv('./lecture_nb_permutation_data/X_data_normalized_perm1.csv',index_col=0)

# Let's print the first 10 cells and compare with the baseline table above.
# All are in the same place except for the values in the Age column
X_permuted_1.head(10)<jupyter_output><empty_output><jupyter_text>Now, let's see how our model performs with this shuffled input:<jupyter_code># get the patient's risk scores by feeding the baseline data with shuffled Age column
scores = model_X.predict_proba(X_permuted_1)[:, 1]

# measure the c-index given the patient outcome and newly computed risk scores
c_index = cindex(y.values, scores)
print(f"c-index for 1st permutation is {c_index:.4f}")<jupyter_output>c-index for 1st permutation is 0.6749<jupyter_text>We see a big drop in the performance. To quantify the importance of this feature, we simply subtract this drop from the baseline performance. This results in `0.8182 - 0.6749 = 0.1433`  Additional Permutations The result above (i.e. `0.1433`) describes the feature importance of `Age` in our model given the shuffled dataset. However, you might deduce that the predicted risk scores are greatly affected by how the Age column is rearranged. If we arrange it differently, then the model will predict different values and this might lead to a different c-index, and consequently a different value of the feature importance. To illustrate, let's import two more datasets with different permutations of the `Age` column. Since there is a random element when doing the permutation, the resulting order of the elements in the `Age` column will be different for each shuffle. You can see that in the two datasets below.<jupyter_code># 2nd permutation
X_permuted_2 = pd.read_csv('./lecture_nb_permutation_data/X_data_normalized_perm2.csv',index_col=0)

X_permuted_2.head(10)
# 3rd permutation
X_permuted_3 = pd.read_csv('./lecture_nb_permutation_data/X_data_normalized_perm3.csv',index_col=0)

X_permuted_3.head(10)<jupyter_output><empty_output><jupyter_text>Now let's evaluate the performance of the model with these new datasets:<jupyter_code># get the patient's risk scores by feeding a dataset with a different permutation of the Age column
scores = model_X.predict_proba(X_permuted_2)[:, 1]

# measure c-index
c_index = cindex(y.values, scores)

print(f"c-index for 2nd permutation: {c_index:.4f}")
# get the patient's risk scores by feeding a dataset with a different permutation of the Age column
scores = model_X.predict_proba(X_permuted_3)[:, 1]

# measure c-index
c_index = cindex(y.values, scores)

print(f"c-index for 3rd permutation: {c_index:.4f}")<jupyter_output>c-index for 3rd permutation: 0.6428
{'fraction_non_alphanumeric': 0.07035412096521466, 'fraction_numerical': 0.014572234409276089, 'mean_word_length': 5.621369294605809, 'pattern_counts': {'":': 0, '<': 34, '<?xml version=': 0, '>': 34, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '303479', 'license': 'MIT', 'n_tokens_mistral': 1950, 'n_tokens_neox': 1785, 'n_words': 792, 'path': 'Treatment/Week 3/C3W3_L1_Permutation method.ipynb', 'repo_name': 'hassancpu/Coursera-AI-for-Medicine-Specialization', 'size': '6382'}