<jupyter_start><jupyter_text>SSIM for 3 different modelsIn this notebook, we compare the outputs of the explainability methods on three different models, all of which have a similar AUC. We look at output from the explanations on the same images evaluated on the three different models. These images are in `Explanations_ModelSearch2_Seed5`, `Explanations_ModelSearch2_Seed8`, and `Explanations_ModelSearch2_Seed5`.This is for sanity check 3, the sensitivity of the methods.<jupyter_code>%matplotlib inline
import pandas as pd
import numpy as np 
from PIL import Image

from skimage.measure import compare_ssim as ssim
import os
from glob import glob
import matplotlib.pyplot as plt<jupyter_output><empty_output><jupyter_text>Load in images from the 3 different seeds<jupyter_code>path_dict = {}

for i in [5,8,11]:
    path_dict[i] = glob('Explanations_ModelSearch2_Seed{}/*.png'.format(i),recursive=True)
path_frame = pd.DataFrame.from_dict(path_dict)
for i in path_dict.keys():
    image_list = []
    path_list = path_frame[i].tolist()
    for j in path_list:
        image_list.append(np.asarray(Image.open(j)))
    path_frame['Folder_{}_image'.format(i)] = image_list
    
path_frame.head()<jupyter_output><empty_output><jupyter_text>Create placeholder for our results<jupyter_code>for a in [5,8,11]:
    path_frame['SSIM_5_to_{}'.format(a)] = np.nan<jupyter_output><empty_output><jupyter_text>Check - SSIM of the same image should be 1.<jupyter_code>ssim(path_frame['Folder_5_image'][0], path_frame['Folder_5_image'][0], win_size = 3)<jupyter_output><empty_output><jupyter_text>Calculate all SSIM Calculate SSIM's between all 3 models<jupyter_code>for i in path_frame.index:
    for j in [5,8,11]:
        path_frame['SSIM_5_to_{}'.format(j)][i] = ssim(path_frame['Folder_5_image'][i], path_frame['Folder_{}_image'.format(j)][i],
                                                       win_size=9, multichannel = True, use_sample_covariance = True)
path_frame.head()<jupyter_output><empty_output><jupyter_text>Separate SHAP and GradCAM results, and remove extra columns<jupyter_code>shap_df = ['SHAP.png' in path_frame.iloc[i][5] for i in path_frame.index] 
shap_df = path_frame[shap_df].drop([5, 8,11,'Folder_5_image','Folder_8_image','Folder_11_image'], axis = 1)
grad_df = ['Grad.png' in path_frame.iloc[i][5] for i in path_frame.index] 
grad_df = path_frame[grad_df].drop([5, 8,11,'Folder_5_image','Folder_8_image','Folder_11_image'], axis = 1)
grad_df.describe().to_csv('grad_sensitivity_stat.csv')
shap_df.describe().to_csv('shap_sensitivity_stat.csv')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11680801850424055, 'fraction_numerical': 0.020431765612952967, 'mean_word_length': 6.051630434782608, 'pattern_counts': {'":': 0, '<': 25, '<?xml version=': 0, '>': 25, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '6', 'dir': 'jupyter-structured-clean-dedup', 'id': '604641', 'license': 'MIT', 'n_tokens_mistral': 937, 'n_tokens_neox': 906, 'n_words': 199, 'path': 'Model_Sensitivity_Experiments/SSIM_3_Different_Models.ipynb', 'repo_name': 'yelk314/DNNorDermatologist', 'size': '2594'}