<jupyter_start><jupyter_text>SSIM for the same model and imagesIn this notebook, we calculate SSIM on images of SHAP and GradCAM explanations which were produced from the same model running on the same images. This is for sanity check 1, the reproducibility of the methods.The output from the explainability methods are located in the folders `Explanations_ModelSearch2_Seed5_1` and `Explanations_ModelSearch2_Seed5_2`.<jupyter_code>%matplotlib inline
import pandas as pd
import numpy as np 
from PIL import Image

from skimage.measure import compare_ssim as ssim
import os
from glob import glob
import matplotlib.pyplot as plt
path_dict = {}
for i in range(1,3):
    path_dict[i-1] = glob('Explanations_ModelSearch2_Seed5_{}/*.png'.format(i),recursive=True)
len(path_dict[0])
path_frame = pd.DataFrame.from_dict(path_dict, orient = 'index')
path_frame = path_frame.transpose()<jupyter_output><empty_output><jupyter_text>Load all images<jupyter_code>for i in path_dict.keys():
    image_list = []
    path_list = path_frame[i].tolist()
    for j in path_list:
        image_list.append(np.asarray(Image.open(j)))
    path_frame['Folder_{}_image'.format(i)] = image_list
path_frame.head()
for a in range(2):
    path_frame['SSIM_0_to_{}'.format(a)] = np.nan<jupyter_output><empty_output><jupyter_text>Calculate all SSIM values<jupyter_code>for i in path_frame.index:
    for j in range(2):
        path_frame['SSIM_0_to_{}'.format(j)][i] = ssim(path_frame['Folder_0_image'][i], 
                                                       path_frame['Folder_{}_image'.format(j)][i],
                                                       win_size=3, multichannel = True, use_sample_covariance = False
                                                       #, win_size = 3
                                                      )
path_frame.head()<jupyter_output><empty_output><jupyter_text>Describe the SSIM from of the images from the two different folders for GradCAM<jupyter_code>grad_df = ['Grad.png' in path_frame.iloc[i][0] for i in path_frame.index]
grad_df = path_frame[grad_df]
len(grad_df)
grad_df = grad_df.drop([1,'Folder_0_image','Folder_1_image'], axis = 1)
grad_df['SSIM_Difference'] = abs(grad_df['SSIM_0_to_0'] - grad_df['SSIM_0_to_1'])
grad_df['SSIM_0_to_1'].describe()<jupyter_output><empty_output><jupyter_text>Do the same for SHAP<jupyter_code>shap_df = ['SHAP.png' in path_frame.iloc[i][0] for i in path_frame.index] 
shap_df = path_frame[shap_df].drop(['Folder_0_image','Folder_1_image'], axis = 1)
shap_df.head()
shap_df['SSIM_Difference'] = abs(shap_df['SSIM_0_to_0'] - shap_df['SSIM_0_to_1'])
shap_df['SSIM_0_to_1'].describe()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11289719626168224, 'fraction_numerical': 0.015327102803738318, 'mean_word_length': 4.097142857142857, 'pattern_counts': {'":': 0, '<': 21, '<?xml version=': 0, '>': 21, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 8, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '5', 'dir': 'jupyter-structured-clean-dedup', 'id': '595303', 'license': 'MIT', 'n_tokens_mistral': 940, 'n_tokens_neox': 922, 'n_words': 192, 'path': 'Model_Sensitivity_Experiments/SSIM_SameModelAndImages.ipynb', 'repo_name': 'yelk314/DNNorDermatologist', 'size': '2675'}