title model interpretability with integrated gradients aknainhttps twitter comaknain date created 20200602 last modified 20200602 description how to obtain integrated gradients for a classification model accelerator none integrated gradients integrated gradientshttps arxiv orgabs1703 01365 is a technique for attributing a classification model s prediction to its input features it is a model interpretability technique you can use it to visualize the relationship between input features and model predictions integrated gradients is a variation on computing the gradient of the prediction output with regard to features of the input to compute integrated gradients we need to perform the following steps 1 identify the input and the output in our case the input is an image and the output is the last layer of our model dense layer with softmax activation 2 compute which features are important to a neural network when making a prediction on a particular data point to identify these features we need to choose a baseline input a baseline input can be a black image all pixel values set to zero or random noise the shape of the baseline input needs to be the same as our input image e g 299 299 3 3 interpolate the baseline for a given number of steps the number of steps represents the steps we need in the gradient approximation for a given input image the number of steps is a hyperparameter the s recommend using anywhere between 20 and 1000 steps 4 preprocess these interpolated images and do a forward pass 5 get the gradients for these interpolated images 6 approximate the gradients integral using the trapezoidal rule to read indepth about integrated gradients and why this method works consider reading this excellent articlehttps distill pub2020attributionbaselines references integrated gradients original paperhttps arxiv orgabs1703 01365 original implementationhttps github comankurtalyintegratedgradients setup size of the input image load xception model with imagenet weights the local path to our target image integrated gradients algorithm img is a pil image of size 299x299 array is a float32 numpy array of shape 299 299 3 we add a dimension to transform our array into a batch of size 1 299 299 3 computes the gradients of outputs w r t input image args imginput 4d image tensor toppredidx predicted label for the input image returns gradients of the predictions w r t imginput computes integrated gradients for a predicted label args imginput ndarray original image toppredidx predicted label for the input image baseline ndarray the baseline image to start with for interpolation numsteps number of interpolation steps between the baseline and the input used in the computation of integrated gradients these steps along determine the integral approximation error by default numsteps is set to 50 returns integrated gradients w r t input image if baseline is not provided start with a black image having same size as the input image 1 do interpolation 2 preprocess the interpolated images 3 get the gradients 4 approximate the integral using the trapezoidal rule 5 calculate integrated gradients and return generates a number of random baseline images args imginput ndarray 3d image toppredidx predicted label for the input image numsteps number of interpolation steps between the baseline and the input used in the computation of integrated gradients these steps along determine the integral approximation error by default numsteps is set to 50 numruns number of baseline images to generate returns averaged integrated gradients for numruns baseline images 1 list to keep track of integrated gradients ig for all the images 2 get the integrated gradients for all the baselines 3 return the average integrated gradients for the image helper class for visualizing gradients and integrated gradients plot gradients of the outputs w r t an input image def initself positivechannelnone negativechannelnone if positivechannel is none self positivechannel 0 255 0 else self positivechannel positivechannel if negativechannel is none self negativechannel 255 0 0 else self negativechannel negativechannel def applypolarityself attributions polarity if polarity positive return np clipattributions 0 1 else return np clipattributions 1 0 def applylineartransformation self attributions clipabovepercentile99 9 clipbelowpercentile70 0 lowerend0 2 1 get the thresholds m self getthresholdedattributions attributions percentage100 clipabovepercentile e self getthresholdedattributions attributions percentage100 clipbelowpercentile 2 transform the attributions by a linear function fx ax b such that fm 1 0 and fe lowerend transformedattributions 1 lowerend np absattributions e m e lowerend 3 make sure that the sign of transformed attributions is the same as original attributions transformedattributions np signattributions 4 only keep values that are bigger than the lowerend transformedattributions transformedattributions lowerend 5 clip values and return transformedattributions np cliptransformedattributions 0 0 1 0 return transformedattributions def getthresholdedattributionsself attributions percentage if percentage 100 0 return np minattributions 1 flatten the attributions flattenattr attributions flatten 2 get the sum of the attributions total np sumflattenattr 3 sort the attributions from largest to smallest sortedattributions np sortnp absflattenattr 1 4 calculate the percentage of the total sum that each attribution and the values about it contribute cumsum 100 0 np cumsumsortedattributions total 5 threshold the attributions by the percentage indicestoconsider np wherecumsum percentage00 6 select the desired attributions and return attributions sortedattributionsindicestoconsider return attributions def binarizeself attributions threshold0 001 return attributions threshold def morphologicalcleanupfnself attributions structurenp ones4 4 closed ndimage greyclosingattributions structurestructure opened ndimage greyopeningclosed structurestructure return opened def drawoutlines self attributions percentage90 connectedcomponentstructurenp ones3 3 1 binarize the attributions attributions self binarizeattributions 2 fill the gaps attributions ndimage binaryfillholesattributions 3 compute connected components connectedcomponents numcomp ndimage label attributions structureconnectedcomponentstructure 4 sum up the attributions for each component total np sumattributionsconnectedcomponents 0 componentsums for comp in range1 numcomp 1 mask connectedcomponents comp componentsum np sumattributionsmask componentsums appendcomponentsum mask 5 compute the percentage of top components to keep sortedsumsandmasks sorted componentsums keylambda x x0 reversetrue sortedsums listzipsortedsumsandmasks0 cumulativesortedsums np cumsumsortedsums cutoffthreshold percentage total 100 cutoffidx np wherecumulativesortedsums cutoffthreshold00 if cutoffidx 2 cutoffidx 2 6 set the values for the kept components bordermask np zeroslikeattributions for i in rangecutoffidx 1 bordermasksortedsumsandmasksi1 1 7 make the mask hollow and show only the border erodedmask ndimage binaryerosionbordermask iterations1 bordermaskerodedmask 0 8 return the outlined mask return bordermask def processgrads self image attributions polaritypositive clipabovepercentile99 9 clipbelowpercentile0 morphologicalcleanupfalse structurenp ones3 3 outlinesfalse outlinescomponentpercentage90 overlaytrue if polarity not in positive negative raise valueerror f allowed polarity values positive or negative but provided polarity if clipabovepercentile 0 or clipabovepercentile 100 raise valueerrorclipabovepercentile must be in 0 100 if clipbelowpercentile 0 or clipbelowpercentile 100 raise valueerrorclipbelowpercentile must be in 0 100 1 apply polarity if polarity positive attributions self applypolarityattributions polaritypolarity channel self positivechannel else attributions self applypolarityattributions polaritypolarity attributions np absattributions channel self negativechannel 2 take average over the channels attributions np averageattributions axis2 3 apply linear transformation to the attributions attributions self applylineartransformation attributions clipabovepercentileclipabovepercentile clipbelowpercentileclipbelowpercentile lowerend0 0 4 cleanup if morphologicalcleanup attributions self morphologicalcleanupfn attributions structurestructure 5 draw the outlines if outlines attributions self drawoutlines attributions percentageoutlinescomponentpercentage 6 expand the channel axis and convert to rgb attributions np expanddimsattributions 2 channel 7 superimpose on the original image if overlay attributions np clipattributions 0 8 image 0 255 return attributions def visualize self image gradients integratedgradients polaritypositive clipabovepercentile99 9 clipbelowpercentile0 morphologicalcleanupfalse structurenp ones3 3 outlinesfalse outlinescomponentpercentage90 overlaytrue figsize15 8 1 make two copies of the original image img1 np copyimage img2 np copyimage 2 process the normal gradients gradsattr self processgrads imageimg1 attributionsgradients polaritypolarity clipabovepercentileclipabovepercentile clipbelowpercentileclipbelowpercentile morphologicalcleanupmorphologicalcleanup structurestructure outlinesoutlines outlinescomponentpercentageoutlinescomponentpercentage overlayoverlay 3 process the integrated gradients igradsattr self processgrads imageimg2 attributionsintegratedgradients polaritypolarity clipabovepercentileclipabovepercentile clipbelowpercentileclipbelowpercentile morphologicalcleanupmorphologicalcleanup structurestructure outlinesoutlines outlinescomponentpercentageoutlinescomponentpercentage overlayoverlay ax plt subplots1 3 figsizefigsize ax0 imshowimage ax1 imshowgradsattr astypenp uint8 ax2 imshowigradsattr astypenp uint8 ax0 settitleinput ax1 settitlenormal gradients ax2 settitleintegrated gradients plt show let s testdrive it 1 convert the image to numpy array img getimgarrayimgpath 2 keep a copy of the original image origimg np copyimg0 astypenp uint8 3 preprocess the image imgprocessed tf castxception preprocessinputimg dtypetf float32 4 get model predictions preds model predictimgprocessed toppredidx tf argmaxpreds0 printpredicted toppredidx xception decodepredictionspreds top10 5 get the gradients of the last layer for the predicted label grads getgradientsimgprocessed toppredidxtoppredidx 6 get the integrated gradients igrads randombaselineintegratedgradients np copyorigimg toppredidxtoppredidx numsteps50 numruns2 7 process the gradients and plot vis gradvisualizer vis visualize imageorigimg gradientsgrads0 numpy integratedgradientsigrads numpy clipabovepercentile99 clipbelowpercentile0 vis visualize imageorigimg gradientsgrads0 numpy integratedgradientsigrads numpy clipabovepercentile95 clipbelowpercentile28 morphologicalcleanuptrue outlinestrue integrated gradients integrated gradients https arxiv org abs 1703 01365 is a technique for attributing a classification model s prediction to its input features it is a model interpretability technique you can use it to visualize the relationship between input features and model predictions integrated gradients is a variation on computing the gradient of the prediction output with regard to features of the input to compute integrated gradients we need to perform the following steps 1 identify the input and the output in our case the input is an image and the output is the last layer of our model dense layer with softmax activation 2 compute which features are important to a neural network when making a prediction on a particular data point to identify these features we need to choose a baseline input a baseline input can be a black image all pixel values set to zero or random noise the shape of the baseline input needs to be the same as our input image e g 299 299 3 3 interpolate the baseline for a given number of steps the number of steps represents the steps we need in the gradient approximation for a given input image the number of steps is a hyperparameter the s recommend using anywhere between 20 and 1000 steps 4 preprocess these interpolated images and do a forward pass 5 get the gradients for these interpolated images 6 approximate the gradients integral using the trapezoidal rule to read in depth about integrated gradients and why this method works consider reading this excellent article https distill pub 2020 attribution baselines references integrated gradients original paper https arxiv org abs 1703 01365 original implementation https github com ankurtaly integrated gradients setup size of the input image load xception model with imagenet weights the local path to our target image integrated gradients algorithm img is a pil image of size 299x299 array is a float32 numpy array of shape 299 299 3 we add a dimension to transform our array into a batch of size 1 299 299 3 computes the gradients of outputs w r t input image args img_input 4d image tensor top_pred_idx predicted label for the input image returns gradients of the predictions w r t img_input computes integrated gradients for a predicted label args img_input ndarray original image top_pred_idx predicted label for the input image baseline ndarray the baseline image to start with for interpolation num_steps number of interpolation steps between the baseline and the input used in the computation of integrated gradients these steps along determine the integral approximation error by default num_steps is set to 50 returns integrated gradients w r t input image if baseline is not provided start with a black image having same size as the input image 1 do interpolation 2 preprocess the interpolated images 3 get the gradients 4 approximate the integral using the trapezoidal rule 5 calculate integrated gradients and return generates a number of random baseline images args img_input ndarray 3d image top_pred_idx predicted label for the input image num_steps number of interpolation steps between the baseline and the input used in the computation of integrated gradients these steps along determine the integral approximation error by default num_steps is set to 50 num_runs number of baseline images to generate returns averaged integrated gradients for num_runs baseline images 1 list to keep track of integrated gradients ig for all the images 2 get the integrated gradients for all the baselines 3 return the average integrated gradients for the image helper class for visualizing gradients and integrated gradients plot gradients of the outputs w r t an input image 1 get the thresholds 2 transform the attributions by a linear function f x a x b such that f m 1 0 and f e lower_end 3 make sure that the sign of transformed attributions is the same as original attributions 4 only keep values that are bigger than the lower_end 5 clip values and return 1 flatten the attributions 2 get the sum of the attributions 3 sort the attributions from largest to smallest 4 calculate the percentage of the total sum that each attribution and the values about it contribute 5 threshold the attributions by the percentage 6 select the desired attributions and return 1 binarize the attributions 2 fill the gaps 3 compute connected components 4 sum up the attributions for each component 5 compute the percentage of top components to keep 6 set the values for the kept components 7 make the mask hollow and show only the border 8 return the outlined mask allowed polarity values positive or negative but provided polarity 1 apply polarity 2 take average over the channels 3 apply linear transformation to the attributions 4 cleanup 5 draw the outlines 6 expand the channel axis and convert to rgb 7 superimpose on the original image 1 make two copies of the original image 2 process the normal gradients 3 process the integrated gradients let s test drive it 1 convert the image to numpy array 2 keep a copy of the original image 3 preprocess the image 4 get model predictions 5 get the gradients of the last layer for the predicted label 6 get the integrated gradients 7 process the gradients and plot
import numpy as np
import matplotlib.pyplot as plt
from scipy import ndimage
from IPython.display import Image, display

import tensorflow as tf
import keras
from keras import layers
from keras.applications import xception

keras.config.disable_traceback_filtering()



img_size = (299, 299, 3)


model = xception.Xception(weights="imagenet")


img_path = keras.utils.get_file(
    "elephant.jpg", "https://i.imgur.com/Bvro0YD.png"
)
display(Image(img_path))




def get_img_array(img_path, size=(299, 299)):
    
    img = keras.utils.load_img(img_path, target_size=size)
    
    array = keras.utils.img_to_array(img)
    
    
    array = np.expand_dims(array, axis=0)
    return array


def get_gradients(img_input, top_pred_idx):
    
    images = tf.cast(img_input, tf.float32)

    with tf.GradientTape() as tape:
        tape.watch(images)
        preds = model(images)
        top_class = preds[:, top_pred_idx]

    grads = tape.gradient(top_class, images)
    return grads


def get_integrated_gradients(
    img_input, top_pred_idx, baseline=None, num_steps=50
):
    
    
    
    if baseline is None:
        baseline = np.zeros(img_size).astype(np.float32)
    else:
        baseline = baseline.astype(np.float32)

    
    img_input = img_input.astype(np.float32)
    interpolated_image = [
        baseline + (step / num_steps) * (img_input - baseline)
        for step in range(num_steps + 1)
    ]
    interpolated_image = np.array(interpolated_image).astype(np.float32)

    
    interpolated_image = xception.preprocess_input(interpolated_image)

    
    grads = []
    for i, img in enumerate(interpolated_image):
        img = tf.expand_dims(img, axis=0)
        grad = get_gradients(img, top_pred_idx=top_pred_idx)
        grads.append(grad[0])
    grads = tf.convert_to_tensor(grads, dtype=tf.float32)

    
    grads = (grads[:-1] + grads[1:]) / 2.0
    avg_grads = tf.reduce_mean(grads, axis=0)

    
    integrated_grads = (img_input - baseline) * avg_grads
    return integrated_grads


def random_baseline_integrated_gradients(
    img_input, top_pred_idx, num_steps=50, num_runs=2
):
    
    
    integrated_grads = []

    
    for run in range(num_runs):
        baseline = np.random.random(img_size) * 255
        igrads = get_integrated_gradients(
            img_input=img_input,
            top_pred_idx=top_pred_idx,
            baseline=baseline,
            num_steps=num_steps,
        )
        integrated_grads.append(igrads)

    
    integrated_grads = tf.convert_to_tensor(integrated_grads)
    return tf.reduce_mean(integrated_grads, axis=0)





class GradVisualizer:
    

    def __init__(self, positive_channel=None, negative_channel=None):
        if positive_channel is None:
            self.positive_channel = [0, 255, 0]
        else:
            self.positive_channel = positive_channel

        if negative_channel is None:
            self.negative_channel = [255, 0, 0]
        else:
            self.negative_channel = negative_channel

    def apply_polarity(self, attributions, polarity):
        if polarity == "positive":
            return np.clip(attributions, 0, 1)
        else:
            return np.clip(attributions, -1, 0)

    def apply_linear_transformation(
        self,
        attributions,
        clip_above_percentile=99.9,
        clip_below_percentile=70.0,
        lower_end=0.2,
    ):
        
        m = self.get_thresholded_attributions(
            attributions, percentage=100 - clip_above_percentile
        )
        e = self.get_thresholded_attributions(
            attributions, percentage=100 - clip_below_percentile
        )

        
        
        transformed_attributions = (1 - lower_end) * (
            np.abs(attributions) - e
        ) / (m - e) + lower_end

        
        transformed_attributions *= np.sign(attributions)

        
        transformed_attributions *= transformed_attributions >= lower_end

        
        transformed_attributions = np.clip(transformed_attributions, 0.0, 1.0)
        return transformed_attributions

    def get_thresholded_attributions(self, attributions, percentage):
        if percentage == 100.0:
            return np.min(attributions)

        
        flatten_attr = attributions.flatten()

        
        total = np.sum(flatten_attr)

        
        sorted_attributions = np.sort(np.abs(flatten_attr))[::-1]

        
        
        cum_sum = 100.0 * np.cumsum(sorted_attributions) / total

        
        indices_to_consider = np.where(cum_sum >= percentage)[0][0]

        
        attributions = sorted_attributions[indices_to_consider]
        return attributions

    def binarize(self, attributions, threshold=0.001):
        return attributions > threshold

    def morphological_cleanup_fn(self, attributions, structure=np.ones((4, 4))):
        closed = ndimage.grey_closing(attributions, structure=structure)
        opened = ndimage.grey_opening(closed, structure=structure)
        return opened

    def draw_outlines(
        self,
        attributions,
        percentage=90,
        connected_component_structure=np.ones((3, 3)),
    ):
        
        attributions = self.binarize(attributions)

        
        attributions = ndimage.binary_fill_holes(attributions)

        
        connected_components, num_comp = ndimage.label(
            attributions, structure=connected_component_structure
        )

        
        total = np.sum(attributions[connected_components > 0])
        component_sums = []
        for comp in range(1, num_comp + 1):
            mask = connected_components == comp
            component_sum = np.sum(attributions[mask])
            component_sums.append((component_sum, mask))

        
        sorted_sums_and_masks = sorted(
            component_sums, key=lambda x: x[0], reverse=True
        )
        sorted_sums = list(zip(*sorted_sums_and_masks))[0]
        cumulative_sorted_sums = np.cumsum(sorted_sums)
        cutoff_threshold = percentage * total / 100
        cutoff_idx = np.where(cumulative_sorted_sums >= cutoff_threshold)[0][0]
        if cutoff_idx > 2:
            cutoff_idx = 2

        
        border_mask = np.zeros_like(attributions)
        for i in range(cutoff_idx + 1):
            border_mask[sorted_sums_and_masks[i][1]] = 1

        
        eroded_mask = ndimage.binary_erosion(border_mask, iterations=1)
        border_mask[eroded_mask] = 0

        
        return border_mask

    def process_grads(
        self,
        image,
        attributions,
        polarity="positive",
        clip_above_percentile=99.9,
        clip_below_percentile=0,
        morphological_cleanup=False,
        structure=np.ones((3, 3)),
        outlines=False,
        outlines_component_percentage=90,
        overlay=True,
    ):
        if polarity not in ["positive", "negative"]:
            raise ValueError(
                f
            )
        if clip_above_percentile < 0 or clip_above_percentile > 100:
            raise ValueError("clip_above_percentile must be in [0, 100]")

        if clip_below_percentile < 0 or clip_below_percentile > 100:
            raise ValueError("clip_below_percentile must be in [0, 100]")

        
        if polarity == "positive":
            attributions = self.apply_polarity(attributions, polarity=polarity)
            channel = self.positive_channel
        else:
            attributions = self.apply_polarity(attributions, polarity=polarity)
            attributions = np.abs(attributions)
            channel = self.negative_channel

        
        attributions = np.average(attributions, axis=2)

        
        attributions = self.apply_linear_transformation(
            attributions,
            clip_above_percentile=clip_above_percentile,
            clip_below_percentile=clip_below_percentile,
            lower_end=0.0,
        )

        
        if morphological_cleanup:
            attributions = self.morphological_cleanup_fn(
                attributions, structure=structure
            )
        
        if outlines:
            attributions = self.draw_outlines(
                attributions, percentage=outlines_component_percentage
            )

        
        attributions = np.expand_dims(attributions, 2) * channel

        
        if overlay:
            attributions = np.clip((attributions * 0.8 + image), 0, 255)
        return attributions

    def visualize(
        self,
        image,
        gradients,
        integrated_gradients,
        polarity="positive",
        clip_above_percentile=99.9,
        clip_below_percentile=0,
        morphological_cleanup=False,
        structure=np.ones((3, 3)),
        outlines=False,
        outlines_component_percentage=90,
        overlay=True,
        figsize=(15, 8),
    ):
        
        img1 = np.copy(image)
        img2 = np.copy(image)

        
        grads_attr = self.process_grads(
            image=img1,
            attributions=gradients,
            polarity=polarity,
            clip_above_percentile=clip_above_percentile,
            clip_below_percentile=clip_below_percentile,
            morphological_cleanup=morphological_cleanup,
            structure=structure,
            outlines=outlines,
            outlines_component_percentage=outlines_component_percentage,
            overlay=overlay,
        )

        
        igrads_attr = self.process_grads(
            image=img2,
            attributions=integrated_gradients,
            polarity=polarity,
            clip_above_percentile=clip_above_percentile,
            clip_below_percentile=clip_below_percentile,
            morphological_cleanup=morphological_cleanup,
            structure=structure,
            outlines=outlines,
            outlines_component_percentage=outlines_component_percentage,
            overlay=overlay,
        )

        _, ax = plt.subplots(1, 3, figsize=figsize)
        ax[0].imshow(image)
        ax[1].imshow(grads_attr.astype(np.uint8))
        ax[2].imshow(igrads_attr.astype(np.uint8))

        ax[0].set_title("Input")
        ax[1].set_title("Normal gradients")
        ax[2].set_title("Integrated gradients")
        plt.show()





img = get_img_array(img_path)


orig_img = np.copy(img[0]).astype(np.uint8)


img_processed = tf.cast(xception.preprocess_input(img), dtype=tf.float32)


preds = model.predict(img_processed)
top_pred_idx = tf.argmax(preds[0])
print("Predicted:", top_pred_idx, xception.decode_predictions(preds, top=1)[0])


grads = get_gradients(img_processed, top_pred_idx=top_pred_idx)


igrads = random_baseline_integrated_gradients(
    np.copy(orig_img), top_pred_idx=top_pred_idx, num_steps=50, num_runs=2
)


vis = GradVisualizer()
vis.visualize(
    image=orig_img,
    gradients=grads[0].numpy(),
    integrated_gradients=igrads.numpy(),
    clip_above_percentile=99,
    clip_below_percentile=0,
)

vis.visualize(
    image=orig_img,
    gradients=grads[0].numpy(),
    integrated_gradients=igrads.numpy(),
    clip_above_percentile=95,
    clip_below_percentile=28,
    morphological_cleanup=True,
    outlines=True,
)