<jupyter_start><jupyter_text>Explainable AI in *ktrain*Deep neural networks are sometimes called "black boxes", as it is not always clear how such models are using data to make a decision or prediction.  **Explainable AI** (XAI) involves methods and techniques to help understand how an AI model reach particular conclusions.  Although XAI is an [open and actively researched problem](https://www.darpa.mil/program/explainable-artificial-intelligence), there are some existing methods that can be practically applied now.In previous tutorials, we saw that, for both image classification and text classification, we can invoke the `ktrain.get_predictor` function and obtain a `Predictor` object to make predictions on new raw data.  For instance, with text data, one can easily make predictions from the raw and unprocessed text of a document as follows:```predictor = ktrain.get_predictor(learner.model, preproc=preproc)predictor.predict(document_text) ```In this notebook, we  show how one can invoke the `explain` method of `Predictor` objects to help understand how those predictions were made.  This is particularly useful in understanding **misclassifications**. We start with image classification.  Explaining Image Classification Let's begin by using *ktrain* to train an image classifier for a single epoch on the publicly available [Kaggle Dogs vs. Cats dataset](https://www.kaggle.com/c/dogs-vs-cats), as we did in the [tutorial 3 notebook](https://github.com/amaiya/ktrain/blob/master/tutorials/tutorial-03-image-classification.ipynb).<jupyter_code># imports
import ktrain
from ktrain import vision as vis

# STEP 1: load data
(train_data, val_data, preproc) = vis.images_from_folder(
                                              datadir='data/dogscats',
                                              data_aug = vis.get_data_aug(horizontal_flip=True),
                                              train_test_names=['train', 'valid'], 
                                              target_size=(224,224), color_mode='rgb')

# STEP 2: define model
model = vis.image_classifier('pretrained_resnet50', train_data, val_data, freeze_layers=15)
learner = ktrain.get_learner(model=model, train_data=train_data, val_data=val_data, 
                             workers=8, use_multiprocessing=False, batch_size=64)

# STEP 3:  train
learner.fit_onecycle(1e-4, 1)<jupyter_output>Using TensorFlow backend.<jupyter_text>The `view_top_losses` method in *ktrain* identifies those examples in the validation set that were misclassified and sorts them by loss such that examples at the top of the list are the most severely misclassified.  Let's invoke `view_top_losses` to see the top 3 most misclassified images by our model.<jupyter_code>learner.view_top_losses(n=1, preproc=preproc)<jupyter_output><empty_output><jupyter_text>The top most misclassified image depicts both a dog and a cat but is labeled as belonging to only the **cats** category.  Our classifier's prediction for this image is **dogs** probably because it is focusing mostly on the dog in this image.  This can be verified by invoking the `explain` method.<jupyter_code>predictor = ktrain.get_predictor(learner.model, preproc)
predictor.explain('data/dogscats/valid/cats/cat.5583.jpg')<jupyter_output><empty_output><jupyter_text>The `explain` method displays the image and highlights the area on which the classifier focuses to make the prediction.  As expected, our model is focusing on the dog in this picture when predicting the label **dogs**.  In this case, **Explainable AI** was not really necessary to understand why our classifier generated its prediction.  However, such techniques can be helpful in other cases where the decision-making process is not as evident.  Consider this image in the validation set, for example:<jupyter_code>vis.show_image('data/dogscats/valid/cats/cat.92.jpg')<jupyter_output><empty_output><jupyter_text>The image (i.e., *cat.92.jpg*) is interesting in that there are no real dogs or cats in the picture.  Nevertheless, the image has a ground truth label of **cats**.  Perhaps this is because there is a stuffed cat in the photo.  There is, however, also a cartoon dog in the background.  Our classifier is understandably confused and placed the picture in the **dogs** category.<jupyter_code>predictor.predict_filename('data/dogscats/valid/cats/cat.92.jpg')<jupyter_output><empty_output><jupyter_text>But, **why** has our classifier predicted **dogs** for this photo?  Is it mistaking the stuffed cat for a dog or is it focusing on the cartoon dog in the background?  We can find the answer by creating a `Predictor` object and invoking the `explain` method.<jupyter_code>predictor.explain('data/dogscats/valid/cats/cat.92.jpg')<jupyter_output><empty_output><jupyter_text>It is clear from the visualization that our classifier is focusing on the *stuffed cat* and mistaking it for a dog.  Moreover, it appears to be particuarly to focused on the ears of the stuffed animal, which do sort of look like dog ears.These visualizations are based on the [Grad-CAM technique](https://arxiv.org/abs/1610.02391) and is supported in *ktrain* via the [eli5](https://github.com/TeamHG-Memex/eli5) library.  Explaining Text ClassificationWe can also visualize which words **text classifiers** tend to focus on when making predictions.  As before, let's use *ktrain* to quickly build a text classifier to classify [IMDb movie reivews](https://ai.stanford.edu/~amaas/data/sentiment/) as positive or negative.<jupyter_code># imports
import ktrain
from ktrain import text

# STEP 1: load and preprocess text data
(x_train, y_train), (x_test, y_test), preproc = text.texts_from_folder('data/aclImdb', 
                                                                       max_features=20000, maxlen=400, 
                                                                       ngram_range=1, 
                                                                       train_test_names=['train', 'test'],
                                                                       classes=['pos', 'neg'])
                
# STEP 2: define a Keras text classification model
from tensorflow.keras.models import Sequential
from tensorflow.keras.layers import Dense, Embedding, GlobalAveragePooling1D
model = Sequential()
model.add(Embedding(20000+1, 50, input_length=400)) # add 1 for padding token
model.add(GlobalAveragePooling1D())
model.add(Dense(2, activation='softmax'))
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['accuracy'])
learner = ktrain.get_learner(model, train_data=(x_train, y_train), val_data=(x_test, y_test))

# STEP 3: train
learner.autofit(0.005, 2)<jupyter_output>Word Counts: 88582
Nrows: 25000
25000 train sequences
Average train sequence length: 231
x_train shape: (25000,400)
y_train shape: (25000,2)
25000 test sequences
Average test sequence length: 224
x_test shape: (25000,400)
y_test shape: (25000,2)


begin training using triangular learning rate policy with max lr of 0.005...
Train on 25000 samples, validate on 25000 samples
Epoch 1/2
25000/25000 [==============================] - 9s 357us/step - loss: 0.4473 - acc: 0.7881 - val_loss: 0.3067 - val_acc: 0.8809
Epoch 2/2
25000/25000 [==============================] - 8s 317us/step - loss: 0.2328 - acc: 0.9122 - val_loss: 0.2794 - val_acc: 0.8890<jupyter_text>As we did above, we invoke `view_top_losses` to view the most misclassified review in the validation set.<jupyter_code>learner.view_top_losses(n=1, preproc=preproc)<jupyter_output>----------
id:8244 | loss:11.57 | true:neg | pred:pos)

is not like mickey rourke ever really disappeared he has had a steady string of appearances before he burst back on the scene he was memorable in domino sin city man on fire once upon a time in mexico and get carter but in his powerful dramatic performance in the wrestler 2008 we see a full blown presentation of the character only hinted at in get carter whenever we get to know him rourke remains a cool but sleazy muscle bound slim ball br br this is an leonard story and production leonard wrote such notable movies as western thriller 3 10 to yuma be cool jackie brown get 52 pick up and joe this means that we get tough guys some good some not so good br br it also means we get tight realistic plots with characters doing what is best for them in each situation weaving complications into violent conclusions is no different tough slim ball killer rourke stalks unhappily married witness lane think history of violence meets no country [...]<jupyter_text>The review is evidently about a movie starring <NAME>, but it is difficult to follow. This is partly because the text displayed is post-processed in that punctuation and rare words are omitted (including the actual title of the movie!).  Let's use the ID of this document (i.e., id=8244 as shown above) to retrieve the original text and supply that as input to `predictor.explain`.<jupyter_code>from sklearn.datasets import load_files
test_b = load_files(os.path.join('data/aclImdb',  'test'), shuffle=False, categories=['neg', 'pos'])
doc = test_b.data[8244].decode('utf-8')
predictor = ktrain.get_predictor(learner.model, preproc)<jupyter_output><empty_output><jupyter_text>**NOTE:** Output of `explain` below does not render correctly through GitHub's notebook viewer (colors are .  View directly through Jupyter notebook for proper display.<jupyter_code>predictor.explain(doc)<jupyter_output><empty_output><jupyter_text>The visualization is generated using a technique called [LIME](https://arxiv.org/abs/1602.04938).  The input is randomly perturbed to examine how the prediction changes.  This is used to infer the relative importance of different words to the final prediction using a linear interpretable model.  The GREEN words contribute to the incorrect classification. The RED (and PINK) words detract from our final prediction. (Shade of color denotes the strength or size of the coefficients in the inferred linear model.)   By examining the GREEN words, we can see that the review is overall positive despite being given a ground truth label of negative.  In fact, the last line recommends this movie as a rental. The review is also odd in that it spends a lot of time praising past performances of the actors in previous movies. The PINK words show us that there is not very much there in terms of negative words that would lead us to conclude that this is a negative review.  For these reasons, we can probably forgive our simple model for misclassifying this particular review as positive, as many humans would probably classify this as positive, as well.  Explaining Tabular ModelsLet's train a model to predict **Survival** using [Kaggle's Titatnic dataset](https://www.kaggle.com/c/titanic).After training the model, we will **explain** the model's prediction for a specific example.<jupyter_code>import pandas as pd
import numpy as np
train_df = pd.read_csv('data/titanic/train.csv', index_col=0)
train_df = train_df.drop('Name', 1)
train_df = train_df.drop('Ticket', 1)
train_df = train_df.drop('Cabin', 1)

np.random.seed(42)
p = 0.1 # 10% for test set
prop = 1-p
df = train_df.copy()
msk = np.random.rand(len(df)) < prop
train_df = df[msk]
test_df = df[~msk]


import ktrain
from ktrain import tabular
trn, val, preproc = tabular.tabular_from_df(train_df, label_columns=['Survived'], random_state=42)
model = tabular.tabular_classifier('mlp', trn)
learner = ktrain.get_learner(model, train_data=trn, val_data=val, batch_size=32)
learner.fit_onecycle(1e-3, 25)
predictor = ktrain.get_predictor(learner.model, preproc)
preds = predictor.predict(test_df, return_proba=True)
df = test_df.copy()[[c for c in test_df.columns.values if c != 'Survived']]
df['Survived'] = test_df['Survived']
df['predicted_Survived'] = np.argmax(preds, axis=1)
df.head()<jupyter_output><empty_output><jupyter_text>Let's explain the male passenger with PassengerID=5 using the [shape](https://github.com/slundberg/shap) library.<jupyter_code>predictor.explain(test_df, row_index=35, class_id=1)<jupyter_output>Explanation for class = Survived (PassengerId=35):
{'fraction_non_alphanumeric': 0.08545949962843696, 'fraction_numerical': 0.023202047725208487, 'mean_word_length': 4.383111111111111, 'pattern_counts': {'":': 0, '<': 47, '<?xml version=': 0, '>': 46, 'https://': 9, 'lorem ipsum': 0, 'www.': 3, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 20, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '12', 'dir': 'jupyter-structured-clean-dedup', 'id': '180707', 'license': 'Apache-2.0', 'n_tokens_mistral': 3660, 'n_tokens_neox': 3367, 'n_words': 1367, 'path': 'tutorials/tutorial-A2-explaining-predictions.ipynb', 'repo_name': 'xirect/ktrain', 'size': '12118'}