<jupyter_start><jupyter_text>Anchor explanations for movie sentiment In this example, we will explain why a certain sentence is classified by a logistic regression as having negative or positive sentiment. The logistic regression is trained on negative and positive movie reviews.<jupyter_code>import numpy as np
from sklearn.feature_extraction.text import CountVectorizer
from sklearn.linear_model import LogisticRegression
from sklearn.metrics import accuracy_score
from sklearn.model_selection import train_test_split
import spacy
from alibi.explainers import AnchorText
from alibi.datasets import fetch_movie_sentiment
from alibi.utils.download import spacy_model<jupyter_output><empty_output><jupyter_text>Load movie review dataset The `fetch_movie_sentiment` function returns a `Bunch` object containing the features, the targets and the target names for the dataset.<jupyter_code>movies = fetch_movie_sentiment()
movies.keys()
data = movies.data
labels = movies.target
target_names = movies.target_names<jupyter_output><empty_output><jupyter_text>Define shuffled training, validation and test set<jupyter_code>train, test, train_labels, test_labels = train_test_split(data, labels, test_size=.2, random_state=42)
train, val, train_labels, val_labels = train_test_split(train, train_labels, test_size=.1, random_state=42)
train_labels = np.array(train_labels)
test_labels = np.array(test_labels)
val_labels = np.array(val_labels)<jupyter_output><empty_output><jupyter_text>Apply CountVectorizer to training set<jupyter_code>vectorizer = CountVectorizer(min_df=1)
vectorizer.fit(train)<jupyter_output><empty_output><jupyter_text>Fit model<jupyter_code>np.random.seed(0)
clf = LogisticRegression(solver='liblinear')
clf.fit(vectorizer.transform(train), train_labels)<jupyter_output><empty_output><jupyter_text>Define prediction function<jupyter_code>predict_fn = lambda x: clf.predict(vectorizer.transform(x))<jupyter_output><empty_output><jupyter_text>Make predictions on train and test sets<jupyter_code>preds_train = predict_fn(train)
preds_val = predict_fn(val)
preds_test = predict_fn(test)
print('Train accuracy', accuracy_score(train_labels, preds_train))
print('Validation accuracy', accuracy_score(val_labels, preds_val))
print('Test accuracy', accuracy_score(test_labels, preds_test))<jupyter_output>Train accuracy 0.9801624284382905
Validation accuracy 0.7544910179640718
Test accuracy 0.7589841878294202<jupyter_text>Load spaCy modelEnglish multi-task CNN trained on OntoNotes, with GloVe vectors trained on Common Crawl. Assigns word vectors, context-specific token vectors, POS tags, dependency parse and named entities.<jupyter_code>model = 'en_core_web_md'
spacy_model(model=model)
nlp = spacy.load(model)<jupyter_output><empty_output><jupyter_text>Initialize anchor text explainer<jupyter_code>explainer = AnchorText(nlp, predict_fn)<jupyter_output><empty_output><jupyter_text>Explain a prediction<jupyter_code>class_names = movies.target_names
text = data[4]
print(text)<jupyter_output>a visually flashy but narratively opaque and emotionally vapid exercise in style and mystification .<jupyter_text>Prediction:<jupyter_code>pred = class_names[predict_fn([text])[0]]
alternative =  class_names[1 - predict_fn([text])[0]]
print('Prediction: %s' % pred)<jupyter_output>Prediction: negative<jupyter_text>Explanation:<jupyter_code>np.random.seed(0)
explanation = explainer.explain(text, threshold=0.95, use_unk=True)<jupyter_output><empty_output><jupyter_text>use_unk=True means we will perturb examples by replacing words with UNKs. Let us now take a look at the anchor. The word 'exercise' basically guarantees a negative prediction.<jupyter_code>print('Anchor: %s' % (' AND '.join(explanation['names'])))
print('Precision: %.2f' % explanation['precision'])
print('\nExamples where anchor applies and model predicts %s:' % pred)
print('\n'.join([x[0] for x in explanation['raw']['examples'][-1]['covered_true']]))
print('\nExamples where anchor applies and model predicts %s:' % alternative)
print('\n'.join([x[0] for x in explanation['raw']['examples'][-1]['covered_false']]))<jupyter_output>Anchor: exercise
Precision: 0.99

Examples where anchor applies and model predicts negative:
a visually UNK UNK UNK opaque and emotionally vapid exercise UNK UNK and mystification UNK
UNK visually UNK UNK narratively opaque UNK emotionally UNK exercise UNK style UNK mystification .
UNK UNK UNK but UNK UNK and emotionally UNK exercise UNK UNK and UNK UNK
UNK UNK flashy but narratively UNK UNK emotionally vapid exercise UNK style and mystification .
UNK UNK UNK UNK narratively opaque and UNK UNK exercise UNK UNK and UNK .
a UNK flashy UNK UNK UNK and emotionally vapid exercise in UNK and mystification UNK
a visually UNK but UNK opaque UNK UNK vapid exercise UNK style and mystification UNK
a visually UNK but UNK opaque UNK UNK vapid exercise UNK UNK and mystification UNK
a UNK flashy but UNK UNK UNK emotionally UNK exercise in UNK and mystification .
a UNK UNK but UNK opaque UNK UNK UNK exercise UNK UNK UNK mystification .

Examples where anchor applies and model predicts positive:
UNK vi[...]<jupyter_text>Changing the perturbation distributionLet's try this with another perturbation distribution, namely one that replaces words by similar words instead of UNKs.Explanation:<jupyter_code>np.random.seed(0)
explanation = explainer.explain(text, threshold=0.95, use_unk=False, sample_proba=0.5)<jupyter_output><empty_output><jupyter_text>The anchor now shows that we need more to guarantee the negative prediction:<jupyter_code>print('Anchor: %s' % (' AND '.join(explanation['names'])))
print('Precision: %.2f' % explanation['precision'])
print('\nExamples where anchor applies and model predicts %s:' % pred)
print('\n'.join([x[0] for x in explanation['raw']['examples'][-1]['covered_true']]))
print('\nExamples where anchor applies and model predicts %s:' % alternative)
print('\n'.join([x[0] for x in explanation['raw']['examples'][-1]['covered_false']]))<jupyter_output>Anchor: flashy AND emotionally
Precision: 0.96

Examples where anchor applies and model predicts negative:
a visually flashy but psychologically preferable and emotionally unintelligible action behind style and laziness .
the anatomically flashy but narratively renderer and emotionally silly exercise in type and madness .
a properly flashy but narratively greenish and emotionally ludicrous blueprint although style and anathema .
a impressively flashy but extraordinarily bright and emotionally unfunny exercise in accent and anathema .
a visually flashy but supremely opaque and emotionally vapid exercise except bossa and empiricism .
a visually flashy but narratively opaque and emotionally vapid exercise in style and mystification .
a visually flashy but similarly readable and emotionally vapid exercise beyond practicality and bravado .
both incredibly flashy but narratively greenish and emotionally vapid meditation than style and falsehood .
every visually flashy but gloriously opaque a[...]<jupyter_text>We can make the token perturbation distribution sample words that are more similar to the ground truth word via the `top_n` argument. Smaller values (default=100) should result in sentences that are more coherent and thus more in the distribution of natural language which could influence the returned anchor. By setting the `use_probability_proba` to True, the sampling distribution for perturbed tokens is proportional to the similarity score between the possible perturbations and the original word. We can also put more weight on similar words via the `temperature` argument. Lower values of `temperature` increase the sampling weight of more similar words. The following example will perturb tokens in the original sentence with probability equal to `sample_proba`. The sampling distribution for the perturbed tokens is proportional to the similarity score between the ground truth word and each of the `top_n` words.<jupyter_code>np.random.seed(0)
explanation = explainer.explain(text, threshold=0.95, use_similarity_proba=True, sample_proba=0.5,
                                use_unk=False, top_n=20, temperature=.2)

print('Anchor: %s' % (' AND '.join(explanation['names'])))
print('Precision: %.2f' % explanation['precision'])
print('\nExamples where anchor applies and model predicts %s:' % pred)
print('\n'.join([x[0] for x in explanation['raw']['examples'][-1]['covered_true']]))
print('\nExamples where anchor applies and model predicts %s:' % alternative)
print('\n'.join([x[0] for x in explanation['raw']['examples'][-1]['covered_false']]))<jupyter_output>Anchor: flashy
Precision: 0.99

Examples where anchor applies and model predicts negative:
this visually flashy but stylistically thin and emotionally vapid exercise into fashion and delusion .
another visually flashy but narratively opaque and intellectually vapid exercise into style and mystification .
another visually flashy but artistically translucent and mentally vapid exercise in style and mystification .
a stylistically flashy but artistically opaque and psychologically vapid exercise in style and selfishness .
a visually flashy but narratively transparent and emotionally vapid exercise to style and mystification .
another functionally flashy but thematically opaque and emotionally vapid exercise in style and delusion .
a artistically flashy but narratively resilient and emotionally classless exercise in retro and delusion .
a concurrently flashy but narratively translucent and emotionally vapid exercise in style and mystification .
a visually flashy but narratively translucent[...]
{'fraction_non_alphanumeric': 0.07784493538974573, 'fraction_numerical': 0.011358899541475615, 'mean_word_length': 6.92485549132948, 'pattern_counts': {'":': 0, '<': 59, '<?xml version=': 0, '>': 59, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '424021', 'license': 'Apache-2.0', 'n_tokens_mistral': 2835, 'n_tokens_neox': 2638, 'n_words': 982, 'path': 'examples/anchor_text_movie.ipynb', 'repo_name': 'axsaucedo/alibi', 'size': '9596'}