<jupyter_start><jupyter_text>Optimizing Zmap for "on-the-fly" text classificationThis notebook is the first of two to accompany our [Few-Shot Text Classification](https://few-shot-text-classification.fastforwardlabs.com/) report in which we dive into the ins and outs of latent text embeddings for text classifications in several limited-labeled-data regimes. One such regime is what we call "on-the-fly" text classification in which no labeled data exists. To classify text with no labeled training data we rely on the inherent meaning of the labels themselves and embed both the label name and the text document with Sentence-BERT. Classification proceeds by assiging the label name whose embedding maximizes the cosine similarity to the text document embedding.  To enhance the classification accuracy of this method, this notebook focuses on how to optimize a mapping (Zmap) between Sentence-BERT (SBERT) embedding space and word2vec (w2v) embedding space since SBERT is optimized to generate representations for *sentences*, while w2v is optimized for *words*.   In the following experiments we use two datasets: AG News and the Reddit dataset, both of which can be found on the [HuggingFace Dataset Repository](link) (CAUTION: the Reddit dataset is *very* large and takes a looooong time to download. In our experiments we have already downsampled the dataset to fit in memory. Details on how we did that can be found in [this](link) notebook.)    **While this notebook _can_ be run from start to finish we do NOT recommend you do so unless you have GPUs. Instead, this notebook serves as additional documentation that a user can read through to develop a better understanding of the analysis.** Notes* This notebook requires the use of the original word2vec embeddings (which take a while to download for the first time)* Because this notebook makes use of a Transformer model, **GPUs are strongly prefered**. * You can access free GPUs by running this notebook with Google Colaboratory, a Jupyter Notebook environment that runs in the browser using Google Cloud. [Open this notebook as a Colab.](https://colab.research.google.com/github/fastforwardlabs/few-shot-text-classification/blob/master/notebooks/Zmap_Experiments.ipynb)   Installation in a Colab environmentIf you followed the installation instructions in the README and are running this notebook locally or in your CML/CDSW environment, you can skip this section. However, if you'd like to take advantage of the free GPUs provided by a Colab environment follow the steps below. 1. Open this notebook with Google's Colaboratory. 2. In the top menu select Runtime --> Change runtime type3. A pop-up menu will appear.  Under Hardware accelerator, select GPUNow follow the installation and data download steps below.<jupyter_code># clone the few-shot repo 
!git clone https://github.com/fastforwardlabs/few-shot-text-classification.git
!mv few-shot-text-classification/fewshot . 
!mv few-shot-text-classification/setup.py .
!mv few-shot-text-classification/data .
!pip3 install datasets==1.1.2
!pip3 install transformers==3.3.0
!pip3 install -e .<jupyter_output><empty_output><jupyter_text>Getting Started<jupyter_code>import os
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt

from fewshot.data.loaders import load_or_cache_data

from fewshot.embeddings.word_embeddings import (
    load_word_vector_model,
    get_topk_w2v_vectors,
    get_word_embeddings
)
from fewshot.embeddings.sentence_embeddings import (
    load_transformer_model_and_tokenizer,
    get_sentence_embeddings,
)

from fewshot.models.on_the_fly import OLS_with_l2_regularization

from fewshot.eval import predict_and_score, simple_topk_accuracy

from fewshot.utils import (
    fewshot_filename,
    torch_load,
    torch_save,
    pickle_load,
    pickle_save,
    to_tensor,
    to_list,
)

import nltk
nltk.download('stopwords')
nltk.download('punkt')<jupyter_output><empty_output><jupyter_text>Load some dataThe datasets below are included in this repo to help get you started more quickly.<jupyter_code># ag news dataset contains `distilbert/sentence_bert` embeddings
news_data = load_or_cache_data("data/agnews", "agnews")
reddit_data = load_or_cache_data("data/reddit", "reddit")<jupyter_output><empty_output><jupyter_text>Load SentenceBERT and W2V embedding modelsExecuting the following two cells for the first time will initiate a series of downloads and preprocessing. The first cell will download a SentenceBERT Transformer model from the HuggingFace repo (if this step hasn't already been done elsewhere).  The second cell will download the original word2vec vectors, a rather large file, and this can take some time. Once completed, all future calls will load these resources from cache first.<jupyter_code>## Load SBERT embedding model
model, tokenizer = load_transformer_model_and_tokenizer("deepset/sentence_bert")
## Load W2V embedding model
w2v_model = load_word_vector_model(small=True) #, cache_dir="data"<jupyter_output><empty_output><jupyter_text>Experiment: How many words should we use to create Zmap? For this experiment we'll use the W2V vocabulary, which includes millions of words. As we'll see, we won't need nearly this many to maximize the contribution of Zmap to our latext text classification task.   Get SBERT and W2V embedding vectors for the topk words in the W2V vocabularyWe'll use the most frequent words as computed over the corpus that was used to train the w2v embeddings. This was a very large corpus so the most frequent words should be those that are used ... well, frequently!  The assumption is that these common words will create an optimal mapping between SBERT space and W2V space.<jupyter_code># NOTE: this cell computes SBERT embeddings for thousands of words which are then cached
#       running this cell without a GPU can take several hours!

# Get w2v embeddings and the vocabulary of most frequent words
w2v_vectors, vocabulary = get_topk_w2v_vectors(w2v_model, k=100000, return_word_list=True)
vocabulary_filename = fewshot_filename("my_data/w2v", "w2v_vocab_sbert_embeddings.pt")

try:
    sbert_vectors = torch_load(vocabulary_filename)['embeddings']
except:
    # get sbert embeddings for each word in vocabulary
    sbert_vectors = get_sentence_embeddings(vocabulary, 
                                            model, 
                                            tokenizer, 
                                            output_filename=vocabulary_filename)<jupyter_output><empty_output><jupyter_text>We'll compute Z map with *up to* 100,000 words, starting with just 100 words and increasing in steps of 100 to 1000 until we cover the range up to 100,000.<jupyter_code>num_words = np.concatenate((np.arange(10**2, 10**3, 100), np.arange(10**3, 10**5+990, 1000)))<jupyter_output><empty_output><jupyter_text>We'll iterate over the `num_words`, selecting the same words from both the sbert_embeddings and w2v_embeddings. These embeddings will be used to compute Z map. The Z map matrix is then used to compute label predictions for a dataset. The resulting score is stored to an array so we can see how the score changes as a function of the number of words.<jupyter_code>def how_many_words(dataset, num_words):
    num_categories = len(dataset.categories)
    scores = []
    for i in num_words:
        # Use Ordinary Least Squares with l2 regularization to compute Zmap 
        # between the sbert embeddings and w2v embeddings for the first i words
        Zmap = OLS_with_l2_regularization(sbert_vectors[:i], to_tensor(w2v_vectors[:i]))
        
        # Our model (Zmap) is then used as a projection before performing cosine similarity
        # the predict_and_score function wraps this process
        score = predict_and_score(dataset, [Zmap])
        # save the score 
        scores.append(score)
    return scores
news_scores = how_many_words(news_data, num_words)
reddit_scores = how_many_words(reddit_data, num_words)
# We've already run this for both agnews and reddit datasets and saved the resulting scores
news_scores = pickle_load(fewshot_filename("my_data", "howmanywords_agnews_scores.pkl"))
reddit_scores = pickle_load(fewshot_filename("my_data","howmanywords_redditcurated_scores.pkl"))

fig = plt.figure(figsize=(9,7))
ax = fig.add_subplot()

ax.plot(num_words, np.array(news_scores)/max(news_scores), lw=2, label='AG News');
ax.plot(num_words, np.array(reddit_scores)/max(reddit_scores), lw=2, label='Reddit posts');

ax.set_yticks([0.35, 1])
ax.set_yticklabels(["worst", "best"])
ax.tick_params(axis='both', labelsize=16)
ax.set_xlabel("number of words used", labelpad=20, fontsize=20)
ax.set_ylabel("classification accuracy", fontsize=20)
ax.set_ylim(.32, 1.03)

plt.legend(loc="right", bbox_to_anchor=(1., 0.3),  frameon=False, fontsize=16);
plt.tight_layout()
plt.savefig("Zmaps_howmanywords.png", dpi=150)<jupyter_output><empty_output><jupyter_text>As we discussed in detail in the accompanying report, we see that both datasets peak around 20,000 words. Using more words to compute Zmap actually results in worse performance.   Experiment: vocabulary from the w2v corpus vs your specific corpus?In the previous experiment we used the most frequent words as measured over the vast corpus that the w2v model was trained on, but that's not the only option. Perhaps it's better to use the most frequent words as measured over the dataset you're trying to classify! The intuition here is that words most common to your dataset might provide a better mapping between SBERT and w2v space. For this we'll first want to set a baseline--in this case, classification performance without using any kind of Zmap.   Baseline: SBERT embeddings onlyA Zmap can enhance classification accuracy by providing better modeling of individual words making it easier to map sentences to their labels in the embedding space. However, we can use SBERT embeddings only and this will provide a solid baseline to compare for model improvement.<jupyter_code>baseline_news_acc, news_preds = predict_and_score(news_data, linear_maps=None, return_predictions=True)
baseline_news_acc_top3 = simple_topk_accuracy(news_data.labels, news_preds)

print(baseline_news_acc, baseline_news_acc_top3)
baseline_reddit_acc, reddit_preds = predict_and_score(reddit_data, linear_maps=None, return_predictions=True)
baseline_reddit_acc_top3 = simple_topk_accuracy(reddit_data.labels, reddit_preds)

print(baseline_reddit_acc, baseline_reddit_acc_top3)<jupyter_output><empty_output><jupyter_text>Zmap: Construct from most frequent words in w2v training corpusWe'll construct a Zmap using the w2v vocabulary consisting of the most frequently used 20,000 words, since we found in the previous section that this provided an optimal Zmap.<jupyter_code>k_words = 20000

# if above cells have already been run, no need to recompute sbert embeddings
try:    
    sbert_vectors_20k = sbert_vectors[:k_words]
    w2v_vectors_20k = to_tensor(w2v_vectors[:k_words])

# otherwise, compute sbert embeddings from the top 20k w2v words
except:
    vocabulary_filename = fewshot_filename("my_data/w2v", "w2v_vocab20k_sbert_embeddings.pt")
    try:
        # Load 20k sbert vectors if we've already created them
        sbert_vectors_20k = torch_load(vocabulary_filename)['embeddings']
    except:
        # get sbert embeddings for each word in vocabulary
        w2v_vectors_20k, vocabulary = get_topk_w2v_vectors(w2v_model, k=k_words, return_word_list=True)
        sbert_vectors_20k = get_sentence_embeddings(vocabulary, 
                                                    model, 
                                                    tokenizer, 
                                                    output_filename=vocabulary_filename)

Zmap_w2v = OLS_with_l2_regularization(sbert_vectors_20k, to_tensor(w2v_vectors[:k_words]))
# We save this Zmap for use in our few-shot application
torch_save(Zmap_w2v, fewshot_filename("data/maps", "Zmap_20k_w2v_words_alpha0.pt"))
zw2v_news_acc, news_preds = predict_and_score(news_data, linear_maps=[Zmap_w2v], return_predictions=True)
zw2v_news_acc_top3 = simple_topk_accuracy(news_data.labels, news_preds)

print(zw2v_news_acc, zw2v_news_acc_top3)
zw2v_reddit_acc, reddit_preds = predict_and_score(reddit_data, linear_maps=[Zmap_w2v], return_predictions=True)
zw2v_reddit_acc_top3 = simple_topk_accuracy(reddit_data.labels, reddit_preds)

print(zw2v_reddit_acc, zw2v_reddit_acc_top3)<jupyter_output>33.52307692307692 60.107692307692304<jupyter_text>Using a Zmap constructed from the most frequent 20k words as measured over the corpus that w2v embeddings were trained on definitely improved performance for the AG news dataset (increasing accuracy by nearly 15 points!), but it made our classification worse for the reddit dataset.word2vec embeddings were trained on 100 billion words of the Google News dataset and the most frequent words are based on the distribution contained with the Google News dataset. It is perhaps not surprising, then, that choosing the most frequent words from a news corpus improves performance on another news corpus. Perhaps using the most frequently used words on Reddit will yield a better choice of Zmap for the reddit dataset.   Side Quest: fine-tuning the regularization parameterTurns out you can increase the performance by about another 5 points depending on the regularization parameter. The drawback, however, is that fine-tuning parameters requires *some*  labeled data in order to properly evaluate between different values. In our experiments we find that values beteen 5 and 12 tend to work best for most datasets.<jupyter_code>for alpha in [0.1, 1, 5, 10, 15, 20]:
    Zmap = OLS_with_l2_regularization(sbert_vectors_20k, to_tensor(w2v_vectors_20k), alpha=alpha)
    news_acc = predict_and_score(news_data, linear_maps=[Zmap], return_predictions=False)
    reddit_acc = predict_and_score(reddit_data, linear_maps=[Zmap], return_predictions=False)
    print(f"alpha: {alpha} \t news_acc: {news_acc} \t reddit_acc: {reddit_acc}")<jupyter_output>alpha: 0.1 	 news_acc: 66.05263157894737 	 reddit_acc: 35.58461538461539
alpha: 1 	 news_acc: 68.38157894736842 	 reddit_acc: 36.69230769230769
alpha: 5 	 news_acc: 71.05263157894737 	 reddit_acc: 37.06153846153846
alpha: 10 	 news_acc: 71.39473684210526 	 reddit_acc: 36.54615384615384
alpha: 15 	 news_acc: 71.01315789473685 	 reddit_acc: 36.16153846153846
alpha: 20 	 news_acc: 70.47368421052632 	 reddit_acc: 36.03846153846153<jupyter_text>Zmap: Construct from most frequent words in a specific corpus (e.g., Reddit)Next, we'll construct a Zmap using the most frequent words as measured over the Reddit dataset. The most common words in the Reddit dataset was explored in [this notebook](https://github.com/fastforwardlabs/few-shot-text-classification/blob/master/notebooks/CreatingRedditDataset.ipynb). **This file is not included in the repo so the cells in this section will not run.**However, we leave these cells in to provide a walk-through of how to perform the analysis.<jupyter_code>most_common_reddit_words = pickle_load(fewshot_filename("my_data/reddit", "most_common_words.pkl"))
# not all words have embeddings in the canonical version of w2v
# identify top 20k words in the reddit word list that also have w2v embeddings

w2v_vectors_redditwords, words_not_found = get_word_embeddings(
    most_common_reddit_words['no_punc_no_stop'], 
    w2v_model)

#select the top20k w2v embeddings (corresponding to the top 20k words in common)
w2v_vectors_redditwords = w2v_vectors_redditwords[:20000]

# identify the words in common
words_not_found = set(words_not_found)
common_words = []
for word in most_common_reddit_words['no_punc_no_stop']:
    if word not in words_not_found:
        common_words.append(word)

top20k_common_words = common_words[:20000]
vocabulary_filename = fewshot_filename("my_data/reddit", "reddit_vocab20k_sbert_embeddings.pt")

try:
    sbert_vectors_redditwords = torch_load(vocabulary_filename)['embeddings']
except: 
    # compute sbert vectors for the top20_common_words
    sbert_vectors_redditwords = get_sentence_embeddings(top20k_common_words, 
                                                        model, 
                                                        tokenizer, 
                                                        output_filename=vocabulary_filename)
Zmap_redditwords = OLS_with_l2_regularization(sbert_vectors_redditwords, 
                                              to_tensor(w2v_vectors_redditwords))

zred_reddit_acc, reddit_preds = predict_and_score(reddit_data, linear_maps=[Zmap_redditwords], 
                                                  return_predictions=True)
zred_reddit_acc_top3 = simple_topk_accuracy(reddit_data.labels, reddit_preds)

print(zred_reddit_acc, zred_reddit_acc_top3)<jupyter_output>25.6 59.315384615384616<jupyter_text>Unfortunately, constructing Zmap from a vocabulary of the most frequent words in the Reddit dataset doesn't help; in fact, it hurts performance even more! On the bright side, even a small amount of regularization will improve the performance significantly (even if it doesn't outperform using SBERT embeddings alone).  Below, we see that even having `alpha=1` increases the score by 7 points.<jupyter_code>def fine_tune_alpha(alpha_values, dataset, sbert_vectors, w2v_vectors, verbose=True):

    for alpha in alpha_values:
        Zmap = OLS_with_l2_regularization(sbert_vectors,to_tensor(w2v_vectors), alpha=alpha)
        acc = predict_and_score(dataset, linear_maps=[Zmap], return_predictions=False)
        if verbose:
            print(f"alpha: {alpha} \t score: {acc}")
alpha = [0, 0.1, 1, 5, 10, 15]
fine_tune_alpha(alpha, reddit_data, sbert_vectors_redditwords, w2v_vectors_redditwords)<jupyter_output>alpha: 0 	 score: 25.6
alpha: 0.1 	 score: 29.861538461538462
alpha: 1 	 score: 32.684615384615384
alpha: 5 	 score: 34.19230769230769
alpha: 10 	 score: 34.43846153846154
alpha: 15 	 score: 34.51538461538461<jupyter_text>Summary figureThe figure below summarizes our findings so far. 1. SBERT embeddings alone can be a quick and intuitive way to compare the similarity of a block of text to a set of possible labels2. Learning a mapping between SBERT embedding space and w2v embeddings space yields a projection matrix that, once applied to the SBERT embeddings, can often increase the classification accuracy of this simpmle method. 3. These mappings don't always work, however, as in the case of the Reddit dataset, where overall performance decreased.<jupyter_code>plt.style.use("seaborn-white")
cldr_colors = ['#00b6b5', '#f7955b','#6c8cc7']

fig = plt.figure(figsize=(8,6))
width = 0.25
plt.bar(np.array([0, 1])-width/2, 
        [baseline_news_acc, baseline_reddit_acc], 
        width=width, 
        color=cldr_colors[0],  
        label='SBERT'
        )
plt.bar(np.array([0, 1])+width/2, 
        [zw2v_news_acc, zw2v_reddit_acc], 
        width=width, 
        color=cldr_colors[1], 
        label='SBERT*Zmap'
        )
plt.xticks([0,1], ['AG News', 'Reddit'], fontsize=16)#
plt.yticks(fontsize=16)
plt.ylabel("Accuracy", fontsize=20);
plt.legend(frameon=False, fontsize=16);
plt.tight_layout()
plt.savefig("accuracy_agnews_reddit.png", transparent=True, dpi=150)<jupyter_output><empty_output><jupyter_text>Error analysisLet's take a closer look at how these datasets are composed to see if we can understand why AG News performs well under a Zmap transformation while Reddit performs worse!   Number of labelsThe AG News dataset has only four categories, while our subset of Reddit contains posts from 10 different subreddits. This makes the Reddit dataset more difficult from the outset.<jupyter_code>news_data.categories
reddit_data.categories<jupyter_output><empty_output><jupyter_text>More importantly, we can see that the category names are different.  The latent text embedding method relies heavily on the semantic meaning of category names so perhaps subreddit names are more difficult to predict on than AG News category names. Below we explore the classification predictions for each subreddit using SBERT embeddings alone, and after applying the two Zmaps we learned: the one using w2v's vocabulary, and the other using Reddit's vocabulary.<jupyter_code>acc, preds = predict_and_score(reddit_data, linear_maps=None, return_predictions=True)
acc, preds_zw2v = predict_and_score(reddit_data, linear_maps=[Zmap_w2v], return_predictions=True)
acc, preds_zred = predict_and_score(reddit_data, linear_maps=[Zmap_redditwords], return_predictions=True)
# create lists of label predictions from the Prediction object returned by predict_and_score
best = [p.best for p in preds]
best_zw2v = [p.best for p in preds_zw2v]
best_zred = [p.best for p in preds_zred]
# count how many examples were assigned a given label
thing, best_counts = np.unique(best, return_counts=True)
_, best_zw2v_counts = np.unique(best_zw2v, return_counts=True)
_, best_zred_counts = np.unique(best_zred, return_counts=True)
plt.style.use("seaborn-white")

fig = plt.figure(figsize=(9,7))
ax = fig.add_subplot(111)

width = .25
ax.barh(np.array(thing)-width,
         best_counts,  
         height=width,
         color=cldr_colors[0],
         label='SBERT'
        )
ax.barh(np.array(thing),
         best_zw2v_counts, 
         height=width, 
         color=cldr_colors[1],
         label='SBERT*Zmap (w2v words)'
         )
ax.barh(np.array(thing)+width,
         best_zred_counts,
         height=width, 
         color=cldr_colors[2],
         label='SBERT*Zmap (Reddit words)'
         )
ax.set_yticks(thing)
ax.set_yticklabels(reddit_data.categories)
ax.invert_yaxis()
ax.tick_params(axis="both", labelsize=16)
ax.set_xlabel("Examples", fontsize=20);
ax.legend(frameon=False, fontsize=16);
ax.set_title("[Reddit] Number of predictions in each subreddit", fontsize=20)
plt.tight_layout()
plt.savefig("reddit_bycategory.png", transparent=True, dpi=150)<jupyter_output><empty_output><jupyter_text>The Reddit dataset contains 1300 examples in each subreddit. If our classification were perfect, each of these bars would fall at exactly 1300 -- but they obviously do not. We see that when we apply various Zmaps to the SBERT embeddings, more and more examples are predicted to fall into the `funny` subreddit. The shift is most noticeable when using a Zmap that has been constructed with words from Reddit's own vocabulary. This makes sense because "funny" is a very general and subjective word and humans love humor. We speculate that, rather than being a flaw in the method itself, what we see here is a consequence of over-generalized subreddit names. *Anything* can be funny, and as we construct better Zmaps, we only serve to accentuate this trend.<jupyter_code>news_acc, news_preds = predict_and_score(news_data, linear_maps=None, return_predictions=True)

news_acc, news_preds_zw2v = predict_and_score(news_data, linear_maps=[Zmap_w2v], return_predictions=True)

news_best = [p.best for p in news_preds]
news_best_zw2v = [p.best for p in news_preds_zw2v]

other_thing, news_best_counts = np.unique(news_best, return_counts=True)
_, news_best_zw2v_counts = np.unique(news_best_zw2v, return_counts=True)
plt.style.use("seaborn-white")

fig = plt.figure(figsize=(8,6))
ax = fig.add_subplot(111)

width = .25
ax.barh(np.array(other_thing)-width/2, 
        news_best_counts, 
        height=width, 
        color=cldr_colors[0], 
        label='SBERT'
        )
ax.barh(np.array(other_thing)+width/2, 
        news_best_zw2v_counts, 
        height=width, 
        color=cldr_colors[1], 
        label='SBERT*Zmap'
        )

ax.set_yticks(other_thing)
ax.set_yticklabels(news_data.categories)
ax.invert_yaxis()
ax.tick_params(axis='both', labelsize=16)
ax.set_xlabel("Examples", fontsize=20)
ax.set_xlim(0, 3800)
ax.set_title("[AG News] Number of predictions in each news category", fontsize=20)
ax.legend(frameon=False, fontsize=16);
plt.tight_layout()
plt.savefig("agnews_bycategory.png", transparent=True, dpi=150)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.07127078731296628, 'fraction_numerical': 0.03059225607468845, 'mean_word_length': 4.781686746987952, 'pattern_counts': {'":': 0, '<': 61, '<?xml version=': 0, '>': 62, 'https://': 4, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 30, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '16', 'dir': 'jupyter-structured-clean-dedup', 'id': '378164', 'license': 'Apache-2.0', 'n_tokens_mistral': 7751, 'n_tokens_neox': 7154, 'n_words': 2395, 'path': 'notebooks/Zmap_Experiments.ipynb', 'repo_name': 'amenityllc/few-shot-text-classification', 'size': '23993'}