<jupyter_start><jupyter_text>1. Prepare IMDB data  1.1 Load, filter, and sample data Load Yelp data. Dataset can be downloaded [here](https://www.yelp.com/dataset/download). After entering your contact information, select "Download JSON."<jupyter_code>%%time

PATH = "../data/yelp_dataset/"
yelp_data = []
r_dtypes = {"review_id":str,
            "user_id":str,
            "business_id":str,
            "stars": np.int32, 
            "date":str,
            "text":str,
            "useful": np.int32, 
            "funny": np.int32,
            "cool": np.int32}
drop = ['review_id', 'user_id', 'useful', 'funny', 'cool']
query = "date >= '2017-12-01' and (stars==1 or stars ==5)"

with open(f"{PATH}yelp_academic_dataset_review.json", "r") as f:
    reader = pd.read_json(f, orient="records", lines=True, dtype=r_dtypes, chunksize=1000)
    for chunk in reader:
        reduced_chunk = chunk.drop(columns=drop).query(query)
        yelp_data.append(reduced_chunk)
    
yelp_data = pd.concat(yelp_data, ignore_index=True)

yelp_data.shape<jupyter_output><empty_output><jupyter_text>Merge reviews and metadata on businesses.<jupyter_code>yelp_business = pd.read_json(f"{PATH}yelp_academic_dataset_business.json", orient="records", lines=True)
yelp_business.shape<jupyter_output><empty_output><jupyter_text>Select businesses in Portland and Atlanta only.<jupyter_code>yelp_business = yelp_business[yelp_business.city.isin(["Portland", "Atlanta"])]
yelp_business.shape<jupyter_output><empty_output><jupyter_text>Merge review and business data.<jupyter_code>yelp_merged = yelp_data.merge(yelp_business, on='business_id')
yelp_merged.shape
yelp_merged = yelp_merged.rename(columns={"stars_x":"stars"})<jupyter_output><empty_output><jupyter_text>Split string category column into list.<jupyter_code>categories = [c.split(",") for c in yelp_merged.categories if c is not None]
c_count = Counter(c.strip() for c_list in categories for c in set(c_list))
c_count.most_common()[:30]
yelp_categorized = yelp_merged.assign(categories=yelp_merged['categories'].str.split(',')).explode('categories')
yelp_categorized.categories = yelp_categorized.categories.str.strip()
yelp_categorized.shape<jupyter_output><empty_output><jupyter_text>Filter data on category for Restaurants and 'Health & Medical'.<jupyter_code>yelp_filtered = yelp_categorized[yelp_categorized.categories.isin(['Restaurants','Health & Medical'])]
yelp_filtered.shape<jupyter_output><empty_output><jupyter_text>Sample data, so that we get 1000 of each category and of 1 or 5 stars.<jupyter_code>s_size = 1000
rs = 42
sample = yelp_filtered.groupby(["stars", "categories"]).sample(n=s_size, random_state=rs).reset_index()
sample.shape
stopword_list=stopwords.words('english')<jupyter_output><empty_output><jupyter_text>1.2 Phrase data, remove special formatting and stopwords Initialize tokenizer.<jupyter_code>tokenizer = ToktokTokenizer()<jupyter_output><empty_output><jupyter_text>Remove stopwords before denoising, lemmatizing and removing special characters.<jupyter_code>%%time 

sample['review_clean']= [remove_stopwords(r, stopword_list, tokenizer) for r in sample['text']]<jupyter_output>CPU times: user 1.53 s, sys: 10.9 ms, total: 1.54 s
Wall time: 1.55 s<jupyter_text>Denoise, remove special characters, lemmatize.<jupyter_code>%%time

sample['review_clean']=sample['review_clean'].apply(denoise_text)
sample['review_clean']=sample['review_clean'].apply(remove_special_characters)
sample['review_clean']=sample['review_clean'].apply(simple_lemmatizer)<jupyter_output>CPU times: user 2.65 s, sys: 74.7 ms, total: 2.73 s
Wall time: 2.76 s<jupyter_text>Remove stopwords again, after other preprocessing.<jupyter_code>%%time 

sample['review_clean']= [remove_stopwords(r, stopword_list, tokenizer) for r in sample['review_clean']]<jupyter_output>CPU times: user 888 ms, sys: 5.44 ms, total: 894 ms
Wall time: 902 ms<jupyter_text>Find phrases. Load phrases found in data during finetuning. This will require that you have a folder called 'embeddings' in the root of the 'paper' folder with files that can be found here. Alternatively, you can finetune and phrase vectors yourself, using the 'FinetuneYelp.py' script.<jupyter_code>PHRASING = True
MIN = 500
THRESHOLD = 200
%%time

if PHRASING:
    sample['review_clean']= get_phrases([tokenizer.tokenize(i) for i in sample['review_clean']],
                                        min_count = MIN, 
                                        threshold = THRESHOLD,
                                        save=False,
                                        load=True,
                                        PATH="../embeddings/")<jupyter_output>Finding bigrams in data.
Finding trigrams in data.
CPU times: user 17.9 s, sys: 4.69 s, total: 22.6 s
Wall time: 23.3 s<jupyter_text>Data _before_ preprocessing and phrasing.<jupyter_code>sample['text'][6]<jupyter_output><empty_output><jupyter_text>Data _after_ preprocessing and phrasing.<jupyter_code>#sample['review_clean'][0]
" ".join(sample['review_clean'][6])
sample["sentiment"] = ['positive' if s == 5 else 'negative' for s in sample['stars']]
sample[["sentiment", "stars"]]
sample.head()
sample_sorted = sample.sort_values(["sentiment", "categories"]).reset_index()
sample_sorted<jupyter_output><empty_output><jupyter_text>1.3 Separate pos and neg reviews<jupyter_code>pos = sample_sorted[sample_sorted.sentiment == "positive"].reset_index(drop=True)
neg = sample_sorted[sample_sorted.sentiment == "negative"].reset_index(drop=True)
pos = pos.review_clean.tolist()
neg = neg.review_clean.tolist()<jupyter_output><empty_output><jupyter_text>2. WMD  2.1 Tokenize data Separate data into positive and negative reviews and tokenize each review.<jupyter_code>pos_tok = list(map(lambda x: tokenize(x, tokenizer), pos))
neg_tok = list(map(lambda x: tokenize(x, tokenizer), neg))
pos_sample = [" ".join(doc) for doc in pos_tok]
neg_sample = [" ".join(doc) for doc in neg_tok]
print(len(pos_sample))
print(len(neg_sample))<jupyter_output>2000
2000<jupyter_text>2.2 Load pretrained Google News W2V model Load word vectors. You can use vectors finetuned for the paper (found here), pretrained vectors from Google (found here), or finetune vectors yourself using the 'FineTuneYelp.py' script.<jupyter_code>finetuned = True
if not finetuned:
    print("Loading GoogleNews Vectors")
    %time model = KeyedVectors.load_word2vec_format('../embeddings/GoogleNews-vectors-negative300.bin.gz', binary=True)
else:
    print("Loading GoogleNews Vectors finetuned using Yelp review data.")
    %time model = KeyedVectors.load_word2vec_format('../embeddings/yelp_w2v.txt', binary=False)
model.distance("taco", "burrito")
model.distance("quesadilla","tostada")
model.distance("sushi", "sashimi")<jupyter_output><empty_output><jupyter_text>2.3 Load corpus and remove OOV words Form a corpus for running the TfIdf vectorizer with l1 normalization.<jupyter_code>%%time

corpus = pos_sample + neg_sample
vectorizer = TfidfVectorizer(use_idf=True, tokenizer=tfidf_tokenize, norm='l1')
vectorizer.fit(corpus)<jupyter_output>/Users/mikaelbrunila/Documents/Code/wmdecompose/venv-wmdecomp/lib/python3.8/site-packages/sklearn/feature_extraction/text.py:489: UserWarning: The parameter 'token_pattern' will not be used since 'tokenizer' is not None'
  warnings.warn("The parameter 'token_pattern' will not be used"<jupyter_text>Find lingering out-of-vocabulary words.<jupyter_code>%time oov = [word for word in vectorizer.get_feature_names() if word not in model.key_to_index.keys()]
len(oov)
print(oov[:50])<jupyter_output>["'", ',', '\\\\___', '_community_', 'aahhhhhhhhmazing', 'aboutlastnight', 'ackwkowlegement', 'acutaly', 'addictivethere', 'adonika', 'afterwardscould', 'againmethinks', 'agosta', 'airopro', 'airoproindigopro', 'akosua', 'aksanaiva', 'albertathe', 'alcoholptsd', 'aligeti', 'alliera', 'almostttt', 'amberbeautyatl', 'amberrose', 'analysishistory', 'andtheh', 'anethistician', 'anplaces', 'anshante', 'antgolotti', 'antidentite', 'antiiconic', 'anxietydepression', 'apppts', 'areacentral', 'arnoud', 'aromantic', 'aroundtheclockcare', 'athina', 'athletetherapist', 'atlantaedgewood', 'atrovent', 'autocharge', 'availity', 'babycare', 'baedner', 'bajraktarevi', 'barnese', 'battaglia', 'bazoria']<jupyter_text>Remove oov words.<jupyter_code>%time pos_sample = list(map(lambda x: remove_oov(x, tokenizer, oov), pos_sample[:2000]))
%time neg_sample = list(map(lambda x: remove_oov(x, tokenizer, oov), neg_sample[:2000]))
pos_sample[5]<jupyter_output><empty_output><jupyter_text>Now transform the corpus into bag-of-words.<jupyter_code>%time
pos_nbow = vectorizer.transform(pos_sample)
neg_nbow = vectorizer.transform(negT_sample)<jupyter_output>CPU times: user 2 µs, sys: 0 ns, total: 2 µs
Wall time: 5.01 µs<jupyter_text>Tokenize samples.<jupyter_code>pos_tok = list(map(lambda x: tokenize(x, tokenizer), pos_sample))
neg_tok =list(map(lambda x: tokenize(x, tokenizer), neg_sample))
print(pos_tok[5][:20])
%time oov_ = [word for word in vectorizer.get_feature_names() if word not in model.key_to_index.keys()]
len(oov_)<jupyter_output><empty_output><jupyter_text>2.4 Get features and embeddings<jupyter_code>features = vectorizer.get_feature_names()
word2idx = {word: idx for idx, word in enumerate(vectorizer.get_feature_names())}
idx2word = {idx: word for idx, word in enumerate(vectorizer.get_feature_names())}<jupyter_output><empty_output><jupyter_text>Get the embedding matrix "E" for all features.<jupyter_code>E = model[features]<jupyter_output><empty_output><jupyter_text>2.5 Cluster In order to make the results of the WMD model more interpretable, we add the option to inspect the output not only by individual words, but also by *word clusters*. We do this by clustering the input words with two different algorithmsand assigning each word to a cluster.  2.5.1 Kmeans First, we select the number of clusters we want to search to determine the optimal size of K.<jupyter_code>%%time

K = range(10,210, 10)<jupyter_output>CPU times: user 5 µs, sys: 1e+03 ns, total: 6 µs
Wall time: 8.11 µs<jupyter_text>Now, let's search through the different Ks to see what the optimal K is using the [silhouette score](https://towardsdatascience.com/silhouette-coefficient-validating-clustering-techniques-e976bb81d10c). The silhouette score can take on values between -1 and 1, with values closer to 1 indicating nicely separated clusters. We are also getting the sum of squared differences in order to plot the so called [elbow plot](https://blog.cambridgespark.com/how-to-determine-the-optimal-number-of-clusters-for-k-means-clustering-14f27070048f).<jupyter_code>%%time

wvec_ssd, wvec_silhouette = kmeans_search(E, K)<jupyter_output><empty_output><jupyter_text>Plotting the elbow plot. A good heuritic for K is if there are any clear and "sharp" corners in the plot.<jupyter_code>plot_kmeans(K,wvec_ssd,"elbow")<jupyter_output><empty_output><jupyter_text>Plotting the silhouette score, with values closer to 1 indicating nicely separated clusters.<jupyter_code>plot_kmeans(K,wvec_silhouette,"silhouette")<jupyter_output><empty_output><jupyter_text>2.5.2 T-SNE + Kmeans As we saw above, using Kmeans on the raw embeddings did not produce very satisfactory results. To get clusters that are better separated, we will try to first reduce the vectors into two dimensions using the t-SNE algorithm, and then running Kmeans on these reduced embeddings.<jupyter_code>method='barnes_hut'
n_components = 2
verbose = 1
E_tsne = TSNE(n_components=n_components, method=method, verbose=verbose).fit_transform(E)<jupyter_output>[t-SNE] Computing 91 nearest neighbors...
[t-SNE] Indexed 19498 samples in 0.004s...
[t-SNE] Computed neighbors for 19498 samples in 8.943s...
[t-SNE] Computed conditional probabilities for sample 1000 / 19498
[t-SNE] Computed conditional probabilities for sample 2000 / 19498
[t-SNE] Computed conditional probabilities for sample 3000 / 19498
[t-SNE] Computed conditional probabilities for sample 4000 / 19498
[t-SNE] Computed conditional probabilities for sample 5000 / 19498
[t-SNE] Computed conditional probabilities for sample 6000 / 19498
[t-SNE] Computed conditional probabilities for sample 7000 / 19498
[t-SNE] Computed conditional probabilities for sample 8000 / 19498
[t-SNE] Computed conditional probabilities for sample 9000 / 19498
[t-SNE] Computed conditional probabilities for sample 10000 / 19498
[t-SNE] Computed conditional probabilities for sample 11000 / 19498
[t-SNE] Computed conditional probabilities for sample 12000 / 19498
[t-SNE] Computed conditional probabilities for sam[...]<jupyter_text>A scatterplot of the reduced vectors indicates that there are indeed quite clear (albeit very many) clusters in the data.<jupyter_code>plt.scatter(E_tsne[:, 0], E_tsne[:, 1], s=1);<jupyter_output><empty_output><jupyter_text>Run the same diagnostics as above, using the silhouette score and the elbow plot.<jupyter_code>%%time

tsne_ssd, tsne_silhouette = kmeans_search(E_tsne, K)<jupyter_output>For n_clusters = 10 The average silhouette_score is : 0.37478644
For n_clusters = 20 The average silhouette_score is : 0.36666763
For n_clusters = 30 The average silhouette_score is : 0.36414522
For n_clusters = 40 The average silhouette_score is : 0.35804102
For n_clusters = 50 The average silhouette_score is : 0.36096376
For n_clusters = 60 The average silhouette_score is : 0.3575868
For n_clusters = 70 The average silhouette_score is : 0.35891905
For n_clusters = 80 The average silhouette_score is : 0.36087665
For n_clusters = 90 The average silhouette_score is : 0.36161482
For n_clusters = 100 The average silhouette_score is : 0.3613771
For n_clusters = 110 The average silhouette_score is : 0.36354688
For n_clusters = 120 The average silhouette_score is : 0.36354893
For n_clusters = 130 The average silhouette_score is : 0.36461017
For n_clusters = 140 The average silhouette_score is : 0.365778
For n_clusters = 150 The average silhouette_score is : 0.36173385
For n_clusters = 160 Th[...]<jupyter_text>Elbow plot is much clearer than above, with a break at 20 or 25, or alternatively closer to 100.<jupyter_code>plot_kmeans(K,tsne_ssd,"elbow")<jupyter_output><empty_output><jupyter_text>The silhouette score indicates that results are better than above, but the differences in values are quite nominal.<jupyter_code>plot_kmeans(K,tsne_silhouette,"silhouette")<jupyter_output><empty_output><jupyter_text>2.5.3 PCA + Kmeans While t-SNE gave us promising diagnostics, another approach would be to use dimensionality reduction with PCA. However, as can be seen inspecting the diagnostics below, PCA reduced vectors are actually much less clearly separated than the t-SNE reduced vectors.<jupyter_code>n_components = 0.9
verbose = 1
pca_fit = PCA(n_components = n_components).fit(E)
print(len(pca_fit.explained_variance_ratio_))
print(pca_fit.explained_variance_ratio_)
print(np.sum(pca_fit.explained_variance_ratio_))
E_pca = pca_fit.transform(E)
plt.scatter(E_pca[:, 0], E_pca[:, 1], s=1);
%%time

pca_ssd, pca_silhouette = kmeans_search(E_pca, K)
plot_kmeans(K,pca_ssd,"elbow")
plot_kmeans(K,pca_silhouette,"silhouette")<jupyter_output><empty_output><jupyter_text>2.5.4 UMAP + Kmeans Finally, we also try the increasingly popular UMAP dimensionality reduction technique. The results are comparible to the t-SNE reduced vectors, but because t-SNE is better known, we stick with that algorithm. However, UMAP could be a good alternative to explore in more detail in the future.<jupyter_code>%%time

metric = 'cosine'
dm = cdist(E, E, metric)
np.fill_diagonal(dm, 0)
is_valid_dm(dm)
mean, std = np.mean(dm), np.std(dm)
print(mean, std)
min_dist=mean - 2*std
n_neighbors = int(0.001*len(E))
n_components=2
print(f"Min distance: {min_dist}")
print(f"N. neighbors: {n_neighbors}")
print(f"N. compontents: {n_components}")
%%time

E_umap = umap.UMAP(
    n_neighbors=n_neighbors,
    min_dist=min_dist,
    n_components=n_components,
    random_state=42,
    verbose=verbose
).fit_transform(E)
plt.scatter(E_umap[:, 0], E_umap[:, 1], s=1);
%%time

umap_ssd, umap_silhouette = kmeans_search(E_umap, K)
plot_kmeans(K,umap_ssd,"elbow")
plot_kmeans(K,umap_silhouette,"silhouette")<jupyter_output><empty_output><jupyter_text>2.5.5 Choose clustering model Becaus silhouette scores were so even across K and because the elbow plots were ultimately quite ambigious, we use 100 clusters as a simple heuristic.<jupyter_code>k_base = 100
k_pca = k_base
k_tsne = k_base
k_umap = k_base
%%time

km_base = cluster.KMeans(n_clusters=k_base,max_iter=300).fit(E)
labels = km_base.labels_
centroids = km_base.cluster_centers_

km_pca = cluster.KMeans(n_clusters=k_pca,max_iter=300).fit(E_pca)
labels_pca = km_pca.labels_

km_tsne = cluster.KMeans(n_clusters=k_tsne,max_iter=300).fit(E_tsne)
labels_tsne = km_tsne.labels_

km_umap = cluster.KMeans(n_clusters=k_umap,max_iter=300).fit(E_umap)
labels_umap=km_umap.labels_<jupyter_output><empty_output><jupyter_text>Create an index that maps each word to a cluster.<jupyter_code>word2cluster = {features[idx]: cl for idx, cl in enumerate(labels)}
print(take(10, word2cluster.items()))<jupyter_output>[('[', 9), (']', 93), ('^', 9), ('_____', 9), ('aa_bowl', 5), ('aaaaaand', 85), ('aaron', 21), ('aarp', 10), ('abandoned', 27), ('abandoning', 19)]<jupyter_text>Now, conversely, create an index that maps each cluster to a word.<jupyter_code>cluster2words = defaultdict(list)
for key, value in word2cluster.items():
    cluster2words[value].append(key)
print(len(cluster2words[0]))
print(cluster2words[0][:100])
print(len(cluster2words[1]))
print(cluster2words[1][:100])<jupyter_output>59
['ahi', 'albacore', 'bento', 'bento_box', 'california_roll', 'caterpillar_roll', 'conveyor_belt_sushi', 'edamame', 'eel', 'eel_sauce', 'fish', 'gyoza', 'handrolls', 'hibachi', 'japanese', 'karaage', 'katsu', 'miso', 'miso_soup', 'monkey_brain', 'nigiri', 'nigiri_sashimi', 'omakase', 'onigiri', 'poke', 'poke_bowl', 'rainbow_roll', 'ramen', 'raw_fish', 'roe', 'roll', 'sake', 'sake_nigiri', 'salmon_roe', 'salmon_sashimi', 'sashimi', 'scallop_nigiri', 'seaweed_salad', 'shrimp_tempura', 'soft_shell', 'spicy_ahi', 'spicy_mayo', 'spicy_tuna', 'sushi', 'sushi_chef', 'sushi_roll', 'tempura', 'teriyaki', 'tomo', 'toro_fatty', 'tuna', 'tuna_nigiri', 'udon', 'unagi', 'uni', 'wasabi', 'yakitori', 'yellowtail', 'yuki']<jupyter_text>2.6 Initialize documents Transform all reviews into "documents", each with a set of weights per word in the corpus ("nbow"), the sum of these weights ("weights_sum"), the indeces of the words in the documents ("idxs") and the word vectors corresponding to each word ("vecs").<jupyter_code>pos_nbow[0]
%%time 

pos_docs, neg_docs = [], []

for idx, doc in enumerate(pos_tok):
    pos_docs.append(Document(doc, pos_nbow[idx], word2idx, E))
    
for idx, doc in enumerate(neg_tok):
    neg_docs.append(Document(doc, neg_nbow[idx], word2idx, E))
pos_docs[0].nbow
pos_docs[0].weights_sum
pos_docs[0].idxs[:10]
pos_docs[0].vecs[:1][0][:10]<jupyter_output><empty_output><jupyter_text>2.7 Linear-Complexity Relaxed WMD (LC-RWMD) Run the [Linear-Complexity Relaxed WMD](https://arxiv.org/abs/1711.07227) to get the distances between all positive and all negative reviews.<jupyter_code>metric = "cosine"
lc_rwmd = LC_RWMD(pos_docs, neg_docs,pos_nbow,neg_nbow,E)
%time lc_rwmd.get_D(metric=metric)<jupyter_output>CPU times: user 26min 22s, sys: 11min 28s, total: 37min 50s
Wall time: 10min 39s<jupyter_text>2.8 Gale-Shapeley Pairing Use the [Gale-Shapeley matching algorithm](https://en.wikipedia.org/wiki/Gale%E2%80%93Shapley_algorithm) to find the optimal pairs between positive and negative reviews. This iterates over all the reviews and finds the set of matches that pairs each review with its optimal match given that all positive reviews have to be matched with a negative review and vice versa. The output is a dictionary of key-value pairs, where each pair represents an optimal match.Alternatively, you can run full pairs or random pairs.<jupyter_code># Options: 'gale_shapeley','random','full'

pairing = 'gale_shapeley'
%%time

if pairing == 'gale_shapeley':
    print("Running Gale-Shapeley pairing.")
    matcher = Matcher(lc_rwmd.D)
    engaged = matcher.matchmaker()
    print(f"Pairing is stable: {matcher.check()}")
    pairs = [(k, v) for k, v in engaged.items()]
if pairing == 'random':
    print("Running random pairing.")
    pos_idx = list(range(0,len(pos_docs)))
    neg_idx = list(range(0,len(neg_docs)))
    shuffle(pos_idx)
    shuffle(neg_idx)
    pairs = list(zip(pos_idx, neg_idx))
if pairing == 'full':
    print("Running full pairing.")
    pos_idx = list(range(0,len(pos_docs)))
    neg_idx = list(range(0,len(neg_docs)))
    pairs = [(i,j) for i in pos_idx for j in neg_idx]<jupyter_output>Running Gale-Shapeley pairing.
Pairing is stable: True
CPU times: user 18.2 s, sys: 1.56 s, total: 19.7 s
Wall time: 21.5 s<jupyter_text>Let's look at the output of our pairing (by default Gale-Shapeley, see above):<jupyter_code>print(pairs[:20])
example_pairs = [(' '.join(word for word in pos_docs[p[0]].words), 
                  ' '.join(word for word in neg_docs[p[1]].words))
                 for p in pairs]
example_pairs[1][0]
example_pairs[1][1]
len(pairs)<jupyter_output><empty_output><jupyter_text>2.9 Pairwise WMD with Baseline Kmeans Calculate the pairwise distances between the documents selected by the Galey-Shapeley algorithm _without_ returning the flow between individual words.<jupyter_code>%%time 

wmd_pairs = WMDPairs(pos_docs,neg_docs,pairs,E,idx2word, metric=metric)
wmd_pairs.get_distances(decompose=False,thread=False,relax=True)<jupyter_output>Calculated distances between approximately 0 documents.00h00m00s elapsed.
Calculated distances between approximately 1000 documents.00h00m13s elapsed.
CPU times: user 17.3 s, sys: 7.94 s, total: 25.3 s
Wall time: 8.09 s<jupyter_text>The return value is a matrix of distances between the document pairs.<jupyter_code>np.max(wmd_pairs.distances)
wmd_dists = np.concatenate(wmd_pairs.distances)
wmd_dists[wmd_dists != 0].mean()<jupyter_output><empty_output><jupyter_text>Calculate the pairwise distances between the documents selected by the Galey-Shapeley algorithm, this time also returning the flow between individual words.<jupyter_code>%%time 

wmd_pairs_flow = WMDPairs(pos_docs,neg_docs,pairs,E,idx2word, metric=metric)
wmd_pairs_flow.get_distances(decompose = True, 
                             sum_clusters = True, 
                             w2c = word2cluster, 
                             c2w = cluster2words,
                             thread = False,
                             relax = True)<jupyter_output>Calculated distances between approximately 0 documents.00h00m00s elapsed.
Calculated distances between approximately 1000 documents.00h00m17s elapsed.
CPU times: user 24.6 s, sys: 9.68 s, total: 34.3 s
Wall time: 10.5 s<jupyter_text>Now we have three return values.The first one is again a matrix of distances between the document pairs.<jupyter_code>np.max(wmd_pairs_flow.distances)
wmd_flow_dists = np.concatenate(wmd_pairs_flow.distances)
wmd_flow_dists[wmd_flow_dists != 0].mean()<jupyter_output><empty_output><jupyter_text>The second return value is a list of tuples with all the words that contributed the most to the distance from the positive documents to the negative ones. These are _not_ sorted from high to low or vice versa.<jupyter_code>take(10, wmd_pairs_flow.wd_source.items())<jupyter_output><empty_output><jupyter_text>The third return value is a list of tuples with all the words that contributed the most to the distance from the negative documents to the positive ones. Again, these are _not_ sorted from high to low or vice versa.<jupyter_code>take(10, wmd_pairs_flow.wd_sink.items())
take(10, wmd_pairs_flow.cd_source.items())
take(10, wmd_pairs_flow.cd_sink.items())
{k: v for k, v in sorted(wmd_pairs_flow.cd_source.items(), key=lambda item: item[1], reverse=True)[:10]}<jupyter_output><empty_output><jupyter_text>Calculate the WMD with difference (see paper).<jupyter_code>wmd_pairs_flow.get_differences()<jupyter_output><empty_output><jupyter_text>3.1 Intepreting pairwise WMD flows Now, let's sort the distances of the words that created the most distance from the positive to the negative reviews.<jupyter_code>top_words_source = {k: v for k, v in sorted(wmd_pairs_flow.wd_source_diff.items(), key=lambda item: item[1], reverse=True)[:30]}
top_words_source_df = pd.DataFrame.from_dict(top_words_source, orient='index', columns = ["distance"])
top_words_source_df['cost'] = top_words_source_df.distance.round(2)
top_words_source_df<jupyter_output><empty_output><jupyter_text>Next, let's see what added most distance when moving from the negative to the positive reviews.<jupyter_code>top_words_sink = {k: v for k, v in sorted(wmd_pairs_flow.wd_sink_diff.items(), key=lambda item: item[1], reverse=True)[:30]}
top_words_sink_df = pd.DataFrame.from_dict(top_words_sink, orient='index', columns = ["distance"])
top_words_sink_df['distance'] = top_words_sink_df.distance.round(2)
top_words_sink_df<jupyter_output><empty_output><jupyter_text>Finally, let's look at the distances between the two sets by clustering similar words, in order to get a better sense of what kind of "topics" that separate them. Each cluster has a weight that matches the sum of the words belonging to that cluster. We choose *n* top clusters to inspect. To make the clusters interpretable, we also represent each of them by *m* keywords, selected based on the cost they individually add between the two sets.<jupyter_code>n_clusters = 10
n_words = 10
c1 = output_clusters(wd=wmd_pairs_flow.wd_source_diff.items(), 
                     cd=wmd_pairs_flow.cd_source.items(), 
                     c2w=cluster2words, 
                     n_clusters=n_clusters, 
                     n_words=n_words)
c2 = output_clusters(wd=wmd_pairs_flow.wd_sink_diff.items(), 
                     cd=wmd_pairs_flow.cd_sink.items(), 
                     c2w=cluster2words, 
                     n_clusters=n_clusters, 
                     n_words=n_words)<jupyter_output><empty_output><jupyter_text>Positive to negative clusters.<jupyter_code>c1<jupyter_output><empty_output><jupyter_text>Negative to positive clusters.<jupyter_code>c2 
sample.shape<jupyter_output><empty_output><jupyter_text>3.2. Explore differences by category Because our sample is stratified by city and by category, we also offer this small example of how to explore the differences by city and category using facetted boxplots.<jupyter_code>source_dists = pd.DataFrame(wmd_pairs_flow.source_feat)
source_dists.index = [p[0] for p in pairs]
source_dists = source_dists.sort_index()
source_dists = source_dists[c1.columns]
source_dists['categories'] = sample[:2000].categories
Counter(source_dists.categories)
source_dists_long = pd.melt(source_dists, id_vars=['categories']).rename(columns={"variable":"cluster"})
source_dists_long.head()
source_dists_long = source_dists_long[source_dists_long.value != 0]
g = sns.catplot(x="categories", 
                y="value", 
                col="cluster", 
                data=source_dists_long, 
                kind="box",
                height=5, 
                aspect=.7,
                col_wrap=5,
                margin_titles=True);
g.map_dataframe(sns.stripplot, 
                x="categories", 
                y="value", 
                palette=["#404040"], 
                alpha=0.2, dodge=True)
g.set_axis_labels("Category", "Distance")
for ax in g.axes.flatten():
    ax.tick_params(labelbottom=True)

sink_dists = pd.DataFrame(wmd_pairs_flow.sink_feat)
sink_dists.index = [p[1] for p in pairs]
sink_dists = sink_dists.sort_index()
sink_dists = sink_dists[c2.columns]
sink_dists['categories'] = sample[2000:4000].categories.tolist()
sink_dists_long = pd.melt(sink_dists, id_vars=['categories']).rename(columns={"variable":"cluster"})
sink_dists_long.head()
sink_dists_long = sink_dists_long[sink_dists_long.value != 0]
g = sns.catplot(x="categories", 
                y="value", 
                col="cluster", 
                data=sink_dists_long, 
                kind="box",
                height=5, 
                aspect=.7,
                col_wrap=5,
                margin_titles=True);
g.map_dataframe(sns.stripplot, 
                x="categories", 
                y="value", 
                palette=["#404040"], 
                alpha=0.2, dodge=True)
g.set_axis_labels("Category", "Distance")
for ax in g.axes.flatten():
    ax.tick_params(labelbottom=True)<jupyter_output><empty_output><jupyter_text>4 Better clusters with t-SNE Repeat the steps in section 3, but with clusters based on t-SNE reduced embeddings.<jupyter_code>word2cluster_t = {features[idx]: cl for idx, cl in enumerate(labels_tsne)}
print(take(10, word2cluster_t.items()))<jupyter_output>[('[', 31), (']', 80), ('^', 31), ('_____', 31), ('aa_bowl', 1), ('aaaaaand', 31), ('aaron', 48), ('aarp', 87), ('abandoned', 91), ('abandoning', 61)]<jupyter_text>Now, conversely, create an index that maps each cluster to a word.<jupyter_code>cluster2words_t = defaultdict(list)
for key, value in word2cluster_t.items():
    cluster2words_t[value].append(key)
%%time 

wmd_pairs_flow_t = WMDPairs(pos_docs,neg_docs,pairs,E,idx2word)
wmd_pairs_flow_t.get_distances(decompose = True, 
                               sum_clusters = True, 
                               w2c = word2cluster_t, 
                               c2w = cluster2words_t,
                               thread = False,
                               relax = True)
wmd_pairs_flow_t.get_differences()
c1_t = output_clusters(wd=wmd_pairs_flow_t.wd_source_diff.items(), 
                       cd=wmd_pairs_flow_t.cd_source.items(), 
                       c2w=cluster2words_t, 
                       n_clusters=n_clusters, 
                       n_words=n_words)
c2_t = output_clusters(wd=wmd_pairs_flow_t.wd_sink_diff.items(), 
                       cd=wmd_pairs_flow_t.cd_sink.items(), 
                       c2w=cluster2words_t, 
                       n_clusters=n_clusters, 
                       n_words=n_words)
c1_t
c2_t<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11392488978753025, 'fraction_numerical': 0.028240909542908282, 'mean_word_length': 5.164691458929301, 'pattern_counts': {'":': 12, '<': 211, '<?xml version=': 0, '>': 212, 'https://': 5, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 1, 'word_list_counts': {'cursed_substrings.json': 12, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '57', 'dir': 'jupyter-structured-clean-dedup', 'id': '499362', 'license': 'MIT', 'n_tokens_mistral': 11121, 'n_tokens_neox': 10206, 'n_words': 2516, 'path': 'paper/notebooks/Yelp_example.ipynb', 'repo_name': 'maybemkl/wmdecompose', 'size': '30169'}