<jupyter_start><jupyter_text>Evaluating doc2vec vectors for wiki10 documents (wikipedia article text with multiple class labels)<jupyter_code># import modules & set up logging
from pathlib import Path
import re
import string 
from itertools import islice
import multiprocessing
import json
import numpy as np
import pandas as pd 
from sklearn.manifold import TSNE, MDS
from plotly.offline import init_notebook_mode, plot, iplot
import plotly.graph_objs as go
import plotly.plotly as py
import seaborn as sns
import matplotlib.pyplot as plt 
from gensim.models import Doc2Vec
from sklearn.metrics import pairwise 
init_notebook_mode(connected=True)
%matplotlib inline 
DATA_DIR = Path('../../data/wiki10')
TEXT_DIR = DATA_DIR / 'text' 
TAGS_PATH = DATA_DIR / 'tags.json'
MODEL_PATH = DATA_DIR / 'doc2vec.model'<jupyter_output><empty_output><jupyter_text>EvaluationWe will now evaluate distance between documents by reducing dimensions with T-SNE and plotting the result. We can also get an idea of how these "clusters" compare to human categorisation by loading our document labels and overlaying.Loading model and meta data containing tag information<jupyter_code>model = Doc2Vec.load(str(MODEL_PATH))
with TAGS_PATH.open() as input_fileobj:
    tags = json.load(input_fileobj)<jupyter_output><empty_output><jupyter_text>Create DataFrame from meta dictionary containing, ensuring row order is the same as doc_vectors matrix* id: document filename* title: wikipedia document title* user_count: number of people which tagged the document* tags: dict of {'tag': tag_score} where tag score is the proportion of users which assigned that tag* tag: tag with highest score * mapped_tag: mapped_tag created in 01_cluster_rags<jupyter_code>meta_records = [tags[doc_id] for doc_id in model.docvecs.offset2doctag]
df = pd.DataFrame.from_records(meta_records)<jupyter_output><empty_output><jupyter_text>Append doc_vectors as a new column to simplify sampling/reordering data. Can recover ndarray with vstack. (Alternative would be to keep objects separate and pass df index as row index to doc vectors after df manipulation)<jupyter_code>df['doc_vectors'] = list(model.docvecs.doctag_syn0)  
type(df.doc_vectors[0]) 
df.head()
df.tag.value_counts()<jupyter_output><empty_output><jupyter_text>We still have too many classes and observations to visualize effectively so we will sample each of our classes and split them into two groups of tag types, then run TSNE dimensionality reduction and plot.<jupyter_code>group_1 = {'religion', 'economics', 'art', 'music', 'math', 'science', 'film', 'people', 'language'}
group_2 = {'politics', 'history', 'tv', 'programming', 'technology', 'philosophy', 'literature', 'games', 'food' }
def sample_tags(df, tags, n):
    ''' return a view of dataframe containing rows from sample of tag types'''
    df_sample = (df.loc[df.tag.isin(tags)]
                   .groupby('tag')
                   .apply(lambda x: x.sample(n))
                   .reset_index(level=0, drop=True))  # removing tags from multindex
    return df_sample
def plot_tsne(df):
    # compute TSNE
    tsne = TSNE(n_components=2, verbose=1, perplexity=40, n_iter=300, random_state=0)
    tsne_results = tsne.fit_transform(np.vstack(df['doc_vectors']))

    # plot results
    traces = []
    for tag_type in df.tag.unique():
        trace = go.Scatter(
            x=tsne_results[df.tag==tag_type,0],
            y=tsne_results[df.tag==tag_type,1],
            text=df.loc[df.tag==tag_type,"title"],
            hoverinfo='text',
            mode='markers',
            name=tag_type, 
            marker=dict(
                size=5,
                opacity=0.5
            )
        )
        traces.append(trace)
        
    layout = go.Layout(
        hovermode='closest',
        width=650,
        height=500,
        margin=go.Margin(
            t=10
        )
    )

    fig = go.Figure(data=traces, layout=layout)
    iplot(fig)
    #py.image.ishow(fig)
%%time
plot_tsne(sample_tags(df, tags=group_1, n=200))
%%time
plot_tsne(sample_tags(df, tags=group_2, n=200))<jupyter_output>[t-SNE] Computing 121 nearest neighbors...
[t-SNE] Indexed 1800 samples in 0.004s...
[t-SNE] Computed neighbors for 1800 samples in 1.247s...
[t-SNE] Computed conditional probabilities for sample 1000 / 1800
[t-SNE] Computed conditional probabilities for sample 1800 / 1800
[t-SNE] Mean sigma: 1.638291
[t-SNE] KL divergence after 250 iterations with early exaggeration: 70.237312
[t-SNE] Error after 300 iterations: 1.421418<jupyter_text>Even in 2 dimensions there is a clear separation between some manually labelled classes from the unsupervised doc2vec. Also notice the tight groups of varyiable tag types which do not really belong to any one particular "cluster". Hovering over them reveals they often refer to articles which contain lists of items e.g. **"list of literary awards"**, **"list of software development philosophies"**. Whilst the lists do appear to be very specific to a single domain, the sentences input to doc2vec algorithm for these pages will not be full sentences, only single words/phrases and so the learnt weights for the document vector may suffer, and as a result we see them sitting in "no-mans land".We will now look at the distance between the labelled tags types and their document vectors in more detail.Finding all tag types which appear >= 200 times to reduce the number of classes we consider.<jupyter_code>tags_200 = (df.groupby(['tag'])
              .filter(lambda x: len(x) >= 200)
              .tag
              .unique())<jupyter_output><empty_output><jupyter_text>Sampling all tag types in tags_200 and finding "centroid" vector for each tag type.<jupyter_code>centroids = (sample_tags(df, tags=tags_200, n=200)
             .groupby('tag')  
             .apply(lambda x: np.vstack(x.doc_vectors).mean(axis=0)))
df_centroids = pd.DataFrame({
    'tag':centroids.index,
    'doc_vectors':centroids.values
   }) 
df_centroids.head()<jupyter_output><empty_output><jupyter_text>Calculating and plotting TSNE for each centroid to visualize distance between each tag<jupyter_code>def plot_scatter_labels(X, Y, labels):
    plt.rcParams["figure.figsize"] = (8,8)
    for x, y, label in zip(X, Y, labels):
        plt.scatter(x, y) 
        plt.text(x, y, label)
    plt.show()
tsne = TSNE(n_components=2, perplexity=40, n_iter=300, random_state=0)
tsne_2d = tsne.fit_transform(np.vstack(df_centroids['doc_vectors']))
X = tsne_2d[:,0]
Y = tsne_2d[:,1]
labels = df_centroids.tag.values

plot_scatter_labels(X, Y, labels)<jupyter_output><empty_output><jupyter_text>The TSNE algorithm tries to find local trends, but is less effective at capturing global trends. We can see that problem illustrated here as there is a large distance between "physics" and "science", which is counterintuitive. https://stats.stackexchange.com/questions/238538We will try applying a different algorithm for dimensionality reduction (Multi-dimensional scaling) which should better preserve the actual distance between observations.<jupyter_code>mds = MDS(n_components=2, random_state=1)
mds_2d = mds.fit_transform(np.vstack(df_centroids['doc_vectors']))
X = mds_2d[:,0]
Y = mds_2d[:,1]
labels = df_centroids.tag.values

plot_scatter_labels(X, Y, labels)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09577387486278814, 'fraction_numerical': 0.021542261251372118, 'mean_word_length': 4.343841642228739, 'pattern_counts': {'":': 0, '<': 36, '<?xml version=': 0, '>': 38, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '262542', 'license': 'MIT', 'n_tokens_mistral': 2397, 'n_tokens_neox': 2236, 'n_words': 723, 'path': '03_doc2vec-evaluate.ipynb', 'repo_name': 'david-macleod/wiki10', 'size': '7288'}