<jupyter_start><jupyter_text>Sklearn $\ge$ 0.19 is necessary for the efficient implementation of tsne.<jupyter_code>import sklearn.manifold  
sklearn.__version__<jupyter_output><empty_output><jupyter_text>Pull the data from the internet and write it to a fileDon't bother running this if you've already downloaded the dataset.<jupyter_code>#%%time
#getData = True  # Change this to false the first time you run the notebook
#if(getData): 
#    results = requests.get('https://drive.google.com/file/d/0B7XkCwpI5KDYNlNUTTlSS21pQmM/edit?usp=sharing')
#    model = KeyedVectors.load_word2vec_format('GoogleNews-vectors-negative300.bin', binary=True)
#    model.save_word2vec_format('GoogleNews-vectors-negative300.txt', binary=False)<jupyter_output><empty_output><jupyter_text>Read in from file and transform into a vector space<jupyter_code>%%time
from gensim.models.keyedvectors import KeyedVectors

model = KeyedVectors.load_word2vec_format('GoogleNews-vectors-negative300.bin', binary=True)
data = model.syn0
print(data.shape)
model = KeyedVectors.load_word2vec_format('GoogleNews-vectors-negative300.bin', binary=True)
model.save_word2vec_format('GoogleNews-vectors-negative300.txt', binary=False)<jupyter_output><empty_output><jupyter_text>Now we have our data in a list of vectors.  Let's extract the object id's from the files and cast to data frame (in case we want to explore things further)  Now let's use UMAP to embed these points into a two dimensional space.<jupyter_code>num_points = 100000
selected_index = np.random.choice(data.shape[0],size=num_points, replace=False)
data = data[selected_index,:]
words = np.array(model.index2word)[selected_index]
fit = umap.UMAP(n_neighbors=15, random_state=42, metric='cosine')
%time u = fit.fit_transform(data)
embedding = pd.DataFrame({'word':words, 'x':u[:,0], 'y':u[:,1]})
embedding.to_csv("~/python/UMAP/umap_paper_notebooks/embedding_word_100_umap.csv")
plt.scatter(u[:,0], u[:,1], s=0.1);<jupyter_output><empty_output><jupyter_text>100,000 datapoints subsampled from our file results give a run time of (2min 47sec) and results in lovely topological structure.  T-SNE It is largely accepted that cosine distance is a good measure to use on text.  Unfortunately, t-SNE cannot be efficiently run with cosine distance.  Fortunately, cosine distance can be approximated via L2-normalized Euclidean distance<jupyter_code>sklearn.preprocessing.normalize(data, norm='l2', copy=False);
fit_tsne = sklearn.manifold.TSNE(perplexity=60)
%time u_tsne = fit_tsne.fit_transform(data)
embedding = pd.DataFrame({'word':words, 'x':u_tsne[:,0], 'y':u_tsne[:,1]})
embedding.to_csv("~/python/UMAP/umap_paper_notebooks/embedding_word_100_tsne.csv")
plt.scatter(u_tsne[:,0], u_tsne[:,1], s=0.1);<jupyter_output><empty_output><jupyter_text>t-SNE reveals very the small clumps contained within the data but fails to highlight the relationship between these small clumps as was revealed via the UMAP algorithm earlier.Additionally, t-SNE took 2 hours a 30 minutes to complete while UMAP finished the task in only 2 minutes and 47 seconds.  PCAThe old standby of PCA which is blindingly fast to compute and often used as an initialization to many of the more complex algorithms.  This isn't really a competitor but instead should be thought of as a strawman.  Given that other algorithms initialize with these values one would hope that they can do better.<jupyter_code>from sklearn.decomposition import PCA
pca = PCA(n_components=2)
%time u_pca = pca.fit_transform(data)
plt.scatter(u_pca[:,0], u_pca[:,1],  s=1);<jupyter_output><empty_output><jupyter_text>MDSFor old times sake we attempt to embed with multi-dimensional scaling.  Probably not worth running for the 100,000 use case.  Might try an overnight run though.Nope killed my kernel on an 8 GB laptop on an overnight run.<jupyter_code>fit_mds = sklearn.manifold.MDS()
%time u_mds = fit_mds.fit_transform(data)
plt.scatter(u_mds[:,0], u_mds[:,1], s=10);<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.10314465408805032, 'fraction_numerical': 0.02339622641509434, 'mean_word_length': 6.720388349514563, 'pattern_counts': {'":': 0, '<': 29, '<?xml version=': 0, '>': 29, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 1, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '7', 'dir': 'jupyter-structured-clean-dedup', 'id': '147503', 'license': 'MIT', 'n_tokens_mistral': 1384, 'n_tokens_neox': 1308, 'n_words': 371, 'path': 'UMAP WordEmbedding hundredThousand.ipynb', 'repo_name': 'lmcinnes/umap_paper_notebooks', 'size': '3975'}