<jupyter_start><jupyter_text>Finding entity classes in embeddingsIn this notebook we're going to use embeddings to find entity classes and how they correlate with other things<jupyter_code>%matplotlib inline

from sklearn import svm
from keras.utils import get_file
import os
import gensim
import numpy as np
import random
import requests
import geopandas as gpd
from IPython.core.pylabtools import figsize
figsize(12, 8)
import pycountry
import csv<jupyter_output>Using TensorFlow backend.<jupyter_text>as before, let's load up the model<jupyter_code>MODEL = 'GoogleNews-vectors-negative300.bin'
path = get_file(MODEL + '.gz', 'https://s3.amazonaws.com/dl4j-distribution/%s.gz' % MODEL)
unzipped = os.path.join('generated', MODEL)
if not os.path.isfile(unzipped):
    with open(unzipped, 'wb') as fout:
        zcat = subprocess.Popen(['zcat'],
                          stdin=open(path),
                          stdout=fout
                         )
        zcat.wait()<jupyter_output><empty_output><jupyter_text>Most similar to a bunch of countries are some other countries!<jupyter_code>model = gensim.models.KeyedVectors.load_word2vec_format(unzipped, binary=True)
model.most_similar(positive=['Germany'])
model.most_similar(positive=['Annita_Kirsten'])<jupyter_output><empty_output><jupyter_text>No we'll create a training set with countries and non countries and get a support vector machine to learn the difference.<jupyter_code>countries = list(csv.DictReader(open('data/countries.csv')))
countries[:10]
positive = [x['name'] for x in random.sample(countries, 40)]
negative = random.sample(model.vocab.keys(), 5000)
negative[:4]
labelled = [(p, 1) for p in positive] + [(n, 0) for n in negative]
random.shuffle(labelled)
X = np.asarray([model[w] for w, l in labelled])
y = np.asarray([l for w, l in labelled])
X.shape, y.shape
TRAINING_FRACTION = 0.3
cut_off = int(TRAINING_FRACTION * len(labelled))
clf = svm.SVC(kernel='linear')
clf.fit(X[:cut_off], y[:cut_off])<jupyter_output><empty_output><jupyter_text>We did alright, 99.9% precision:<jupyter_code>res = clf.predict(X[cut_off:])

missed = [country for (pred, truth, country) in 
 zip(res, y[cut_off:], labelled[cut_off:]) if pred != truth]

100 - 100 * float(len(missed)) / len(res), missed
all_predictions = clf.predict(model.syn0)
res = []
for word, pred in zip(model.index2word, all_predictions):
    if pred:
        res.append(word)
        if len(res) == 150:
            break
random.sample(res, 10)
country_to_idx = {country['name']: idx for idx, country in enumerate(countries)}
country_vecs = np.asarray([model[c['name']] for c in countries])
country_vecs.shape<jupyter_output><empty_output><jupyter_text>Quick sanity check to see what is similar to Canada:<jupyter_code>dists = np.dot(country_vecs, country_vecs[country_to_idx['Canada']])
for idx in reversed(np.argsort(dists)[-10:]):
    print(countries[idx]['name'], dists[idx])<jupyter_output>Canada 7.5440245
New_Zealand 3.9619699
Finland 3.9392405
Puerto_Rico 3.838145
Jamaica 3.8102934
Sweden 3.8042784
Slovakia 3.7038736
Australia 3.6711009
Bahamas 3.6240418
United_States 3.537434<jupyter_text>Ranking countries for a specific term:<jupyter_code>def rank_countries(term, topn=10, field='name'):
    if not term in model:
        return []
    vec = model[term]
    dists = np.dot(country_vecs, vec)
    return [(countries[idx][field], float(dists[idx])) 
            for idx in reversed(np.argsort(dists)[-topn:])]
rank_countries('cricket')<jupyter_output><empty_output><jupyter_text>Now let's visualize this on a world map:<jupyter_code>world = gpd.read_file(gpd.datasets.get_path('naturalearth_lowres'))
world.head()<jupyter_output><empty_output><jupyter_text>We can now plot some maps!<jupyter_code>def map_term(term):
    d = {k.upper(): v for k, v in rank_countries(term, topn=0, field='cc3')}
    world[term] = world['iso_a3'].map(d)
    world[term] /= world[term].max()
    world.dropna().plot(term, cmap='OrRd')

map_term('coffee')
map_term('cricket')
map_term('China')<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.1298219584569733, 'fraction_numerical': 0.030415430267062313, 'mean_word_length': 5.34012539184953, 'pattern_counts': {'":': 0, '<': 35, '<?xml version=': 0, '>': 35, 'https://': 1, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '9', 'dir': 'jupyter-structured-clean-dedup', 'id': '8642', 'license': 'Apache-2.0', 'n_tokens_mistral': 1561, 'n_tokens_neox': 1442, 'n_words': 327, 'path': '03.2 Domain specific ranking using word2vec cosine distance.ipynb', 'repo_name': 'bartbroere/deep_learning_cookbook', 'size': '4044'}