<jupyter_start><jupyter_text>60.単語ベクトルの読み込みと表示<jupyter_code>import gensim
import numpy as np

# その他
import warnings
# configs
GOOGLE_MODEL_PATH = './ch7_folder/model/GoogleNews-vectors-negative300.bin'<jupyter_output><empty_output><jupyter_text>Warningを消すために以下のコードを追加します。<jupyter_code>warnings.simplefilter('ignore')
model = gensim.models.KeyedVectors.load_word2vec_format(GOOGLE_MODEL_PATH, 
                                                        binary=True)
model['United_States']
model['United_States'].shape
type(model['United_States'])<jupyter_output><empty_output><jupyter_text>61.単語の類似度 & 62.類似度の高い単語10件 & 63.加法構成性によるアナロジー<jupyter_code>from common_utils import cosine_similarity
vec_united_states = model['United_States']
vec_us = model['U.S.']
cosine_similarity(vec_united_states, vec_us)
model.wv.most_similar([vec_united_states],[],10)
vec_spain = model['Spain']
vec_madrid = model['Madrid']
vec_athens = model['Athens']
model.wv.most_similar([vec_spain - vec_madrid + vec_athens],[],10)
model.wv.most_similar([vec_spain - vec_madrid + vec_athens],[],1)[0]<jupyter_output><empty_output><jupyter_text>64.アナロジーデータでの実験 & 65.アナロジータスクでの正解率 　単語アナロジーの評価データには6章とちがって`readme.txt`が存在しないため、少し補足をします。`: var`と書いてある文字列があります、これは次の`: foo`までの文字列が何のジャンルに関するかを表しています。辞書のキーみたいなものだと思ってください。`:var`の下には半角スペースで区切られた文字が4つ並んでいます。`var`に関する文字列です。　他の文字列については、具体例を用いて説明します。`: capital-common-countries`というジャンルの下に`Athens Greece Baghdad Iraq`という文字列があります。これらの関係は意味的な演算として、$$  \text{Greece} -\text{Athens} + \text{Baghdad} = \text{Iraq}$$という関係性になってます。これを先ほど使ったモデルを使って解析していきます。<jupyter_code>import os
import pandas as pd
from common_utils import make_csv
# configs
QUESTIONS_WORDS_PATH = './ch7_folder/questions-words.txt'
SAVE_DATA_PATH = './ch7_folder/save_data/'
if not os.path.exists(SAVE_DATA_PATH):
    os.makedirs(SAVE_DATA_PATH)
with open(QUESTIONS_WORDS_PATH, mode='r', encoding='utf-8') as f:
    questions_words_readlines_list = [s.strip() for s in f.readlines()]<jupyter_output><empty_output><jupyter_text>問題には書いてありませんが、`: var`ごとにCSVを作ると便利そうなのでCSVに変換します<jupyter_code>HEADERS = ['FIRST', 'SECOND', 'THIRD', 'ANSWER']
key_indexes_list = [questions_words_readlines_list.index(word) 
              for word in questions_words_readlines_list if ':' in word]

for i in range(len(key_indexes_list)):
    index_num = key_indexes_list[i]
    csv_name = questions_words_readlines_list[index_num][2:]
    keyword_index = key_indexes_list.index(index_num)
    
    if keyword_index < 12:
        next_index_num = key_indexes_list[i + 1]
        tmp_list = questions_words_readlines_list[(index_num + 1):next_index_num]
    else:
        tmp_list = questions_words_readlines_list[(index_num + 1):]
    tmp_split_list = [word.split() for word in tmp_list]
    
    csv_path = SAVE_DATA_PATH + csv_name + '.csv'
    make_csv(csv_path, tmp_split_list, HEADERS)<jupyter_output>./ch7_folder/save_data/capital-common-countries.csvはもう存在しています
./ch7_folder/save_data/capital-world.csvはもう存在しています
./ch7_folder/save_data/currency.csvはもう存在しています
./ch7_folder/save_data/city-in-state.csvはもう存在しています
./ch7_folder/save_data/family.csvはもう存在しています
./ch7_folder/save_data/gram1-adjective-to-adverb.csvはもう存在しています
./ch7_folder/save_data/gram2-opposite.csvはもう存在しています
./ch7_folder/save_data/gram3-comparative.csvはもう存在しています
./ch7_folder/save_data/gram4-superlative.csvはもう存在しています
./ch7_folder/save_data/gram5-present-participle.csvはもう存在しています
./ch7_folder/save_data/gram6-nationality-adjective.csvはもう存在しています
./ch7_folder/save_data/gram7-past-tense.csvはもう存在しています
./ch7_folder/save_data/gram8-plural.csvはもう存在しています
./ch7_folder/save_data/gram9-plural-verbs.csvはもう存在しています<jupyter_text>準備をいろいろしていきます。<jupyter_code>CAPITAL_COMMON_CSV_PATH = './ch7_folder/save_data/capital-common-countries.csv'
CAPITAL_WORLD_CSV_PATH = './ch7_folder/save_data/capital-world.csv'
CURRENCY_CSV_PATH = './ch7_folder/save_data/currency.csv'
CITY_CSV_PATH = './ch7_folder/save_data/city-in-state.csv'
FAMILY_CSV_PATH = './ch7_folder/save_data/family.csv'
GRAM1_CSV_PATH = './ch7_folder/save_data/gram1-adjective-to-adverb.csv'
GRAM2_CSV_PATH = './ch7_folder/save_data/gram2-opposite.csv'
GRAM3_CSV_PATH = './ch7_folder/save_data/gram3-comparative.csv'
GRAM4_CSV_PATH = './ch7_folder/save_data/gram4-superlative.csv'
GRAM5_CSV_PATH = './ch7_folder/save_data/gram5-present-participle.csv'
GRAM6_CSV_PATH = './ch7_folder/save_data/gram6-nationality-adjective.csv'
GRAM7_CSV_PATH = './ch7_folder/save_data/gram7-past-tense.csv'
GRAM8_CSV_PATH = './ch7_folder/save_data/gram8-plural.csv'
GRAM9_CSV_PATH = './ch7_folder/save_data/gram9-plural-verbs.csv'
df_capital_common = pd.read_csv(CAPITAL_COMMON_CSV_PATH)
df_capital_world = pd.read_csv(CAPITAL_WORLD_CSV_PATH)
df_gram1 = pd.read_csv(GRAM1_CSV_PATH)<jupyter_output><empty_output><jupyter_text>CSVの中身をみてみるとこんな感じになってます。<jupyter_code>df_capital_common[:10]
df_gram1[:10]<jupyter_output><empty_output><jupyter_text>64で書いてある課題を行うための準備をします。<jupyter_code>def check_analogy(row):
    vec_predict = model[row['SECOND']] - model[row['FIRST']] + model[row['THIRD']]
    most_similarity_word, similarity_value = model.wv.most_similar([vec_predict],[],1)[0]
    return pd.Series([most_similarity_word, similarity_value])
%%time
df_capital_common[['PREDICT_WORD', 'SIMILARITY_VALUE']] = df_capital_common.apply(check_analogy, axis=1)
%%time
df_gram1[['PREDICT_WORD', 'SIMILARITY_VALUE']] = df_gram1.apply(check_analogy, axis=1)
df_capital_common[:10]
def analogy_accurancy(df):
    denominator = len(df)
    numerator = (df['ANSWER']  == df['PREDICT_WORD']).sum()
    return (numerator / denominator) * 100
analogy_accurancy(df_capital_common)
analogy_accurancy(df_gram1)<jupyter_output><empty_output><jupyter_text>意味的アナロジーの方が正解率は高そうですね。  66. WordSimilarity-353での評価<jupyter_code>from scipy.stats import spearmanr
# configs
WORD_SIM_PATH = './ch7_folder/wordsim353/'
combined_csv_path = WORD_SIM_PATH + 'combined.csv'
df_combined = pd.read_csv(combined_csv_path)
df_combined[:10]
def words2similarity(row):
    vec1 = model[row['Word 1']]
    vec2 = model[row['Word 2']]
    return cosine_similarity(vec1,vec2)
df_combined['SIMILARITY_VALUE'] = df_combined.apply(words2similarity ,axis=1)
human_similarity_list = df_combined['Human (mean)'].values.tolist()
cosine_similarity_list = df_combined['SIMILARITY_VALUE'].values.tolist()
spearmanr(human_similarity_list, cosine_similarity_list)
type(spearmanr(human_similarity_list, cosine_similarity_list))<jupyter_output><empty_output><jupyter_text>67.k-meansクラスタリング ただの余談なんですが、問題文にある『国名に関する単語ベクトルを抽出し，...』と書いてあるんですが、これって全ての国名って意味なんですかね。とりあえずここから国名を取ってきました。`https://www.jpo.go.jp/system/patent/pct/tetuzuki/document/kuni-meisho/kuni-meisho-ichiran.pdf`<jupyter_code>from sklearn.cluster import KMeans
from sklearn.decomposition import PCA
import matplotlib.pyplot as plt
import japanize_matplotlib
COUNTRY_NAME_PATH = './ch7_folder/country.txt'
with open(COUNTRY_NAME_PATH, mode='r') as f:
    country_names_list = [s.strip().replace(' ', '_') for s in f.readlines()]
country_names_list.append('United_States')
model_vocab_list = list(model.wv.vocab.keys())
model_country_names_list = [ word for word in country_names_list if word in model_vocab_list ]
X = np.array([model[word] for word in model_country_names_list])
kmeans_model = KMeans(n_clusters=5, random_state=10).fit(X)
kmeans_model.labels_<jupyter_output><empty_output><jupyter_text>この結果を主成分分析して二次元データにプロットします。<jupyter_code>labels = kmeans_model.labels_
color_codes = {0: '#DD1F30', 1:'#2058DB', 2:'#32D820', 3:'#D69F20', 4:'#7FFFD4'}
colors = [color_codes[x] for x in labels]
pca = PCA()
pca.fit(X)
feature = pca.transform(X)
plt.figure(figsize=(10,10))
for x, y, country_name in zip(feature[:, 0], feature[:,1], model_country_names_list):
    plt.text(x, y, country_name, alpha=0.8, size=10)
plt.scatter(feature[:, 0], feature[:, 1], alpha=0.8, color=colors, s=100)
plt.title('主成分分析結果')
plt.xlabel("1st")
plt.ylabel("2nd")
plt.show()<jupyter_output><empty_output><jupyter_text>68.Ward法によるクラスタリング<jupyter_code>from scipy.cluster.hierarchy import dendrogram, linkage, fcluster
Z = linkage(X,method='ward',metric='euclidean')
plt.figure(figsize=(30,10))
dendrogram_dict = dendrogram(Z, leaf_font_size=12,leaf_rotation=82 ,labels=model_country_names_list)<jupyter_output><empty_output><jupyter_text>69.t-SNEによる可視化<jupyter_code>from sklearn.manifold import TSNE
X_reduced = TSNE(n_components=2, random_state=10).fit_transform(X)
X_reduced.shape
plt.figure(figsize=(10,10))
for x, y, country_name in zip(X_reduced[:, 0], X_reduced[:, 1], model_country_names_list):
    plt.text(x, y, country_name, alpha=0.8, size=10)
plt.scatter(X_reduced[:, 0], X_reduced[:, 1], s=100)
plt.xlabel("1st")
plt.ylabel("2nd")
plt.show()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.13254397057133005, 'fraction_numerical': 0.024370617312334752, 'mean_word_length': 7.596837944664031, 'pattern_counts': {'":': 0, '<': 53, '<?xml version=': 0, '>': 52, 'https://': 1, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 2, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '13', 'dir': 'jupyter-structured-clean-dedup', 'id': '90489', 'license': 'MIT', 'n_tokens_mistral': 4118, 'n_tokens_neox': 3810, 'n_words': 391, 'path': 'ch7.ipynb', 'repo_name': 'reonyanarticle/100pon_nock_2020', 'size': '8699'}