<jupyter_start><jupyter_text>自然言語処理* 普段が私たちが使用している言葉を自然言語と呼ぶ。 シソーラス（類義語）* WordNet ・問題点* 時代の変化に対応するのが困難* 人の作業コストが高い* 単語の細かなニュアンスを表現できない カウントベースの手法* コーパス：大量のテキストデータ<jupyter_code>import numpy as np
import matplotlib.pyplot as plt
import os
%matplotlib inline
text = "You say Hello and how are you and I say Hello and I'm fine."
text = text.replace(".", " .")#空白を作ることで一つの文字として認識される
lower_text = text.lower()#小文字
print(lower_text)
upper_text = text.upper()#大文字
print(upper_text)
split_text = text.split(' ')
print(split_text)
#カウントベースの処理
word_to_id = {}
id_to_word = {}
for split in split_text:
    new_id = len(word_to_id)#現在の長さを代入
    word_to_id[split] = new_id 
    id_to_word[new_id] = split
    
print(id_to_word)
print(word_to_id)
print(id_to_word[5])
#コーパスとはテキストデータのこと
corpus = [word_to_id[w] for w in split_text]
corpus = np.array(corpus)
corpus
def preprocess(text):
    text = text.lower()
    text = text.replace(".", " .")
    words = text.split(" ")
    
    word_to_id, id_to_word = {}, {}
    for word in words:
        new_id = len(word_to_id)
        word_to_id[word] = new_id
        id_to_word[new_id] = word
    corpus = np.array([word_to_id[w] for w in words])
    
    return corpus, word_to_id, id_to_word
text = "You say goodbye and I say hello."
corpus, word_to_id, id_to_word = preprocess(text)<jupyter_output><empty_output><jupyter_text>単語分散表現* RGBのようなベクトル表現 共起行列* 分布仮説に基づいて、単語をベクトルで表現する。（カウント）<jupyter_code>os.chdir("/work/NaturalProcessing/deep-learning-from-scratch-2-master/")
from common.util import preprocess
text = "You say goodbye and I say hello. Hello World."
corpus, word_to_id, id_to_word = preprocess(text)
print(corpus)
print(id_to_word)
print(word_to_id)
#共起行列：分布仮説に基づいて単語をベクトルで表現する。
def create_co_matrix(corpus, vocab_size, window_size=1):
    corpus_size = len(corpus)
    co_matrix = np.zeros((vocab_size, vocab_size), dtype=np.int32)
    
    for idx, word_id in enumerate(corpus):
        for i in range(1, window_size + 1):
            left_idx = idx - i
            right_idx = idx + i
            
            if left_idx >= 0:
                left_word_id = corpus[right_idx]
                co_matrix[word_id, left_word_id] += 1
                
            if right_idx < corpus_size:
                right_word_id = corpus[right_idx]
                co_matrix[word_id, right_word_id] += 1
    return co_matrix
#コサイン類似度（二つのベクトルがどれだけ同じ方向を向いているかを求める）
#完全に同じ方向を向いている場合は１逆は−１
def cos_similarity(x, y, eps=1e+8):
    nx = x / (np.sqrt(np.sum(x ** 2)) + eps)
    ny = y / (np.sqrt(np.sum(y ** 2)) + eps)
    return np.dot(nx, ny)
from common.util import preprocess, create_co_matrix, cos_similarity

text = "You say goodbye and I say hello."
corpus, word_to_id, id_to_word = preprocess(text)
vocab_size = len(word_to_id)
C = create_co_matrix(corpus, vocab_size)

c0 = C[word_to_id["you"]]
c1 = C[word_to_id["i"]]
print(cos_similarity(c0, c1))
def most_similar(query, word_to_id, id_to_word, word_matrix, top=5):
    if query not in word_to_id:#queryが取り出す。もしなければ, not foundを返す
        print("%s is not found" % query)
        return 
    print("\n[query]" + query)
    query_id = word_to_id[query]
    query_vec = word_matrix[query_id]
    
    vocab_size = len(id_to_word)#コサイン類似度の算出
    similarity = np.zeros(vocab_size)
    for i in range(vocab_size):
        similarity[i] = cos_similarity(word_matrix[i], query_vec)
        
    count = 0
    for i in (-1 * similarity).argsort():#コサイン類似度の結果から、その値が高い順に出力
        if id_to_word[i] == query:
            continue
        print("%s: %s" % (id_to_word[i], similarity[i]))
        
        count += 1
        if count >= top:
            return
x = np.array([100, -20, 2])
x.argsort()
(-x).argsort()
text = "You say goodbye and I say hello."

corpus, word_to_id, id_to_word = preprocess(text)
vocab_size = len(word_to_id)
C = create_co_matrix(corpus, vocab_size)

most_similar("you", word_to_id, id_to_word, C, top=5)<jupyter_output>[query]you
goodbye: 0.7071067691154799
i: 0.7071067691154799
hello: 0.7071067691154799
say: 0.0
and: 0.0<jupyter_text>カウントベースの欠点と改善* theやaなどの冠詞はよく文中で出現する。→本来関係のある単語のコサイン類似度の値が低くなってしまう。 相互情報量(Pointwise Mutual Information)$$PMI(x,y)=\log_{2}\frac{P(x,y)}{P(x)P(y)}$$* P(x)がxが起こる確率、P(y)はyが起こる確率* 先程の式に共起行列を付け加える。* 最終的な式は次のようになる。(Cを共起行列、Nをコーパスに含まれる単語数)$$PMI(x,y)=\log_{2}\frac{C(x,y) \cdot N}{C(x)C(y)}$$* 実際に使用する際は**正の相互情報量**という指標を使用する。（共起する回数が０の場合、ー∞になっていしまうため）$$PPMI(x, y)=max(0, PMI(x, y))$$<jupyter_code>def ppmi(C, verbose=False, eps = 1e-8):
    '''PPMI（正の相互情報量）の作成

    :param C: 共起行列
    :param verbose: 進行状況を出力するかどうか
    :return:
    '''
    M = np.zeros_like(C, dtype=np.float32)
    N = np.sum(C)
    S = np.sum(C, axis=0)
    total = C.shape[0] * C.shape[1]
    cnt = 0

    for i in range(C.shape[0]):
        for j in range(C.shape[1]):
            pmi = np.log2(C[i, j] * N / (S[j]*S[i]) + eps)
            M[i, j] = max(0, pmi)

            if verbose:
                cnt += 1
                if cnt % (total//100 + 1) == 0:
                    print('%.1f%% done' % (100*cnt/total))
    return M
text = "You say goodbye and I say hello."
corpus, word_to_id, id_to_word = preprocess(text)
vocab_size = len(word_to_id)
C = create_co_matrix(corpus, vocab_size)
W = ppmi(C)

np.set_printoptions(precision=3)
print("covariance matrix")
print(C)
print("_" * 50)
print("PPMI")
print(W)<jupyter_output>covariance matrix
[[0 1 0 0 0 0 0]
 [1 0 1 0 1 1 0]
 [0 1 0 1 0 0 0]
 [0 0 1 0 1 0 0]
 [0 1 0 1 0 0 0]
 [0 1 0 0 0 0 1]
 [0 0 0 0 0 1 0]]
__________________________________________________
PPMI
[[0.    1.807 0.    0.    0.    0.    0.   ]
 [1.807 0.    0.807 0.    0.807 0.807 0.   ]
 [0.    0.807 0.    1.807 0.    0.    0.   ]
 [0.    0.    1.807 0.    1.807 0.    0.   ]
 [0.    0.807 0.    1.807 0.    0.    0.   ]
 [0.    0.807 0.    0.    0.    0.    2.807]
 [0.    0.    0.    0.    0.    2.807 0.   ]]<jupyter_text>次元削除* 次元削除の一つの方法として**特異値分解(Singular Value Decomposition:SVD)**がある$$X=USV^T$$* 任意の行列Xを、U,S,Vの３つの行列の積に分解する。* UとVは直交行列、Sは対角行列<jupyter_code>text = "You say goodbye and I say hello."
corpus, word_to_id, id_to_word = preprocess(text)
vocab_size = len(id_to_word)
C = create_co_matrix(corpus, vocab_size, window_size=1)
W = ppmi(C)

U, S, V = np.linalg.svd(W)

print(C[0])
print(W[0])
print(U[0])
for word, word_id in word_to_id.items():
    plt.annotate(word, (U[word_id, 0], U[word_id, 1]))
plt.scatter(U[:,0], U[:,1], alpha=0.5)
plt.show()
from dataset import ptb
corpus, word_to_id, id_to_word = ptb.load_data("train")
print('corpus_size:', len(corpus))
print("corpus[:30]", corpus[:30])
print()
print("id_to_word[0]:", id_to_word[0])
print("id_to_word[1]:", id_to_word[1])
print("id_to_word[2]:", id_to_word[2])
print()
print("word_to_id['car']:", word_to_id["car"])
print("word_to_id['happy']:", word_to_id["happy"])
print("word_to_id['lexus']:", word_to_id["lexus"])
window_size = 2
wordvec_size = 100

corpus, word_to_id, id_to_word = ptb.load_data('train')
vocab_size = len(word_to_id)
print('counting  co-occurrence ...')
C = create_co_matrix(corpus, vocab_size, window_size)
print('calculating PPMI ...')
W = ppmi(C, verbose=True)

print('calculating SVD ...')
try:
    # truncated SVD (fast!)
    from sklearn.utils.extmath import randomized_svd
    U, S, V = randomized_svd(W, n_components=wordvec_size, n_iter=5,
                             random_state=None)
except ImportError:
    # SVD (slow)
    U, S, V = np.linalg.svd(W)

word_vecs = U[:, :wordvec_size]

querys = ['you', 'year', 'car', 'toyota']
for query in querys:
    most_similar(query, word_to_id, id_to_word, word_vecs, top=5)<jupyter_output>counting  co-occurrence ...
calculating PPMI ...
1.0% done
2.0% done
3.0% done
4.0% done
5.0% done
6.0% done
7.0% done
8.0% done
9.0% done
10.0% done
11.0% done
12.0% done
13.0% done
14.0% done
15.0% done
16.0% done
17.0% done
18.0% done
19.0% done
20.0% done
21.0% done
22.0% done
23.0% done
24.0% done
25.0% done
26.0% done
27.0% done
28.0% done
29.0% done
30.0% done
31.0% done
32.0% done
33.0% done
34.0% done
35.0% done
36.0% done
37.0% done
38.0% done
39.0% done
40.0% done
41.0% done
42.0% done
43.0% done
44.0% done
45.0% done
46.0% done
47.0% done
48.0% done
49.0% done
50.0% done
51.0% done
52.0% done
53.0% done
54.0% done
55.0% done
56.0% done
57.0% done
58.0% done
59.0% done
60.0% done
61.0% done
62.0% done
63.0% done
64.0% done
65.0% done
66.0% done
67.0% done
68.0% done
69.0% done
70.0% done
71.0% done
72.0% done
73.0% done
74.0% done
75.0% done
76.0% done
77.0% done
78.0% done
79.0% done
80.0% done
81.0% done
82.0% done
83.0% done
84.0% done
85.0% done
86.0% done
87.0% done
88.[...]
{'fraction_non_alphanumeric': 0.1515858208955224, 'fraction_numerical': 0.0625, 'mean_word_length': 3.396207073295746, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 16, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '4', 'dir': 'jupyter-structured-clean-dedup', 'id': '571059', 'license': 'MIT', 'n_tokens_mistral': 4392, 'n_tokens_neox': 4077, 'n_words': 839, 'path': 'Basic_NL.ipynb', 'repo_name': 'ju-ki/Deep_learning_from_scratch_2', 'size': '8576'}