<jupyter_start><jupyter_text>第8章: ニューラルネット  70.単語ベクトルの和による特徴量<jupyter_code>import gensim
import numpy as np
import pandas as pd
import re

# その他
import warnings
#configs
TRAIN_DATA_PATH ='./ch6_folder/save_data/train.csv'
VALID_DATA_PATH = './ch6_folder/save_data/valid.csv'
TEST_DATA_PATH = './ch6_folder/save_data/test.csv'
GOOGLE_MODEL_PATH = './ch7_folder/model/GoogleNews-vectors-negative300.bin'<jupyter_output><empty_output><jupyter_text>Warningを消すために以下のコードを追加します。<jupyter_code>warnings.simplefilter('ignore')
model = gensim.models.KeyedVectors.load_word2vec_format(GOOGLE_MODEL_PATH, binary=True)
model_vocab_list = list(model.wv.vocab.keys())<jupyter_output><empty_output><jupyter_text>記事の見出しの単語列 $(w_{i,1}, \ldots, w_{i,T_i})$ に対して、embeddingをおこない、以下のような特徴ベクトルを生成すると書いてあります。$$    {x}_i = \frac{1}{T_i}\sum_{1 \leq t \leq T_i} {emb}(w_{i,t}) $$これは、各単語数に関する平均を特徴量にするということですね。<jupyter_code>CATEGORY_TAG = {'b': 0, 't': 1, 'e': 2, 'm': 3}
df_train = pd.read_csv(TRAIN_DATA_PATH)
df_valid = pd.read_csv(VALID_DATA_PATH)
df_test = pd.read_csv(TEST_DATA_PATH)
train_titles_list = [title.split() for title in df_train['TITLE'].values.tolist()]
valid_titles_list = [title.split() for title in df_valid['TITLE'].values.tolist()]
test_titles_list = [title.split() for title in df_test['TITLE'].values.tolist()]
train_length = len(train_titles_list)
valid_length = len(valid_titles_list)
test_length = len(test_titles_list)
print(train_length, valid_length, test_length)
def tokenize(word: str) -> str:
    """
    """
    word = word.replace('-', '')
    word = word.replace('.', '')
    word = word.replace('\'', '')
    word = word.replace(':', '')
    return word
def get_embedding_matrix(titles_list):
    feature_vecs_list = []
    
    for title in titles_list:
        tmp_vecs_list =[]
        for word in title:
            word = tokenize(word)
            try:
                vec = model[word]
                tmp_vecs_list.append(vec)
            except KeyError as e:
                pass
        
        if len(tmp_vecs_list) == 1:
            feature_vec = tmp_vecs_list[0]
        else:
            np_vecs = np.array(tmp_vecs_list)
            feature_vec = np.mean(np_vecs, axis=0)
        feature_vecs_list.append(feature_vec)
    return np.array(feature_vecs_list)
X_train = get_embedding_matrix(train_titles_list)
X_valid = get_embedding_matrix(valid_titles_list)
X_test = get_embedding_matrix(test_titles_list)
X_train.shape
X_valid.shape
X_test.shape
y_train= pd.read_csv(TRAIN_DATA_PATH)['CATEGORY'].map(CATEGORY_TAG).values
y_valid = pd.read_csv(VALID_DATA_PATH)['CATEGORY'].map(CATEGORY_TAG).values
y_test = pd.read_csv(TEST_DATA_PATH)['CATEGORY'].map(CATEGORY_TAG).values
Y_train = np.eye(4)[y_train]
Y_valid = np.eye(4)[y_valid]
Y_test = np.eye(4)[y_test]
y_train[1]
y_train.shape
Y_train.shape
y_valid.shape
Y_valid.shape
y_test.shape
Y_test.shape<jupyter_output><empty_output><jupyter_text>71. 単層NNによる予測 個人的にこういう時にKerasを使うのが好きなのでKerasを使っていきます。重みの初期化のところ、毎回同じ値を取りたいのでrandom seedみたいなことをしたいのですが、どうすればいいですかね。<jupyter_code>from tensorflow.keras.layers import Dense, Input
from tensorflow.keras.models import Model
import tensorflow as tf
def predict_nn():
    inputs = Input(shape=(300,))
    outputs = Dense(4, activation='softmax') (inputs)
    model = Model(inputs=inputs, outputs=outputs)
    return model
x_1 = X_train[:1]
X_1_4 = X_train[:4]
model_71 = predict_nn()
model_71.summary()
y_1 = model_71.predict(x_1)
Y_1_4 = model_71.predict(X_1_4)
y_1
Y_1_4<jupyter_output><empty_output><jupyter_text>72. 損失と勾配の計算 文面からは、実際自分でクロスエントロピー損失関数や、勾配を`numpy`や`math`を用いて作成して、作成した関数を用いて求めてほしいのか、71とかで使ったモジュールを用いてやって欲しいのかイマイチわからないのですが、自作はせずにkerasを使ってやります。<jupyter_code>from tensorflow.keras.losses import CategoricalCrossentropy
from tensorflow.keras.backend import gradients
y_train_1 = Y_train[:1]
Y_train_1_4 = Y_train[:4]
CategoricalCrossentropy?
cce = CategoricalCrossentropy()<jupyter_output><empty_output><jupyter_text>$\hat{y}_1$のクロスエントロピー損失は、<jupyter_code>cce(y_train_1,y_1).numpy()<jupyter_output><empty_output><jupyter_text>$\hat{Y}$のクロスエントロピー損失は、<jupyter_code>cce(Y_train_1_4,Y_1_4).numpy()<jupyter_output><empty_output><jupyter_text>損失を求められたので、次は勾配を求めます。実は、`keras`の関数である`keras.backend.gradient`を用いると`RuntimeError`がでます。そのため、今回は`tf.GradientTape()`を用いて求めます。ちなみに、先ほど定義したモデルから、weightの行列を取得し、そのweight行列と`numpy`の`gradient`を用いて計算して、gradientを求めようと思ったのですが、`numpy.gradient`の仕様がよくわからなくてやめました。誰か書ける人がいたら教えてください。<jupyter_code>def calculate_grad(x,y):
    
    def _loss_fn(y_predict, y):
        return cce(y_predict, y)

    inputs = tf.convert_to_tensor(x)
    targets = tf.convert_to_tensor(y)
    with tf.GradientTape() as tape:
        y_predict = model_71(inputs)
        loss = _loss_fn(y_predict, targets)
    grads = tape.gradient(loss, model_71.variables)
    
    return grads
calculate_grad(x_1, y_train_1)
calculate_grad(x_1, y_train_1)[0].shape
calculate_grad(X_1_4, Y_train_1_4)
calculate_grad(X_1_4, Y_train_1_4)[0].shape<jupyter_output><empty_output><jupyter_text>73. 確率的勾配降下法による学習 >確率的勾配降下法（SGD: Stochastic Gradient Descent）を用いて，行列Wを学習せよ.とのことなので、さっきのモデルを用いて、行いたいと思います。<jupyter_code>from tensorflow.keras.optimizers import SGD
model_73 = predict_nn()
%%time
opt = SGD(learning_rate=0.01)
model_73.compile(optimizer=opt, loss='categorical_crossentropy',metrics=['accuracy'])
history = model_73.fit(X_train, Y_train, epochs=100, validation_data=(X_valid, Y_valid))<jupyter_output>Train on 10672 samples, validate on 1334 samples
Epoch 1/100
10672/10672 [==============================] - 0s 46us/sample - loss: 1.1910 - accuracy: 0.6330 - val_loss: 1.0789 - val_accuracy: 0.7534
Epoch 2/100
10672/10672 [==============================] - 0s 26us/sample - loss: 1.0184 - accuracy: 0.7687 - val_loss: 0.9790 - val_accuracy: 0.7631
Epoch 3/100
10672/10672 [==============================] - 0s 25us/sample - loss: 0.9390 - accuracy: 0.7792 - val_loss: 0.9151 - val_accuracy: 0.7706
Epoch 4/100
10672/10672 [==============================] - 0s 26us/sample - loss: 0.8838 - accuracy: 0.7797 - val_loss: 0.8668 - val_accuracy: 0.7691
Epoch 5/100
10672/10672 [==============================] - 0s 26us/sample - loss: 0.8411 - accuracy: 0.7795 - val_loss: 0.8282 - val_accuracy: 0.7691
Epoch 6/100
10672/10672 [==============================] - 0s 26us/sample - loss: 0.8065 - accuracy: 0.7803 - val_loss: 0.7964 - val_accuracy: 0.7661
Epoch 7/100
10672/10672 [==========================[...]<jupyter_text>74.正解率の計測<jupyter_code>score = model_73.evaluate(X_test, Y_test, verbose=0)
print('Test loss:{0}'.format(score[0]))
print('Test accuracy:{0}'.format(score[1]))<jupyter_output>Test loss:0.41655058169114717
Test accuracy:0.8605697154998779<jupyter_text>75. 損失と正解率のプロット Kerasでは、`Callback`というクラスがあり、それを使うことによって、訓練中に関数を適用させることができます。<jupyter_code>import os
import matplotlib.pyplot as plt
import japanize_matplotlib
from IPython.display import clear_output
from tensorflow.keras.callbacks import Callback
class RealTimePlot(Callback):
    def __init__(self, name=False, absolute=False, log_scale=False, path=os.getcwd() ,save=False):
        super().__init__()
        self.name = name
        self.absolute = absolute
        self.log_scale = log_scale
        self.path = path
        self.save = save

    def on_train_begin(self, logs={}):
        self.i = 0
        self.x = []
        self.losses = []
        self.val_losses = []
        self.acc = []
        self.val_acc = []
        self.fig = plt.figure()
        
        self.logs = []

    def on_epoch_end(self, epoch, logs={}):
        
        self.logs.append(logs)
        self.x.append(self.i)
        self.losses.append(logs.get('loss'))
        self.val_losses.append(logs.get('val_loss'))
        self.acc.append(logs.get('accuracy'))
        self.val_acc.append(logs.get('val_accuracy'))
        self.i += 1
        
        fig, (ax1, ax2) = plt.subplots(1, 2, sharex=True, figsize=(10,5))
        clear_output(wait=True)
        
        # fig全体の設定
        if self.name:
            fig.suptitle('{0}の学習'.format(self.name))
            
        
        # fig内のax1(サブグラフ)の設定
        if self.log_scale:
            ax1.set_yscale('log')
            
        ax1.plot(self.x, self.losses, label="loss")
        ax1.plot(self.x, self.val_losses, label="val_loss")
        ax1.legend()
        
        # fig内のax2(サブグラフ)の設定
        if self.absolute:
            ax2.set_ylim(0.6, 0.98)

        ax2.plot(self.x, self.acc, label="accuracy")
        ax2.plot(self.x, self.val_acc, label="val_accuracy")
        ax2.legend()
        plt.show();
        
    def on_train_end(self, logs={}):
        fig, (ax1, ax2) = plt.subplots(1, 2, sharex=True, figsize=(10,5))
        clear_output(wait=True)
        
        # fig全体の設定
        if self.name:
            fig.suptitle('{0}の学習'.format(self.name))
            
        
        # fig内のax1(サブグラフ)の設定
        if self.log_scale:
            ax1.set_yscale('log')
    
        ax1.plot(self.x, self.losses, label="loss")
        ax1.plot(self.x, self.val_losses, label="val_loss")
        ax1.legend()
        
        # fig内のax2(サブグラフ)の設定
        if self.absolute:
            ax2.set_ylim(0.6, 0.98)

        ax2.plot(self.x, self.acc, label="accuracy")
        ax2.plot(self.x, self.val_acc, label="val_accuracy")
        ax2.legend()
        
        if self.save:
            output_fig_name = os.path.join(self.path, self.name + '.png')
            plt.savefig(output_fig_name)
        plt.show();
real_time_plot = RealTimePlot(name='75番')
model_75 = predict_nn()
%%time
opt = SGD(learning_rate=0.01)
model_75.compile(optimizer=opt, loss='categorical_crossentropy',metrics=['accuracy'])
history = model_75.fit(X_train, Y_train, epochs=100,
                    validation_data=(X_valid, Y_valid),
                    callbacks=[real_time_plot])<jupyter_output><empty_output><jupyter_text>lossもvalidation lossも同じように下がっているのが分かります。しかし奇妙な動きをしてますね。。。  76.チェックポイント こちらも75と同様に、`Callbacks`を用います。<jupyter_code>import os
from tensorflow.keras.callbacks import ModelCheckpoint
# configs
MODEL_DIR = './ch8_folder/models'
if not os.path.exists(MODEL_DIR):  # ディレクトリが存在しない場合、作成する。
    os.makedirs(MODEL_DIR)
checkpoint = ModelCheckpoint(filepath=os.path.join(MODEL_DIR, "model-{epoch:02d}.h5"), verbose=0,save_weights_only=True, save_freq=1)
model_76 = predict_nn()<jupyter_output><empty_output><jupyter_text>先ほどまでエポックは100で行っていましたが、パラメータをたくさん出力したくないため、エポックを10で行います。<jupyter_code>opt = SGD(learning_rate=0.01)
model_76.compile(optimizer=opt, loss='categorical_crossentropy',metrics=['accuracy'])
history = model_76.fit(X_train, Y_train, epochs=10,
                    validation_data=(X_valid, Y_valid),
                    callbacks=[checkpoint])<jupyter_output>Train on 10672 samples, validate on 1334 samples
Epoch 1/10
10672/10672 [==============================] - 2s 181us/sample - loss: 1.2007 - accuracy: 0.6371 - val_loss: 1.0789 - val_accuracy: 0.7444
Epoch 2/10
10672/10672 [==============================] - 2s 150us/sample - loss: 1.0174 - accuracy: 0.7623 - val_loss: 0.9774 - val_accuracy: 0.7609
Epoch 3/10
10672/10672 [==============================] - 2s 149us/sample - loss: 0.9374 - accuracy: 0.7718 - val_loss: 0.9133 - val_accuracy: 0.7624
Epoch 4/10
10672/10672 [==============================] - 2s 151us/sample - loss: 0.8824 - accuracy: 0.7756 - val_loss: 0.8652 - val_accuracy: 0.7624
Epoch 5/10
10672/10672 [==============================] - 2s 149us/sample - loss: 0.8398 - accuracy: 0.7765 - val_loss: 0.8267 - val_accuracy: 0.7654
Epoch 6/10
10672/10672 [==============================] - 2s 148us/sample - loss: 0.8053 - accuracy: 0.7772 - val_loss: 0.7950 - val_accuracy: 0.7661
Epoch 7/10
10672/10672 [===========================[...]<jupyter_text>これで`MODEL_DIR`内にモデルが生成されていると思います。  77.ミニバッチ化 kerasをもちいたミニバッチ化は簡単です。`model.fit`のオプション変数である、`batch_size`で変更が可能です。問題ではBの値を1,2,4,8,...と書いてあるのですが、1,8,16あたりでやってみます。<jupyter_code>model_77_1 = predict_nn()
real_time_plot = RealTimePlot()
%%time
opt = SGD(learning_rate=0.01)
model_77_1.compile(optimizer=opt, loss='categorical_crossentropy',metrics=['accuracy'])
history = model_77_1.fit(X_train, Y_train, epochs=100, batch_size=1,
                    validation_data=(X_valid, Y_valid),
                    callbacks=[real_time_plot])
score1 = model_77_1.evaluate(X_test, Y_test, verbose=0)
print('Test loss:{0}'.format(score1[0]))
print('Test accuracy:{0}'.format(score1[1]))
model_77_8 = predict_nn()
%%time
opt = SGD(learning_rate=0.01)
model_77_8.compile(optimizer=opt, loss='categorical_crossentropy',metrics=['accuracy'])
history = model_77_8.fit(X_train, Y_train, epochs=100, batch_size=8,
                    validation_data=(X_valid, Y_valid),
                    callbacks=[real_time_plot])
score2 = model_77_8.evaluate(X_test, Y_test, verbose=0)
print('Test loss:{0}'.format(score2[0]))
print('Test accuracy:{0}'.format(score2[1]))
model_77_16 = predict_nn()
%%time
opt = SGD(learning_rate=0.01)
model_77_16.compile(optimizer=opt, loss='categorical_crossentropy',metrics=['accuracy'])
history = model_77_16.fit(X_train, Y_train, epochs=100, batch_size=16,
                    validation_data=(X_valid, Y_valid),
                    callbacks=[real_time_plot])
score3 = model_77_16.evaluate(X_test, Y_test, verbose=0)
print('Test loss:{0}'.format(score3[0]))
print('Test accuracy:{0}'.format(score3[1]))<jupyter_output>Test loss:0.36266527352840644
Test accuracy:0.8785607218742371<jupyter_text>78.GPU上での学習 こちらは手元の環境ではできないので行いません。  79.多層ニューラルネットワーク 先ほどまでは一層のニューラルネットワークをさわってたわけですが、今度は多層のニューラルネットとかを作ってみましょう。今までの結果から、- epochが100のままだと過適合しそう- バッチサイズは小さいとよくなさそうなので、16ぐらいでみたいが見えるので、そこら辺を意識しながらパラメータを編集します。<jupyter_code>from tensorflow.keras.layers import Dropout
def many_nn():
    inputs = Input(shape=(300,))
    x = Dense(200, activation='relu')(inputs)
    x = Dropout(0.5)(x)
    outputs = Dense(4, activation='softmax') (x)
    model = Model(inputs=inputs, outputs=outputs)
    return model
model_79 = many_nn()
%%time
opt = SGD(learning_rate=0.01)
model_79.compile(optimizer=opt, loss='categorical_crossentropy',metrics=['accuracy'])
history = model_79.fit(X_train, Y_train, epochs=45, batch_size=16,
                    validation_data=(X_valid, Y_valid),
                    callbacks=[real_time_plot])
score4 = model_79.evaluate(X_test, Y_test, verbose=0)
print('Test loss:{0}'.format(score4[0]))
print('Test accuracy:{0}'.format(score4[1]))<jupyter_output>Test loss:0.2823952407851212
Test accuracy:0.9002998471260071
{'fraction_non_alphanumeric': 0.15837950138504156, 'fraction_numerical': 0.06440443213296398, 'mean_word_length': 4.437123493975903, 'pattern_counts': {'":': 0, '<': 56, '<?xml version=': 0, '>': 58, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '292791', 'license': 'MIT', 'n_tokens_mistral': 6724, 'n_tokens_neox': 6044, 'n_words': 782, 'path': 'ch8.ipynb', 'repo_name': 'reonyanarticle/100pon_nock_2020', 'size': '14440'}