<jupyter_start><jupyter_text>Learning Word Vectors with TensorFlow: Optimizer Selection Stochastic Gradient Descent (with Momentum) *<NAME> (<EMAIL>)*This notebook explores (with Momentum optimizer):1. Learning rates2. Momentum settingsA learning rate of 0.01 with an aggressive momentum of 0.95 does best. A higher learning rate of 0.1 converges in fewer epochs, but consistently has worse training and validation loss.SGD with momentum consistently achieves better overall performance than RMSProp in this application. With both faster training/validation loss convergence. And, arriving a better solution (i.e. lower training and validation loss).<jupyter_code>from wordvector import WordVector
from windowmodel import WindowModel
import docload
from plot_util import plot_results

import numpy as np
import sklearn.utils
files = ['../data/adventures_of_sherlock_holmes.txt',
        '../data/hound_of_the_baskervilles.txt',
        '../data/sign_of_the_four.txt']
word_array, dictionary, num_lines, num_words = docload.build_word_array(
    files, vocab_size=50000, gutenberg=True)
print('Document loaded and processed: {} lines, {} words.'
      .format(num_lines, num_words))
x, y = WindowModel.build_training_set(word_array)

# shuffle and split 10% validation data
x_shuf, y_shuf = sklearn.utils.shuffle(x, y, random_state=0)
split = round(x_shuf.shape[0]*0.9)
x_val, y_val = (x_shuf[split:, :], y_shuf[split:, :])
x_train, y_train = (x[:split, :], y[:split, :])<jupyter_output><empty_output><jupyter_text>Learning Rate and Momentumlearn_rate = {0.0001, 0.001, 0.01}  momentum = {0.8, 0.9, 0.95}<jupyter_code>results_list = []
count = 0
for learn_rate in [0.0001, 0.001, 0.01]:
    for momentum in [0.8, 0.9, 0.95]: 
        print('{}) learn_rate = {}, momentum = {}'
              .format(count, learn_rate, momentum))
        count += 1
        graph_params = {'batch_size': 32,
                        'vocab_size': np.max(x)+1,
                        'embed_size': 128,
                        'hid_size': 128,
                        'neg_samples': 64,
                        'learn_rate': learn_rate,
                        'momentum': momentum,
                        'embed_noise': 0.1,
                        'hid_noise': 0.1,
                        'optimizer': 'Momentum'}
        model = WindowModel(graph_params)
        results = model.train(x_train, y_train, x_val, y_val, epochs=80, verbose=False)
        results_list.append((graph_params, results))
plot_results(results_list)<jupyter_output><empty_output><jupyter_text>Check Larger Learning Rates**First round found best learn rate was 0.01, another round to check even larger learning rates**  learn_rate = {0.1, 1, 10}  momentum = {0.8, 0.9, 0.95}<jupyter_code>results_list2= []
count = 0
for learn_rate in [0.1, 1, 10]:
    for momentum in [0.8, 0.9, 0.95]: 
        print('{}) learn_rate = {}, momentum = {}'
              .format(count, learn_rate, momentum))
        count += 1
        graph_params = {'batch_size': 32,
                        'vocab_size': np.max(x)+1,
                        'embed_size': 128,
                        'hid_size': 128,
                        'neg_samples': 64,
                        'learn_rate': learn_rate,
                        'momentum': momentum,
                        'embed_noise': 0.1,
                        'hid_noise': 0.1,
                        'optimizer': 'Momentum'}
        model = WindowModel(graph_params)
        results = model.train(x_train, y_train, x_val, y_val, epochs=80, verbose=False)
        results_list2.append((graph_params, results))
plot_results(results_list2)<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.11591033351558229, 'fraction_numerical': 0.034718425369054125, 'mean_word_length': 2.6700100300902707, 'pattern_counts': {'":': 0, '<': 15, '<?xml version=': 0, '>': 15, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '3', 'dir': 'jupyter-structured-clean-dedup', 'id': '567741', 'license': 'MIT', 'n_tokens_mistral': 1234, 'n_tokens_neox': 1115, 'n_words': 292, 'path': 'src/tune_sherlock3.ipynb', 'repo_name': 'pat-coady/word2vec', 'size': '3677'}