<jupyter_start><jupyter_text>Observing the effect of learning rateAfter observing the effect of number of hidden units, in this part the effect of learning rate is observed. Here in the experiments, for each learning rate value, a different hyper-parameter set has been used to see whether the model can be trained with such a rate or not. Necessary ImportsThe used libraries were identical to the ones that are used for the previous part. In addition to personal impelementations *ann.py* and *dataset_setup.py* numpy and matplotlib are the libraries used.<jupyter_code>import numpy as np
import matplotlib.pyplot as plt
from ann import *
from dataset_setup import *<jupyter_output><empty_output><jupyter_text>Importing the dataHere the dataset that is going to be used is retreved from the data files that already exist. For importing and normalizing the data, the utilities provided in *Dataset* class are used.<jupyter_code>file_paths = ['data/train1.txt', 
              'data/test1.txt', 
              'data/train2.txt', 
              'data/test2.txt']
dataset_1 = Dataset(file_paths[0], file_paths[1])
dataset_2 = Dataset(file_paths[2], file_paths[3])<jupyter_output><empty_output><jupyter_text>Helper FunctionsIn order to plot and demonstrate the results obtained for the trials made, the following helper functions are used. The explanations regarding these functions are provided in the file *Finding a Configuration for Each Dataset (1 - A).ipynb*.<jupyter_code>def draw_uniform_samples(data, no_of_samples):
    max_values = np.max(data, axis=0)
    min_values = np.min(data, axis=0)
    dist = np.random.uniform(min_values[0], max_values[0], no_of_samples)
    return dist
def plot_losses(losses):
    epoch_count = len(losses)
    plt.plot(range(1, epoch_count + 1), losses)
    plt.title("Loss vs Epoch Count", fontsize=14)
    plt.ylabel("Loss", fontsize=10)
    plt.xlabel("Epoch", fontsize=10)
def plot_predictions(samples, dist_count, model, title, data_label, loc):
    dist = draw_uniform_samples(samples, dist_count)
    dist = np.reshape(dist, (len(dist), 1))
    y_pred = model.predict(dist)
    sort = list(sorted(zip(dist, y_pred)))
    dist, y_pred = zip(*sort)
    plt.title(title, fontsize=14)
    plt.xlabel("x", fontsize=10)
    plt.ylabel("y", fontsize=10)
    plt.scatter(samples[:, 0], samples[:, 1], label=data_label)
    plt.plot(dist, y_pred, label="Learned model", c="red", lw=2.5)
    plt.legend(loc=loc, borderaxespad=1.0)
def get_loss(data, model):
    labels = data[:, 1]
    samples = data[:, 0]
    samples = np.reshape(samples, (len(samples), 1))
    y_pred = model.predict(samples)
    print("Standard Deviation of loss: {:.4f}".format(np.std(model.loss(labels, y_pred, stage="forward"), axis=0)))
    loss = np.sum(model.loss(labels, y_pred, stage="forward"), axis=0) / len(labels)
    return loss
normalized_train_2, normalized_test_2 = dataset_2.normalize_data()<jupyter_output><empty_output><jupyter_text>Learning rate: 1With a learning rate this high, the model is not expected to learn any meaningful information since it does not have the ability of healthily moving towards a global optimum point in terms of gradients. Due to this behavior it is expected from the model to flunctuate between loss values that are far from 0. The loss curve plotted for this experiment also proves this finding. The loss curve and the learned data distribution is given below.<jupyter_code>lr_1_model = ANN(input_dim=1, weight_range=0.0003, loss="mse", hidden_layer_enabled=True, hidden_units=8, activation="sigmoid")

lr_1_losses = lr_1_model.fit(normalized_train_2, 100, learning_rate=1, update="batch", momentum_enabled=True, alpha=0.1, threshold=0.9)

plot_losses(lr_1_losses)
plot_predictions(normalized_train_2, 500, lr_1_model, "Model with lr=1 on Dataset 2", "Training data", "lower left")
print("Training Loss: {:.4f}".format(get_loss(normalized_train_2, lr_1_model)))
print("Threshold: 0.9, Epoch count: {}".format(len(lr_1_losses)))<jupyter_output>Threshold: 0.9, Epoch count: 56<jupyter_text>Learning Rate: 0.1By lowering the learning rate to 0.1, a value that is suitable for batch learning achieved. Since for each update, batch learning performs one update with such a value convergence is achieved. Comparing with the previous experiment, the ability that using a lower learning rate gives to the model in terms of finding an optimum set of gradients is observed. Originating from the experiments conducted in part c, the threshold for convergence is set around 0.1 as it is the point where optimal model with 8 hidden units achieve. The loss curve and the learned data distributipn is given below.<jupyter_code>lr_0_1_model = ANN(input_dim=1, weight_range=0.0001, loss="mse", hidden_layer_enabled=True, hidden_units=8, activation="tanh")

lr_0_1_losses = lr_0_1_model.fit(normalized_train_2, 10000, learning_rate=0.1, update="batch", momentum_enabled=True, alpha=0.1, threshold=0.11)

plot_losses(lr_0_1_losses)
plot_predictions(normalized_train_2, 500, lr_0_1_model, "Model with lr=0.1 on Dataset 2", "Training data", "lower left")
print("Training Loss: {:.4f}".format(get_loss(normalized_train_2, lr_0_1_model)))
print("Threshold value: 0.11, No of epochs: {}".format(len(lr_0_1_losses)))<jupyter_output>Threshold value: 0.11, No of epochs: 6173<jupyter_text>Learning rate: 0.01By even lowering the learning rate, a rate that is suitable for Stochastic learning is found. Since SGD algorithm moves along the gradient space with smaller updates in purpose, this type of a learning rate would be more suitable. After initially training the model by 10000 epochs and together with the experiments performed in part c, 0.25 is found as a suitable threshold. However with such a learning rate SGD algorithm gets stuck at a local minima due to the relatively large value of learning rate. This is the main reason why learning stops at such an early stage. The loss curves and the learned data distribution is given below.<jupyter_code>lr_0_01_model = ANN(input_dim=1, weight_range=0.003, loss="mse", hidden_layer_enabled=True, hidden_units=8, activation="tanh")

lr_0_01_losses = lr_0_01_model.fit(normalized_train_2, 8000, learning_rate=0.01, update="sgd", momentum_enabled=True, alpha=0.4, threshold=0.25)

plot_losses(lr_0_01_losses)
plot_predictions(normalized_train_2, 500, lr_0_01_model, "Model with lr=0.01 on Dataset 2", "Training data", "lower left")
print("Training Loss: {:.4f}".format(get_loss(normalized_train_2, lr_0_01_model)))
print("Threshold: 0.2, Epoch count: {}".format(len(lr_0_01_losses)))<jupyter_output>Threshold: 0.2, Epoch count: 73<jupyter_text>Learning Rate: 0.001Following the discussion regarding learning rate 0.01 for SGD learning algorithm, by even loweing the learning rat, an algorithm that is able to bypass local minimas can be found. By keeping the learning rate not too high or not too low consdiering the learning algorithm, a model that fits well to the give data distribution can be found. The loss curve and the learned data distribution is given below.<jupyter_code>lr_0_001_model = ANN(input_dim=1, weight_range=0.001, loss="mse", hidden_layer_enabled=True, hidden_units=8, activation="tanh")

lr_0_001_losses = lr_0_001_model.fit(normalized_train_2, 10000, learning_rate=0.001, update="sgd", momentum_enabled=True, alpha=0.3, threshold=0.1)

plot_losses(lr_0_001_losses)
plot_predictions(normalized_train_2, 500, lr_0_001_model, "Model with lr=0.001 on Dataset 2", "Training data", "lower left")
print("Training Loss: {:.4f}".format(get_loss(normalized_train_2, lr_0_001_model)))
print("Threshold: 0.10, Epoch count: {}".format(len(lr_0_001_losses)))<jupyter_output>Threshold: 0.10, Epoch count: 8664<jupyter_text>Learning Rate: 0.0001 As a learning rate too small, the value 0.0001 has been tried out. As it can be seen from the change of loss value ang epochs, the learning confinues in an extremely stable way but it is considerably smaller than the other alternatives. The loss curve and the learned data distribution is given below.<jupyter_code>lr_0_0001_model = ANN(input_dim=1, weight_range=0.001, loss="mse", hidden_layer_enabled=True, hidden_units=8, activation="tanh")

lr_0_0001_losses = lr_0_0001_model.fit(normalized_train_2, 50000, learning_rate=0.0001, update="sgd", momentum_enabled=True, alpha=0.3, threshold=0.8)

plot_losses(lr_0_0001_losses)
print("Training Loss: {:.4f}".format(get_loss(normalized_train_2, lr_0_0001_model)))
plot_predictions(normalized_train_2, 500, lr_0_0001_model, "Model with lr=0.001 on Dataset 2", "Training data", "lower left")
print("Threshold: 0.8, Epoch Count: {}".format(len(lr_0_0001_losses)))<jupyter_output>Threshold: 0.8, Epoch Count: 6286
{'fraction_non_alphanumeric': 0.09803244735933725, 'fraction_numerical': 0.042802899551259925, 'mean_word_length': 6.021001615508886, 'pattern_counts': {'":': 0, '<': 28, '<?xml version=': 0, '>': 28, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '93663', 'license': 'Apache-2.0', 'n_tokens_mistral': 2993, 'n_tokens_neox': 2760, 'n_words': 910, 'path': 'HW2/Effect of learning rate (1-D).ipynb', 'repo_name': 'yusufdalva/CS550_Assignments', 'size': '8691'}