<jupyter_start><jupyter_text>Lecture 3.13: Learning Better Pt. 3[**Lecture Slides**](https://docs.google.com/presentation/d/1gCJQZkepnwXhu-IUAYsWZJrD4eFJDNwuhqyhQMs1P4w/edit?usp=sharing)This lecture, we are going to experiment with batch sizes, learning rates, & optimizers in keras, in an attempt to better understand neural network optimization.**Learning goals:**- examine the effect of batch size on training- compare the effect of learning rates on training- contrast the choice of optimizers- visualize loss curves vs epochs, batches, and time  1. SetupThis notebook uses the keras and tensorflow deep learning libraries. If you haven't already, please follow the setup steps in notebook 3.11 to correctly install these dependencies.  2. Data MungingWe'll use the same banknote authentication dataset to explore neural network optimization hyperparameters. 💸 We load it into a `DataFrame`:<jupyter_code>import pandas as pd

df = pd.read_csv('bank_note.csv')
df.head()<jupyter_output><empty_output><jupyter_text>Our features are scaled and ready to go! 🏋️‍♀️We'll use all 4 features and put them in a feature matrix:<jupyter_code>X = df[['feature_1', 'feature_2', 'feature_3', 'feature_4']].values
y = df['is_fake'].values<jupyter_output><empty_output><jupyter_text>3. Batch Size 3.1 loss vs epochsNow that we understand what `batch_size` means, let's test different flavours of gradient descent on our neural network training.We'll stick to the 2 hidden layers of 6 neurons with ReLU activation from last lecture. Let's wrap the neural network creation and training in one helper function called `.train_neural_network()`. That way, we can iterate through hyperparameters quickly to compare their effects.The function takes a feature matrix, `X`, a label vector, `y`, and optimization hyperparameters. It returns the loss `history` and the training `time`:<jupyter_code>from keras.models import Sequential
from keras.layers import Dense
from timeit import default_timer as timer
import numpy as np
import tensorflow as tf

def train_neural_network(X, y, optimizer='adam', **kwargs):
    # create model
    model = Sequential([
        Dense(6, activation='relu', input_dim=4),
        Dense(6, activation='relu'),
        Dense(1, activation='sigmoid')
    ])
    model.compile(loss='binary_crossentropy', optimizer=optimizer)
    
    # training reproducibility
    np.random.seed(1337)
    tf.random.set_seed(666)
    
    # train and time model
    start = timer()
    history = model.fit(X, y, **kwargs)
    end = timer()
    
    time = end - start
    return history, time<jupyter_output><empty_output><jupyter_text>ℹ️ Note how `**kwargs` prevents us from explicitly listing and passing on all the `.fit()` arguments.Now we can train neural networks with different `batch_size` to compare vanilla gradient descent, stochastic gradient descent, and various sizes of mini-batch gradient descent.This is the first time that we will meet the excellent `namedtuple`: Python is flexible and fast, and therefore has a tendency to end up cluttered with hundreds of complicated nested dictionaries 🐍. One solution is to create custom _classes_ to hold this data. However it's not very "pythonic" make hundreds of dedicated tiny classes for every single data object. Instead, we can use a `namedtuple`. It implements the `tuple` interface, and thus can be instantiated and unpacked easily. However, it also has named fields, so it is readable and safe like a real class. More details about this handy object [here](https://pymotw.com/2/collections/namedtuple.html).We want to group _settings_ and _results_ together, and this happens often in machine learning experiments. `namedtuple` offers a terse and immutable alternative to dictionaries. We therefore create two named tuples:- `Setting`, with fields `batch_size` and `epochs`- `Result`, with fields `batch_size`, `history`, and `time`We also change the `epochs` for each `batch_size` so that the training procedure doesn't take too long. Small batch sizes have more steps per epoch, and can last a while. The setting pairs chosen below aren't special in any way: they were chosen retroactively to keep the total training time under control 👮🏻.Let's train some neural networks!<jupyter_code>from collections import namedtuple

Setting = namedtuple('Setting', ['batch_size', 'epochs'])
settings = [Setting(1, 10), 
            Setting(2, 20), 
            Setting(8, 80), 
            Setting(32, 100), 
            Setting(128, 200), 
            Setting(len(X), 1500)]

Result = namedtuple('Result', ['batch_size', 'history', 'time'])

results = []

for s in settings:
    history, time = train_neural_network(X, y, batch_size=s.batch_size, epochs=s.epochs)
    results.append(Result(s.batch_size, history, time))<jupyter_output><empty_output><jupyter_text>The losses of each neural network are stored under `results`. We can iterate through the list to plot them on the same graph:<jupyter_code>import matplotlib.pyplot as plt
import seaborn as sns

sns.set()
colors = sns.dark_palette("palegreen")

fig = plt.figure(figsize=(8, 6), dpi=120)
ax = fig.add_subplot(111)

for i, r in enumerate(results):
    ax.plot(r.history.history['loss'], label=f'batch_size={r.batch_size}', c=colors[i], lw=3, alpha=0.8)
    
ax.set_xlabel('epochs')
ax.set_ylabel('loss')
ax.set_xlim(left= -5, right=100)
ax.legend(loc = 'upper right', ncol=3);<jupyter_output><empty_output><jupyter_text>🤤 That's a lot of information! Small batch sizes take fewer epochs to converge. In our case, this is mostly due to the fact they have more gradient updates per epochs, even if each step is less precise than bigger batches.🧠 How many steps are in 1 epoch of vanilla gradient descent?🧠 The smaller the batch size, the lower the starting loss value. Why?We learned that stochastic gradient descent has a "bouncy" loss 🏀, but here it is smooth. This is because we are plotting _epochs_ on the x-axis, not individual _steps_. Let's try it out!  3.2 loss vs stepsBy default, keras saves the loss at the end of each _epoch_. If we want it for each _batch_ , we need to write a custom [callback](https://keras.io/api/callbacks/). Remember callbacks are functions used to extend training functionality.We create a `LossPerBatch` callback, which stores the neural network loss at the end of each batch. We're extending the [`Callback`](https://www.tensorflow.org/api_docs/python/tf/keras/callbacks/Callback) abstract base class, meaning we only have to override the methods we're interested in:<jupyter_code>from keras.callbacks import Callback

class LossPerBatch(Callback):
    def on_train_begin(self, logs={}):
        self.history = {}
        self.history['loss'] = []

    def on_batch_end(self, batch, logs={}):
        self.history['loss'].append(logs['loss'])<jupyter_output><empty_output><jupyter_text>We can use the callback by passing it as an argument to `.fit()`. We can repeat the previous experiments, this time recording the loss at each gradient descent step:<jupyter_code>settings = [Setting(1, 10), 
            Setting(2, 20), 
            Setting(8, 80), 
            Setting(32, 100), 
            Setting(128, 200), 
            Setting(len(X), 1500)]

results_per_batch = []

for s in settings:
    history = LossPerBatch()
    _, time = train_neural_network(X, y, batch_size=s.batch_size, epochs=s.epochs, callbacks=[history])
    results_per_batch.append(Result(s.batch_size, history, time))<jupyter_output><empty_output><jupyter_text>The `history` variable stored in our `result` is now the `LossPerBatch` history, so let's visualize it:<jupyter_code>fig = plt.figure(figsize=(8, 6), dpi=120)
ax = fig.add_subplot(111)

for i, r in enumerate(results_per_batch):
    ax.plot(r.history.history['loss'], label=f'batch_size={r.batch_size}', alpha=0.6, lw=1, c=colors[i])
    
ax.set_xlabel('steps')
ax.set_ylabel('loss')
ax.legend(loc = 'upper right', ncol=3);<jupyter_output><empty_output><jupyter_text>This is quite a confusing graph 😟 The stochasticity of some of the loss curves (looking at you `batch_size=1` 😡) has rendered the graph illegible. Let's pick a few batch sizes to compare side by side:<jupyter_code>def compare_batch_sizes(results):
    height = len(results) * 4
    fig = plt.figure(figsize=(8, height), dpi=120)
    for i, r in enumerate(results):
        r = results[i]
        ax = fig.add_subplot(len(results), 1, i+1)
        ax.plot(r.history.history['loss'], label=f'batch_size={r.batch_size}', alpha=0.8, lw=1, c=colors[3])    
        ax.set_xlabel('steps')
        ax.set_ylabel('loss')
        ax.legend()
    
compare_batch_sizes([results_per_batch[0], results_per_batch[3], results_per_batch[5]])<jupyter_output><empty_output><jupyter_text>- The single example updates of stochastic gradient descent create a lot of noise, which seems to slow down its convergence. It struggles to stay close to the global minimum even after reaching it.- Vanilla gradient descent's updates are precise, but the smoothness might get it stuck in local minima for non-convex loss surfaces.- Mini-batch gradient descent has just enough stochasticity to be robust yet fast.The per-step efficiency of vanilla gradient descent is still the best here, and it seems to converge to the same final loss value as the other optimizers. This suggests that the loss surface isn't very "rough" and that bad local minima are only a minor concern for this dataset and neural architecture.  3.3 loss vs timeWe have shown which batch sizes are most efficient per epoch and per step. However, we described earlier that batch size influences the _time_ spent on each step. ⏱ Let's find out which batch size is the _fastest_ to converge. Let's assume that each epoch takes roughly the same amount of time. Since we know the _total_ training time, we can convert the _epochs_ unit to _seconds_. An easy way of converting the units of a regularly spaced vector, is rescaling the values using `np.linspace()`.<jupyter_code>fig = plt.figure(figsize=(8, 6), dpi=120)
ax = fig.add_subplot(111)

for i, r in enumerate(results):
    loss = r.history.history['loss']
    x_time = np.linspace(0, r.time, len(loss))
    ax.plot(x_time, loss, label=f'batch_size={r.batch_size}', lw=2, alpha=0.8)
    
ax.set_xlabel('time / s')
ax.set_ylabel('loss')
ax.set_xlim(left= -0.5, right=6)
ax.legend(loc = 'upper right', ncol=3);<jupyter_output><empty_output><jupyter_text>Interestingly, the fastest batch size is neither 1 or 1372. It seems that 128 examples per gradient update is the right balance between the step speed and accuracy to converge the quickest.Of course speed isn't the only factor, and one should also pick hyperparameters which find 'good' minima. In this example however, all these settings seem to converge well.ℹ️ Next lecture we will cover GPU & parallelisation in machine learning. This will speed up training even further and change the relationship between batch size and training time. Remember to choose the best and fastest settings for a given dataset, task, and _hardware_.  4. Learning RatesNow that we have a feel of what batch size is best for our optimization problem, let's investigate the effects of the _learning rate_ on training. We reuse the `train_neural_network()` helper function to iterate through a range of learning rates. The learning rate must be set on an [`Optimizer`](https://keras.io/api/optimizers/), and provided as argument to the model compilation step.We use `SGD` here, because we want to showcase the effects of a constant learning rate on gradient descent. Adaptive learning rate optimizers like `adam` will change the learning rate throughout the training, and muddle our results.We modify our `Result` namedtuple to hold the `learning_rate` instead of the `batch_size`:<jupyter_code>from keras.optimizers import SGD

learning_rates = [0.001, 0.01, 0.1, 1, 10]
Result = namedtuple('Result', ['learning_rate', 'history', 'time'])

results = []

for lr in learning_rates:
    optimizer = SGD(learning_rate=lr)
    history, time = train_neural_network(X, y, optimizer=optimizer, batch_size=32, epochs=100)
    results.append(Result(lr, history, time))<jupyter_output><empty_output><jupyter_text>We can now compare our results:<jupyter_code>fig = plt.figure(figsize=(8, 6), dpi=120)
ax = fig.add_subplot(111)

for i, r in enumerate(results):
    ax.plot(r.history.history['loss'], label=r.learning_rate, alpha=0.8, lw=3, c=colors[i])
    
ax.set_xlabel('epochs')
ax.set_ylabel('loss')
ax.legend(loc = 'upper right', ncol=6);<jupyter_output><empty_output><jupyter_text>This resembles the graph shown in the lecture slides:- 0.001 is too slow- 10 is too high, and fails to find the loss surface minimum- 1 minimizes the loss the fastest  These profiles are typical: one step size hits the right balance between convergence quality and speed.ℹ️ Note however that these _values_ aren't very common. For more non-convex loss surfaces, learning rates $< 0.001$ are more the norm. However, just like batch size, this is a _hyperparameter_ that must be tuned to the task at hand.Keep in mind that these experiments were run with the same `batch_size` and `epochs`. Therefore the time spent per epoch is the same for all these curves (see keras logs). The steepest curve on this graph is therefore also the fastest in seconds.  5. OptimizersIn the lecture slides, we understood the principles of momentum and adaptive learning rates behind the ✨**adam**✨ optimizer. We also explained that there _many_ other optimizers, some of which might be better suited to our learning task. Let's try _all_ of keras' optimizers and see which minimizes our loss function the best.We'll want to peek at the loss per epoch and per step. We therefore update our `Result` namedtuple to hold both, so we don't have to run the experiments twice. As pointed out last lecture, the `optimizer` is given to the `.compile()` method.<jupyter_code>optimizers = ['sgd', 'rmsprop', 'adam', 'adadelta', 'adagrad', 'adamax', 'nadam']
Result = namedtuple('Result', ['optimizer', 'history', 'history_per_batch', 'time'])

results = []

for o in optimizers:
    history_per_batch = LossPerBatch()
    history, time = train_neural_network(X, y, optimizer=o, batch_size=32, epochs=100, callbacks=[history_per_batch])
    results.append(Result(o, history, history_per_batch, time))
fig = plt.figure(figsize=(8, 6), dpi=120)
ax = fig.add_subplot(111)

for r in results:
    ax.plot(r.history.history['loss'], label=r.optimizer, alpha=0.8)
    
ax.set_xlabel('epochs')
ax.set_ylabel('loss')
ax.legend(loc = 'upper right', ncol=3);<jupyter_output><empty_output><jupyter_text>There is a wide range of convergence speeds. - the constant learning rate `sgd` is far behind, not having converged by 100 epochs- `nadam` leads with an impressive ~ 30 epochs to convergence 🏎 - `adam` also seems to be a solid  optimizer in 3rd placeWe'd like to see the loss per-step, but we anticipate those curves to be noisy. So let's pick three to compare side by side:- regular ol' `sgd` (mini-batch gradient descent with constant learning rate) 👴- the messiah, `adam` 😇- the prophet's nemesis, `nadam` 😈<jupyter_code>def compare_optimizers(results):
    height = len(results) * 4
    fig = plt.figure(figsize=(8, height), dpi=120)
    for i, r in enumerate(results):
        r = results[i]
        ax = fig.add_subplot(len(results), 1, i+1)
        ax.plot(r.history_per_batch.history['loss'], label=r.optimizer, alpha=0.8, lw=1, c=colors[3])    
        ax.set_xlabel('steps')
        ax.set_ylabel('loss')
        ax.legend()
    
compare_optimizers([results[0], results[2], results[6]])<jupyter_output><empty_output><jupyter_text>`adam` might have lost this race, but we can clearly see that advanced optimizers speed up loss function minimization. 🎊In fact, notice that the adaptive learning rate methods also reduce the stochasticity later in the training, meaning they also improve the _quality_ of the optimization.The analysis above only proves that `nadam` is the fastest optimizer per _epoch_ and per _step_. Indeed, one of these methods could take longer to_calculate, and result in a longer total training _time_. 💪💪 Plot the loss curves of each optimizer versus training time.- see section 3.3 for an example- the graph is the unit test 🙃<jupyter_code># INSERT YOUR CODE HERE<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.09337496938525594, 'fraction_numerical': 0.014388929708547637, 'mean_word_length': 4.779547062986554, 'pattern_counts': {'":': 0, '<': 62, '<?xml version=': 0, '>': 61, 'https://': 5, 'lorem ipsum': 0, 'www.': 1, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '15', 'dir': 'jupyter-structured-clean-dedup', 'id': '77062', 'license': 'CC-BY-4.0', 'n_tokens_mistral': 5041, 'n_tokens_neox': 4762, 'n_words': 1953, 'path': 'data_analysis/3.13_learning_better_pt.3/learning_better_pt.3.ipynb', 'repo_name': 'camille-vanhoffelen/modern-ML-engineer', 'size': '16332'}