<jupyter_start><jupyter_text>Scaling your Deep Learning TrainingWelcome to the third lab of the course. So far, you've learned about stochastic gradient descent and the effects of batch size on training. You've also learned how to scale training from a single GPU to multiple GPUs. You should have seen a considerable speedup that we'll be measuring in more detail.You've seen that as we scale to multiple GPUs, we effectively increase the batch size. From our testing in lab one, we know that increasing the batch size can increase throughput, but it can also affect a network's ability to train effectively. In this lab, you'll be exploring how training is affected when scaling to multiple GPUs. We will start with measuring the effect of multiple GPUs on throughput, in our case images per second. You'll then go on to learn tools and techniques that can help to maintain high accuracy as you scale.  Scaling of Performance: Images/Second In the first lab, you evaluated images/second as a measurement of performance of the training. We saw that throughput increased with batch size, but only to a certain point. As the batch size increased to saturate the GPU memory, we saw diminishing returns. Now that we are scaling to multiple GPUs, it would make sense that our throughput would increase once again. How do you think throughput will scale with respect the the number of GPUs used in the training?As in Lab 1, we have a callback that measures throughput in images/second after each epoch. Run the cell below __on one GPU__ and record the throughput in the code cell down below. As you might recall from Lab 1, throughput in the first epoch should be ignored because we are paying one-time costs, so choose a throughput from the second or third epoch.<jupyter_code>!mpirun -np 1 python fashion_mnist.py --epochs 3 --batch-size 128<jupyter_output>2022-01-19 12:23:08.647365: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2022-01-19 12:23:10.047071: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2022-01-19 12:23:10.048023: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcuda.so.1
2022-01-19 12:23:10.210998: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:1038] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
2022-01-19 12:23:10.212288: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1746] Found device 0 with properties: 
pciBusID: 0000:00:1b.0 name: Tesla V100-SXM2-16GB computeCapability: 7.0
coreClock: 1.53GHz coreCount: 80 deviceMemorySize: 15.78GiB deviceMemoryBandwidth: 836.37GiB/s
2022-01-19 12:23:10.212378: I tensorflow/stream_executor/cuda/cuda_gpu[...]<jupyter_text>Next, alter the training to run __on all GPUs__. You may notice that throughput will fluctuate between epochs. Pick a throughput value that seems closest to the average, or feel free to calculate the average of a few epochs. Remember not to include the first epoch's value.<jupyter_code>!nvidia-smi
num_gpus = 4
!mpirun -np $num_gpus python fashion_mnist.py --epochs 5 --batch-size 128<jupyter_output>2022-01-19 12:27:14.530147: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2022-01-19 12:27:14.530152: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2022-01-19 12:27:14.530149: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2022-01-19 12:27:14.530150: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2022-01-19 12:27:15.994916: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2022-01-19 12:27:15.995399: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2022-01-19 12:27:15.995534: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2022-01-[...]<jupyter_text>Record the values you obtained in the cell below, and then execute it to see the scaling ratio. Think about the number you get for a minute, and then reveal the block below by clicking on the three dots.<jupyter_code>one_gpu_throughput = 1401
multi_gpu_throughput = 5236

print("Multi GPU speedup factor: {}".format(multi_gpu_throughput/one_gpu_throughput))<jupyter_output>Multi GPU speedup factor: 3.7373304782298358<jupyter_text>You should see scaling at nearly a linear rate. This is a much more promising result than the scaling we saw in Lab 1 when increasing batch size. It stands to reason that each GPU would add a significant boost to the training process. We don't get perfectly linear scaling. A significant component of this is due to communication between the GPUs when updating weights. There can also be other factors, such as waiting for slower GPUs to finish processing before the weights are averaged. But still, this is pretty good.Production DL training at scale is usually benchmarked against the ideal case of linear scaling (N GPUs should be N times faster than 1 GPU). Horovod, and the NCCL library, do a good job of maintaining high throughput, but performance is intricately tied to the hardware in use. The requirement to have high bandwidth and low latency between GPUs on the same node is what led to the development of the NVLink interconnect, which is what the server used for this course deploys for linking up the GPUs on the node. The NVIDIA DGX-2 carried this forward with NVSwitch, a fabric for connecting up to 16 GPUs at a peak bidirectional bandwidth of 300 GB/s.As you scale to more GPUs, multi-node training is required, and further hardware considerations are needed to effectively scale.  Comparing Validation Accuracy Over TimeAs with the first lab, throughput is just one way of measuring performance. It's important to evaluate the training of the network, specifically its ability to effectively make predictions. To show clear comparisons between different trainings, we'll be plotting the validation accuracy of the network over time.To accomplish this you will be saving off validation accuracy, as well as the total elapsed time after each epoch. Each training will create a CSV file, which will be used to plot the trainings against each other. We provide you with a script to plot the training sesssions below.   Implementing the CallbackFirst, let's make a copy of the training script in case you need it as a reference.<jupyter_code>!cp fashion_mnist.py fashion_mnist_original.py<jupyter_output><empty_output><jupyter_text>Next, take a look at `callbacks/save_training_data.py`. Follow the steps to implement the callback in `fashion_mnist.py`. You'll need to pass the callback a name for the CSV file, which will be made up of the hyperparameters that you use in your training. If you have trouble and need help, you can find the solution at `solutions/save_training_data.py`.   Running a Training SessionAfter you've implemented the callback, try training on multiple GPUs. We suggest stopping after a certain amount of time (100-200 seconds). You can do this by clicking the stop button in the top bar (or, Kernel > Interrupt in the menu). You can also use the `--target-accuracy` and `--patience` arguments to stop training as you did in Lab 1. The CSV file will save correctly even if you end the training early.<jupyter_code># Add the training_data folder
!mkdir training_data
!mpirun -np $num_gpus python fashion_mnist.py --batch-size 32 --target-accuracy 0.7<jupyter_output>2022-01-19 12:33:37.156250: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2022-01-19 12:33:37.156250: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2022-01-19 12:33:37.156251: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2022-01-19 12:33:37.156254: I tensorflow/stream_executor/platform/default/dso_loader.cc:49] Successfully opened dynamic library libcudart.so.11.0
2022-01-19 12:33:38.692826: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2022-01-19 12:33:38.692848: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2022-01-19 12:33:38.693056: I tensorflow/compiler/jit/xla_cpu_device.cc:41] Not creating XLA devices, tf_xla_enable_xla_devices not set
2022-01-[...]<jupyter_text>Plotting the DataIf the callback worked, you should have a CSV file with the data in the `training_data` folder. Feel free to navigate to it in the file browser in the pane on the left, where you can open the file. Then look over the code in the cell below. It will plot all the CSV files in the `training_data` folder and use the names in the legend to identify each training run. Once you have a sense of the code, execute the cell to plot your training.<jupyter_code>%matplotlib widget
import os
import numpy as np
import matplotlib.pyplot as plt

# By default we skip the first row, which contains the headers
# By skipping 2 rows, you can disregard the first data-point (0,0) to get a closer look
def plot_trainings(skiprows=1):
    plt.close()
    for filename in os.listdir("training_data"):
        if filename == ".ipynb_checkpoints": continue
        x, y = np.loadtxt("training_data/" + filename, delimiter=',', unpack=True, skiprows=skiprows)
        plt.plot(x,y, label=filename.split('.csv')[0])

    plt.xlabel('Time (s)')
    plt.ylabel('Validation Accuracy')
    plt.title('Training Comparison')
    plt.legend()
    plt.show()
    
plot_trainings()<jupyter_output><empty_output><jupyter_text>Clearing up the DataAs you go through the following exercises in this lab, you may find that the graph will get cluttered and make it hard to compare trainings against each other. When that happens, we suggest that you go into the `training_data` folder and remove files as you see fit. You can clear up everything in the directory with:<jupyter_code># Only run this if you want to delete all training data
!rm -rf training_data/*<jupyter_output><empty_output><jupyter_text>Also note that if you run a second training session with all the same hyperparameters (and thus uses the same name for the CSV file) it will overwrite the original training file.  Adjusting the Optimizer: Adding MomentumSo far, we've modified our training by altering batch size, as well as distributing across multiple GPUs. We'll explore other hyperparameters, such as learning rate later in the lab. In this section, we'll explore another key tool in improving our training process: selecting and tuning the optimizer that we use. As you may recall in Lab 1, the optimizer updates the weights of the network in order to minimize the loss function. In the stochastic gradient descent optimizer that we've used so far, weights are updated based on their gradient with respect to the loss function of a mini-batch. In other words, we determine how altering a weight will affect the loss, and move a small step in the direction that minimizes that loss for the mini-batch. By taking those steps with each back-propagation, we slowly make our way toward a global minima, decreasing our loss, and increasing our accuracy.Though this process works well, it can be improved upon. One downside is that if the network gets near a local minimum or saddle point, the gradient can be quite small, and the training will slow down in that area. The noise introduced by using minibatches might help the model find its way out, but it might take some time. Additionally, there may be areas where the algorithm keeps taking steps in the roughly the same direction. It would be advantageous in those areas if the optimizer helped us take larger steps to move toward the global minimum faster.A good solution to these issues is to use momentum. Instead of the algorithm taking a small independent step each time, adding momentum to our optimizer allows the process to retain a memory of the last several steps. If the weights have been moving in a certain direction on average, momentum will help continue to propel the updates in the same direction. If the training is also fluctuating back and forth, it can smooth out this movement. A decent analogy is a ball rolling down a hill, which will pick up and retain momentum. ImplementationIn this case implementing momentum is quite simple, as we do not have to alter the code of the optimizer. SGD algorithms will often accept a momentum parameter, as is the case in the Keras implementation.Step 1: Alter the argument parser to accept a momentum argument:```Pythonparser.add_argument('--momentum', type=float, default=0.9,                    help='SGD momentum')```Step 2: Pass the momentum into the optimizer:```Pythonopt = tf.keras.optimizers.SGD(lr=args.base_lr, momentum=args.momentum)```Step 3: Update the CSV filename to include the momentum hyperparameter:```Pythondata_filepath = "training_data/{}ranks-{}bs-{}lr-{}m.csv".format(hvd.size(), args.batch_size, args.base_lr, args.momentum)```If you need any help, you can find the solution at `solutions/add_momentum.py`. Training with MomentumRun the training again either for the same amount of time as before, or stopping at a certain accuracy using `--target-accuracy`. After the training completes, execute the plotting script to see the difference between training with and without momentum.<jupyter_code>!mpirun -np $num_gpus python fashion_mnist.py --batch-size 32 --momentum .9
plot_trainings()<jupyter_output><empty_output>
{'fraction_non_alphanumeric': 0.06033679684700824, 'fraction_numerical': 0.04184879971336439, 'mean_word_length': 4.979434447300771, 'pattern_counts': {'":': 1, '<': 29, '<?xml version=': 0, '>': 30, 'https://': 0, 'lorem ipsum': 0, 'www.': 0, 'xml': 0}, 'pii_count': 0, 'substrings_counts': 0, 'word_list_counts': {'cursed_substrings.json': 0, 'profanity_word_list.json': 0, 'sexual_word_list.json': 0, 'zh_pornsignals.json': 0}}
{'chain_length': '8', 'dir': 'jupyter-structured-clean-dedup', 'id': '86442', 'license': 'MIT', 'n_tokens_mistral': 4243, 'n_tokens_neox': 3772, 'n_words': 1856, 'path': 'Lab 3/01_Notebook_Scaling_Training.ipynb', 'repo_name': 'ksetdekov/Fundamentals-of-Deep-Learning-for-Multi-GPUs', 'size': '13955'}