The SGD algorithm has two major problems. First, the value of \(\eta \)  has to be chosen manually. If it is set too high, then the network may not converge and if it is set too low, convergence will be very slow. Second, all parameters are updated with the same learning rate. However, different parameters in a neural network might need different learning rates to be optimized properly. To solve these problems, several adaptive optimization algorithms are proposed such as Adagrad [1]}, RMSprop (unpublished work by Geoffrey Hinton, 2012), and Adam [2]}. These optimization algorithms adapt the learning rate \(\eta \)  for each parameter differently during training. We have used Adagrad and Adam optimization algorithms for our models.
