This question is part of a sample exam that I'm working on to prepare for the real one. I've been stuck on this one for quite a while and can't really motivate my answers since it keeps referring to guarantees, can someone explain what guarantees we can have in a neural network such as the one described?Network description:Assume a deep neural network (DNN) that takes as input images images of handwritten digits (e.g., MNIST). The input network consists of 28x28 = 784 input units, a hidden layer of logistic units, and a softmax group of 10 units as the output layer. The loss function is the cross-entropy. We have many training cases and we always compute our weight update based on the entire training set, using the error backpropagation algorithm. We use a learning rate thats small enough for all practical purposes, but not so small that the network doesnt learn. We stop when the weight update becomes zero. For each of the following questions, answer yes or no, and explain very briefly.Questions:Will this DNN configuration enable the the weights to minimize the loss value (there may be multiple global optima)?Does this DNN configuration guarantee that the loss reduces on every step? We use our network to classify images by simply seeing which of the 10 output units gets the largest probability when the network is presented with the image, and declaring the number of that output unit to be the networks guessed label.Does this DNN configuration guarantee that the number of mistakes that the network makes on training data (by following this classification strategy) never increases?
Neural Network accuracy and loss guarantees?
machine learning;neural network
For my answers, I assume you are talking about batch (not mini-batch or stochastic) gradient descent.No. Assume you initialize all weights with the same value. Then all gradients (in the same layer) will be the same. Always. Hence the network effectively only learns one parameter per layer. It is possible (and likely) that this is neither a gloabl nor a local minimum of the network (which has more parameters).Yes, as the learning rate is small enough for all practical purposes. (No, if you use SGD or mini-batch gradient descent)Usure. I think the correct answer is No, the network can make more mistakes in between with cross entropy.. It is certainly sure to improve CE loss while at the same time getting worse at accuracy (see proof below). However, I'm not sure if the gradient would ever lead to such a result.Example for 3#!/usr/bin/env pythonfrom math import logdef ce(vec):    index 0 is the true class.    return -(log(vec[0]) + sum([log(1-el) for el in vec[1:]]))a = [0.1001, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.1, 0.0999]print(ce(a))b = [0.49, 0.51, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0]print(ce(b))gives:ce(a) = 3.24971912864ce(b) = 1.42669977575Hence the cross entropy loss dropped (as expected), the probability for the correct class increased (as expecte) but it makes a mistake (if you simply take the argmax).