The theoretical analysis of practical DNNs is generally complicated, and thus the linear activation (no non-linearities) function along with other necessary assumptions are often made for tractability. Our formal analysis in this paper assume the linear activation (i.e. linear DNNs) as well. As such, following existing literature, we first emphasize the relevance of studying linear DNNs, including their usefulness for understanding DNNs with non-linear activation functions (non-linear DNNs). In [1]}, it was observed that analytically results obtained using linear DNNs conform with results from non-linear DNNs. This is expected given that the loss function of a linear multilayer DNN is non-convex with respect to the model parameters, similar to non-linear DNNs. Furthermore, the relevance of linear DNNs for analytical study is seen in the work [2]}. Interestingly, we note that much stronger assumptions are common in the literature. For example, the assumption of no nonlinearities and no batch-norm can be found in [3]}. In [4]}, the assumptions of (i) no nonlinearities, (ii) no batch-norm, (iii) the number of units in the different layers are more than the number of units in the input or output layers, and (iv) data is whitened.
The assumptions of (i) no batch-norm, (ii) infinite number of hidden units, and (iii) infinite number of training samples can be found in [5]}. In [6]}, the assumptions of (i) no nonlinearities, (ii) no batch-norm, (iii) the thinnest layer is either the input layer or the output layer, and (iv) arbitrary convex differentiable loss can be found. Nonetheless, the results from the aforementioned works have proven quite useful in practice. We show later on in our experiments that both non-linear and linear DNNs exhibit similar training characteristics for the generalization gap problem, which is not surprising.
