I'mtakingAndrew Ng's machine learningcourse andweek 5 covers the training of neural networks. The modified cost function for neural network training is derived from the logisticregression cost function, and is described as follows:$$\begin{gather*} J(\Theta) = - \frac{1}{m} \sum_{i=1}^m \sum_{k=1}^K \left[y^{(i)}_k \log ((h_\Theta (x^{(i)}))_k) + (1 - y^{(i)}_k)\log (1 - (h_\Theta(x^{(i)}))_k)\right] + \frac{\lambda}{2m}\sum_{l=1}^{L-1} \sum_{i=1}^{s_l} \sum_{j=1}^{s_{l+1}} ( \Theta_{j,i}^{(l)})^2\end{gather*}$$Here, K is the number of output units, L is thetotal number of layers in thenetwork, and ${s_l}$ is the number of units in layer l. I don't understand why thesecond half of the cost function,intended toprevent over-fitting by minimizing the values of thetheta parameters, is summed across theentire network while the first half of the equation which actuallydetermines theta is only for a specific layer.Ifeachlayer's cost function already includesaminimizationparameter for that layer's theta values,why is it necessary toperform this minimization globally for every layer in thecost function J?As Iunderstand it,intraining a neural networkyoumainly treat each layer as completely separate from the next (and in fact, that is one of the major selling points of the NN approach, whereinsolvingL layers independently of one-another lets you use a simple approach to obtain powerful results) - so why arewesumming thesecond half of the equation over all layers?
Why doesthe neural network logistic regression cost function sum for all layers only for lambda?
machine learning;neural networks
The first term depends on the output of the entire network, i.e., on all the parameters for all layers, not just a single layer's parameters.This is an application of the empirical risk minimization framework for training a classifier, with regularization.  The cost function $J(\Theta)$ has the form$$J(\Theta) = \text{empirical risk} + \text{regularization penalty}.$$Denote the empirical risk by $R(\Theta)$; it depends on the parameters $\Theta$ (for all layers).  Denote the regularization penalty by $P(\Theta)$; it depends on the parameters $\Theta$ (for all layers).The empirical risk $R(\Theta)$ is the sum of the loss for each instance in the training set, i.e., it has the form$$R(\Theta) = {1 \over m} \sum_{i=1}^m \ell(h_\Theta(x^{(i)}), y^{(i)}),$$where $\ell$ is a loss function, $x^{(i)},y^{(i)}$ is the $i$th instance in the training set, and $h_\Theta(x)$ denotes the output of the classifier on input $x$ when the parameters are $\Theta$.  Note that the output of the classifier depends on all the parameters for all the layers, so $R(\Theta)$ depends on all the parameters for all layers (not just a single layer).A comment on notation: We use $\Theta$ to denote all of the parameters, i.e., a single vector that concatenates the parameters for each layer.  Here it appears $\Theta^{(l)}$ has been used to denote the parameters for the $l$th layer, so $\Theta = (\Theta^{(1)}, \dots, \Theta^{(L)})$.In your specific example, we use the cross-entropy loss for $\ell$ (this is also known as the logistic loss function, where there are only two classes).  That's how we get the first term.Also the regularization penalty $P(\Theta)$ depends on all of the parameters, for all layers; it is a sum of penalties for each layer.If you work through the math, you'll see how we get something of the form you've shown in the question.  As this answer hopefully makes clear, each term depends on the parameters for all the layers, not just a single layer.