By comparing the above two theorems and following the argument of [1]}, we see that the generalization benefit of NFM comes from two mechanisms. The first mechanism is based on the term \(Q_\epsilon (f)\) . Assuming that the Rademacher complexity term is the same for both methods, then NFM has a better generalization bound than that of standard method if \(Q_\epsilon (f) > 0\) . The second mechanism is based on the Rademacher complexity term \(R_n(l \circ \mathcal {F})\) . For certain families of neural networks, this term can be bounded by the norms of the hidden layers of the network and the norms of the Jacobians of each layer with respect to
all previous layers [2]}, [3]}. Therefore, this term differs for the case of training using NFM and the case of standard training. Since NFM implicitly reduces the feature-output Jacobians (see Theorem REF ), we can argue that NFM leads to a smaller Rademacher complexity term and hence a better generalization bound.
