Revealing the Structure of Deep Neural Networks via Convex Duality
We study regularized deep neural networks (DNNs) and introduce a convex analytic framework to characterize the structure of the hidden layers. We show that a set of optimal hidden layer weights for a norm regularized DNN training problem can be explicitly found as the extreme points of a convex set. For the special case of deep linear networks with $K$ outputs, we prove that each optimal weight matrix is rank $K$ and aligns with the previous layers via duality. More importantly, we apply the same characterization to deep ReLU networks with whitened data and prove the same weight alignment holds. As a corollary, we prove that norm regularized deep ReLU networks yield spline interpolation for one dimensional datasets which was previously known only for two layer networks. Furthermore, we provide closed form solutions for the optimal layer weights when data is rank one or whitened. We then verify our theory via numerical experiments.
ICML