We prove that a particular deep network architecture is more efficient at approximating radially symmetric functions than the best known 2 or 3 layer networks. We use this architecture to approximate Gaussian kernel SVMs, and subsequently improve upon them with further training. The architecture and initial weights of the Deep Radial Kernel Network are completely specified by the SVM and therefore sidesteps the problem of empirically choosing an appropriate deep network architecture.
Therefore evidence is building that deep networks are more powerful than their shallow counterparts in terms of the number of parameters or neurons required. Nevertheless, we shouldn't stop there because there is relatively little work linking this theory with practical applications of the same. In this work we directly extend the work of and . The latter work is extended to deeper networks for approximating radially symmetric functions that require fewer parameters than their construction. The former @cite_1 is extended by generalising their notion of folding transformations to work in multiple dimensions and more simply with ReLU networks. The proofs are constructive and allow us to build networks for approximating radially symmetric functions. These networks are used to approximate Gaussian kernel SVMs and the results show how we can further train these approximations to do better than the original SVM in many cases.
@cite_1
2033337358
We present a comparative theoretical analysis of representation in artificial neural networks with two extreme architectures, a shallow wide network and a deep narrow network, devised to maximally decouple their representative power due to layer width and network depth. We show that, given a specific activation function, models with comparable VC-dimension are required to guarantee zero error modeling of real functions over a binary input. However, functions that exhibit repeating patterns can be encoded much more efficiently in the deep representation, resulting in significant reduction in complexity. This paper provides some initial theoretical evidence of when and how depth can be extremely effective.