A common issue with these alternate views of the problem such as sparse linear regression or sparse WHT is that the resulting algorithms are not suited for function approximation. In particular, real-world physical and biological functions often have additional structure which is either unknown a priori or cannot be described succinctly, and which nonlinear function classes are often implicitly biased towards learning. Indeed, several deep neural networks (DNNs) have been shown to exhibit strong generalization performance in biological and physical problems [1]}, [2]}, [3]}. This leads to a fundamental disconnect between algorithms for which strong theoretical guarantees are known (e.g., LASSO) and practical ML models based on nonlinear and complex function classes which are well-suited for function approximation.
