MLPs have the universal approximation capability to an arbitrary
function [1]}, [2]}. Without
loss of generality, consider the mapping from an \(n\) -dimensional input
to an arbitrary 1D function. The latter can be approximated by a union of
piece-wise low-order polynomials. The basic unit of a piece-wise
constant (or linear) approximation is a box (or a triangular-shaped)
function of finite support. The form of activation determines the shape,
e.g., two step activations can be used to synthesize a box function.
Users are referred to [3]} for more detail. For each
partitioned interval, the mean value of observed output samples (or the
labels of the majority class) can be assigned in a regression problem
(or a classification problem). Decision making with space partitioning
is essential to all ML classifiers/regressors, e.g., the support vector
machine/regression (SVM/R), decision trees, random forests, gradient
boosting classifiers/regressors. Yet, no nonlinear activation is
required by them but neural networks. This difference will be explained
in Sec. REF .
