A neural network of arbitrary width and bounded depth has universal approximation property (UAP). The classical UAP is that continuous functions on a compact subset on \(\mathbb {R}^n\)  can be approximated by a linear combination of activation functions. It has been shown that the UAP for the neural networks holds by choosing a sigmoidal function [1]}, [2]}, [3]}, [4]}, any bounded function that is not a polynomial function [5]}, and any function in Lizorkin space including a ReLU [6]} function as an activation function. The UAP for neural network and its proof for each activation function are presented in Table REF .
<TABLE>