A non-polynomial activation function in a neural network with three layers has a universal approximation property. Such result was shown by Leshno [1]} using functional analysis and later by Sonoda and Murata [2]} using Ridgelet transform.
