We explored a wide array of choices for activation functions, such as Tanh, Leaky-ReLU, SoftPlus [1]}, Shifted SoftPlus [2]}, and Swish [3]}. In our experiments, we find Swish, i.e., \({\rm act}(x) = x\cdot {\rm sigmoid}(x)\) , to perform particularly well. As illustrated in Figure REF , Swish provides a smoother output landscape and has non-zero activation for negative inputs. As we demonstrate in Section REF  and Figure REF , the replacement of ReLU with Swish consistently and significantly improves the predictive accuracy while maintaining scalability across all choices of basis functions.
