I am a graduate student in mathematics, but very new to the field, but excited to start studying AI in greater depth. Wikipedia lists a relatively large list of activation functions. I know that certain activation functions are better suited for neural nets in certain cases. I know differentiable and monotone functions are very desirable because you can use gradient descent for training the neural net (as I understand it).Each activation function has its pros and cons for a given situation. As an example, maybe a function is differentiable but not monotonic. Let's take this same function, and through appropriate modifications, make it differentiable and monotonic, would this modified function be a better' activation function than the original? I do have a mathematical method to custom design an activation function with the desirable properties of differentiability and monotonicity, I just am unsure how applicable it is to neural nets. 
Analysis and Neural Networks
neural networks;deep learning