Our final key component is simple but crucial: the choice of non-linear activation function plays a central role in modeling complex non-linearities of atomic intereactions.
The ReLU activation [1]} is widely used in many deep learning models, including the existing GNS model [2]}, [3]}.
However, ReLU may not be ideal in modeling atomic forces, since it results in outputs being modeled as piece-wise linear hyper-planes with sharp boundaries. Ideally, we desire a smooth and expressive non-linear activation function.
