Here, we briefly discuss neural network architectures suitable for our proposed approach. We require two neural networks for our framework, one for learning the implicit representation \(_\theta ^{\textsf {I}}\)  and the second one \(_\theta ^{\textsf {Dyn}}\)  is to learn the vector field. For implicit representation, we use a fully connected multi-layer perceptron (MLP) as depicted in fig:NNarchi(a) with periodic activation functions (e.g., \(\sin \) ) [1]} which has shown its ability to capture finely detailed features as well as the gradients of a function. To approximate the vector field, we consider two possibilities depending on applications. If the data do not have any spatial dependency, then we consider a simple residual-type network as illustrated in fig:NNarchi(b) with exponential linear unit (ELU) as an activation function [2]}. We choose ELU as the activation function since it is continuous and differentiable and resembles a widely used activation function, namely rectified linear unit (ReLU).
On the other hand, when the data has spatial correlations, e.g., dynamics in data are governed by a partial differential equation, then it is more intuitive to use a convolutional neural network (CNN) with residual connections as depicted in fig:NNarchi(c). It explicitly makes use of the spatial correlation. For CNN, we also employ the batch normalization scheme [3]} after each convolution step for a better distribution of the input to the next layer and use ELU as an activation function.
<FIGURE>