However, this poses a major challenge: deciphering the reasoning behind the model's predictions. For instance, typical NN architectures for classification or regression problems incrementally transform the representation of the input data in the so-called latent space (or feature space) and then use this transformed representation to make their predictions, as summarized in Fig. REF .
Each step of this incremental data processing pipeline (or feature extraction chain) is carried out by a so-called layer, which is mathematically a non-linear function (blue rectangle in Fig. REF ). It is typically made of a linear transformation followed by a non-linear activation function [1]}, [2]}, but more complex alternatives exist – e.g. the residual block layers of ResNet models [3]} or the self-attention layers [4]} in transformer models.
These first stages of the model (Fig. REF ) often rely on the bottlenecking of the information that's passing through it by sequentially decreasing the size of the feature maps and applying non-linear transformations – e.g. the widely used ReLU activation function [5]}. To summarize, these first stages project the input data into a latent space.
Therefore, the neural network's data extraction pipeline is driven by the training data that were used to optimize its parameters.
The second part of the network (Fig. REF ), which is standard for classifiers or regressors, is generally simpler to understand than the first, as it is often composed of matrix-vector products (often denoted as dense or fully-connected layers) followed by ReLU activation functions. Consequently, it is mathematically equivalent to a piece-wise linear transformation [6]}. More importantly, these non-linear transformations depend on parameters that are optimized to make accurate predictions for a particular task when training the NN.
<FIGURE>