A typical problem that appear during training ML models is over-fitting, where a model accuracy is quite high on training but when inference is applied to a similar dataset, a very poor accuracy is achieved. This is the consequence of the model learning or capture the dataset features but also its noise, e.g. features that do not provide any information and that are caused by randomness. Hence, in this type of NN models, it is quite common to use regularization techniques defined as “any modification we make to a learning algorithm that is intended to reduce its generalization error, but not its training error”, according to Goodfellow et al.[1]}.
