The inference procedure (\(I\) ) is what enables learning in machine learning methods. It is widely believed that modern inference methods, specifically gradient descent and variants, “implicitly" bias the solutions of the networks towards those that generalize well and away from those that generalize poorly [1]}, [2]}, [3]}. The effects of the inference algorithm are intimately tied to the specifics of the model (e.g., weight-sharing) and the data (e.g., data augmentation), and might not be fully understood with a fixed-data, fixed-model analysis. Indeed, good performance may derive from interactions between \((\mathcal {M}, I)\) , or \(( I)\) , or even \(( \mathcal {M}, I)\) . In Sec. , we demonstrate the DIDE effect for a particular choice of \((\mathcal {M}, I)\)  and show that this effect disappears if any one of \(, \) M\(, or \) I\( is altered.\) The above discussion highlights the insufficiency of treating \(, \) M\(, and \) I\( as separate non-interacting modules. They must be considered as an integrated system. Throughout this paper, we will refer to the triplet \) ( M, I)\( ~as a (machine) learning system and the tuple \) (M, I)\( as the learning algorithm of the system that operates on \) .
