Then on the training side, stochastic gradient descent methods introduce their own biases. Sometimes, this bias helps with the capacity issue by pushing the optimization away from the correlates and enabling generalization [1]}. However, if there is an easy correlate in the data that helps get most examples right, training might "lazily" converge to that instead of identifying the (more complex) causal variables that explain the data. For instance, a model might learn to predict braking only once the brake light is on, instead of solving the more complex problem of identifying the need for human to begin braking. This is the more problematic the more information the model has access to, because more correlates exist (the "causal misindentification" [2]}.
