The cosine similarity between easy examples is much higher than that of difficult ones, as shown in Figure REF .
Easy examples reside at the head of the long tail distribution and belong to similar subpopulations, and this similarity between images engenders similar gradients. These similar gradients strengthen each other to make a SGD update that can benefit them all, which results in the faster optimization speed for easy examples.
This observation is in agreement with [1]}, who uses a measure of gradient coherence to show that the parameter update of the network can benefit many examples when such similarity exists.
