Table 12 shows the prediction results of T5-Vanilla and T5-Modular on three knowledge probing tasks: InvariantLAMA, UpdatedLAMA, and NewLAMA. We show the prediction for every training epoch for each model. The instances are selected from the predictions that T5-Modular got correct but T5-Initial got wrong on the final prediction, in order to see where the gap of the EM comes from.