Do these architectural variations impact the models' maximum capacities? Our findings suggest that, in terms of knowledge capacity, GPT2 -- when enhanced with rotary embedding and without dropout -- performs no worse than any other architecture choice above in the sufficient training regime.