The Indo-Aryans were nomadic so weren't huge in number. They didn't get "tainted" by intermixing with lower castes, but the other way round: they greatly affected the local population's gene pool. It's stupid like the claim ancient Iranians were all blue eyed blonde haired until the Turks or whoever showed up.