Race is a social construction -- zero out skin color and some people will just find new and equally stupid reasons to hate.  See, for instance, sectarian violence in Northern Ireland, genocide in Rwanda, etc., for cases where idiots rose above common skin color to express their needs for bigotry.