How common is it for grad school admissions to use a whitelist of undergrad institutions? I was talking to a professor recently and he mentioned something that I found surprising. Namely, for graduate admissions in his department (top-30, US public) they had a whitelist of universities and they only consider applicants whose undergraduate studies were at one of those universities. He went on to complain that initially the list was too US-centric and it had to be expanded because they missed good international candidates.

I was very surprised by this. I did not know that this whitelist approach is used, but evidently it is. My question: How common is this? Have others heard of this practice?

I'm pretty sure my undergraduate institution would not make it onto any such list. So I'm sad to hear about such practices. Fortunately, I did get admitted to several very good US grad schools, so evidently the practice is not universal.
I think your use of "white-list" is not accurate. Namely, usually it does not mean the exclusion of all others, but only the inclusion of those on the list.

As far as graduate admissions, yes, prior experience with letter writers from a given undergrad institution, especially via the people admitted to our program from that institution, establish a track record for those letter-writers, and for those programs. That's not to say that every individual from a good program is equally good... but that we have data points about the meaning of the letters of recommendation, and about prior preparation. 

In fact, our graduate program pays considerable attention to individuals who most likely fall outside the doctrinaire "best undergrad institutions", exactly because we know that many grad programs will under-value them, so that (if we can see a good hint of potential) we have a better chance of recruiting them. So, no, there is absolutely no a-priori exclusion of people from ... odd... places.

But, again, sure, prior experience (for better or for worse) with institutions, their faculty, and their programs, obviously gives us information. Yes, it seems to be true that some institutions are a bit delusional/arrogant, but we don't hold that against their students. :)