Going to a lower-ranked school for better advisor fit Computer Science PhD admit here. 

Is it crazy of me to turn down a top school (MIT/Stanford) for a lower-ranked school (think Cornell/UW/Columbia/Michigan) if I think there is better advisor fit?

Obviously all of these are fantastic schools and I am very lucky to be choosing from them. However, I felt like I got along really well with my would-be advisor at the lower-ranked school than I did at MIT/Stanford. People have told me that it's crazy of me to turn down MIT/Stanford since they are, along with Berkeley/CMU, on a different "level" than the other schools.

(My field is somewhat narrow so there is only 1~2 faculty at each of the schools doing research in the area. So regardless of which school I go to, if the advisor doesn't work out, I would be in a dire position.)

EDIT: Both advisors are well known in the field (and both have produced outstanding graduates), but the MIT/Stanford professor is good deal more senior. Also, talking with their current students, the MIT/Stanford professor is known to be pushy and have a "strong" personality (which may work for some people, but not sure whether it will work for me, as I've never worked with such people before).
All of the schools that you are discussing are first-rank computer science schools with global leadership in their areas of specialty.  In this area at least, the only real difference between the "top" and "top of the top" schools that you are considering is the number of different areas in which the school is a global leader.  Thus, if you already have a clear, narrow focus that you know that you want to work in, and there are 1-2 people in that area at all of the schools you are choosing between, the distinction between "top" and "top of the top" is pretty much null.  

Furthermore, your graduate career will be primarily determined by your advisor relationship rather than the school within which it will occur.  Beware of being misled by first impressions, but a good match with your Ph.D. advisor is worth much more than the incremental difference between schools.