Is the gap between lower- and higher-ranked graduate programs in pure math really that big? Initially I asked this question on math stackexchange but it seems academia stackexchange is more appropriate place for this.

Note: a question similar to that I'm going to ask was discussed here, but answers to the questions I'm going to ask were not covered there.

Basically, I'd like to understand how significant is the gap between lower- and higher-ranked graduate schools in mathematics. I'll be referring to the USN graduate schools ranking. Let me fix the notation straight away. By 'higher-ranked schools' (which I will also refer to as 'top schools') I will mean top 10 (or if you want top 20) and by 'lower-ranked schools' I will mean schools ranked 20-40 (all according to USN).

What I understood from the question I provided the link to above is that being amidst 'top students' at a higher ranked school is more beneficial than being amidst 'average students' at a lower-ranked school. Also, higher-ranked schools may be more diverse and provide more opportunities in meeting people from other top schools as well as in obtaining a job in the academia. 

But I still have a couple of questions. 

The first one is about the first two years of the study. Is, in general, the instruction level at lower-ranked schools worse than that in higher-ranked ones? Also, are graduate corses at 'top schools' more difficult to master and to pass? If so, does it imply that one needs a better preparation (i.e., a stronger mathematical background) to succeed in a top graduate program? Also, does it imply that the students enrolled in a top program will eventually have a better mathematical background that will make it easier for them to conduct research? If you have anything else to say about the coursework at top universities in comparison to that at lower-ranked universities, I would appreciate it.

Secondly, in the question I referred to above (or elsewhere), some people mentioned that exposure to new ideas in various branches of mathematics (which is one of the advantages of top programs) is a consequence of the size of the department and the 'quality' of faculty/post-docs/students. Whereas I do agree that students at top universities are more knowledgable and creative, I cannot see why the other assertions hold. Correct me if I am wrong but the the math department of say Stony Brook or Indiana (ranked 25 and 34, resp.) is not smaller than that of Chicago or Columbia (ranked 5 and 9, resp.). Furthermore, the vast majority of professors in all of the places mentioned are alumni of Harvard/Princeton/Berkeley/MIT/Chicago/Stanford (i.e., a top school in my terminology); post-docs also come from very prestigious places to all of the four mentioned universities. So what makes Chicago or Columbia 'better' than Stony Brook or Indiana? Just their name? (Of course Chicago may have more publications than Indiana but I don't think that graduate students feel it.)
Looking at the problem as someone who has been on a variety of hiring committees and grant panels,
I would say that in general, having someone with a good reputation in the field writing a strong letter of support is the most important factor, together with the journals where you have published.

In view of this, the school doesn't really matter: if you do your PhD in a small school but with a supervisor with a solid reputation who produces a multi-page letter highlighting well your contributions to the field, you will usually fare better than if you come from a very well known school with very little endorsement from your supervisor. Of course, some people tend to get a little bit star struck, but the dynamics of a hiring (or funding) committee usually smooths things out. Actually, contrary to what Dave Kaye was saying, I would argue that the school matters less in academia, more if you want to venture out into the real world, where name recognition is more important and might help you land your first job.

So why favor a good school? Well, there will be more visiting speakers of a high caliber. There might be a wider variety of grad courses to choose from. Overall, probably a more enriching experience. The difficulty level will not be very different, if at all, once you have been admitted.