Does rigor/thoroughness of undergraduate program matter (for graduate/PhD applications)? I am an undergraduate at a large public research university in the US, finishing my second year. I have done some theorycrafting around my graduation course requirements, and I have realized that it is possible, with some work (and generous CLEP testing), to graduate with a double major, Math BS + Physics BA, by next May. I would be meeting the bare-minimum requirements for both majors (in particular, for math, I will have only taken the 4-semester calculus series, lin. algebra, and the 2-semester abstract algebra series; plus some extraneous ones e.g. game theory).

If I were to stay the extra year, I would comfortably qualify for a double Math BS + Physics BS, and I would have 3-4 more math classes under my belt (perhaps the 2-semester topology series, graph theory, advanced abstract algebra, real analysis, etc). (edit: this option, Physics BS rather than BA, stresses a more thorough physics courseload; 15 hours of upper level physics classes are required for BS over BA)

edit after several answers: Another 4-year option is to get Physics BA + Math BS, leaving my course load open to taking 3-4 math classes per semester. This, hypothetically, could put my post-lin-algebra courses at 2 sem. abstract algebra, 2 sem. topology, 2 sem. intro analysis, graduate level courses of the above, intro complex analysis, intro graph theory, etc

My intent, if I were to graduate next year, is probably to take a year off from academics, and then apply for graduate/PhD programs in math.

Will it be an issue in being considered for these programs, that I have only taken a meager selection of the course offerings in my undergraduate studies? And might it be offset by the fact that I graduated in three years with a double major?
This would be a great question to discuss with your advisor, or another faculty member at your school whose opinion you respect.

My opinion, for what it's worth, is that the "bare-minimum" 3-year program you describe would be woefully inadequate to prepare you for graduate study in mathematics, and I think most admissions committees would agree.  The fact that you did it in 3 years with a double major won't mitigate that.  A year of analysis is pretty essential, some topology would be very helpful, and taking a significant number of upper-division math electives will be a great way to build what people call "mathematical maturity".

Your 4-year program sounds better, but honestly still a little on the light side.  Keep in mind that many applicants to top grad programs will have taken practically every undergraduate course their department offers, and maybe a couple of graduate courses besides, in addition to often having undergraduate research experience.  You also want to make sure you get to know (and impress) your mathematics professors well enough that they can write you strong letters 
of recommendation, and taking more math courses is the biggest part of that.  

The physics degree on your diploma will be nice, especially if you are looking to do a PhD in mathematical physics, but it may not go a long way to making up for weakness on the math side.