Competitiveness for masters program I will be graduating this Fall with my undergrad (GPA: 3.66) and am planning to go to grad school in math.  Given my early history of poor grades (2 Ds, 1 F), I am planning on only applying to the masters programs.  Ideally, I would like to get admitted to my current university's masters program (top 15 US school in math).  To help boost my application, I have taken some first-year grad math courses. Additionally, I have been doing research under fairly famous applied math professor (h-index above 50.  Though I have heard that h-index is not an objective factor).  While I have read that in some grad courses they hand out all A's, is there a way to prove to the admission's committee that I actually deserved the A's?  Lastly, what do masters admission's committees look for in applicants?
Masters programs would be easier to get into, since they often don’t burden the department with a graduate stipend, that means you’ll be responsible for tuition most likely.

I would apply to PhD programs if I were in your position. If you performed well on your upper division courses and have a gpa over 3.5, then you’d have a good shot of getting in. In this case, you are more likely to get a stipend and a tuition waver. That means you’d get paid to go to school and if you feel that the PhD is too much, you can also leave with a masters degree.

Edit: didn’t have time for a fuller answer earlier.

Don’t worry about the “free grade” in the graduate courses you took. Truly, all that is important there is that you have demonstrated experience and success in a graduate course, and indeed, at an apparently good Institution. No one is going to look at your grade and say “whatever, it’s just a grad course.”

Most grad schools look to your upper division courses as a measure of how you’ll perform in graduate school. I did terrible in my first couple of years in undergrad; I also failed a class or two, but I did well enough in the latter half to end with a 3.5. I was admitted into a good graduate school (in 2008).

As to your peofessor’s h index, that’s actually rather good. It means that he has at least 50 publications with 50 citations. While there is no objective measure of the quality of research, this means that he is well known in his community, his work is frequently cited, and his recommendation will carry with it some weight.

Grad schools look at several things. You need a good GRE score to get past general admissions, but no one cares past that. Your GPA matters, and yours seems good enough (better than mine was). After that they want to know how you did in upper division courses like Analysis, Algebra, etc, since you’ll be taking more intense versions of these classes, and they want you prepared.

Also having experience with graduate courses is a plus, I took a grad number theory course that helped me, and almost as importantly as the rest, they look at your recommendations. You will be part of the school’s community for the next five or more years as a PhD student, and they want to make sure that you can work with someone on research topics.