How does someone with C's and F's in graduate-level courses and A's and B's in undergraduate courses get admitted to a PhD program? I am currently the BA undergraduate math major (4th year), and I got ~3.3 cumulative GPA.  I'm the average student who likes math a lot, so I took more than 10 graduate courses and higher division undergraduate courses in mathematics.  For two courses, I received two C grades in those graduate courses by the same professor (hard exams and finals).  I attempted to get better grade in one of those courses, but received the F (because of super hard final and midterm, and because of the same professor).  For the undergraduate courses, I got mostly A's and one B+.

Though I still have time to maximize my chance to get into math PhD program by (probably) research experience and great letters of recommendation, I believe that I am more likely not to get into the program (because low grades greatly affect my application).  However, I have one question to ask: is it REALLY over for me to get into PhD math program if I receive bad grades in these courses or not?  I don't have lots of time to retake such courses, and if I do I will have to waste all money and time on doing so.

I don't aim to get into top-notch universities with excellent PhD programs, like Harvard University.  All I expect is a good PhD program for me.
In the graduate level courses, was the grading scale the same for undergraduate and graduate students? This varies from school to school, department to department, even course to course. Were you taking these classes without all of the prerequisites? You should consider all extenuating circumstances and tell your letter writers so they can help better explain away your grades and focus on how amazing your research is. Give your letter writers bullet points (suggested for them to edit/ignore as they see fit) to include in their recommendation letters. Faculty don't enjoy nor want to spend that much time on letter writing so make it easy. In your email to a faculty member you might write something like this

Hi Professor X,

Here is a reminder of things you might want to address in my letter of recommendation for __  (of course only if you agree with them)

Evidence of enthusiasm and ambition

-I took many graduate courses out of genuine interest for the subject despite not having taken many of the prerequisite courses. Even though I received the poor/mediocre grades in some of these courses, I was able to successfully translate some of the advanced knowledge and skills learned when I ... during my research project ...

-Some other way you can spin these poor grades into something positive

Thanks for agreeing to write this letter, I hope the above bullets make things easier for you; I know you are busy and have to write many of these letters.

Thanks again,

-Student

The point is you want your letter writers to spin the poor grades into something positive that fits with a cohesive story about you. You personally addressing the grades may sound a bit like excuses. A professor addressing the grades won't. It should be noted that some schools do have strict unspoken GPA cutoffs, but many really good schools do not.