How difficult Is a PhD Program? I am a Master's student in math who is doing pretty well so far. 
I aced my first two classes (real analysis) 
and my summer session in probability theory is going well, too. 
Even before my Master's study, I already thought of a PhD. 
Nonetheless, there are concerns:


My good grades were the result of grit, not brain, 
meaning that I had to spend time to understand what the book is telling me. 
Furthermore, a lot of times during exam and homework, 
I had to try and fail a few times before arriving at the solution.
I did not study math during undergrad.  Before my Master's study,
I gulped down three semesters of calculus, plus LA and DE in one year. 
So compared to other students, I am already behind. 
In the next two months, I will learn complex numbers on my own
and review my linear algebra.
I am a professional in a field that has nothing to do with math or research. 
After the holiday, I will speak to my advisor about taking a thesis class. 
It's not an insurmountable obstacle, 
but, in an unrelated field, I do have less time to concentrate on math.


At the end, how big is the jump from Master's to Phd? 
Especially after what I said in (1), I do worry that it is beyond my ability. 
I do not hold anyone responsible for my decision. 
So, please kindly offer your best assessment of my situation.
I'll answer your title question and sub-questions in three words (then add to that): Difficulty is relative. 

Noam Elkies finished his PhD in math at the age of 20. The average graduate student would probably take anywhere from 4 - 5 years. That doesn't mean Noam was a god among graduate students when he was a graduate student, and that's the reason he was able to finish his thesis faster than Joe Schmoe (though he is quite exceptional) and Joe Schmoe should just give up. Far from it. The fact that you're doing well in your master's courses is a good indicator you're well qualified for the coursework part. Working on a PhD thesis means producing original research. Producing original research means dealing with not so cut-and-neat problems like ones you'd encounter in your homework or textbooks. It means working with problems that you're not sure are close-ended, open-ended, or neither. In other words, different skill sets are needed for research in addition to the problem-solving heuristics which you've already acquired from your coursework. 

Since you've never wet your foot it seems to me you have no indication of your ability to do research . So what do you do? Just go for it! You only live once as far as we know, and it seems to me that your real problem is your self-imposed doubt. Get that out of your head. Research is about staring and thinking about a problem long enough then coming up with an answer which turns out to be completely wrong and in that, the process repeats until you get your "Eureka!" moment. 

You should also stop worrying about others being ahead of you. There'll always be someone faster and better than you (unless your name is John Von Neumann :-) especially in academia. A good algebraic geometer from UGA by the name of Roy Smith didn't get his start in advanced mathematics until his late 20's, after working in industry as a meat lugger. He received his PhD at 35. Over 30 years later, he's still going strong. What you should learn from him is if passion calls for it, then there should be no stopping you. If you fail at a career, then that's that. It's not the end of the world. But if you never bother to even try, then you'll never know and you'll always have a "What if?" moment. 

So just to sum all of this up: Apply to a PhD program and see what happens.

https://mathoverflow.net/questions/7120/too-old-for-advanced-mathematics/45644#45644 (if you're interested in Roy's post)