My master thesis supervisor refused to recommend me for a PhD He reasoned that I lacked the understanding of the subject and so not competent enough for a PhD. I agree to his concerns, since I was studying this concept for the first time (for my thesis) and was a bit slow in my approach. On the other hand I'm very passionate about this particular subject and intend to pursue an academic career in this field. I did manage to get a decent score in my thesis and graduated with cum laude. The question is should I give up and take up some job since nothing will move forward without a recommendation.
From what you've written, it sounds like your adviser doesn't think you know enough to be able to research a subject. Knowledge can always be learned. It's actually a very important trait for a PhD student to be able to sit down and learn what they need on their own. 

There's no reason not to apply to PhD programs. It's going to be very hard to get into one without a letter from your adviser, but it's not theoretically impossible. The worst thing that can happen is that you spent some time and money and fail to get accepted anywhere. The best thing that can happen is that you get into a program and have the chance to follow your passion.

On advice from Pete L. Clark, I'm adding a caveat: You should be careful as to what sort of programs you're applying to and think carefully about whether those programs will lead to your desired outcome. Going into a PhD program is not something to be taken lightly. It will takes years of your life and be a potential drain on you mentally, emotionally, and economically. To land a decent academic position you need to graduate from  a top program and produce high level research. Afterwards you may have to put in more time and effort as a post-doc. Once you do finally get a position, you'll have to work your way through the tenure process.