How do I know if I am fit to do a PhD in my area of interest, specifically, theoretical CS, when my background was mostly applied EE? I have a Master's degree from a good (top-15) school in Signal Processing and my Master's research involved mainly helping out PhD students in very applied projects. The idea of the project was nice (and it even resulted in a patent and several publications), but the individual contributions to the team - and this includes mine - were not novel, just implementing existing algorithms.

I was dissatisfied with my MS research lacking any real novelty (but at that time I joined that lab because the professor had funding), so after graduating, I worked for a bit in the industry and then came back to (a different) grad school for a PhD, hoping to do theoretical work.

Over the summer, I am reading papers with potential advisors in math/applied math/theoretical CS departments at my university. I am now wondering - while I am liking the ideas in the papers I am reading, am I trained enough to make novel contributions in theory? By aiming to do theoretical work for my PhD while having a such an applied background, did I gravely confuse liking an area with ability to do something new in it?

My question is: How do I know I can do something new in the theoretical field I am reading papers in? Of course I like these papers - they have been written by leaders in the field who know how to write well; the ideas are great, which is why they are famous papers. But I have been so stupid to have assumed, all this while, that liking these papers is some sort of indicator that I can be in this field. People who actually wrote these papers are mathematicians and theoretical computer scientists who, after years and years of training, have come up with these novel ideas. Will I ever be able to do that? 

I will start my second year in a few months; I just don't know if I should stick to trying to solve theoretical CS/math problems of the kind I am reading now, or if I should stick to what I have training in - applied signal processing.
Until you try, you wouldn't know if you can or cannot do it. There are plenty of people who make more radical switch of fields and make meaningful contributions. If you are very good at applied signal processing, you probably can get good at applied discret math and theoretical CS. One way to ease into the field is to take relevant grad-level courses. Try taking a theoretical CS class and put your best efforts to excel.

(I switched from CS to EE. I was very much into logic and computation theory before starting my PhD in adaptive signal processing.)