Can I do a thesis project in an area if I haven't taken a course on it? To provide a background, I am currently a Physics major and along the way I ended up getting really interested in Mathematics, so much that I'm now planning to do pure math in graduate school. Now, the problem is that, we are required to do a senior's thesis before we graduate, and I still don't have a topic in mind.

I actually worked a little bit with a professor in Theoretical Physics, I did bits of grunt work under him. I got exposed in Asymptotic methods when I was in his lab, and I got the idea that maybe I could do an undergraduate thesis about/using asymptotics. During the time I was working there, I was under the impression that he'll eventually talk about my thesis topic but he never even mentioned it (to be fair though, I never talked about it to him either), so now I'm kinda panicking that I might end up going solo in my research.

Now, in the case that he'll not accept to advice me on this , my question is, is it naive to even think about doing an undergraduate research that mainly involves Asymptotic Analysis/Methods when I've never even taken a formal course in it? To make the odds worse, it seems it's a graduate level topic, I am willing self study it though. 

I want to know just how far I can go with doing research in a topic I haven't formally studied in a classroom setting, it doesn't really need to be Asymptotics or anything, I just wanted to do  something that's theoretical, and that's borderline math, just so I'd be forced to learn as much math as I can during my last year in undergrad. I figured doing research in applied math is the best area to do work on since I'm still a Physics major.
Of course you can! (I did. My advisor did. Most of my students did.)

Someone will have to agree to supervise your research, but that's true even if you have taken a course in your proposed research area.