Is it feasible to take an unsure PhD topic? I was working towards converting to a PhD but was unsure of what research I would do. Suddenly an idea striked my mind and I discussed it with my advisor. He doesn't have much experience in that exact field but has been in the area for around 6 months. 

He said that he thinks that idea is feasible but doesn't know what would be the approach, and not sure whether the results would be positive in terms of PhD because there several parallel domains. Moreover, literature doesn't have much details on that exact topic. And my literature survey (pretty limited though) has only enhanced my confusion. Probably because my idea deals with application of a recent class of techniques to optimize a problem. 

Is it worth to convert to a PhD with such an idea ?
This is exactly how most of the really great PhD students I've seen have all started.  Working on a problem with unknowns is a process called "research."  

Clearly, you are going to have to find some collaborators with more experience in the field than your advisor, but that's part of the process as well.  Hopefully, your advisor can assist with networking / introductions / etc.  The support of your advisor will be critical (as it is for all PhD students).

The trick is to make sure you have a backup plan and a way to convert to it if necessary.  Don't spend three years mucking around if you're not making progress.  But, spending 6 months investigating if there is something there -- that's well worth it.