Is it wise to select a PhD topic before choosing an advisor? I am a student of  Pure Mathematics interested in Linear Algebra and Abstract Algebra. I was browsing the Internet for good research topics containing both a flavor of linear algebra and also having some real life applications when I came to learn about Spectral Graph Theory.

I have searched for professors who are actively involved in this field, but I have only found a few names whose topic of interest is Spectral Graph Theory. There are also a very few number of books available in this field.

Is this a good topic to work on? Is it very difficult to find an advisor in this field? Though I am able to  learn the topic on my own, I am finding it difficult to approach someone in case I am having a problem.

Also, is it wise to first decide a topic and then find an institute which has professors in this field or it should be the other way round i.e., I should select a good institute according to QS rankings first and then decide a topic accordingly?
In brief: no, in mathematics in the U.S., you do not "have to choose" a research topic prior to beginning your PhD. At the same time it is _of_course_ good to have something to be enthusiastic about, curious about, fixated upon. But that can change, and probably should change as you learn more. It is apparently the case that in the U.S., and to some degree elsewhere, in mathematics, undergrads and even M.S. students really don't get enough background to understand contemporary mathematics at all, except very superficially. Fine, that's not a moral failing... maybe just evidence that there's a lot happening in mathematics... (!?!) But, then, in that light, the enthusiasms that novices have are necessarily (by-far-typically) ill-informed... and will change if/when they learn more.

This is not to say that one should not have an opinion, or should not admit interest in things because one knows one's interest will change... but only that one should anticipate that change, rather than believe that one "has arrived" at age 22 or so, and there's nothing left to learn, and now it's just "trying to solve problems/do research". I understand the appeal of that, but, if we think about it, it's better in the long run if mathematics is not that shallow or superficial.

More specifically, spectral graph theory is a quite viable research area. "Expander graphs" are popular and google-able. Lots of people work on this, both from the "spectral side", and from the "graph theory" side. 

And, perhaps counter-intuitively, the possible fact that there are not so many "books" treating a topic can in fact be a very positive sign... it's unclear. That is, if there are zillions of books on something, then chances are good that it's an older topic and has been worked over pretty hard... leaving less room for beginners to contribute.

So, sure, it's entirely reasonable to have specific and fine-grained interest... as opposed to "general" interest. Like having a specific lunch rather than a lunch-in-general. :) But be utterly open to changes in your viewpoint... as you learn more, which will be for many years. :)