How does your area of research affect your chances of getting job in academia? I've heard that the trendiness of your area of work is one of the important factor that has am impact your chances of getting job in academia. How far is this true?

Also, how do I decide what areas are trendy and what areas are not in the field I'm interested that is mathematics?
Trendiness can be factor, but a much more important factor is what needs the hiring department has. If they're chock full of topologists, it doesn't matter how "trendy" your work in topology is: there's a good chance they might be looking to expand in other areas. 

OTOH, if they're looking in your area, then being at the cutting edge can't hurt. The danger of course is timing: trends rise and fall quickly, and what might be trendy right now might not be so in a year or two (maybe in mathematics, trends move more slowly). 

So I'd argue that trendiness, while one potential factor, is not a universal key factor at all. Every department has their own needs. 

As regards your second question, usual measures of "trendiness" include the relative frequency of publication in that subarea. In CS, it's relatively easy to determine this via conference acceptances, but in math I'm not sure how you'd go about measuring this frequency. It also helps to attend big meetings and listen to where the buzz is. An advisor can also be helpful in this regard (especially if they're pre-tenure and need to keep track of trends themselves :))