At the age of 27 is it worth changing graduate schools and starting again? I am interested in areas of theoretical physics/mathematics which simply don't exist in my institute. I tried 2 or 3 different groups here which were not in my interest (but I felt I had some transferable skills) and it didn't work out.
There is no age limit for graduate studies. People are free to apply at whatever stage of life they choose, if they feel it's the right move for them.

As an example of this, a very good friend of mine was a social sciences major as an undergraduate, and worked in Washington, D.C., for a number of years before leaving politics and starting a PhD program in medical physics—and he did this in his mid-thirties. I've worked in the same department as other postdocs who made the career choice even later—they were in their early fifties!

So I would not look at your case as hopeless at all. If you find something else that inspires you, go for it.