When changing fields from undergrad to PhD, which department should recommendation letters come from? I am a physics undergrad who has worked with profs mostly on areas in Quantum field theory, string theory. However, my interests have changed slightly over the areas, and now I want to pursue a PhD in Pure Mathematics, perhaps in algebraic geometry or topology. Is it OK, if I apply for a math grad school with recommendation letters from physics profs, or would this diminish my chances to get selected? Should the recommendation letter be given by a prof working in the same area as that you want to apply to?
I agree with blackace that you should probably be OK, but let me sketch what some of the drawbacks are:

Ideally, an application to math grad school will have recommendations from mathematicians.  The further you get away from that, the less meaningful the letters are.  (For example, at least once per year I see a letter from an English professor, which is utterly unhelpful.)  The basic issue is that you need recommenders who really understand what it takes to succeed in math grad school and as a mathematician.  Fortunately, physics is close enough that physicists can do a pretty good job of judging this, so you should be OK.  In my experience, the admissions committee will worry about two things:

One is that physicists may not appreciate certain math-specific issues.  For example, the expected coursework and background.  A physicist may not fully understand the extent to which someone's background is nonstandard or deficient for the math program they are applying to.

A second reason is the belief that most people's standards go down a little when making recommendations for other fields.  If someone is applying to the top schools in your field, you know very well what the standards and competition are like, and you have something invested in the system and your own reputation as a recommender.  In practice, recommenders from other fields seem to be a little more cavalier about making strong recommendations based on a feeling that the applicant is smart, rather than a comparison with the rest of the applicant pool.  This means recommendations with be taken with a grain of salt.

So if you have equally good prospects for letters writers from math and physics, you should choose the mathematicians for math applications.  On the other hand, a physicist who knows you is still a good choice, much better than a mathematician who doesn't know you.  (But a mathematician who doesn't know you is a better choice than an English professor who does.)