Can principal investigators use leftover grant money to take a vacation with their families? Can one use leftover grant funding to take a vacation?

Or does leftover funding get returned to the funding agency? 

For reference, I am asking about research labs in American universities.
Without knowing the actual laws or seeing the grant proposal, I think I can pretty definitively say no, you can't do that. If it is noticed, it would likely generate an audit of the entire grant. In an extreme case (NSF funding) it could result in jail time as fraud or embezzlement. 

However, if the grant is from some private party, the rules might be very loose, though I doubt it. 

However, if the topic of study in the grant were somehow related to leisure time or such then what seems like a vacation might actually be research. (We should all be so lucky.) 

What happens with unused funds is likely spelled out in the grant itself or in the regulations of the funding agency. They might go back to that agency, or perhaps remain with some other institution. 

But vacations? No. Pizza night for the research team, maybe. Unlikely.