Why are grant indirect costs so high? I am preparing an NSF grant proposal in mathematics. I am requesting summer salary and a travel budget (and not much else), and after I compute the total I have to add an extra 50% of it and add it as the "indirect costs" of my proposed research.

This feels absurd to me (although it is the same, sometimes with an even higher rate, everywhere in the United States) -- especially since I do not require any special facilities or equipment to conduct my work. Is there a convincing argument to be made that this is reasonable?
The indirect costs help to pay for all of the other resources you have access to which are not directly paid for by other costs. For instance, these indirect costs help to pay for administrative assistants, information technology, library subscriptions, utilities, and other costs.

They can also go to pay for the costs of administrators (including the fine folks in your university grants and contracts office who actually submit your NSF proposal).

Whether or not the rate is reasonable for your university is hard to judge, but a 50% overhead rate overall is not that excessive. Compare that to industry or the US national laboratories, where the overhead rate can exceed 200%, and things don't look so bad in comparison.