It has come to this because people advocating a violent response to vile words fail to understand that is exactly the response the groups touting racism and bigotry want.  They are a few (really a few in the context of 300 million in USA) folks who believe if they can incite you to commit violence then 1) They will gain more attention and supporters who might now be on the fence because it is all just talk; and 2) They can then react with violence and say: "All we were doing is talking, you started it.  We will respond with more violence to defend ourselves."   

Yes punching these fools would feel good.  Resist temptation, it is exactly what they want you to do.