It would depend on what you mean by "none the worse for it." Strangers grabbing your hand when you'd just as soon not, for example. And it seems to me that the ones making a big deal out of it, are the ones who want it when it's not called for. Doing nothing when nothing is called for is what "not a big deal" looks like.