It's weird what people are offended by. My mates make terrorist jokes at me quite often and I think it's hilarious but if they referred to me as a paki I'd be pretty annoyed. In your case though it's about the intention. You know it isn't with malice from your friends and family but it might be when it's from a stranger.