It's weird what people are offended by. My mates make terrorist jokes at me quite often and I think it's hilarious but if they referred to me as a paki I'd be pretty annoyed, as i am not one. In your case though it's about the intention. You know it isn't with malice from your friends and family but it might be when it's from a stranger.