I'm more bothered by people smoking cigars (or even cigarettes) near me.  If I can't tell my neighbours that they're not allowed to smoke tobacco in their backyards why should they be able to tell anyone that they can't smoke a J?