If apparently the mainstream agreement is that 'freedom means you can do whatever' so  people can have rights then why can no one state they dislike certain people? It's a preference that 'harms no one' as you said, so it should be perfectly acceptable. But it's not. So then we only apply freedom to some people, not others?