You're right that people shouldn't judge you by your name, but life's not always fair. For example, a woman who claims to have been sexually assaulted should not be told it was a result of what she was wearing - very true. But, I would tell my daughter to be careful about what she wears because some will think that's a licence to assault. It's just 'playing safe'. Is it fair? Of course not.
You should be willing to change your name in order to appeal to those who have been traumatized. It's not fair, but that's life.