I don't know of anyone who tolerates violence. Which happens anywhere you have large numbers of people. Violent people need to be locked up, but we have filled our jails and prisons with people who use or sell drugs and other nonviolent offenders who do things the middle class disapproves of.  Which is very expensive and doesn't leave enough capacity for the violent. Incarceration has also become our default mental health system, because the middle class is not willing to pay taxes for mental health treatment.

Being attacked, downtown or anywhere else, is a fairly random occurrence, like being struck by lightning. I live near downtown and have walked around it for over ten years. I have never been attacked and only once has someone said something rude to me. I don't feel any more threatened there than I would anywhere else. We live in a violent society.

What I will not tolerate is excluding poor people from downtown because of irrational fears. They have rights.