"Loitering" on public property is not a criminal offense. That's why it's called "public" property. The homeless are US citizens and part of the public. Under the US Constitution you cannot ban people from public property because they're poor. Of course, that hasn't stopped the cops from continually harassing them, even when they have no legal basis to do so.

As for drug use, I don't see that. Maybe it's because I don't care and mind my own business.

Middle-class comfort is not high on my list of priorities. If you don't want to see poor people, do something about our economy. High rents and low wages are killing working-class people.