Please don't go all self-righteous on this. I have no hesitation in judging slavery to be an evil thing. But that doesn't mean that every monument erected in the era, or afterwards depicting an individual from that era, is evil as well. It's not. It's an expression of a collective sentiment in the day. 

By reading the surviving record, one can get a feel for what prominent people may have thought at the time. But we have no idea what each person who democratically went along thought of it. If we remove that statue, we censor that collective expression of free speech. And we do so not because that expression is hate speech, as a mere statue can never be. We do it self-righteously, to condemn what we, in ignorance of all that was involved, interpret as a negative thing.

The analogy to free expression today is that we should suppress the speech of an entire group, because we disapprove of the comments of some members of that group.