I think you are too good a man to cast aside a woman you have loved,--like a soiled glove,--because ill-natured words have been spoken of her by men, or perhaps by women, who know nothing of her life. 
I think you are too good a man to cast aside a woman like a soiled glove, because ill-natured words have been spoken of her by people who know nothing of her life.
