"Very, very few people had the 'luxury' of slaves [...]"

Just once more to puncture this assertion:  very few people had _ownership_ of slaves, because ownership of all property was concentrated in the male heads of households.  So, the line gets repeated about how R.E. Lee never owned a slave -- true, he just lived (when not on military postings) in the household of his father-in-law, who owned more than 200.  In this family, only one white southerner in eleven actually owned slaves, but they all lived off the avails of human slavery.