I told a co-worker that, and he laughed at me. He put all the blame on the government.
While government was indeed complicit in it, I reminded him that the Constitution wasn't written until slavery was already well-established here. It took government action, and a VERY bloody war to end it. Well, at least "end it" nominally and in spirit.
There's long existed money-grubby psychopaths willing to crush others underfoot, and "We The People" shouldn't overlook them for a second!
Here's a "liberal narrative" that I read a few years ago (which I liked):
http://www.saturdayeveningpost.com/2012/08/21/in-the-magazine/trends-and-opinions/americas-painful-divide.html
Once upon a time, the vast majority of human persons suffered in societies that were unjust, unhealthy, repressive, and oppressive. These traditional societies were reprehensible because of their deep-rooted inequality, exploitation, and irrational traditionalism.
But the noble human aspiration for autonomy, equality, and prosperity struggled mightily against the forces of misery and oppression, and eventually succeeded in establishing modern, liberal, democratic, capitalist, welfare societies. While modern social conditions hold the potential to maximize the individual freedom and pleasure of all, there is much work to be done to dismantle the powerful vestiges of inequality, exploitation, and repression. This struggle for the good society in which individuals are equal and free to pursue their self-defined happiness is the one mission truly worth dedicating ones life to achieving.