The Roman Empire routinely enslaved fair skinned Germans and Celts, and referred to those people derogatively as barbarians.

Pretty much all the ancient Mediterranean and Near East empires including Egypt, Mesopotamia, etc. practiced slavery, and like the Romans might also have drawn from more northerly, fairer skinned peoples.

However, we should be careful drawing too broad of comparisons.  Slavery worked differently in different cultures.  As bloodthirsty as the Roman Empire was, they actually had certain [laws protecting slaves][1], that simply did not exist for [slaves in the United States][2].  In the Roman Empire, slavery was much more of a class-based institution; *anyone* could become a slave potentially - your excess children, prisoners of war, and debtors could all be sold into slavery.  So, while the Romans did enslave Germans and Celts, it wasn't race-based as happened in the Americas; the Romans were pretty much an equal opportunity enslavement empire.

Anyway, slavery has been rife throughout history.  It's hard to find any culture that didn't enslave others in some form or other, and that hadn't existed as slaves for someone else.  It's just one of those nasty things humans do to each other until they learn better; there's nothing fundamentally inherent about skin color in any of this.  It's just the random quirks of history and geography why things turned out one way rather than another.