The Roman Empire routinely enslaved fair skinned Germans and Celts, and referred to those people derogatively as barbarians.
Pretty much all the ancient Mediterranean and Near East empires including Egypt, Mesopotamia, etc. practiced slavery, and like the Romans might also have drawn from more northerly, fairer skinned peoples.
However, we should be careful drawing too broad of comparisons. Slavery worked differently in different cultures. As bloodthirsty as the Roman Empire was, they actually had certain laws protecting slaves, that simply did not exist for slaves in the United States. In the Roman Empire, slavery was much more of a class-based institution; anyone could become a slave potentially - your excess children, prisoners of war, and debtors could all be sold into slavery. So, while the Romans did enslave Germans and Celts, it wasn't race-based as happened in the Americas; the Romans were pretty much an equal opportunity enslavement empire.
Anyway, slavery has been rife throughout history. It's hard to find any culture that didn't enslave others in some form or other, and that hadn't existed as slaves for someone else. It's just one of those nasty things humans do to each other until they learn better; there's nothing fundamentally inherent about skin color in any of this. It's just the random quirks of history and geography why things turned out one way rather than another.
As to today's current correlation of skin color and social status, this is most likely an outcome of the several hundred years of European colonialism applied to every corner of the world between 1492 and 1945. It was pretty intense, and unfortunately left quite an impression, involving not just slavery but also a deliberate attempt to erect a racist social hierarchy based on skin tone. These days colonialism as a governance principal may largely be gone, but in some sense the global media carries on many of those same biases in mass-marketed entertainment and advertisement.