The Roman Empire routinely enslaved fair skinned Germans and Celts, and referred to those people derogatively as barbarians.
Pretty much all the ancient Mediterranean and Near East empires including Egypt, Mesopotamia, etc. practiced slavery, and like the Romans might also have drawn from more northerly, fairer skinned peoples.