There are a lot of theological/semi-theological answers that exist, but there is a purely secular answer too. Almost all cultures like the Arabs, Goths, Gauls, Turks, Chinese, Greeks etc. historically existed as multiple distinct tribes, who did not inter-marry that much, had significant differences around culture, religion and politics, but nevertheless formed a single cohesive social grouping for the rest of the outside world. Also, almost every culture had a very feudal order that severely concentrated wealth with a few elites. The fundamentals of the Hindu caste system is hence nothing unique to India, and was easily the status quo in almost all nations historically.
Even regarding class/caste mobility, it is somewhat incorrect to say that Hindu society did not have that. The rules of caste interactions were themselves an evolving feature, and castes and sub-castes did keep getting pushed up and down, along with new political and economic changes. In the south, there are a quite a few sub-castes, among the upper-caste communities who became untouchables for various reasons, and there were many traditionally middle caste communities who went on to have more de-facto privileges than the traditional upper caste communities, because of trading rights, spice cultivation etc. You would be surprised that even inter-religious marriages, in the case of Kerala and most of the West Coast for example, have been legal for over a millennium, whereas even in Europe they were largely illegal till the renaissance.
However, there are a few unique aspects to the Indian caste system too. Whereas the Goths form only two tribes (Protestants/Catholics) today, and the Gauls just one, Hindus still comprise 100 or more tribes. One could say that the tribal divisions have persisted much more in India than in almost all other cultures. Even in countries like Indonesia, which technically have over a hundred ethnic groups, society tends to more cohesive de-facto (more intermarriages) than in India. Also there is a continuing antagonism largely over the historical memory of that feudal order, in the form of modern caste politics. However it could easily be argued that such antagonisms are significantly less intense than those caused by other social factors like in say, Bosnia or Kashmir.
In many ways, asking about the origin of the caste system, is as broad as say, asking about the origin of the Slavic cultures. The origin of these orders can't be fixed to any particular historical periods, let alone to divine revelations or religious texts, and there almost certainly exists some continuity going back to even the pre-neolithic age. Texts like Manusmriti and others are significantly descriptive of an already existing social order rather than something that started off the caste issue. Also every social order in the world, has intersected to some extent with the religious views of the parent society, like justification of racial theories in the West against some Biblical details, or the politics of the Yasukuni Shrine in Japan. But many like me would argue that the underlying social details are much more fundamental than the religious veneer that they seemingly exhibit, and the caste system in India is no different in that regard. The best one can hope to do is understand as much as possible about recorded history, and try to extrapolate trends to get a clearer picture. Indian social history is quite well recorded, and one can read early writings like Xuanzang's records, and Akbarnama to get started.