I understand some were priests...etc. from one tribe, but was there a hierarchy to the whole thing?
That is, were certain groups assigned to certain jobs as part of a "caste" system?
Answering this question depends on what you mean by "ancient Israel". If you are referring to the so-called Biblical period, the biblical literature presents us with what appears to be a three-fold stratification: Priests (kohanim), other levites, then regular Israelites. Priests, who comprise a subset of all levites, face various restrictions as regards whom they are allowed to marry and are recipients of special taxes. Levites face no such restrictions, but are entitled to certain charities. This is resultant of the theoretical prohibition on their owning land outside of the six levitical cities.
In addition, ancient Israel was a society in which slavery was not uncommon, and you could consider slaves therefore to denote a distinct caste as well. The Bible differentiates between Israelite slaves and non-Israelite slaves, the latter of which is human property: they are inherited after the deaths of their owners along with all other assets, and only go free if actively given their freedom. Their condition is hereditary.
If, however, you are referring to the period best represented by the Mishna (a period that comprises the latter years of the second temple, through to approximately 220 CE), then the presence of distinct castes is quite pronounced. The Mishna (Qiddushin 4:1) itself mentions no fewer than ten:
Priests, other levites and regular Israelites (points 1-3 above) - barring anything that might render them forbidden to one another - are allowed to get married. Barring the same exceptions, non-priestly levites, regular Israelites, priests born of illegitimate unions, converts and freed slaves (points 2-6) are allowed to get married. And so too, the individuals mentioned in points 5-10 are allowed to get married, barring reasons to the contrary. (Reasons to the contrary include incest and, in the case of women, already being married; since priests face various marital restrictions of their own, the women whom they are not allowed to marry constitute a broader demographic.)
Early rabbinic society, despite this social stratification, was also a meritocracy. In the words of the Mishna (Horayot 3:8), a sage born of an impermissible marriage takes precedence even over the high priest if he be an ignoramus (ממזר תלמיד חכם קודם לכהן גדול עם הארץ). As such, while the authors of the Mishna recognise the existence of diverse castes, they effectively replace it with two: the scholar and the ignoramus. These positions, needless to say, were not hereditary.
Ancient Israel was not a "caste" society, at least not in the sense that we know it in India.
You appear to be referring to the fact that the priests were drawn from the Levites, one of 12 tribes of Israel. That appeared to have been a "convenience" thing, to formalize the fact that a disproportionate number of priests (and temple workers) came from that tribe.
But that was exceptional; no other occupation was reserved for one tribe, and no other tribe "dominated" one occupation. And the Levites were not considered "better" than anyone else; if anything, they may have been consigned to that status for the tribes "past sins". So there wasn't a caste hierarchy built upon such specialization.
This is going to sound simplistic and possibly self-evident but Israel was a tribal society. And especially in light of Israel's ancient Hebrew texts this wasn't just a general 'tribal' notion but was delineated specifically historically, and by name, according to a fixed number of tribes. There is some debate whether there were 12 or 13 tribes since Ephraim and Manasseh, as Joseph's descendants, double Joseph's tribal representation (Joseph being one of Jacob's [Israel's] 12 sons). So there were in reality 13 tribes but only 12 were allotted land and the Levites in contrast were assigned specific cities in each of the 12 tribal lands of Israel and were subject to the hospitality of the host tribe.
The Levites indeed had a special status, but were the only tribe singled out in such a way for special roles in relation to temple worship, handling of agrarian tithes, and having no land inheritance, among other things.
The reason I go into this is because the Torah (Pentateuch or Law of Moses) spends far more time on inter-tribal relations than anything that could be considered 'caste' assignments in some heirarchical manner. For example the book of Numbers takes under consideration what happens when a woman in one tribe marries a man from another tribe in terms of inheritance transfer - out of a concern that one tribe may become so small from inheritance transfers over time that it would one day cease to exist. There was even a story (found in Numbers 27) of a man named Zelophahad who only had daughters, and the matter of the daughters' inheritance was brought before Moses when their father died since the rules at the time were not clear.
Even with the Levites though, they were seen neither as superior nor inferior to the other tribes in terms of the whole, and all tribes were expected to treat their brothers and sisters fairly and with equity; although of course the High Priest would have been one from the Levite ranks, and specifically had to be a descendant of Aaron, who was an individual of note (but then again many of the other tribes also supplied individuals of note in the form of judges, prophets, and kings). But hereditary succession of rulers and priests is a common feature of almost all ancient cultures and nations and can't be used exclusively to make a 'caste' argument.
Another answerer brought up slaves, but in that regard Israel's practice there must be compared to that of the entire Ancient Near East since all nations back then practiced it, especially enslavement of POWs and putting them to menial labor. However, that is neither prescribed by the Hebrew Biblical texts as something that should be done, nor is it discussed as a predetermined "lot in life" for a certain class of people as if it were a manifestation of the ideology of the Israelite religion in obedience to some kind of teaching or commands to enforce slavery.
I don't think Israel can be said to be based on a caste system unless one is willing to say that the majority of ANE nations were also based on a caste system (i.e. Egypt, Babylon, Hittite Empire, Assyria, Aram, Phoenicia, etc.).
Postscript: It is, however, worth noting – by comparison – that the Greek philosopher Plato did come up with some ideas of how society ought to be structured which bears some strong resemblance to caste systems. He even believed that each child was born with a different kind of metal heart, ranked from more precious to less precious metals, with the kind of metal they were born with determining their role in life. See this article on Plato's Five Regimes for more on that.