The phrase "left over" implies slaves doing most of the work. While seemingly intuitive, it is in actuality quite implausible. For most of history in farming based civilisations, the vast majority of humans were engaged in cultivation. Percentage wise this is far higher than the proportion of slaves, even in the most heavily enslaved populations. Thus, while certain professions may become dominated by slaves in some places, farming always relied on free labour.
In fact, the (admittedly arbitrary) standard formulated by Sir Moses Finley uses 20% as the cut off for defining a slave society. By this metric, there are only two non-colonial examples: Roman Italy and Ancient Athens. Hence, although the question ruled it out, I will cite Rome as an example why agricultural labour demand exceeded the supply of enslaved labour.
In Rome, even by the mid Republic,
slaves were relatively scarce at Rome; the working classes in the city were still largely free natives, the farms were usually owned in small plots by working farmers, and the few slaves on them were still treated . . . as members of the household.
Frank, Tenney. Life and literature in the Roman Republic. Vol. 7. Cambridge University Press, 2015.
Roman slavery expanded as Rome expanded around the Mediterranean. Slaves became a majority of the domestic and urban work forces, as well as a significant component of the rural labour in massive aristocratic estates. Yet even at the peak of Roman slavery during the Empire, free peasantry persisted in the countryside.
[I]t is not true that Roman society was based on slavery. The system of large estates cultivated by slave gangs was limited to certain regions such as southern Italy and Sicily. Elsewhere, estates were worked by sharecroppers and hired laborers as well as slaves; in some provinces, such as Egypt, rural slavery was virtually unknown.
Veyne, Paul. The Roman Empire. Harvard University Press, 1997.
Veyne goes on to note that slaves, forming an estimated 25% of the Italian labour force, supplemented free workers, but did not replace them. Hired farm hands, sharecroppers and other free tenants remained a large majority of the rural labour pool, even within Italy, where slavery was much more extensive than the rest of the Roman Empire.
The situation in Ancient Greece was similar to pre-Imperial Rome. Most citizens in most city states were free farmers, although wealthier families may have one or two slaves to help work on their farms. Sparta infamously subjugated the helots, but they were more like serfs than slaves. Spartan society further contained the perioeci, who were non-citzens and yet free.
[The perioeci] enjoyed personal freedom and local autonomy, but had no political rights in Sparta. Many of them were farmers. After Spartan male citizens made themselves an elite whose sole legitimate occupation was fighting, the perioeci filled this vacuum by becoming craftsmen and traders . . . It appears that the helots were sharecroppers, who gave up half of their crops. This form of payment constituted an easier and more flexible burden than fixed taxes.
Roisman, Joseph. Ancient Greece from Homer to Alexander: the evidence. John Wiley & Sons, 2011.
Ancient Athens, the only other "genuine" slave society of antiquity, is a bit of a an exception. Here the picture is complicated by wide disagreements over how many slaves were there in Athens. Classical sources claim a slave population of 400,000, which is widely dismissed by modern scholars, but if true would seem to imply there were no possible jobs for free lower class Athenians. This angle is sometimes held when Athens is described as an aristocracy of 20,000 citizens supported by 400,000 slaves - that is to say, all poor people were slaves.
More credible modern estimates vary wildly, but all generally puts the slave population at about 1/5 to 1/3 of the total Athenian population.
A. H. M. Jones . . . calculated that there were about 20,000 slaves in a total population of circa 144,000. Other estimates for the same period are cited by Victor Ehrenberg. One of them, attribtued to A. W. Gomme, gives a total population of 258,000, including 104,000 slaves. Another estimates the population at between 140,000 and 190,000, including between 30,000 and 60,000 slaves. William Westermann concluded that the slaves of Attitca, during the early part of the Peloponnesian War, which began in 431 B.C., numbered from 60,000 to 80,000 or about a third or possibly a fourt of the population.
Sellin, J. Thorsten. Slavery and the Penal System. Quid Pro Books, 2016.
Assuming they are correct, this leaves plenty of work opportunity for free Athenians labourers. This is indirectly verified by the fact that Athens boasted a large metics population, free immigrants from elsewhere in Greece who had moved to Athens for the economic opportunities present there. This could not have happened if slaves saturated the labour market.
It should be pointed out that slaves were not without costs. Although they were often cruelly exploited, slaves do require certain levels of upkeep, and represent a non-trivial up front investment with a significant risk of early death. Thus, while slaves diminish the value of free workers, an abundance of free workers likewise reduce the attractiveness of enslaved labour. Outside of temporary supply side shocks (e.g. a recent victorious war) or demographic shocks, this functioned as a sort of self-corrective market mechanism that balances labour costs.
As a final note, in slave societies, truly desperate people often could sell themselves into slavery, or be enslaved for their debt.
