Key explanation: The increasing power of the nobles (and the sovereign's increasing dependence on nobles) (Blum, 1959).
Peters (2019):
Some background:
Serfdom first appeared in the Early Middle Ages (ninth through the eleventh centuries) ...
By the High Middle Ages (twelfth through thirteenth centuries) serfdom was breaking down. ... sovereigns were asserting their central authority over the nobility and freeing the peasantry removed a source of the nobility’s power ... At the dawn of the fourteenth century, there were few serfs in Eastern Europe and
serfs in Western Europe were increasingly gaining their freedom. ...
Consequences of the Black Death:
The population decline created two problems for the nobility, who earned most of their income from agriculture: lower prices for agricultural goods and higher labor costs ...
the absence of intervention in labor markets led to increased wages and decreased prices for foodstuffs, squeezing the nobility, who relied on income from their land. It is not surprising, then, that nobility throughout Europe wanted labor tied to the land. ...
Across Europe, nobles increased the demands on their serfs; asked for the sovereigns’ help in tying the peasants to the land; and/or for ordinances in both rural areas and towns to keep wages below market rates ...
So, nobles liked serfdom. In contrast,
serfdom was against the interests of the sovereign ...
Why? Nobles were a threat to the sovereign's power and
serfdom increased the power of the nobility
Serfdom was also an obstacle to the growth of the sovereign/central government's power:
Serfdom also presented a challenge to the centralization of legal authority and administration ... It limited the building of an effective tax bureaucracy as taxes were collected through the lord ... some sovereigns may have had no choice but to turn to the nobility to collect taxes and administer authority ... this hampered the development of state administration.
Sovereigns in the West successfully resisted the (re)imposition of serfdom. Those in the East did not. Why? This is Peters's (2019) main argument:
the sovereign’s decision to reimpose (or impose for the first time) serfdom was based on whether he needed the nobility to provide ... financing ... During this period, wars and defense were becoming vastly more expensive ... Wars could be financed from revenue or loans from towns, loans from foreigners, revenues from foreign colonies or commerce (including privateering/ piracy), support from the Catholic Church, or revenue or in-kind support from the nobility ... Where sovereigns did not have alternatives to support from the nobility, they imposed serfdom. Where sovereigns could finance war from other sources, they were less likely to impose serfdom. ...
Western sovereigns had alternative sources of finance, while Eastern ones did not. Peters gives these examples:
Poland:
In 1496, with the treaty with the Ottoman Empire due to expire the following year, Poland needed to field an army of at least 40,000 to deter Ottoman aggression ... The king did not have the revenue to hire mercenaries or, even, to pay for an army of conscripts. Instead, he had to turn to the nobility for a levee en masse. In return, the nobility forced the king to agree to the Statute of Piotrkow, the Magna Carta of Poland. The Statue [sic] also began the process of enserfing the peasantry. As the nobility gained increasing power through parliament, they were granted additional serfdom laws in 1501, 1503, 1510, 1511, and 1520
England:
while English Kings ... needed the assent of parliament to obtain new revenues, they had several regular sources of revenue. These included duties on exports of wool, which after the Black Death made up at least 50% of revenue; loans from Italian merchant banks; and later loans from English Merchants ... Thus, while as part of the initial reaction to the Black Death the King enacted a wage ordinance in 1349 and the 1351 Statute of Laborers to limit peasant mobility, in 1427 the king enacted a new statute that exempted employers from prosecution for any breaches of the Statute of Laborers over the objections of the nobility ... The unwillingness of the king to enforce existing laws and enact new ones allowed market forces to do their work, leading to the de facto end of serfdom in the fifteenth century
Aragon:
well-organized groups of peasants developed syndicates to raise money for redemptions and lobbied the monarchy to secure more freedoms ... The monarchy used these syndicates against the nobility to further its own ends against the nobility. In 1448, the monarchy formally allowed the peasants to form syndicates to raise money for redemptions ... The Corts, the parliament representing the nobility, denounced the legislation and in response King Alfonso dissolved it ... He could do so because the syndicates financed his military campaign in Florence ... while the Corts offered Alfonso 30,000 florins to pay for his campaign, the syndicates offered 100,000 florins ...
The conflict between the nobility and the monarchy turned into civil war over a succession crisis. ... In return for the peasants’ support, there was a de facto end of serfdom during the civil war.
In all this, there was a feedback loop:
More/harsher serfdom → Increases nobles' wealth and hence power → Sovereign becomes even more dependent on nobles, fails to develop central state and other sources of revenue → Nobles demand more/harsher serfdom (and more generally, more privileges and more power); peasants become more helpless; sovereign/military becomes more inclined to take the side of nobles (this includes ruthlessly crushing peasant revolts) → More/harsher serfdom → Repeat
Decline of towns/cities
In the East, towns/cities (e.g. Hanseatic League) had previously flourished.
But from the 1400s, nobles got the sovereign to promote anti-urban policies and hence their own interests. They succeeded in "break[ing] urban monopolies of foreign and domestic trade", ending "the cities' practice of receiving runaway peasants", "bypass[ing] the cities in trading, selling their grain to foreign merchants who were now allowed to come to their estates, or exporting it themselves", gaining "favorable tariff provisions for their shipments that gave them a price advantage over the shipments of the bourgeois merchants". Altogether they "succeeded in stopping the flow of peasants upon which the cities depended for their continued growth" (Blum, 1957, p. 834).
This decline of towns/cities was thus another important ingredient in the above feedback loop, continually reducing peasants' escape options and increasing nobles' power.
In contrast, in the West, towns/cities/middle classes grew. Sovereigns/governments (who were not from the beginning already dependent on landowning nobles) were happy to nourish their growth, for they could provide growing tax revenues and also help develop banking/public finance (a different and better feedback loop moving away from serfdom/feudalism towards the Industrial Revolution).
Added:
Colonies and trade were also another source of finance for Western sovereigns that was largely unavailable to Eastern ones.
The grain trade did add to the feedback loop in the East: Nobles wanted peasant-serfs to work their land.
But as OP has pointed out, as usual, we must also explain why this didn't encourage serfdom in the West. So the grain trade was an ingredient that added to the feedback loop in the East, but fails to serve as a fundamental explanation for why serfdom died in the West but grew continually harsher in the East.
Again, the fundamental explanation is that in the East but not in the West, nobles were powerful enough to get sovereigns to do their bidding.
See also Schöffer (1959), "The Second Serfdom in Eastern Europe as a Problem of Historical Explanation", who had this to say of the usual Black Death "explanation":
A superficially convincing explanation of the second serfdom is the view that as a reaction against this crisis, and in particular against depopulation and desertion, the landlords in Eastern Europe tried by force to keep the few labourers left to them. ... But this explanation gives rise to another question: why did the landlords in the West react exactly in the opposite way? To deal with Eastern Europe as an isolated problem is to be like a hedgehog who, by rolling himself up, is unconscious of what happens around him.
