In principle, they were not supposed to leave their order because they took a vow for life. In practice, some did leave, and some were subsequently readmitted. However, the time period covered here is very long, and different orders - and individuals within orders - were probably sometimes more, sometimes less inclined to allow people to leave. Consequently, some just ran away. In short, it's a little complicated.
For Franciscans,
When the year of probation is over, let them be received into
obedience, promising to observe this life and rule always; and,
according to the command of the lord pope, it will be absolutely
forbidden to them to leave the order,...
Novices, though, could leave. This next source on Augustinians also makes this clear:
At the end of the probationary period, the novice was brought to the
chapter and asked whether he wished to leave and go his own way or
offer himself to God and the order.
Source: Frances Andrews, 'The Other Friars: The Carmelite, Augustinian, Sack and Pied Friars in the Middle Ages'
For Carmelites, the same 'rule' applied:
Once professed, a brother remained subject to the order for life.
Source: Andrews
...and for Benedictines, whose vows included the vow of stability which
binds the monk in both body and spirit to the community of his
profession for the rest of his life, where he serves under both a Rule
and an Abbot.
Thus Benedictines, unlike Franciscans for example, weren't just bound to the order but to a specific monastery.
However, despite these vows, there were occasions when monks and nuns were either allowed to leave, forced to leave or ran away. The writer (or writers) of the 6th century Regula Magistri (Rule of the Master) were not so naïve as to believe that some recruits would not want to leave:
...in the Rule we read that the new member’s old clothes are to be
“placed in the vestiary to be preserved; so that if, at any time, the
devil persuading him, he shall consent to go forth from the monastery
— may it not happen — then, taking off the garments of the monastery,
he may be cast out”
Source: Sherri Olson, 'Daily Life in a Medieval Monastery' (2013)
Being readmitted was possible, even after leaving more than once. On "Whether brothers who leave the monastery ought again to be received",
A brother who goes out, or is cast out, of the monastery for his own
fault, if he wishes to return, shall first promise every amends for
the fault.… But if he again departs, up to the third time he shall be
received. Knowing that after this every opportunity of return is
denied to him.
Cited in Olson
One such case was recorded at Ouville in Normandy:
In 1249 at Ouville, a house of Augustinian canons, Eudes orders the
prior to prepare a separate place in the house for John Gaul, who had
often withdrawn from the order. He was not to be allowed to be at
large in the monastery, no one was to talk to him without permission
from the prior, and a book should be given to him “that he may sing
his Hours” alone. He is free to leave the house altogether if he
wishes, but if he does so he is to be “permanently expelled from the
order.”
Source: Olson
And also in 7th century Gaul for nuns,
Waldebert adapts for nuns a chapter of the Benedictine rule in which
the author suggests a more relaxed enclosure as it applied to monks.
Benedict allows the monk up to three opportunities to return to the
community if he “through his own evil action departs from the
monastery.” Once readmitted, the monk was expected to make amends
for his faults after which he returned to the community in the lowest
rank.
Breaking rules was one possible way to be forced to leave, but this could also backfire and lead to imprisonment. For example, among Carmelites in the 13th century,
Inevitably, not all recruits were ready to keep the observance or
happy to remain, but any departure from the rule was subject to harsh
discipline. Already in 1227 Gregory IX allowed the prior to deal with
apostates (runaways) from the order and by 1269 the prior general was
authorised to excommunicate and imprison both apostates and the
disobedient.
Source: Andrews
Over time, though, it is clear that different standards were applied and that much depended on the Abbot or Abbess. One could simply ask, as in the case of the story of this monk told by Pope Gregory the Great in his biography of St. Benedict:
A monk who “had become unstable in spirit” continually asked Benedict
to let him leave the monastery and made such a nuisance of himself
that Benedict “angrily ordered him to leave”
Source: Olson
Getting out with the approval of a higher authority clearly wasn't easy sometimes. One nun, Joan of Leeds, resorted to faking her own death. In 1318, the archbishop of York, William Melton, wrote in a letter that Joan,
“with the help of numerous of her accomplices, evildoers, with malice
aforethought, crafted a dummy in the likeness of her
body in order to mislead the devoted faithful, and she had no shame in
procuring its burial in a sacred space amongst the religious of that
place.”
Even for royalty, abandoning one's vows could bring about stern sanctions. On the death of Offa, King of Mercia, Eadberht III Præn of Kent led a rebellion in 796 AD against Mercian control, apparently abandoning his vows in the process. Eadberht claimed otherwise but Pope Leo III recognized
Eadberht’s earlier ordination and consequent ineligibility for the
kingship....The Church had no alternative but to condemn him,
excommunicate and reject him, as Pope Leo put it, ‘having regard to
the safety of his soul’
Source: D. P. Kirby, 'The Earliest English Kings' (2001)
One option to life-long vows for men and women from the 13th to the 16th centuries was to be Beguines (women) and Beghards (men) instead. They
...lived in semi-monastic communities but did not take formal
religious vows....although they promised not to marry "as long as they
lived as Beguines," to quote one of the early Rules, they were free to
leave at any time.
Monks and nuns could also be given the choice to leave, or forced, when larger events came to the fore. A classic example is the Dissolution of the Monasteries in England, when the devotion of the vast majority of monks and nuns became evident:
Nunneries were especially vulnerable as the wheels of the Dissolution
began to turn, because the smallest and poorest houses were targeted
first. In 1536 an act was passed to “suppress” (i.e., take into the
king’s hands) the “lesser monasteries,” defined as houses with an
annual income under £200.... Given the choice to leave the monastic
life or be transferred to another house, most nuns chose the latter:
out of a total of roughly 900 women, only 100 chose to renounce their
vows in 1536–1537.
Source: Andrews
Eventually, though, most of these larger houses were closed as well so thousands of monks and nuns were forced out.
Also subject to powers beyond their control were the some Sack Friars, who survived by begging. The order of prohibition of some mendicant orders was formalised at the Second Council of Lyon in 1274. Otto,
the rector general, wrote to his members to tell them that, as per constitution 23,
...they could now leave the order: those who wished to, should do
‘whatever seemed good in their own eyes’, a direct contradiction of
the normal stipulation that regular religious must submit to the will
of the superior without hesitation.
Source: Olson