The area of Europe is given as 3,931,000 square miles. If between one half and one quarter of that was divided into manors in medieval times, that gives about 982,750 to 1,965,500 square miles of manorial land.
One site says that the average (English) manor was about 1,200 to 1,800 acres. With 640 aces to a square mile the average the average (English) manor was about 1.875 to 2.8125 square miles. Thus manorial Europe might have had as many as 349,422.22 to 1,048,266.6 manors.
The site on medieval manors says that England after the Norman Conquest in 1066 had over nine thousand manorial estates. Medieval England had an area of about 50,346 Square miles, so the total area of manorial Europe should have been about 19.519 to 39.038 times as large as the Kingdom of England and thus should have had about 175,679 to 351,358 manors. http://www.lordsandladies.org/medieval-manors.htm
So there should have been hundreds of thousands of manors in medieval Europe.
Most manors had a manor house where the lord of the manor and his family lived full time if it was his only manor or part time if he owned a bunch of manors.
The total number of castles in medieval Europe is unknown but must have been in the tens of thousands. Each castle was supported by the neighboring population of one or more manors. A castle might have tens of persons defending it and/or taking refuge in it during a siege, or sometimes hundreds, or in a few cases thousands.
If a lord of only one manor was besieged in his only castle, the besiegers might do enough damage to his peasants and his property to ruin him financially.
The original question asks about a ruler, and that seems more important than the lord of only one manor. If "ruler" means an important feudal ruler like a count, Duke, Margrave, etc., there were only a few hundred or a few thousand of them at any one time in medieval Europe, so an average one might have at least tens of castles and at least hundreds of manors in the territory he ruled, though most would be owned by his subordinate lords.
have there been rulers who outlasted a siege but were left with only themselves and the inhabitants of their castle to rule over?
refers to kings, there were only about twenty or fewer kingdoms at a time in castle building parts of Europe from 1000 to 1500 AD, so the average kingdom included hundreds and thousands of castles and manors. So if invaders besieged a king in a castle and slaughtered all the peasants in a few nearby manors, the King would likely have the vast majority of his tax paying subjects left after the invaders gave up.