Why were bastards historically not able to inherit nobility or monarchy from their parents? What about being a bastard made one unfit to rule? For example William IV of the UK was succeeded by his niece Queen Victoria instead of any of his children because all 10 of them were bastards.
Because for society as a whole, a peaceful transition of power is infinitely more important than honoring whatever rights a bastard might claim.
The most dangerous time for any kingdom (or nation, really) is the interregnum between the passing of the old leader and the assumption of power by the next. Kingdoms which had a well-ordered process surely tended to survive better than those that didn't. And the most important aspect of that process was that the successor be recognized as being legitimate by the broadest population and that legitimacy was as unimpeachable as possible. The competence of the new king is secondary to his legitimacy. It usually takes years for the costs of incompetence to become apparent and good advisors can mitigate the damage, but the price of civil war is immediate and extreme.
Most kings understood the importance of a successful transition - not just for their heirs but for their subjects. Therefore they subscribed to the process of making their heir - not just fathering children, but reinforcing the legitimacy of the their heir by every means possible. The relationship with the mother would be sanctified by marriage. The mother must be of as high a station as possible. The heir would would be granted as many titles as possible. In most societies, and particularly in Europe, the clergy would further reinforce the inheritance in every way possible, including general condemnations of bastardy.
There were other proscriptions besides the one against bastardy. In Europe in general, monks and priests were barred from the throne, so one way to eliminate a rival was to force them into the priesthood. (Or, if you were a Merovingian, just kidnap him long enough to give him a tonsure. That seemed to be enough.) From a modern perspective, it might seem silly to prevent a member of the best educated segment of society from reaching the throne. But this was a useful tool for ensuring the main goal: a peaceful transfer of power.
Most "bastards" were sired by their fathers with women of a lower rank than the fathers. Kings and princes would seldom have access to "princesses" (who were protected and treated like vestal virgins until they married) outside of wedlock, but they would have numerous relations with other women, ranging from servants to lesser nobility, who were "too low" to marry. Because of the "ineligibility" of the mothers, the children were not eligible to receive the full inheritance of their fathers. (Bastard children were sometimes allowed a "truncated" inheritance.) James Stewart, Earl of Moray (a truncated inheritance), was the older brother of Mary, Queen of Scots from such a situation.
There was even an institution called morganatic marriage, under which a king would "marry" a non-noble woman, but their children would not inherit his title because the mother was not an "eligible" woman. When Archduke Francis Ferdinand of Austria Hungary married a Bohemian Countess, the marriage itself was (barely) recognized, but their children were barred from ascending to the throne; he was supposed to marry someone "higher." So the countess received the title of "princess," which was lower than "archduchess" in Imperial Austria-Hungary.
What I wrote started with kings and princes, but of course, the principle applied to the rest of society.
In certain countries, at least. Norway and Denmark's early medieval rulers were bastards more often than not; Denmark had six bastard sons becoming kings in a row, and Norway possibly eight. One of the more well-known English kings, William the Conqueror, is also known as William the Bastard, and started out from his inherited Normandy.
Whether being born in wedlock gave you right to inherit or not is a matter of custom, and not something that is inherently better: you can certainly find bitter feuds between siblings all born in wedlock. This custom is something that ties in strongly to Christian ideology surrounding marriage, and the producing of children as the only legitimate reason to have sex. Since Christianity was the dominant ideology in Europe in roughly the same era monarchs and nobles held most power, it was mostly in areas that were not yet or recently converted and where Christian ideas had not fully taken root that bastards could become kings.
Different societies around the world had different succession practices, customs, rules, or laws.
Ancient Roman laws restricted inheritance to the legitimate sons and daughters of a man. Since the marriage of Gaius Julius Caesar with Queen Cleopatra VII Philopater of Egypt was with a foreigner and not recognized by Roman law, their son Ptolemy XV Caesarion was illegitimate by Roman law and not eligible as Caesar's heir.
The tens, hundreds, or thousands of pagan societies in Europe each had their own rules of succession to property and thus to any hereditary lordships and kingdoms they may have had. Thus there was much variation in whether illegitimate children could succeed.
As Christianity spread across Europe and became the majority religion in country after country, the Christian clergy defined marriage as a Christian religious sacrament, though secular non Christian marriages continued to valid for centuries in many parts of Europe - so long as there was no paganism involved.
The Christian clergy also gradually succeeded in limiting inheritance to legitimately born children.
Examples of European rulers of illegitimate birth, and/or problems caused by accusations of illegitimacy include:
1) [Added 07 May 2017] King Pepin of Italy/Lombardy (773-810), second son of Charelmagne, died in 810 and his illegitimate son Bernard (797-818) was appointed King of Itlay or Lombardy despite his illegitimate birth. Bernard's descendants the Counts of Vermandois were the last branch of the Carolingian Dynasty.
[added 05-18-2017] But here it is pointed out that only one source claimed that King Bernard was illegitimate.http://fmg.ac/Projects/MedLands/ITALY,%20Kings%20to%20962.htm#PepinIItalyB1
2) [added 05-18-2017] Emperor Charles III (839-888) tried to make his illegitiamte son Bernard (c. 870-891/92) his heir in 885 and 886.
3) German King and future Emperor Arnulf (c. 850-899) - who may himself have been illegitimate - made his illegitimate son Zwentibold (c. 870-900) King of Lotharingia in 893.
4) [Added 07 May 2017] Edgar the Peaceful (c. 943-975,) King of the English, had two young sons by two different women. Edward (c. 962-978) was possibly the illegitimate son of Edgar and a woman possibly named Aethelflaed, and Ethelred (c. 966-1016) was his legitimate son with Queen Aelfthryth who he married in 964.
When Edgar died one party wanted to make Edward king because he was years older and possibly for all we know legitimately born, while another party wanted to make the certainly legitimate Aethelred king despite being less ready than Edward.
Edward was chosen king, and on March 18, 978, was murdered while visiting Aelfthryth and Aethelred, thus becoming Saint Edward the Martyr while Aethelred became the next king, ready or not.
5) Duke Oldrich of Bohemia (c.975-1034) was Duke from from 1012 to 1032 and again in 1034. Bretislaus I (1002/05-1055), his illegitimate son with a peasant woman named Bozena, daughter of Kresina, became Duke of Bohemia in 1034 and is the ancestor of all later Dukes and kings of Bohemia.
6) Though Denmark became Christian in the 10th century, King Sweyn II Estridsson (died 1076) had no surviving legitimate children. The next five kings until 1134 were his illegitimate sons, and the last illegitimate king was Sweyn III (ruled 1145-1157).
7) The 18 kings and rival kings of Norway from 1093 to 1269 included only three of legitimate birth, and 15 of illegitimate birth, including 3 who showed up and claimed to be foreign born illegitimate sons of former kings.
8) [Added 07 May 2017] When King William II of Sicily died in 1189, the rightful heirs were his aunt Constance and her husband Emperor Henry VI. But William's illegitimately born cousin Tancred, Count of Lecce, became king but died in 1194 and Sicily was conquered by the Emperor.
9) In Ireland and Wales any son recognized by his father was considered a potential successor, legitimate or not. Thus there were many illegitimate born kings and nobles in Ireland and Wales. It was highly unusual and considered rather unjust, for example, for Prince Llywelyn the Great ap Iorwerth (c.1172-1240) to try to make his only legitimate son Dafydd (1212-1246) his sole heir and exclude his several illegitimate sons from succession.
10) [added 4-22-2017] In 1238, Emperor Frederick II married his illegitimate son Enzo (c. 1218-1272) to the heiress of Torres and Galluria in Sardinia and appointed Enzo King of Sardinia.
11) [added 04-22-2017] Manfred (1232-1266), Prince of Taranto, illegitimate son of Emperor Frederick II and Bianca Lancia, who was possibly legitimated by his parent's alleged later marriage, became regent of the Kingdom of Sicily 1254-1258 and made himself King of Sicily in 1258.
12) Margaret the Maid of Norway (1283-1290) became the rightful heir of Scotland when her grandfather Alexander III died in 1286. When Margaret died there was no obvious heir and over 13 competitors sought the crown of Scotland in "the Great Cause".
Seven of the competitors were descended from illegitimate children of members of the Scottish royal family.
13) Robert Stewart (1316-1390) High Steward of Scotland 1326-1371, became King Robert II of Scotland in 1371. He had several illegitimate children with his mistress Elizabeth Mure. He married her in 1336 but some considered the marriage uncanonical, so he got a papal dispensation and married her again in 1349, thus legitimating their children including John Stewart, born in 1337 to 1340.
Elizabeth Mure died and Robert Stewart married again in 1355 to Euphemia de Ross and had several children. Robert also needed a papal dispensation for the second marriage. Apparently both wives may have been too closely related to Robert Stewart. The children of each marriage considered their claim to the throne better than that of their half siblings.
Robert II's son John (1336/40-1406) was selected as heir in 1371 despite any doubts about his legitimacy, and reigned as King Robert III 1390-1406. Robert III's son and heir King James I (born 1394) was murdered on 21 February 1437 by conspirators including Robert Stewart, Master of Atholl, grandson of Walter Stewart (c. 1360-1437), Earl of Atholl, Caithness, and Strathearn, son of Robert II and Euphemia de Ross. Perhaps the Atholl branch hoped to gain the throne from the descendants of Elizabeth Mure.
14) King Alfonso XI of Castile & Leon had an illegitimate son Henry II (1334-1379) who revolted against his legitimate half brother King Pedro I the cruel and finally killed him in 1369, founding the House of Trastamara.
15) When King Ferdinand I of Portugal died in 1383, his daughter and heiress Beatrice was married to King John I of Castile (son of Henry II), but Ferdinand's illegitimate half brother John I was selected King in 1385 and defeated the Castilians, founding the House of Aviz.
16) John of Gaunt (1340-1399) Duke of Lancaster, while married to his second wife Constance of Castile (daughter of Pedro the Cruel), had four children by his mistress Catharine Swynford. Constance died in 1394 and John married Catharine in 1396. Their children, the Beauforts, were legitimated by the church and by King Richard II. Later John's oldest son, King Henry IV decreed that the Beauforts were not legitimate enough to inherit the throne.
In the War of the Roses, King Henry IV's Lancastrian heirs claimed that only legitimate and male lineage descendants were eligible for the English crown, while the Yorkists claimed that they had the senior right through female line descent.
In 1471 the last legitimate descendants of King Henry IV were killed and most Lancastrians decided to accept the Yorkist claim. Those die hard Lancastrians who demanded a claimant of totally legitimate birth would have to take King Alfonso V of Portugal (1432-1481), grandson of John of Gaunt's oldest child Philippa, wife of king John I of Portugal (see number 15 above), or if the claimant also had to be English, Henry Holland (1430-1475), 3rd duke of Exeter, grandson of John of Gaunt's next oldest daughter Elizabeth.
But they were junior in female lineage descent from the Yorkist claimants. So other Lancastrians might have supported the legitimated line of the Beauforts. But the last legitimate male Beauforts were also killed in 1471.
There were plenty of female line descendants of the Beauforts, but of course the Lancastrian claim to the throne denied inheritance through females. Lady Margaret Beaufort (1441/43-1509) was the female with senior Beaufort lineage, and the mother of the eventual usurper King Henry VII (1457-1509), who was thus the great grandson of the illegitimate but legitimated John Beaufort, first Earl of Somerset.
But Margaret Beaufort's first cousin Henry Beaufort (1436-1464), 3rd duke of Somerset, had an illegitimate son Charles Somerset (c. 1460-1526), made first Earl of Worcester in 1514. It is said that he was legitimated, but I don't know at what date. Thus the Dukes of Beaufort descended from him might be considered the rightful heirs of the Lancastrian kings.
17) King Alfonso the Magnanimous (1396-1358) of Aragon, Valencia, Majorca, Sardinia, Corsica, Sicily, etc., and King of Naples, had no legitimate children. His brother John II inherited most of his lands, but his illegitimate son Ferdinand I (1423-1494) inherited the Kingdom of Naples.
18) King Edward IV of England (1442-1483) married Elizabeth Woodville on an unknown date, possibly in 1464, and had her crowned queen on 26 May 1465. She was the acknowledged queen of England, and their son Edward (born 2 November 1470) was made Prince of Wales in 1371.
Edward IV died on 9 April 1483. Edward Prince of Wales was proclaimed king Edward V.
Ralph Shaa preached a sermon on 22 June declaring that Edward IV had made a marriage contract with another woman, making his marriage with Elizabeth Woodville invalid and their children illegitimate. It was a big deal to challenge a long standing public marriage, and the church would try marriage cases carefully. But, perhaps due to the soldiers of Richard Duke of Gloucester who controlled London, an assembly of lords and commons decreed that Edward V and his siblings were illegitimate and proclaimed Richard king. Edward V and his brother Richard disappeared in the Tower of London.
Richard III's first parliament in 1484 passed the act Titulus Regius claiming that Edward IV's children were illegitimate. Already in 1483 Henry Tudor had promised to marry Edward IV's oldest daughter Elizabeth, her brothers seemingly believed dead already. Henry invaded, usurped the throne from the usurper Richard III, married Elizabeth, and repealed the Titulus Regius.
19) [added 04-22-2017] King Matthias I Corvinus (1443-1490) of Hungary, Croatia, and Bohemia declared his illegitimate son by Barbara Edelpok, John Corvinus (1473-1504) to be his successor. But the diets of Hungary and Bohemia elected Vladislaus Jagiellon King when Matthias died.
20) [added 05-18-2017] In 1533 Henry VIII (1491-1547) married Anne Boleyn (c. 1501-1536) and decreed his marriage with Catherine of Aragon (1491-1536) void, and their daughter Mary (1516-1558) was declared illegitimate by the First Succession Act. In 1536 Anne Boleyn was beheaded and her daughter Elizabeth (1533-1603) was declared illegitimate by the Second Succession Act. The Third Succession Act of 1543 returned Mary and Elizabeth to the line of succession while they were still considered illegitimate.
Mary became Queen Mary I in 1553.
21) [added 05-18-2017] Elizabeth became Queen Elizabeth I in 1553 despite still being illegitimate by English law.
22) [added 05-18-2017] King Henry VIII might have even considered making his illegitimate son Henry FitzRoy, Duke of Richmond and Somerset (1519-1536), his heir if need be. The Second Succession Act of 1536 had a clause granting Henry the right to chose his successor, legitimate or not.
23) In 1714 the absolutist King Louis XIV of France and Navarre forced the Parliament of Paris to recognize his illegitimate but legitimated sons Louis Auguste (1670-1736), Duke of Maine, and Louis Alexandre (1678-1737), Count of Toulouse and Duke of Penthievre as Princes of the (royal) Blood and in line to the succession after the legitimate heirs. When Louis XIV died in 1715, the Duke of Orleans, regent for young Louis XV, immediately had the Parliament of Paris reverse that action.
24) Albert I (1848-1922) Prince of Monaco, had a problem in the 1910s, since his only child the future Prince Louis II (1870-1949) had no legitimate children and the next heir was the German Duke of Urach, unpopular with the French government. So Louis's illegitimate daughter Charlotte (1898-1977) was legitimated in 1911 and adopted in 1918 and declared the heir to the throne.
In 1944 Charlotte renounced her rights to the throne in favor of her son, Prince Rainier II (1923-2005).
As these examples show, it was very rare for an illegitimate child to succeed to the throne during the High Middle Ages (c. 1001-1300) or the Late Middle Ages (c. 1301-1500) and later eras. It was also increasingly rare and uncommon as time went on for illegitimate sons to inherit noble titles and fiefdoms.
But haven't I forgotten Duke William the Bastard of Normandy who became King of England in 1066? No. It doesn't matter if William the Bastard was of legitimate or illegitimate birth, because he had exactly zero hereditary (or any other type) claim to the throne of England, except right of conquest. Might makes right and all that.
William the Bastard was not only not first in line for the throne, but he wasn't second, or third, or fourth, or fifth, or 100th or 1,000th, or anything. We can't even say he was last in line for the throne, because he would have to be in line for that to be true. William had no known descent from any king of England or from any member of the house of Wessex or from any Anglo-Saxon king of any kingdom, or even from any Anglo-Saxon serf or slave.
As a foreign invader and usurper William the Bastard doesn't count as someone who inherited a throne despite illegitimate descent from previous monarchs.
But of course, as andejons pointed out, he did inherit the Duchy of Normandy despite being a bastard. William's mother Herleva was the daughter of Fulbert of Falaise whose occupation is uncertain.
And also see the answers to this question: