It seems that until around the 20th century, a relatively large portion of rulers saw it as their goal to acquire more land. What was the main rationale behind this?
There are many factors:
- Economic motives: Many people saw colonies as markets for their finished products, and suppliers for their raw goods. Colonies provided raw materials that often could not be grown or found in the parent country. For example, England's colony in India. India was a good place to grow tea and opium, things that the British could not grow at home.
- Strategic reasons: England controlled the Suez canal in Africa to secure the fastest water route to India. This was a highly coveted position, since India was rich with resources.
- "White man's burden": Many Europeans felt superior to the other races around the world. As such, they made it a goal to assimilate peoples around the world, to a more European lifestyle. Again, I will use England as an example: In India, England built a sewage system, hospitals, schools, etc. In their minds, they were drastically improving life for Indians. However, the Indians did not feel the same way. The schools taught things that were meaningful to the British, but not to Indians. The British also abolished many traditions, one of which was burning a widow alive on her husband's funeral pyre. The Indian people were outraged, as this had been a part of life for many, many years.
Keep in mind that this is New Imperialism, which came after Old Imperialism. Old Imperialistic motives can be summed up quickly:
- Gold: They wanted loot. Essentially, pure greed.
- Glory: Conquistadors and explorers wanted to be famous. Again, very simple.
- God: Many European missionaries felt that it was their duty to convert the entire world to Roman Catholicism. So, they went to South America (primarily) and brought Christianity to the natives.
I think there's a selection bias at play. A ruler that did not care about acquiring more land would typically end up avoiding conflict with the neighbors, and would tend to not be noticed historically. Just because the historically notable rulers were a certain way does not mean that the average rulers were the same way.
Note: this is an opinion question (especially the second part) and probably off-topic here. I'll try to answer it nevertheless. I'm not sure there is a single rationale behind this.
- One reason is psychology. Humans build hierarchies instinctively and try to climb up. A country ruler is no exception, he will also instinctively try to amass more power and make his country stronger than the others - even if this behavior violates common sense. Good rulers obviously have a common sense that is stronger than instinct.
- It wasn't always about territory. In the feudal wars the occupant often didn't expect to keep the territory, he simply got everything valuable out of the country and left it. The colonization of South America followed a similar principle.
- With the rise of nationalism came a competition between nations (same hierarchy but with people identifying themselves with their countries). This should be the reason why Britain amassed huge territories in Africa that were of very little value to it. Russia did the same thing in Asia, other countries behaved similarly.
One can easily find more rationales. In the end, the psychological factors causing countries to compete didn't go away, they are still there. The important difference now might be the globalization: with the economies of all countries interconnected a war between first world countries would be devastating for all parties involved and cannot be rationalized no matter how hard one tries. So the competition shifted into other areas like economy.