Hideyoshi's reasons were not singular. A number of factors motivated his invasion of Korea. Although speculative hypothesis regarding his mental state is popular, domestic pressure for expansion coupled with seemingly-promising opportunities sufficiently explains the decision.
TL;DR: Hideyoshi needed land and to keep his soldiers occupied. Korea was an easy target for both. He was also quite ambitious with grand dreams so conquering Korea had no downsides (or so it seemed).
I. He was Insane
A popular, though somewhat speculative, explanation is that Hideyoshi was simply delusional. As the theory goes, Hideyoshi's triumph after triumph in Japan led him to believe the world was his for the taking. In this view, he launched the invasion of Korea as the first (and logistically inevitable) campaign in a quest for world conquest. Military and resource realities were unimportant, since the megalomania of Hideyoshi's senior years was motivation enough for his foolish adventures.
A slightly toned down version would be to say that Hideyoshi had extremely grandiose ambitions. While not particularly persuasive as a reason in and of itself, ambitious aspirations certainly would provides an overall context for Hideyoshi's decisions.
Yet the Korean expedition did not come from nowhere, and the most important trigger to action was Hideyoshi's own grandiose dream of overseas military conquest ... in fact his personal ambitions went further than Korea and China, and included the conquest of Taiwan, the Philippines and even India.
- Turnbull, Stephen R. Samurai Invasion: Japan's Korean War, 1592-98. Cassell & Company, 2002.
These ambitions are well documented in many of his communications. For example, he spoke of his dreams to invade China when making requests via Portuguese missionaries for galleons to support the same. Of course, he rose from near-commoner status to the highest office in the Imperial Court. That sort of thing don't usually happen without more than a healthy dose of ambitions.
II. A Short, Victorious War
Now, conquering Korea was not that unrealistic. Hideyoshi had every reason to believe an invasion of Korea would go well. And in fact, it did. His well trained, battle-hardened veterans rapidly overran most of the peninsula, capturing even Pyongyang after just two short months. The Koreans were no match.
The reality was that Korea was extremely ill prepared for war. Its military was poorly trained, poorly equipped, and inexperienced. Divisive factional bickering occupied the national attention, while military defense was allowed to decay with neglect. What little preparations were made largely went to the northern borders, where the nomadic tribes of Manchuria were seen as the real enemies.
Korea was weakened politically by feuding bureaucratic factions in the capital, fiscally by the corruption of the bureaucratic apparatus and the failure to maintain tax revenues by the land and tribute taxes, and militarily by the neglect of defense, the under-registration of the adult male population for service, the exemption of slaves and yangban from military duty, the lack of training, and the failure to adopt firearms.
- Palais, James B. Confucian statecraft and Korean institutions: Yu Hyŏngwŏn and the late Chosŏn dynasty. University of Washington Press, 1996.
To a large extent, the Ming intervention dramatically halted a conquest that was nearly fait accompli. While we might think it obvious for China to step in, this was not necessarily true. In reality, the Ming Court was filled with incredulous voices when Korea's plea for help arrived. The Imperial Cabinet suspected that the rapid Korean collapse was a ruse, laid in cahoots with Japan to ambush Ming reinforcements. It took months of intense lobbying by the Korean government before a minor unit was dispatched to reconnoiter the peninsula.
[A]t the beginning of the Imjin Was China deeply suspected that very possibility. It took three months of intense Korean diplomacy to convince Ming China that Korea was not conniving with Japan against China.
- Beeson, Mark, and Richard Stubbs, eds. Routledge Handbook of Asian Regionalism. Routledge, 2012.
Moreover, the Japanese invasion occurred when the Ming Empire was heavily distracted. While Japanese troops landed in Korea, Ming forces were at Ningxia, preoccupied with a major border revolt. Bā Bài, a Mongolian commander, seized 47 forts in the region with support from foreign Mongolian tribes. The pacification of Ningxia was serious enough to take most of the year, and would go down in history as one of the Three Major Conquests of Wanli Emperor (intervention in Korea being another).
Again this is not a very persuasive reason in and of itself. The weakness of Korea does, however, mean that it was (seemingly) a great target of opportunity.
III. Political Capital
On a more practical level, Hideyoshi wanted to conquer Korea (and beyond) for the sake of his domestic political position. A successful invasion of the Korean peninsula would have provided him with important resources. Namely, land - and lots of it. Its value in the Japanese political landscape should not be overlooked. Land (or more precisely, the rice productivity of the land) was the measure of power, and the unit of rewards. It was the object of the samurai's desires.
[T]he invasion of Korea can be seen as the necessary next stage after the conquest of Kyushu in 1587 and the defeat of the Hojo in 1590. The invasion would provide new lands for his warriors and enable Hideyoshi to retain his control over them.
Van Derven, H. J., ed. Warfare in Chinese History. Vol. 47. Brill, 2000.
Hideyoshi was too generous a conqueror. Even while subjugating lord after lord in rapid succession, he generally allowed those who submitted to retain their core holdings. What lands he confiscated land were typically distributed to other samurai lords in return for their pledge of allegiance (as opposed to going to traditional vassals and family - both of which he had none/little). This strategy helped ensure the rapid capitulation of most of Japan, but it also meant a horde of powerful daimyos who had the means (of a large standing army) and the inevitable greed to expand their holdings.
Independent military initiatives by the daimyo to expand their domains at one another's expense would collide directly with Hideyoshi's aim of unification. Hideyoshi felt that he could achieve domestic peace and unity and bring an end to the daimyo's struggles among themselves, by redirecting their tremendous energy for independent action toward the conquest of the continent.
- Hall, John Whitney, ed. The Cambridge History of Japan. Vol. 4. Cambridge University Press, 1991.
Therefore, a nearly defenseless (or so it seemed) Korea provided a logical target for Hideyoshi. An invasion would serve his domestic political needs for land, a highly limited commodity, and maintain his rule. If they failed, they would have whittled their strength in a land war in Asia. In the event of a successful conquest, he could also transfer daimyos into ostensibly bigger territories in Korea, and out of Japan, as rewards. That Hideyoshi perhaps harboured grandiose dreams of being a continental conqueror was a big bonus, too.
Hideyoshi was inspired to invade the continent to keep the military momentum of his unification of Japan going. He saw continental warfare as a means to assert even firmer control over Japan's warrior class, and also to provide new lands to meet the still strong demands of the daimyos for territorial acquisition.
- Black, Jeremy, ed. War in the Early Modern World. Taylor & Francis, 1999.
IV. Unemployment Relief
This is closely linked to the previous section, but with the focus on the huge surplus military manpower Japan had upon unification. Late 15th century Japan was a highly militarised society that has spent more than a century in near-constant warfare before Hideyoshi's unification wars. The speed of his meteoric rise bears stressing. The last hold out, the Hōjō clan of Odawara, was conquered in 1590. It had only been eight years since the assassination of Oda Nobunaga.
Continuing a doctrine of forbidding private warfare, Hideyoshi issued sō buji rei to subjugated clans. This forbade the latter from engaging in warfare. Almost overnight, Hideyoshi's rapid pacification of Japan rendered massive, battle-hardened feudal armies out of work. The country had, for 130 years, oriented itself around fighting; there was no institution in place to absorb the hundreds of thousands who had heretofore made a living out of warfare.
Unemployed samurai, known as rōnin, were in particular a major security concern throughout Medieval Japan. The explosion in their numbers after the Sengoku period exacerbated the situation. Ronin revolts continued to rock the Tokugawa Shogunate well into the 1650, for example. Rather than sit tight and hope for the best, Hideyoshi found a massive public employment scheme in the form of an invasion of Korea.
V. Prestige
Some scholars have argued that Hideyoshi was primarily concerned with international fame. In this view, he didn't have to worry about guerrilla or garrisoning concerns; he only wanted to win. It is of course much easier to defeat a Chinese army than to occupy the whole of China. The latter might be too unrealistic to contemplate, but the former was well within the realm of capabilities.
[M]ost scholars point to status or economic - not military - consideration. For example, Swope notes that, `Hideyoshi craved recognition and homage from foreign rulers. This goal should not be trivialized', and Elizabeth Berry also concluded that, '[Hideyoshi] was clearly less interested in military dominion abroad than in fame'.
- Beeson, Mark, and Richard Stubbs, eds. Routledge Handbook of Asian Regionalism. Routledge, 2012.
This would be entirely in keeping with his past experiences. His modus operandi in the unification of Japan was to crush his enemy in decisive battles. Often this shock and awe convinced the independent-spirited daimyo to render homage, which point Hideoyhi generously confirm them in their possessions.
Of course, in reality this is not that applicable outside of Japan, but Hideyoshi was (take your pick) delusional/overconfident/naive/ignorant/hopeful. He did issues demands to his neighbours, to the effect of supplanting China as the centre of the tributary system.
