Think about the global power balance at the time. Imperial Japan was a rising power that wants to expand against older, more entrenched alliances. Towards the south, it faced the maritime power of the United States which has a strong interest in keeping the Pacific and SE-Asia under her control. Towards the north, it faced the USSR. The Japanese leadership at the time had the confidence to either defeat USSR on land or the USA at sea - but not both. Hence the birth of the "Northern" and "Southern" doctrines.
For most of the time until 1936, the "Northern" doctrine was the leading doctrine in the military. This was primarily because of two reasons. First, Japan had just recently conquered Northeastern China (Manchuria), which it sought to build as a new homeland for the Japanese people. In fact, immigrants had already began arriving, and the risk of losing Manchuria was becoming unacceptable. Second, the Army believed itself to be the creator of Japanese destiny and thus supported the Northern doctrine fervently - just as the Navy supported the Southern doctrine. At the time, the Army had more influence at the top echelons of Japanese decision-makers (in particular the Imperial General Headquarters). Finally, Japan was forming rather close ties with Germany which was (nominally at least) very hostile to the USSR; and at the same time Japan benefited from its trade relationship with the USA. This made USSR the natural enemy.
A few things changed during 1936-1939. First, the London Naval Treaty of 1930 ended, and the Imperial Navy began a period of drastic expansion. And - as anyone familiar with history surely knows - the more military capacity you have, the more you begin to look for enemies. In addition, as Germany began to take over Europe, there was a power vacuum in Southeast Asia - particularly Indonesia, with its rich natural resources. To make a compromise between the Army and the Navy, the Japanese leadership ultimately decided to appease both sides - by pursuing a strategy of expanding in both directions.
By the time 1939 arrived, however, the Northern doctrine had already lost most of its appeal. The army lost a lot of credibility with its drastic over-expectations of "conquering China in three months". With much of the Army trapped in China, creating a second front against USSR become infeasible. The Imperial Army apparently did not see this, however, and decided to launch a major offensive (without the Emperor's permission) against the USSR after some casual harassment from Mongolian troops. This resulted in the disaster in the Battles of Khalkhin Gol, and the Army lost its credibility completely. And while Japanese troops were being slaughtered, Hitler signed a non-Aggression Pact with Stalin - and thus the Northern Doctrine fell apart.
Main source: http://big5.ifeng.com/gate/big5/news.ifeng.com/history/shijieshi/detail_2010_09/25/2618652_1.shtml (in Chinese)