It is somewhat important to realize that even Hitler was not mad enough to consider invading all of Europe, and getting away with it.
Hitler's target -- "the plan" -- was Russia. That's where his ideological enemy was: Bolshevism. That's where his whole screwed "Lebensraum" vision played out: land, slaves, and resources.
As such, the pre-war invasions and occupations -- Austria, Sudetenland, Czechoslovakia -- and the attack on Poland were merely setting the stage for the attack on Russia.
But when England and France declared war after the invasion of Poland, he had to consider stabilizing his position to the west as well. That is where the second set of invasions and occupations comes in.
Denmark and Norway (crippling the German navy), to deny the northern flank to England, secure the Swedish iron ore supplies, and give the German navy some wiggle room in the North Sea. France (via the Netherlands, Belgium, and Luxembourg), the most direct threat to Germany and its industrial heart, the Ruhr area.
That last one actually succeeded so much beyond expectations that an invasion of Britain itself was considered. But at the same time Hitler was still deluded enough to make peace offers to Britain... which would have allowed him to focus all his powers on Russia.
Instead, he had to fortify France, fought the Battle of Britain (crippling the Luftwaffe) in hopes to get Britain out of the war, and had to help his ally Italy in North Africa, and handle the situation in Yugoslavia, and help Italy again in Greece (and Crete, crippling the German paratrooper force) -- to deny England the Balkans and secure the supply of Romanian oil.
All this was a severe strain on available forces, and crucially delayed the attack on Russia.
Note that the first set of invasions was mostly offensive in nature: Getting at Russia. The second set was mostly defensive in nature: Protecting his back and flanks, securing crucial resources. The second set was not originally part of "the plan".
With Norway held by Germany, and Finland still free, Sweden was not under threat. It was much easier delivering coal to Sweden in exchange for ore than stretching the German war machine yet further and invade (probably disrupting the supply of ore for several months).
There was nothing to be gained from invading Sweden, and the German forces did not have the resources to spare anyway.