It is somewhat important to realize that even Hitler was not so mad as to actually consider invading all of Europe, and getting away with it. He had to consider cost vs. benefit.
Hitler's target -- "the plan", as early as 1925 -- was Russiathe Soviet Union. That's where his ideological enemy was: Bolshevism. That's where his whole screwed "Lebensraum" vision played out: land, slaves, and resources.
As such, the pre-war invasions and occupations -- Austria, Sudetenland, Czechoslovakia -- and the attack on Poland were merely setting the stage for the attack on Soviet Russia.
But when England and France started switching from a policy of Appeasement to eventually declaring war on Germany after the invasion of Poland, he had to consider stabilizing his position to the west as well. That is where the second set of invasions and occupations comes in.
Denmark and Norway (crippling the German navy), to deny the northern flank to England, secure the Swedish iron ore supplies, and give the German navy some wiggle room in the North Sea. France (via the Netherlands, Belgium, and Luxembourg), the most direct threat to Germany and its industrial heart, the Ruhr area.
That last one succeeded beyond expectations, and rather surprisingly to everyone (including the Germans) an invasion of Britain itself was considered... Although you could say that Operation Sea Lion was (at least) as much an attempt to scare Britain into an armistice as a credible threat. At the same time Hitler was still deluded enough to make peace offers to Britain... which would have allowed him to focus all his powers on Russiathe Soviets.
So now he had to fortify France, fought the Battle of Britain (crippling the Luftwaffe), and had to help his ally Italy in North Africa, and handle the situation on the Balkans -- to deny England access and secure the supply of Romanian oil.
All this was a severe strain on available forces, and crucially delayed the attack on Soviet Russia. Had he been given the choice, he'd rather have done without all this.
Note that the first set of invasions was mostly offensive in nature: Getting at Russia. The second set was mostly defensive in nature ("defensive" in a very lose definition of the word): Securing his back, flanks, and access to crucial resources.
The second set was not originally part of "the plan". The historian in me is wondering what would have happened if France and Britain hadn't declared war, or if Britain had folded after the fall of France...
Now, Sweden... With Norway held by Germany, and Finland still free, Sweden was not under threat. It was much easier delivering coal to Sweden in exchange for ore than stretching the German war machine yet further and invade (probably disrupting the supply of ore for several months).
There was nothing to be gained from invading Sweden, and the German forces did not have the resources to spare anyway.