By expansive, I mean in terms of land area.
On the Wikipedia pages of Norway's, Sweden's, and Denmark's colonies, it shows they had very few parcels of land around the globe, compared to the colonies of the Spanish, Portuguese, French, British, and even nearby Dutch. And the few small lands they did colonize were very quickly taken over by the British or French (i.e. Porto Novo lost in a month).
Why was Scandinavian colonization less expansive?
Short Answer:
There was a sum of factors:
Long Answer:
Population, as highlighted in the comments, is part of the answer. But only a part, because the Netherlands and Portugal had not a big population, and still they developed quite extended colonial empires. Their small population played a more important role, when they had to defend their empires against France, United Kingdom and Spain.
War is linked to the population issue, and the overall capability of a country to undertake a colonial project. If I trust this list, Sweden had a lot of war. So had Denmark/Norway. Two things are important with those wars:
The second thing is that those wars were not conclusive: Spain finished the Reconquista and started colonies. France finished the Italy wars and started colonies. UK did the same after it unified under Elisabeth the 1rst. In Scandinavia, Poland and Russia, wars repeatedly arose because no peace was found. Note that the new set of wars that came with Louis XIV played a role in reducing Spanish and French colonies.
Weather put Scandinavian harbours unable to work all year long. This had some effects on the capacity to colonize lands far away, because the time to travel back and forth was so long: it needed months to reach America, so only a few convoys could have reached the homeland nor the colonized country.
Willingness eventually is a key factor: by those time, population was not well educated and did not know a lot about geography: to put such a population in boats and send them overseas, you need propaganda and efficient enforcement authority. France for example used it police forces to send thiefs, prostitutes and other low-considered urban population in its colonies, because the rest of the population had no willingness to go there. So willingness of the population is commanded by the willingness of the governements: I have no information about the state of mind of Sweden or Denmark's kings, but it could be interesting to search about that.
I would venture that one reason is the alternatingly dominating fleets of the Spanish, English, French and the Dutch over the centuries. They would simply not want this to happen and had the means to stop it. They all have much better access to the Atlantic and are ideally positioned to project naval power.
Given the geography and politics, it is very easy how one could sideline a country by bottling up a navy, and blockading sea trade, within the Baltic Sea. This would have an enormous impact on the ability and desire of countries which would primarily be based out here and would essentially deny them any capacity to maintain a global empire.
Historically it is easy to see that many Scandinavian (e.g. Sweden) and close neighbors (e.g. Germany) have been mainly land armies with not much in terms of naval power. The only realistic way for them to compete in this theater given all their disadvantages would be asymmetrically (u-boats as an example).
Most of this can be exemplified by Germany in WW1/2, as an example of a country being blockaded and bottled up in the Baltic, with grave military and economic consequences.
