There was a sum of factors:
Population, as highlighted in the comments, is part of the answer. But only a part, because the Netherlands and Portugal had not a big population, and still they developed quite extended colonial empires. Their small population played a more important role, when they had to defend their empires against France, United Kingdom and Spain.
War is linked to the population issue, and the overall capability of a country to undertake a colonial project. If I trust this list, Sweden had a lot of war. So had Denmark/Norway. Two things are important with those wars:
- They exhausted the countries
- The battlefield was the Baltic Sea: nobody controlled it, so nobody could use it as a safe base from which to start colonial operations. In contrast France, Spain and United Kingdom had an extensive Western coast from which to operate. The Netherlands had not, but they ruled the sea.
The second thing is that those wars were not conclusive: Spain finished the Reconquista and started colonies. France finished the Italy wars and started colonies. UK did the same after it unified under Elisabeth the 1rst. In Scandinavia, Poland and Russia, wars repeatedly arose because no peace was found.
Note that the new set of wars that came with Louis XIV played a role in reducing Spanish and French colonies.
Weather put Scandinavian harbours unable to work all year long. This had some effects on the capacity to colonize lands far away, because the time to travel back and forth was so long: it needed months to reach America, so only a few convoys could have reached the homeland nor the colonized country.
Willingness eventually is a key factor: by those time, population was not well educated and did not know a lot about geography: to put such a population in boats and send them overseas, you need propaganda and efficient enforcement authority. France for example used it police forces to send thiefs, prostitutes and other low-considered urban population in its colonies, because the rest of the population had no willingness to go there.
So willingness of the population is commanded by the willingness of the governements: I have no information about the state of mind of Sweden or Denmark's kings, but it could be interesting to search about that.