I've found these numbers for the total population counts at the beginning and end of this period.
Now we need to factor in immigration and emigration to get the natural population change. This is very difficult because few records were kept until the turn of the nineteenth century.
The largest migration of Scots to Ireland was in the early 1600's. Due to lack of definitive records, we do not have exact numbers, but in the early 1600's 120,000 are believed to have migrated -- from both England and Scotland. Bailyn says in one 24 month period in the 1630's at least 10,000 Scots migrated to Ireland. Bailyn, Bernard- The Peopling of British North America: An Introduction
If an equal percentage migrated from Scotland and England, and the population in England at this time was 4 million, it stands to reason about 1.5 to 2.5 thousand Scots migrated. The article goes on to say that:
An unknown number of Scots fled back to Scotland in the 1630's to avoid religious persecution in Ireland.
This makes it difficult to conclude if all or none of the Scots returned. Another source states about emigration in the eighteenth and nineteenth century:
The adult experience of emigration in England and Wales paralleled that in Scotland. The bulk of the migrant population went to the United States in the eighteenth century, although Canada and Australia were popular destinations after 1815.... Canada was the most important destination of emigrants during the first half of the nineteenth century and between 1825 and 1835, over 70% of emigrants from Scotland settled there.... Whatever the cause, Scotland lost between 10 and 47% of its natural population increase every decade between 1830 and 1940.
This was counteracted to a slight degree by immigration from Ireland.
The Irish [mostly Catholic] were by far the largest group of immigrants to settle in Scotland.... In the 1820s, 6000–8000 Irish per year were making the harvest migration. By the 1840s this had grown to 25,000 over the agricultural season. Most of the emigration, however, was on a temporary basis, peaking during important times in the farming calendar, such as harvest. In the summer of 1841, 57,651 Irish, mainly male labourers, crossed to England and Scotland to work on the harvest.
The immigrant Catholic Irish and the Protestant Scots did not get along well. For generations, the majority of Scots would not marry the Irish. They remained socially distinct. At times, the relations between the two ethnicities were even violent. Catholics were condemned from the protestant pulpit and newspapers commonly attacked the Irish. However, this did not seriously phase the Irish, who were quite willing to work for cheaper wages than then Scots. Many English also emigrated to Scotland; in certain areas, the English-born were more numerous than the Irish-born.
There were more English-born residents than Irish-born in Edinburgh and this increased as the 19th century wore on. In 1881 there were 11,514 English-born residents and 7875 Irish-born. Forty years later the respective figures were 28,187 and 6382.
In the mid-1840s, the European Potato Failure struck Northern Europe. Many Scots emigrated to other countries permanently.
Particularly harshly affected were the Scottish Highlands and, above all others, Ireland. Many people starved due to lack of access to other staple food sources.... Emigration to escape the famine centred mainly on Ireland and the Scottish Highlands.... Over 1 million emigrated from the Scottish Highlands...
Others sources give higher numbers:
The... famine caused over 1.7 million people to leave Scotland during the period 1846–52.
According to the census of population, the Irish-born population of Scotland stood at 126,321 out of a total of 2,620,184 in 1841, or 4.8%. Ten years later it stood at 207,367, or 7.2%, out of a total of 2,888,742.... During 1848, the average weekly inflow of Irish into Glasgow was estimated at over 1000.
Colonization was also a major reason to emigrate. This is just one example of emigration trends:
Between 1717 and 1775 some 250,000 Ulster Scots emigrated to the American colonies.
During the Highland and the Lowland clearances, an unknown number (probably tens of thousands) of Scots migrated to England and other parts of the UK for work. This was mostly brought about by the Agricultural Revolution.
Evidently, emigration was a major problem for Scotland. As accurate records were not kept until the nineteenth century, the number of immigrations and emigrations are only speculations. Adding up the numbers, the natural growth was probably close to six or seven million. However, because of emigration, the total population only increased by about 3.6 million.
GENUKI: Poulation of Scotland
Looking at History: Adult Migration 1600-1980
Decline or Growth? European Cities and Rural Economies 1300-1600
European Potato Failure
Highland Potato Famine
Irish emigration to Scotland in the 19th and 20th centuries