Short answer
Blue was adopted by some Tories (later to become formally known as Conservatives) from around the middle of the 18th century. Before that (from around 1680) it was used mostly by Whigs. However, it's important to note that were many changes in party colours over time and, also, regional variations. Blue was not formally adopted by the Conservative party until 1949.
Details
The use of blue in British party politics has a long complicated history dating back to at least the Exclusion Crisis of 1679-81 when it was associated with the earliest Whigs who wanted to exclude James II from the throne on the grounds that he was Catholic:
The historical fortunes of the idiom of "True Blue" Protestantism also testifies to the shifting valences of political labels. During the Restoration, in the context of the Popish Plot, the Exclusion Crisis, the Revocation of the Edict of Nantes, and by allusion to the Scottish Presbyterian party (which had adopted blue as its color, in contradistinction to the royal red), the Whigs were known as the "True Blues" and had proclaimed themselves to be the "true blue" defenders of English Protestantism (against the onslaught of Catholicism).
Just prior to this (and overlapping a little in time), Whigs were also associated with green and, notably, the Green Ribbon Club founded around 1675. They "wore on their hats a bow, or bob, of green ribbon" and were in opposition to King's (Charles II) court. Then,
by the summer of 1681 a different colour from green was being worn by
many Whigs in London, some sporting blue ribbons, to mark themselves
as Monmouth's supporters, to which some Tories responded by wearing
red ribbons, to identify themselves as the duke of York's men.
Source: David Allen, 'Political Clubs in Restoration London'. In 'The Historical Journal, Vol. 19, No. 3' (Sep., 1976)
The Whig connection with blue was still evident in the early 18th century, but [it was evolving into buff] in some regions at least. One observer of the 1705 election in Honiton noted:
the two parties were very nicely distinguished . . . Buff was the
symbol of the Whigs. These had box in their hats, doors and windows;
the other [Tories] had laurel leaves
Whigs were also using blue and buff:
...as early as 1715 the diehard Bishop Burnet was referred to as a
“true blue Whig” upon his death, and by mid-century the combination of
buff and blue was widely associated with the Whigs.
Around the same time, there were
...Tories in Newcastle-upon-Tyne in 1710...with ‘red and blue favours
in their hats...
Blue (alone) was later adopted by Tories (who were to evolve into the Conservative Party during the Robert Peel administrations of the 1830s and 1840s). Later in the 18th century,
the "King-and-Church" men were the ones who
proclaimed themselves to be the True Blues (sometimes even in
preference to the "Tory" label), in allusion now only to the
figurative sense in which "true blue" means "faithful, staunch and
unwavering (in one's faith, principles, etc.): sterling, genuine,
real" (OED, sv "blue, adj.," sense 1e). For example, in 1755, the Tory
newspaper Felix Farley's Bristol Journal "alluded to 'the Blues' or
'the Blue interest' rather than to the Tories" and Edward Ward's
British Weekly Intelligencer charged "that the True Blues of Bristol
sang Jacobite songs" (Sack 1993: 50, 54n.35).
Even so, Tory PM William Pitt the Younger's Whig opponents in 1784 (who included the Prince of Wales) still associated themselves with blue, along with buff:
Source: George Wingrove Cooke, 'The History of Party; from the Rise of the Whig and Tory Factions, in the Reign of Charles 2., to the Passing of the Reform Bill, Vol. 3'
Blue and buff was also a connection between the American revolutionaries uniform and the Whigs who supported American independence:
By the time of the Revolution in America, the buff and blue had become
universally understood to represent resolve and faithfulness to a
cause....
...The Whigs of the late 18th Century advocated constitutionalism,
legislative supremacy over the Crown, religious liberty, natural
rights, the separation of powers, due process and property rights --
the very foundation of what would become the American experiment.
When celebrations were called for upon the recovery of George III's health in 1789, the more extreme Whigs complied, but with a 'message':
Brooke’s club, the St. James’s haunt of the Whig party, hosted its own
gala to ‘toast’ the King’s health. Newspapers, however, reported that
the dress required would be court dress in full embroidery but
rendered in ‘blue and buff’ (the colours routinely adopted by the Whig
party, echoing the uniform of the American revolutionary army)
Source: Hannah Greig, 'Faction and Fashion : The Politics of Court Dress in Eighteenth-Century England'. In Isabelle Paresys et Natacha Coquery (eds.), 'Se vêtir à la cour en Europe (1400-1815)'
"Blue and Buff Loyalty. 31 - 31 Dec 1788." "A hand-coloured print of a satire on the Whig's hopes for power with the ascension of the Prince of Wales as Regent." Image & text source: Thomas Rowlandson, Royal Collection Trust.
Moving on to the 19th century,
The association of the color blue with the Tories or Conservatives has
persisted across the 19th century and beyond. Thus, for example,
writing to his sister in 1835, Disraeli proclaims, "I . . . have
gained the show of hands, which no blue candidate ever did before" and
Anthony Trollope, in Farmley Parsonage (1860) states of a conservative
neighborhood, "There was no portion of the county more decided true
blue" (OED, sv "blue, adj.," sense 6a and 6b).
However, there were many regional variations, particularly (it seems) in the first half of the 19th century. For example, Tories or conservatives were blue in Reading, Colchester and Ipswich (where Whigs were yellow or orange), but orange and purple in Norwich, and red in Carmarthen and Great Grimsby (where the Whigs were blue).
Thus, the implication in articles such as this that blue was first associated with the Tories / Conservatives when the Labour Party (founded in 1900) associated itself with red is leaving out a long history of party colour associations which, even after WWII, sometimes varied locally. Further,
It wasn’t until 1949 that the National Union of Conservative and
Unionist Associations voted to formally adopt blue as the official
colour of the party.
All emphasis is mine
