To answer the question we must first consider which countries became Protestant around the time of the Reformation. There were eight (I think): Scotland, England, (Ireland, sort of of), Switzerland (parts), the Netherlands, Germany (parts), Denmark, and Sweden.
Did any of these countries set up “Inquisition-style” tribunals, and if so, how many people lost their lives through them?
I don’t think that any of these countries set up “inquisition-style” tribunals during the sixteenth century, so the answer to the question in that sense would be zero.
If the question is broadened out to ask how many people lost their lives judicially for religion (as opposed to being killed in war, or murdered, or executed for other reasons), then obviously the answer is non-zero, Michael Servetus in Geneva being a case in point.
In England a number of Roman Catholics were put to death under Elizabeth, listed here, but also some Protestants such as John Penry.
In Scotland, there was a Roman Catholic inquisition of sorts in the fifteen century (see Laurence of Lindores), and Paul Kravar (Craw) and others were put to dead. This seems to have lapsed in the sixteenth century, and Cardinal David Beaton, although an occasional persecutor of Protestants, did not attempt anything systematic. His successor, Archbishop John Hamilton, was even less inclined to this, and the two Protestant martyrdoms during his tenure (Adam Wallace and Walter Milne) are rather puzzling.
After the Reformation of 1560, the Mass was prohibited on pain of death, but this was never inflicted (see, for example, Thomas McCrie, Life of John Knox (1855 edn.), pp. 175-6, 226). The Dominican friar John Black was murdered at the same time as David Riccio, but there were no judicial executions. There was a Roman priest executed around the 1580s, but arguably for some political misdemeanour, and then there was the famous case of John Ogilvie in 1615.
In speaking of “inquisition-style” tribunals in Scotland, the OP is probably thinking of the covenanting period of the 1670s and 1680s. There were certainly resemblances, with people being apprehended, brought directly before the supreme court which was the Privy Council (which included bishops), and tortured. But there were also differences, and one would need to think carefully before equating the two. Quakers and Romanists were imprisoned at that stage, but all the people put to death were Presbyterians. However, this was more than a century after the Scottish Reformation, and not very relevant to the question.
Edit: In answer to some of the comments below, I have checked a few works, and scholars of the European Reformation commonly deal with it under the geographical headings broadly associated with modern countries. Thus in ‘The Reformation in National Context’, ed. Bob Scribner et. al. (Cambridge University Press, 1994), the chapters are entitled: 1. Germany; 2. Switzerland; 3. France; 4. The Low Countries; 5. England; 6. Scotland; 7. Scandinavia; 8. Bohemia; 9. Hungary; 10. Poland; 11. Italy; 12. Spain. It is understood, of course, that most of these geographical ‘regions’ had very different political arrangements in the sixteenth century, but it is convenient to work with modern labels.
In eight of these divisions (Germany, Switzerland, The Low Countries, England, Scotland, Scandinavia, Bohemia, Hungary) there were sovereign entities which accepted Protestantism in the sixteenth century, and which would have to be considered in connection with the OP’s question and religious persecution. In addition to what I said above about Scotland, it would appear from the relevant chapter in Scribner (pp. 1191-21) that there was no systematic religious persecution in Lutheran Denmark.
