I think that when war is declared, there is generally a fear of fifth columns operating at home on behalf of the enemy. I can offer a few examples from the first and second world wars.
Ahead of the first world war, MI5 had prepared extensive lists of
potential fifth-columnists who they thought should be interned
should hostilities break out [Andrew, 2009]. In the event, the UK government
decided to only intern a small percentage of that list, but the fact
that the list was made supports your theory. Those who were interred were
held in camps on the Isle of Man (which was also used for Prisoner of War
camps). The government was quite open about the camps, even allowing
journalists to visit and report on conditions there.
During the First World War, there were several large-scale,
anti-German riots in British cities. For example, the riots in
Liverpool in 1915 targeted German families, and German-owned
businesses.
Many businesses that were owned by foreign nationals found it necessary to make that fact clear to avoid being targeted. Images like the one below are all-too-common from that time:
- During the Second World War, internment camps for German, Italian and
Finnish civilians were once again set up on the Isle of Man by the UK
government. Many of those interned on the island were refugees from the
Nazis, so I have often wondered how they must have felt when the saw the
island's emblem on arrival:
On a side-note, one of those interned on the Isle-of-Man during the Second World War was the German archaeologist Gerhard Bersu. He was permitted to continue working as an archaeologist during the war (assisted by other internees), which goes some way towards explaining why we know so much about the archaeology of the occupation of the island during the historic and prehistoric periods.
Records of those interned by the British government during the two World Wars are now held at the UK National Archives in Kew.
- Following the attacks on Pearl Harbor, the United States government
introduced internment camps for Japanese-American civilians in
many states. Between 110,000 and 120,000 American citizens of
Japanese ancestry were effected. They were motivated to do so by the
fear of fifth-columns acting against the US on behalf of Japan.
]
However, the fact that the United States did not set up similar camps for American citizens of German ancestry does suggest that the phenomenon might not be a general one. At the very least, it suggests that the factors that lead to such a "witch-hunt" are rather more complex in nature.
In practice, this phenomenon doesn't actually require war to be declared for the effects to be felt. @SPavel has already mentioned McCarthyism, which was a similar reaction to the fears of the Cold War. In this case, the reaction can be traced back to Executive Order 9835 signed by President Truman in 1947, which required that federal employees (all 2 million of them) should be screened for "loyalty".
But despite all the above examples, the "large scale counter-intelligence-turned witch hunt" you describe in the question seems to be very much the exception rather than the rule.
If we look further back in history, in 17th century England only one torture warrant seems to have been issued in the years leading up to the English Civil War, and none during it [Thomas & Leo, 2012, p31]. Other than that, there was just the Witchfinder General, Matthew Hopkins - which was an actual "witch hunt" rather than a war.
I did an online search looking for examples of torture being used to extract "confessions" to implicate innocents during the American Civil War and the American War of Independence, but couldn't find any examples at all. Obviously, it is probably impossible to prove that it never happened, but it certainly seems it wasn't carried out on a large scale.
Torture was certainly used in France during the Reign of Terror but, for the most part, that wasn't during a war, and so would seem to fall outside your criteria.
Sources