From what we now know of Stalin, he seemed to have believed in the basics of Marxist-Leninism. As a communist, didn't he fear that killing his own military's leadership would seriously endanger the world's only socialist experiment against capitalist rivals? 1936 moreover was several years after the rise of Hitler. Didn't Stalin realize an escalation of political tensions was going to lead to war soon? There is every indication that Stalin was a ruthless, cruel man. He also displayed moments of great opportunism and caution during his rise to political power throughout the 1930s and his later conduct during the war, after 1942. What motivation beyond paranoia led to the purges, which seems like such a self-evident detrimental policy?
It is impossible to tell for sure what was inside Stalin's brain. Historians can only speculate on this. I can outline some principal opinions stated in the process of this speculation:
a) Stalin's primary goal was to consolidate his absolute power. Not only to remove any real, or potential or imaginary opposition, but to make sure that everyone was scared to death, and would blindly obey his orders whatever happened. Essentially he achieved this goal; and he won the war, after all. (Whether he won it by a narrow margin or not, is a subject of another speculation).
b) It is not only clear that he knew about the approaching war, but he planned for it, and wanted it. This can be confirmed by his recorded statements, the whole scale of preparations and by his support of Hitler on various stages of his career.
c) It is equally open to speculation whether any of those who were repressed were indeed traitors, or whether they were well-qualified to win the approaching war. In any case, the scale of the purges is well-known, it is clear that most of those prosecuted were neither traitors, nor inferior officers in comparison with those who took their place.
d) That he was simply a paranoic.
But there is really no way to tell for sure what was in his mind.
It is worth noting that in 1917, in the midst of The Great War (as it was then commonly known) both the French and Russian armies mutinied. That the French mutiny ultimately amounted to little was in no small measure due to both a massive assault by British forces (the Battle of Passchendaele) that occupied German forces on the Western Front, and a declaration of War by the U.S.
However the Russian Amy mutiny led ultimately to the formation of the communist government in Russia that Stalin led. With a repeat conflict against Germany on the horizon in 1936, it is likely that Stalin wanted to so terrorize and cow the officers of the army that none daring enough to organize any such mutiny again would be left . Despite the fact that the Soviet army in 1941 and 1942 was much less successful against the Germans than the Russian Army had been in 1914-1916, no threat of such a mutiny ever materialized during the Second World War.
Stalin developed Lenin's idea of absolute value of the power to the upper limit. Never was he interested in effectiveness for the sake of Russia or even "The Revolution". His only aim was his own power. He had annihilated the lead economists of the USSR, set by Lenin, because they were not his people. He annihilated millions of farmers because his way to annihilate his political competitors needed it.
The army officers were the last people in the USSR who had their own authority independent on Stalin. (Personally they were dependent on him, but their fame already existed and couldn't be cancelled without destroying these officers). Stalin had to bring EXCUSES to the Tukhachevsky pair for years before, agreeing with his views. These people had to die, when it became possible!
Almost all high officers were killed. On the territorial army groups level, for example, EVERY commander was killed.
So, all talk about somebody's treason, or ineffectiveness were pure nonsense. ALL the generals couldn't have been spies. If they were fools, changing them for the arbitrary lower officers doesn't give cleverer men. And talk about Darwin methods is absurd, too. For example, the general, that almost lost the Finnish war, Mesherjakov, got the upper position after the war, and his opponents, on the contrary, went down.
The normal, qualified officers were a constant and real danger to his personal power. For example, if they were left on their positions, then on the start of WWII in USSR, after the terrible losts of the start of the war, they would SURELY change Stalin for another leader. So, he had his reasons.
You need to read the BEST book on the Great Purges, including the purge of the army, written by Robert Conquest, called The Great Terror, a reassessment.
Essentially, Stalin killed off all political socialist rivals within the Communist party and millions of others, including all of the old respected communist guard from the revolution itself, so he could have absolute power. It made NO economic or military sense, but it made political sense so far as it made him absolute dictator.
He did this to his own party through false trials and executions entirely based on tortured confessions. This spread down through the party ranks until the entire communist party was terrorized. Millions were murdered during his purges. He attacked anyone with a power base outside of his control, first, within the country (the attacks on the kulaks and the forced starvation of the Ukraine, then within the communist party itself, then his attack on the army, again, killing tens of thousands of officers. He finished it off by murdering the Secret Police (the NKVD) themselves by the tens of thousands at the end, to kill off witnesses. Often he killed off the best brains, even people in the Census Bureau were murdered for producing populations numbers that were too low! (due to his killings)
You should also read Anne Applebaum's fine book, Gulag: A History.
Stalin, or any other Soviet leaders, did not "
believe in the basics of Marxist-Leninism". For all of them, including Lenin, the ideology of Marxism was a pretext for acquiring power, a fairy tale for the masses to convince them to support you. Stalin even introduced slavery and servitude (under different names of course) under the guise of "socialism"!
IMO the only two major reasons for the massive purges 1936-38, from Stalin's viewpoint, were consolidation of absolute power and producing a slave population for the massive Gulag project. You can kind of measure which of the reasons was dominating by looking at the percentage of prisoners executed versus sentenced to hard labor camps.
In case of military purges, I think consolidation of power was the primary reason. One can find quite a few documents where Stalin amends massive arrest orders to increase the percentage of prisoners to be executed versus sent to labor camps. Here's one in Stalin's own handwriting where he approves the execution of 46 generals on a single page.
The Great Purge in the USSR already in the 1930s became known as Yezhovshchina.
The main drive behind the purge in the military, and followed Great Terror of 1937 was newly-appointed (September, 1936) Commissar of the Internal Affairs Nikolay Yezhov.
It is quite conceivable that he initially convinced Stalin that there were some serious conspiracies in the military.
But less than a year after the start of the purge it seems the Politburo of Central Committee started changing the attitude.
On July 31, 1937 there was issued a directive by the Central Committee to appoint all the purged officers who were fired even on political charges on economic positions.
...And dispite this the NKVD (Commissariat of Internal Affairs, Yezhov) continued to issue repressive orders (August, 11).
On October 17, 1937 there is a directive by Commissariat of Defense, forbidding firing officers and demanding on any suspensions on political charges to consult with the Commissariat of Defense.
In January, 1938 was conducted a Plenum of Central Committe which issued a directive "About the mistakes of the party organizations regarding expulsions of Communists from the party and about the formal and bureaucratic attitude towards appeals of those excluded from the party and fixing those flaws"
January 20, 1938. A directive by Schadenko, ordering to re-consider all fire orders for the whole 1937 and restore all wrongly fired from the Red Army.
November 17, 1938. A joint directive by Stalin and Molotov "About arrests, prosecutor's oversight and conducting the investigations" which effectively put an end to the purge and contained a lot of criticism of those repressions and methods involved, hinting that there were enemies in the NKVD body itself.
Yezhov was appointed the Commissar of the Water Transport in April, received a deputy Beria in August (who effectively took all affairs) and was fired from his Commissar of Internal Affairs position in November.
The majority of those executed during Stalin's rule (about 800 thousand) fall on 1937.
So, it was Stalin, not Khrushchev who started to uncover and criticize the extensive repressions in the USSR. Khrushchev did not invent anything except blaming also Stalin.
We can see that starting from July, 1937 the Central Committee (Stalin's body) started to resist the purges conducted by NKVD.
Trying to describe this purge in terms of Marxism-Leninism: Bolsheviks knew history of French revolution and were scared by possible military coup of "Red Bonoparte". So, Bolshevik revolution, following this role model, should be finished by self-destruction of their party and termidorian coup d'etat. Trotsky, as head of Red Army, was considered as candidate to Bonoparte, in 1920s. For a while, Frunze was considered so, and after his death - it's apparently Tukhachevsky. I can repeat - it did'nt matter at all, were they REALLY going to became bonopartes, they considered as POTENTIAL candidates, it was a phobia. Trotsky himself considered (since 1933) Stalin's rise to power as termidorian coup. First candidate to bonoparte was, apparently, Muravyov (killed at 1918 during his riot).
The reason is very clear. The Red Army was founded by Leon Trotsky and power struggle inside Communist Party at 1920s shows very clearly that lots of high rank military are still loyal to Trotsky (and even more: at winter of 1924-25 Moscow had rumors that Trotsky is going to make a military coup). So, as a part of total paranoia of that years, military were under suspicion.
In general, Stalin realized very well, that war is coming, and prepared very actively (very serious improvement of military productions, at first).
To be more exact, links with Trotsky were not only reason, but one of major reasons. Other reason for suspicion in disloyalty can be, for example, former membership in some non-communist party.
In 1930s Trotsky was still very active, since 1935 trying to found IV International. Sure he was considered as clear danger.
In general, the purge of Army was a part of total hunting for POTENTIALLY disloyal members of so called "fifth column": former members of oppositions, former members of non-communist parties, former military of tzar army (Case Spring at 1930-31), people of foreign origin (considered as indoctrinated by alien ideology) etc.
Leon Trotsky in 1918 was head of Red Army, so we can safely assume that many of high-level commanders were his supporters and proteges. He was also main opponent of Stalin, and Lenin named Trotsky as "most capable man" to replace him (Lenin) in Lenin's testament.
He was also brilliant public speaker and agitator (unlike Stalin), so again this could win him supporters in army.
Trotsky and Stalin had multiple disagreements during the war, where it can be argued that Stalin's mistakes and failures to cooperate with Trocky led to losses (like Battle of Warsaw see "Second Phase"). Likely, many of the purged commanders were aware of Stalin's mistakes, and had to be eliminated to cover the tracks.
Lenin died 1923, Stalin-Bukharin faction blocked Testament's publication during 1924-1927 while Stalin consolidated power.
There was no "great" purge. This is propaganda. 40 thousand were dismissed from the Red Army. I said "fired", but not destroyed. Some of the dismissed are pensioners. Old farts without qualifications and education. Some of the dismissed are alcoholics and loafers. Some of the dismissed were arrested. These are bandits, thieves. A simple example: in the early 30's ready often after drinking, the officers started to shoot each other. Are these innocent victims of Stalin? I beg of you! Some of the dismissed were innocent. For example, Rokosovsky. But he was not shot or destroyed. Moreover, the "bloody devourer of babies" Beria freed Rokosovsky. Beria began a massive revision of the criminal Soviet commanders. Stalin cleared the army of small bandits and alcoholics.
In place of Stalin you would have done the same.
This book describes the beginning of the war. Part of the information about the causes of the defeat and "great purge". https://www.amazon.com/Dubno-1941-Greatest-Battle-Second/dp/1910777749/ You can search for a black cat in a black room for a long time. Especially if it's not there.