Both sides of this tactic has been used throughout human history. On the one hand, mistreating POWs may make soldiers run in fear rather than fight. On the other hand, those same soldiers will be less likely to surrender if they're cornered.
Soldiers when in desperate straits lose the sense of fear. If there is no place of refuge, they will stand firm. If they are in hostile country, they will show a stubborn front. If there is no help for it, they will fight hard.
This answer will be punctuated with quotes from The Art Of War. While it doesn't speak specifically about the treatment of prisoners, it does talk much about the virtues of victory without fighting.
A reputation for poor treatment of POWs was more successful in pre-modern times when most "soldiers" were conscripts and mercenaries formed around a cadre of professionals. Morale, not bloodshed, was the defining factor in battle; victory was to break your opponent's formation and see them flee the field. Battles were often fought on open plains, so soldiers always had the option to run away. Formations like the phalanx were designed as much to prevent soldiers from running as they were for protection. Sun Tzu puts it best...
When you surround an army, leave an outlet free. Do not press a desperate foe too hard.
Another successful version, used throughout antiquity, is a hybrid. You ensure your army has a reputation for bloodlust. Then, when attacking a town, you promise clemency only if a town surrenders without a fight. The attacking army avoids the need for a lengthy and expensive siege, plus they get an intact town to use for logistics, and they don't have to worry about angry townsfolk. The defending townsfolk don't need to face potential starvation followed by being sacked by an angry army, and they can sell their goods to the soldiers. This is particularly successful if the town is only moderately loyal to whatever empire conquered it last time. The Mongols were particularly adept at both sides of this tactic, keeping their word to spare a town which surrendered, and ensuring that towns which resisted were wiped from the Earth. They even allowed people to escape just to spread the fear of the Mongols.
Therefore the skillful leader subdues the enemy's troops without any fighting; he captures their cities without laying siege to them; he overthrows their kingdom without lengthy operations in the field.
The modern version of this is to request surrender in order to save a city from a lengthy bombardment. After an example was made of Rotterdam, this happened throughout WWII. Manila, Paris, Cracow and Brussels were declared open cities in the early days before it was understood what Axis occupation would be like. Rome, Athens and Hamburg later were declared open for the Allies.
One successful version is to use propaganda to convince your own soldiers that the enemy are monsters, whether they are or not. Not only will this embolden your soldiers by making your cause the righteous one, but it will also make them think twice about surrendering. All sides in WWII used this tactic, the Japanese illustrated the most successful and most tragic use of this tactic in the Battle of Okinawa when they fought a hopeless defense to nearly the last man.
Where the tactic of instilling fear backfired dramatically is on the Eastern Front of WWII, first for the Germans, later for the Soviets. First, the Germans squandered their early advantage as "liberators" against Stalin by brutalizing the population. Instead of encouraging an uprising against Stalin, they fueled a partisan campaign against the Germans. Later, the Soviets retaliated against the Germans leading Germans to fight fiercely against the Soviets knowing what sort of vengeance a victorious Soviet army might take on Germany. This lead to many Germans fleeing west and choosing to surrender to the Allies which reaped the Allies a treasure trove of intellectuals, commanders and equipment for the Cold War.
I'll leave the last words for Sun Tzu.
Build your opponent a golden bridge to retreat across.
