I think the key here is to appreciate just how long the front of a phalanx was. A phalanx really only worked in one continuous line. Each man at the front held anywhere from half a meter (synaspismos, shields locked) to one meter (pycne, normal battle formation). 8000 hoplites arrayed 8 deep means a front of 500 to 1000 meters! This was done, in part, because it's extremely difficult for an ancient commander to see what's going on or to issue orders. One, big continuous front was relatively easy to train for and maintain.
In contrast, the flanks are a mere 8 men, 8 meters long. They could provide no significant defense. The enemy would simply flow around to the rear. To prevent this, the flanks of the army would either extend, or they would retreat inward forming a bowed shape to protect the flanks and rear.
The rear ranks could, and probably did, turn to face a threat from the rear. This is why phalanxes weren't regularly defeated by lightly armed skirmishers and cavalry simply racing around to the rear. But if heavier units got behind them, this meant they were now fighting surrounded. Nobody likes to fight surrounded and this would have a crushing effect on morale. This was not how they were trained to fight, the main weakness of a phalanx was its inflexibility, and the infantry square had not yet come into fashion. Being attacked from a new direction also risked damaging the cohesion of the unit as individual soldiers were now being pushed and attacked from multiple directions.
Publius Flavius Vegetius Renatus stated in De re militari
No part of drill is more essential in action than for soldiers to keep their ranks with the greatest exactness, without opening or closing too much. Troops too much crowded can never fight as they ought, and only embarrass one another. If their order is too open and loose, they give the enemy an opportunity of penetrating. Whenever this happens and they are attacked in the rear, universal disorder and confusion are inevitable.
De re militari, Evolutions
Ancient battles weren't particularly lethal, instead it was about breaking their tight, defensive formations. When you see in the movies two armies run at each other and the fight immediately devolves into individual combat, that's movie nonsense. Ancient armies would fight for hours in tight formations until exhaustion, casualties, or clever tactics broke them, or nightfall simply ended the battle.
One of the main improvements of the Macedonian phalanx was the employment of significant amounts of light and heavy cavalry to protect their flanks and harass the enemy. In addition, the men making up the phalanx were professional soldiers and could be drilled in more complex maneuvers. The Macedonian phalanx would typically pin the enemy in place while the light cavalry dealt with skirmishers and the heavy cavalry attacked the enemy's flank.
The Romans improved on this with the maniple as their basic unit, employed to great effect against the Samnites around 300 BC. 120 men where formed into a front 40 men wide and 3 men deep. This smaller, well trained unit allowed for much greater tactical flexibility. Even so, attacks from the rear invited chaos.
The Romans also introduced a circle formation into training to fight when surrounded. Again, De re militari...
They must be taught to form the circle or orb; for well-disciplined troops, after being broken by the enemy, have thrown themselves into this position and have thereby prevented the total rout of the army. These evolutions, often practiced in the field of exercise, will be found easy in execution on actual service.
This evolved into the infantry square employed by the Romans at the Battle of Carrhae 53 BC. At Carrhae each side was 12 cohorts or about 7,200 men per side! While this succeeded in giving them all around protection, it limited their mobility and they were harassed by horse archers throughout the battle.
The infantry square would continue to be infantry's response to being surrounded or cavalry attack all the way up to the Battle of Waterloo until rapid fire rifles made them obsolete.