Actually, the Romans used the same phalanx everyone else did for a very long time. Past Hannibal.
The essence of winning a phalanx battle is to attack the flank of the phalanx. One may achieve that many ways, hence the many ways phalanxes were formed in particular battles - adapted to the width of the battlefield usually, though if one's enemy overdid that, one might go narrow and deep hoping to break them and turn into their new flanks. That kind of calculus.
But ordinarily, the outcome depended upon gaining superiority on one flank or the other in one's mounted forces. I'll use "cavalry" from here, but they were in no way cavalry. Closer to knights in that they fought as individuals. The cavalry that gained superiority then piled in and while slaughtering the hapless foot soldiers, broke the phalanx allowing their own phalanx to complete the main destruction. As everyone fought spear-in-right-hand, a phalanx battle usually turned counterclockwise as if on a pivot making the enemy's right flank the most exposed and therefore an attractive target for trying to gain cavalry superiority.
Side note: elephants were never, ever effective unless held on the flanks for charging into men who were a) distracted by other men trying to kill them, and b) no longer screened by their own knights.
Hence, the spear. To have even the least chance in the world of holding off horsemen, one needs pole arms. Those longer and longer spears one needs of were of no value fighting someone 4-5-6 ranks of men up, but they did allow one to face cavalry. And the most effective method of using a spear in combat is underhanded: ripping up and across; not something one can even attempt with a 23-foot spear. Someone nearby striking over handed at the same man might be handy since he then has to defend up and down, so the 2-3 ranks behind you having slightly longer spears than yours meant they could provide a worthwhile assist.
Spears though are unwieldy - all you really want is the sharp end. Hence the short sword. Why as long/short as it was? Because iron weaponry is brittle, and it couldn't be any longer without a good chance of breaking. Besides, the evolution of the concept started with just using the head of the spear. Bigger shield to hide your strikes and let you concentrate a bit more on the strikes due to its greater protection. A bit less mobile for the larger shield, but one's striking was far more aggressive, able to come from almost any place and direction. Not as effective anymore though against horse from the flanks, but if you can beat the enemy phalanx before his horse gains superiority on a flank, that never matters.
So the maniple. It allows easier maintenance of your line when attacking over broken ground, so your enemy sees his more broken by the ground than you see yours. (Men can speed up or slow down because there aren't 7-49 men directly behind you (picture cars starting up from a traffic light and one driver hits his brakes for even a moment), and they can move up a rank or two if the ground causes a gap to form while the men farther behind can edge over and up to keep the depth uniform. In a phalanx, the men moving about have the wrong weaponry.
Perhaps far more important, picture a man in the first rank getting tired. He just gets more tired until he is killed in the phalanx. The men behind him have completely unsuitable spears for first rank fighting. A man from rank 7 is going to replace him using his 23-foot spear?? There is weapon trading happening as they rotate out?? Not too easy. With the maniple, not only can men move up to rotate out tired fighters, but they can rotate into any role in the entire unit. They form a reserve of ready, rested replacements and use them as needed. (In fact, the concept of a reserve for the overall army may have been born in the maniple. Though it took a while for the Romans to learn how to use reserves properly.)
Oh, and wounded men can be rotated fairly easily and actually live through the battle, perhaps even fight more later on.
As to facing cavalry if/when the enemy gained tactical superiority on a flank, they still had supplies of javelins and actual spears. Though the truth is, no foot unit survives when an intact unit of heavily armored and weaponed knights piles into your formation from the flank.
To make it very short, why fight using the whole heavy spear when all you really want is the short, sharp end? The Romans found they had no answer to that and used just the short, sharp end of the spear. What we call a short sword.