Understanding this development first requires a discussion of various sailing rigs. Further, to understand the consequences of various sail plan decisions, one must need understand the points of sail, from close-hauled, through beam reach, to running.
When running-before-the-wind a sailing vessel is very comfortable, with the deck flat, hull stable in the water, and wave action minimal. However you can only go downwind, and never even as fast as the wind itself. If the breeze is only 3 knots, then 2 or 2.5 is as fast as the vessel can go. Square-rigged vessels are ideal for this point of sail, with the rig itself stable on top of the beneficial attributes mentioned above. To minimize drag, and thus maximize speed, the sail(s) must be centred about the boat. Viking longships typify this type of vessel.
Over time, sailors realized that by using sheets (ie ropes, not sails as the uninitiated might think) to angle the sail and supporting spar into the wind, plus a rear-mounted rudder, points of sail several degrees off of straight downwind could be achieved. Even more remarkable, the boat was now capable of sailing faster than the wind itself. The slight heel and increased wave action of the vessel was a very minor inconvenience by comparison.
For square-rigged vessels, the limit of this gain is obtained at a beam reach or a very slight close reach. Although the vessels could make headway while pointed to within about 60 degrees of the wind, the leeway encountered ensured that the velocity made good to windward was minimal at best. In any sort of unfavourable wave action, VMG would be negative, and the vessel was better to anchor and wait for more favourable conditions. This is why one will so often hear of square-rigged vessels being stuck, either in or outside of a port, waiting on a favourable wind. The large tidal range of Southern England allowed the squadrons of Hawkins and Drake to make sea despite the mildly unfavourable wind conditions when the Armada was spotted!
In order to sail upwind at all a vessel requires a fore-and-aft-rig, with either gaff or lateen sails. These rigs generate substantial aerodynamic lift from the sails which, when the vessel also gains hydrodynamic lift from a well designed keel, can enable efficient sailing to within 45 degrees (30 degrees for some very high-tech modern vessels) of the wind. Top speed relative to wind speed steadily decreases from its peak at beam reach, but if the leeway is managed by a well designed keel the VMG upwind obtained by tacking on the wind shifts can be considerable. The ship is heeled well over at this point of sail, is trickier to steer, and suffers more from wave action, but need not wait on the wind.
So with all that our of the way, why three-masted vessels? Because it allowed a single ship to combine multiple rigs. Theoretically a single massive mast is most efficient, but that only works for carbon-fibre masts with small crews. A wooden ship with wooden masts achieves an even larger sail area with multiple masts. By pushing the fore-mast and bow-sprit further forward on the vessel, balanced by sails on the mizzen mast to keep the bow from digging into the water, even a square-rigger can make some headway upwind.
It was pointed out to me that minimum crew size depends on the size of the largest sail - not of the total sail area. Thus square-rigged vessels with multiple masts and a plethora of smaller sails on every mast could be handled in a sea worthy fashion by quite small crews. Combined with not needing to worry about crash-gybes when running downwind, such vessels were ideal for trade vessels utilizing the trade winds most of the time.
More information on the development of European post-Renaissance navigation. The navigation per se did not require three-masted vessels, but those designs (developed through carrack, caravel and galleon) were the most ocean-worthy yet conceived by Europeans as they entered the Age of Exploration.