The fall (and rise) of lances were tied to other developments regarding horse troops.
It was the (original) "cavalry" that used pointed weapons from the lances, dating back to the Middle Ages. By about the 17th century, there was a new type of horse soldier, dragoons, who were mounted infantry, rather than cavalry. As such, they were "musketeers" on horses, in contrast to "lancers." As time passed, tacticians "rethought" the value of hand weapons, and gave dragoons swords, which were easier to handle than lances, while (in some cases), replacing their heavy muskets with lighter "carbines" for firing. Provided with inferior weapons and horses, dragoons were usually at a disadvantage against both infantry and cavalry in a "stand up" fight, but their combination of speed and firepower made them ideal for patrolling, scouting, seizing and holding key points, etc.
In the 19th century, the introduction of rifles changed the equation further, by making guns longer ranged, and through the addition of the "repeating" feature. At this point, riflemen on horses were at a disadvantage against riflemen on foot, but the cavalry did have the advantage of getting to key points faster. Using this advantage, cavalry would (mostly) fight dismounted, with one-fourth of the men holding the horses of three others. In a war in which (cavalry) general Nathan Bedford Forrest described as "getting there firstest with the mostest," the advantage of faster arrival often outweighed the disadvantage of a one-quarter reduction in manpower.
Even so, traditional cavalry (with lances) was still good for some things, like attacking artillery positions and "running down" fleeing infantrymen from broken formations. These advantages were more apparent in the plains of Eastern Europe (and in Spanish possessions), than in the more broken ground of the rest of Western Europe. So many East European armies reserved a some cavalry for such purposes, while other armies largely switched to infantry. In this regard, the (remaining) use of cavalry was kind of like the old rock-scissors-paper game.