Before the widespread use of firearms, various kinds of plate armour were in widespread use. As muskets (and later, rifles) became widespread, full plate became less effective and fell out of use.
However, mortars and (later) cannon did exist, firing exploding shells. Unit formations would be changed depending on whether the enemy had artillery or not, because a tightly-packed formation would suffer greater casualties from a well-placed shell. Shrapnel was very much a feature of battles (named after Henry Shrapnel, of course).
With this in mind, why was body armour so rarely used? Even the most basic armour would have reduced casualties. More curiously still, effective steel helmets were replaced with caps or leather helmets such as shakos which are more decorative than protective. Steel was still relatively expensive, but not unreasonably so, and wrought iron plate was in mass production.
At the outbreak of WWI, the high number of casualties with head wounds led to swift adoption of helmets such as the Adrian helmet. Other body armour was tried as well, with varying degrees of success, but helmets were a clear win. The only innovative aspect of WWI helmets was that they were pressed from a single steel sheet which made them cheaper to manufacture. As far as their basic design went, they were functionally indistinguishable from helmets of the Middle Ages.
Failing to equip your troops to keep them alive could perhaps be understood if combat effectiveness was allowed to degrade during peacetime. However in a period covering the Napoleonic Wars, the American Civil War, and the maintenance of empires around the globe by the various European powers, this clearly is not the case.
Why did it take 100+ years for the value of effective helmets and body armour to be recognised? And how could the combined militaries of the world forget that it existed for 100+ years?
Updated following comment by Steve Bird: I've seen this question already, and I don't think it answers my question. As answers to that question say, the issue with shields is that using them effectively takes serious training. The same applies for fighting in full plate armour too - whilst I'm not a HEMA player, I'm sure it takes some practise to ignore sword strikes to your extremities and trust in your armour. This is not the case for helmets and light breastplates though - they're simple to put on, they give decent passive protection, and apart from getting used to the weight they don't affect anything else.
Also on another comment by Steve Bird, the scope may seem to be wide, but in fact it applies through the entire 19th century. Whilst the guns available are radically different, from muskets and smooth-bore cannons at the start of the century to recognisably modern rifles and artillery at the end of the century, the uniforms worn by soldiers were unchanged throughout. Headgear in particular is clearly a fashion statement and is not designed for personal protection. I'm curious why that's the case when keeping yourself alive would seem to be more of a priority - and even if keeping your troops alive is not your top priority, why anyone would start outfitting them with shakos or pith helmets (which do cost money) instead of a simple helmet.