In reading McPherson's "Battle Cry of Freedom" he gives several examples of cavalry groups of less than 2000 men causing tremendous damage, seemingly with tremendous leverage.
Why was the North so slow to catch on? Why wasn't this tactic used more frequently during the war? Was this technique used more widely in subsequent wars?
Confederate general Nathan Bedford Forrest famously said, "I git there first with the most men," often misquoted as "I git there firstest with the mostest." A Union cavalry general, John Buford, reportedly said, "A horse is just a form of transportation."
The main advantage of cavalry is that it could get to places, and at speeds and timetables not available to infantry. The disadvantage was that horsemen had to dismount to fight properly against infantry (in an era of rifles, especially "repeating" rifles introduced late in the war). Every fourth cavalryman held the horses of three others, so the fighting strength of the cavalry unit was effectively reduced by one fourth.
Confederates like Forrest understood that it was better to arrive first at a critical location with say, 2,000 men, of which only 1,500 could fight if it came to that. Especially when they were on a "raiding" mission, "hitting and running" would be much better than arriving with 2,000 infantry "later." The northern generals (other than southern-born Buford), were more concerned about the one-quarter reduction in fighting strength. A difference of philosophies leading to a difference in tactics.
This tactic was used less in subsequent wars as the spread of "repeating" rifles further increased the advantage of infantry over cavalry. Cavalry was most effective during the era of single-shot weapons such as muskets during which "contact" weapons such as lances or sabers were most effective.
While I'm no expert, I believe the following to be the explanation for the "why calvary is so effective" part of the question.
Since ancient times up until WW1, the main goal of almost every battle had been to outmaneuver the enemy army (e.g. flanking them) and forcing them to retreat. When this would happen, the calvary (as it moves faster than infantry) would chase them and mow them down from behind. With no effective way of defending, the fleeing army would usually end up decimated. This also explains why ~80% of deaths in most battles of pre-WW1 era was during retreats and not actual battles. And also why Alexander the Great allegedly lost less than 1000 soldiers during his entire career - his army was never forced to retreat (and even if the number is likely overblown, he did indeed lose a surprisingly low number of soldiers).
As of WW1, heavy artillery came into play, allowing the retreating army to simply stop and bombard the calvary chasing after it, rendering it infinitely less useful.
