I think it is also necessary to recognize that the southern states 1860 basically was in the same situation as the small states (and not so small states in Latin America) or the states in Africa after the great depression 1930:

  - dependency on cotton, sugar and rice as cash crops but which is heavily exposed to changes in international commerce.
  - the Northern states push for protectionism after 1800 was disadvantageous - less profits from export-focused agriculture
  - the large plantations was owned by a small white minority who invested in industries in the north, railroads and luxury items - not in small scale industry (or growth of the local economy) in its neighborhood.
  - the plantations was also in debt to the banks (be them in New York or London.) This, after the abolition caused a even larger pressure on the profits. This debt was large even before the civil war... ie profit wasn't as good as they could seem to be.

For example: clothing for everyone was imported from the northern states or more probably England, so also the chains necessary in the slave trade and the food for them.

The british settlers from Barbados who (according to "A History of World Societies" Wiesner-Hanks, Ebrey) colonised South and North Carolina had
already on Barbados built up a number of sugar plantations.
The Carolinas is very well suited to large single-crop plantations exporting rice and later cotton while the nature in the northern states isn't really suited to a system of large-scale slave worked agriculture with a cash crop as output.

One other weakness of slave labour is the capital outlay - the rents on that can pay for a fair amount of labour from free men, men which have a good reason to improve their work.

Harold D. Woodman in the journal of southern history (vol 29 nr 3) mentions a sum of 1.2 billion dollars invested in slaves. The loss in 1865 and emancipation made that sum basically nil.
One other problem which is enumerated is this:
the southern stagnated and was overtaken by the North between 1800 and 1850. Why ? Was it because european emigrants didn't want to work in the South - which would have meant accepting wages only a bit larger than the planters costs for a slave.

Ulrich B Philips views plantations as a method of organizing work and slavery as a way of exploiting labor. One of his points is that slaves simply in the long run costs too much - a planter had to feed and support his slaves their whole life. Slaves is bound capital which can't be liquidated and then re-invested in some other more profitable endeavour. Because of that, slavery was for the South a one-direction road towards a road-block.