Tony Horwitz is paraphrasing the speech that Calhoun gave in the Senate in the debate on 6 February 1837 on the Abolition of Slavery.
Calhoun's speeches have also been collected and published, with this particular speech in Volume 2 (a slightly easier to read version of the text is also available on Wikisource).
What Calhoun actually said was:
"I hold then, that there never has yet existed a wealthy and civilized society in which one portion of the community did not, in point of fact, live on the labor of the other"
"There is and always has been in an advanced stage of wealth and civilization, a conflict between labor and capital. The condition of society in the South exempts us from the disorders and dangers resulting from this conflict; and which explains why it is that the political condition of the slaveholding States has been so much more stable and quiet than that of the North".
The rich whites (who provided the capital) were certainly freed from drudgery, but the poor whites of the South (who had nothing to offer but their labour) were clearly not. Even on the plantations, the slave overseers (who were white) had to work, and so were not able to just do what they wanted.
So, to answer your question, no, there weren't any "small communities in which slaves did all the unpleasant tasks and Whites were able to do what they wanted".
It is perhaps worth noting that Calhoun also said in that speech:
"I may say with truth, that in few countries so much is left to the share of the laborer, and so little exacted from him, or where there is more kind attention paid to him in sickness or infirmities of age."
Compare that claim with Calhoun's treatment of his own slaves.
One event mentioned in his correspondence (and noted on his Wikipedia) page illustrates the point.
In 1831, a house-slave named Aleck belonging to Calhoun ran away when threatened with a severe whipping. Calhoun wrote to his brother-in-law James Edward Calhoun (Letter to James Edward Calhoun, 27 August 1831, Correspondence of John C. Calhoun, p300), asking him to keep a lookout for Aleck. Calhoun wrote that if he was taken, he should be "severely whipped and sent home immediately".
Aleck was caught, and Calhound wrote the following to his captor:
"I am glad to hear that Alick has been apprehended and am much obliged to you for paying the expense of apprehending him . . . . He ran away for no other cause, but to avoid a correction for some misconduct, and as I am desirous to prevent a repetition, I wish you to have him lodged in Jail for one week, to be fed on bread and water and to employ some one for me to give him 30 lashes well laid on, at the end of the time. I hope you will pardon the trouble. I only give it, because I deem it necessary to our proper security to prevent the formation of the habit of running away, and I think it better to punish him before his return home than afterwards".