In that time (the quote comes from 1872), brown-bread was just recently introduced as having any benefits for health. Edward Smith started to promote brown-bread in his book "Practical dietary for families, schools, and the laboring classes", released in 1865 in London. Earlier it was treated as worse (because of ingredients) and commonly used as a meal for peasants and poor people. It had a bad fame among higher classes and when some people in United States started to favor it, it could be seen as just another example of liberal philosophy and lack of respect to traditions of conservative society.
What's more, there were times in history when brown and white bread was restricted for different groups of people. The linked website provides following informations, based on book "In The Devil's Garden. A Sinful History of Forbidden Food", written by Stewart Lee Allen (2002):
The Italians had divided social classes between those that ate white
bread called "Bread Mouths" vs. those that ate dark bread called
"Fodder Mouths." Just like with the French, the aristocrats ate the
white bread only. The Roman elite would attack someone if they
offered them a slice of dark bread. Caesar even made it a law that
stated that anyone who served an aristocrat dark bread was to be
punished with prison time.
In 1775, Philippe Cordelois, a shoemaker, was arrested in his home by
the King's men. He was charged with "possession of a crouton of bread
that was absolutely brown" and taken to the interrogation building
below the du Chatelet (today a metro station). Why was this brown
bread so bad evidence against him?
Just like the Italian peasants, the French peasants also ate only
coarse, dry and barely breads. It was believed that the peasants were
slightly above pigs in those days. While, the aristocrats had very
touchy digestive systems and could only deal with eating the softest
breads that were well-buttered. The only exception is the French
Army, which was allowed to eat white bread only after they revolted
when given rye.
Here's another fragment, regarding connection between brown-bread and political matters, as asked by Mark:
This whole thing was so ridiculous that even Marie Antoinette came out
with a statement. "If the peasants were unhappy with their bread, why
didn't they just eat cake?" Ironically, in 1793, just a month after
she said this, she was beheaded and the National Assembly voted to
create a National Bread of Equality.
Once the revolution got into full force, political correctness took
over. Suddenly, white was out. Proletariat brown was in. Political
groups protested against the class separation caused by la mollesse
(luxury white breads) and urged that it be banned to create some
uniformity. Court records of this time show who some bakers were
arrested for politically incorrect baking.