Felix Frankfuter, before his SCOTUS appointment, was the founder of the ACLU, and an avid New Dealer. As such, the quote doesn't sound at all like him, or at least not the way its being used. He was just not a guy who thought about government in that way.
For reference, here are some documented quotes from him about how he believes the USA operates. Both in outlook and vocabulary, they don't seem similar at all to the one you have.
In a democratic society like ours, relief must come through an aroused popular conscience that sears the conscience of the people's representatives.
Lincoln's appeal to "the better angels of our nature" failed to avert a fratricidal war. But the compassionate wisdom of Lincoln's first and second inaugurals bequeathed to the Union, cemented with blood, a moral heritage which, when drawn upon in times of stress and strife, is sure to find specific ways and means to surmount difficulties that may appear to be insurmountable
- Cooper v. Aaron, 358 U.S. 1 (1958).
This is just not a guy who believed that the people were helpless against some shadowy alternate power structure. Quite to the contrary, he believed strongly in the ultimate power of the electorate.
Some people can't put themselves into another person's head, and thus assume those who they disagree with must have their same perspective, but be stereotypical mustache-twirling "bad guys". This reads a lot like a quote made up by such a person.
I tried running some ngrams on parts of the phrase, but the results were pretty inconclusive. For example, "exercise power" was about twice as popular (and rising) in the early 80's as it was back during Frankfurter's heyday. However, back then there was a big spike in "real rulers". This seems to be due chiefly to a series of books by that name published from the late 30's through the early 60's. (Todo: a search of The Real Rulers of America may well produce the actual source of this quote). Again, these books all seem very conspiratorial in nature, and thus may be more likely to have been read by a 1985 conspiracy book author than by Justice Frankfurter.
So I'd say that it is probably quite fair to take the position that Justice Frankfurter never said any such thing, unless someone can find a documented speech or decision in which he said it. Everything the guy did was heavily documented, so if it exists it should be easy to find.