The most notable non-Christian Roman critic of gladiatorial games was likely the Stoic philosopher Seneca. Of course Christians like Tertullian had good reason to speak out against bloody spectacles in general, and some of the conquered peoples living under Rome took less joy in the games than did the Romans. But for the most part, even enlightened Romans whom we might expect to criticize the games kept quiet, including Stoics like Epictetus and the Emperor Marcus Aurelius. Tacitus mildly disapproved of the games, writing:
"There are the peculiar and characteristic vices of this metropolis of
ours, taken on, it seems to me, almost in the mother's womb--the
passion for play actors and the mania for gladiatorial shows and
horse-racing."
But this is a pretty mild condemnation--he doesn't single out gladiatorial fighting as any worse than plays or horse races.
I see Seneca's name most often associated with criticisms of gladiatorial fights, but he comes off like a lone voice in the wilderness. He thought the games desensitized viewers, and because of them:
Man, an object of reverence in the eyes of men, is now slaughtered for
jest and sport ... and it is a satisfying spectacle to see a man made
a corpse.
As mentioned, early Christians were obviously critical of entertainments that not only had pagan religious origins but which sometimes featured the grisly deaths of fellow Christians. But interestingly, critics like Tertullian didn't object solely to the human slaughter, but to the spirit of all spectacles. Using some Yoda-esque logic, he writes:
There is no public spectacle without violence to the spirit. For where
there is pleasure, there is eagerness, which gives pleasure its
flavor. Where there is eagerness, there is rivalry, which gives its
flavor to eagerness. Yes, and then, where there is rivalry, there also
are madness, bile, anger, pain, and all the things that follow from
them and (like them) are incompatible with moral discipline.
In other words, gladiatorial fights epitomized what was corrupt about all circuses, games, plays, shows, and so on. Later Christian writers may have exaggerated the extent of anti-Christian brutality, but there is at least one passion text that purports to be an eyewitness account of Christians being exposed to gladiators and wild beasts in the early 200s.
Finally, gladiatorial shows were less popular outside of Italy, Roman colonies, and Roman military encampments. In some provinces, there was even overt criticism of the games:
When King Herod wished to offer spectacles in an amphitheater he had
constructed near Jerusalem, "the Jews found such a cruel pleasure to
be impious and an abandonment of their ancestral customs."
Source: Sisella Bok on "The Paradox of Entertainment Violence"