Rutilius was referring to monks, whom he strongly disapproved of because of their ascetic lifestyle. At the time (late antiquity), the abandonment of wealth and family for an ascetic lifestyle was strongly disapproved of by many Christians, especially among the wealthy elite. Rutilius was most likely a pagan but he shared the view of these Christians.
Rutilius Claudius Namatianus (Claudio Rutilio Namaziano) was a poet of the 5th century AD. In an increasingly Christian Roman world, the scholastic consensus is that Rutilius probably remained a pagan, but the poet's dislike of monks was more to do with
the total renunciation of public life advocated by the ascetics
than with Christianity. Rutilius held high positions under Christian emperors so he would be unlikely to write in a way that would offend those he depended on. In fact,
Many Christian members of the elite had reservations enough about sons
and daughters giving away their money and adopting ascetic lifestyle
in cities or on family estates....But the danger of desert islands (on
top of their prison asociations) was something altogether
different....many conservative Christians would have warmly endorsed
such hostility to this alarming development.
Alan Cameron, 'The Last Pagans of Rome'
This English translation of de Reditu suo (A Voyage Home to Gaul) provides more context:
As we advance at sea, Capraria now rears itself — an ill-kept isle
full of men who shun the light. Their own name for themselves is a
Greek one, "monachoi" (monks), because they wish to dwell alone with
none to see. They fear Fortune's boons, as they dread her outrages:
would anyone, to escape misery, live of his own choice in misery? What
silly fanaticism of a distorted brain is it to be unable to endure
even blessings because of your terror of ills? Whether they are like
prisoners who demand the appropriate penalties for their deeds, or
whether their melancholy hearts are swollen with black bile, it was
even so that Homer assigned the ailment of excessive bile as cause of
Bellerophon's troubled soul; for it was after the wounds of a cruel
sorrow that men say the stricken youth conceived his loathing for
human kind.
A little further on in de Reditu suo, Rutilius also writes unfavourably about the ascetic lifestyle when passing the island of Gorgon (between Corsica and modern Livorno). He sees the this abandonment of wealth and family as "incomprehensible".