Some Indian and Chinese emperors had more than 50 mistresses. Wasn't it difficult to please so many women and maintain peace among them? What caused them to have so many women?
I think you're finding this odd because you are looking at it from the point of view of your own culture, and assuming that is "normal". In fact, different cultures evolved with different standards for adult gender relations. Since the rules used appear to be nearly universal within language families, likely the roots of these idioms go back at least to the time their entire language families were formed.
In Afroasiatic cultures, the common historical arrangement was plural marriage. Wives had nearly equal status (subject to the preferences of the husband), and the status of all their offspring was strictly birth order.
Indo-European cultures tended more to a strict exclusive binary marriage. IE Men were generally free to dally outside of that bond (if they could), but both the women and any resulting progeny would not considered "legitimate" at all, so they essentially had no status.
Our records for the ancient Sumerians (who were neither Afroasiatic nor Indo-European) tend to show a surprisingly looser bond, with women able to control their own finances and hold their own positions and generally participate in society completely independent of their husbands.
China appears to have been essentially halfway between the IE and Afroasiatic models. You could look at what high-status men like the emperor had as a plural marriage, with the first wife having a greatly enhanced status. The offspring of concubines were not removed from inheritance, and thus were not considered "illegitimate" in the way a child of a European mistress would be. However, they were lower status than the first wife's children, regardless of birth order. So calling Chinese concubines "mistresses" as this question does is very misleading. Better to think of them as "lesser wives".
As for how this was run, consider it like any large workplace. People are thrown together not necessarily because they are or aren't friends, but because they all are working together on the same job.
In both Afroasiatic and Chinese plural marriage societies, the number of wives was limited by the number that the man was rich enough to support. Of course this meant having a large number of them was an indicator of wealth. As with any wealth indicator, its acceptance as such induces people to acquire more of the status good in question than they really need or can use, just to demonstrate how rich they are.
If you've read The Great Gatsby, you can consider a 50+ harem as like Jay Gatsby's library. The books were all real, so he's not a poser. However they were uncut, openly showing off their purpose is not to be read, but just for you, so you know they are there.
A line in the classic Chinese novel Romance of the Three Kingdoms illustrates the scenario well: After Zhang Fei causes Liu Bei's two wives to be captured, Liu Bei (let me emphasize that he is the hero of the story) tries to aleviate his guilt by saying that wives are like shirts but brothers are like limbs. A shirt can be replaced, a limb cannot. So yeah... that's how things worked at the time.