The main library in Baghdad was Bayt al-Hikma, the House of Wisdom.
A very good article about its content and activities is here.
There were different phases. In the beginning they just interpreted Quran. Then they started translating foreign works. Later they started doing their own research in chemistry, algebra, medicine, and other disciplines.
From Britannica I read:
In that same capital city was founded the great library Bayt al-Ḥikmah
(“House of Wisdom”), which, until the sack of the city by the Mongols
in 1258, served as a huge repository for the series of works from the
Hellenistic tradition that were translated into Arabic. Al-Andalus
became to the rest of Europe a model of a society in which the
religions and cultures of Islam, Christianity, and Judaism could work
together and create a system of scholarship and teaching that could
transmit the heritage of older civilizations and the rich cultural
admixture of Andalusian society. Western science, mathematics,
philosophy, music, and literature were all beneficiaries of this
fascinating era, of whose final stages the fabulous Alhambra palace
complex in Granada, Spain, remains the most visible token.
There also were mathematical texts:
The subsequent acquisition of Greek material was greatly advanced
when the caliph al-Maʾmūn constructed a translation and research
centre, the House of Wisdom, in Baghdad during his reign (813–833).
Most of the translations were done from Greek and Syriac by Christian
scholars, but the impetus and support for this activity came from
Muslim patrons. These included not only the caliph but also wealthy
individuals such as the three brothers known as the Banū Mūsā, whose
treatises on geometry and mechanics formed an important part of the
works studied in the Islamic world.
Of Euclid’s works the Elements, the Data, the Optics, the Phaenomena,
and On Divisions were translated. Of Archimedes’ works only two—Sphere
and Cylinder and Measurement of the Circle—are known to have been
translated, but these were sufficient to stimulate independent
researches from the 9th to the 15th century. On the other hand,
virtually all of Apollonius’s works were translated, and of Diophantus
and Menelaus one book each, the Arithmetica and the Sphaerica,
respectively, were translated into Arabic. Finally, the translation of
Ptolemy’s Almagest furnished important astronomical material.
Of the minor writings, Diocles’ treatise on mirrors, Theodosius’s
Spherics, Pappus’s work on mechanics, Ptolemy’s Planisphaerium, and
Hypsicles’ treatises on regular polyhedra (the so-called Books XIV and
XV of Euclid’s Elements) were among those translated.