Short Answer
An important reason was to destroy those Muslims who opposed the Mongols. This meant that their mosques and Islamic texts were also targeted, especially those of the Isma‘ilis, a Shi‘ite sect which had openly defied the Mongols and which had probably been involved in an attempt on the life Mongke Khan.
It should be noted, though, that the destruction of Baghdad, an 'Arab-Muslim imperial capital' was almost totally indiscriminate and can thus be seen as a message to other cities considering resistance.
Details
Generally, the Mongols did not try to impose their beliefs on those they conquered:
The Mongol rulers presented the clergy of the recognized
religions — Christianity, Buddhism, Taoism (Daoism), and Islam — a deal:
In return for their prayers to God for the Mongol rulers, the rulers
would grant the clergy equal status and exemption from military
service and taxes.
However, there were exceptions and those who resisted could expect harsh treatment.
Certain religious groups were also seen as anti-Mongol or subversive
and hence eliminated. The Isma‘ilis, or “Assassins,” a sect of Shi‘ite
Muslims, had assassinated Mongol officials
The Assassins attack in 1251 was probably aimed at Mongke Khan. Edwin Black in Banking on Baghdad (2004), citing three other sources, says
The Assassins, according to accounts, sent 400 of their best to kill
the ruling khan in the name of Islam. The khan’s many bodyguards and
spies foiled the conspiracy and blamed the entire Islamic
establishment. In 1251, Grand Khan Mongke made the decision. Baghdad
was to be demolished.
The assignment was given to Hulagu Khan, the grandson of Genghis, who
launched his forces two years later, after immense preparation.
Alamut, an Assassin stronghold, was taken by Mongke Khan's younger brother Hulagu Khan in 1256 and its library destroyed. In particular, "every manuscript which related to Ismailian" was destroyed. Thousands of Isma‘ilis were also massacred.
When the Mongols advanced on the Abbasid Caliphate in January 1258, the Caliph not only refused to submit but was openly defiant. As the capital, Baghdad, was ill-equipped to defend itself, this proved to be extremely foolish and all resistance was overcome in February 1258. The destruction wrought was almost complete and included the House of Wisdom. According to witnesses, so many texts were destroyed that the river ran black with ink. Unknown hundreds of thousands were slaughtered (so the river is also described as having run red with blood, but these 'colourful' statements about the river are probably exaggerations); spared, though, were the Christians, one likely reason being that Hulagu's mother (Sorghaghtani Beki) and wife (Doquz Khatun) were both Christians.
Despite the massacres at Baghdad and elsewhere, evidence suggests that there is no reason to suppose that Hulagu had a particular hatred for Muslims as this site claims. His army included Muslim soldiers, he left compliant Muslim communities largely untouched and he allowed the Muslim scholar Nasir al-Din al-Tusi to seek out other scholars who had fled the Mongol onslaughts, and authorized paying them salaries to continue their work in Maraghah in Iran.
Other source:
Abdulhadi Hairi, 'Nasir al-Din Tusi and the Mongol Invasion of Baghdad' (MA thesis, 1968) Note: clicking on the link downloads the document directly)
