First, while Hajj is one of the Five Pillars and is recommended to do it once a year, the actual requirement is once in the lifetime only. If you take into account the spread of Muslim rule in Xth and XIth centuries, only certain percentage of the Muslims would actually go on a Hajj regularly, let alone yearly. There are indeed conditions to be met before one could go: wealthy enough so that family can manage without person's presence, for example (see al-Baqarah 2:158,189,196-199; al-Hajj 22:27-36).
On the other hand, even if the medieval period (specified by the question) was that of the constant warfare between (broadly understood) Islam and Christendom, the Mediterranean was generally safe for Muslim traffic (Ibn Khaldun – "The Christians cannot float a plank on the inland sea that is the Mediterranean"). So it actually could be rather easier to go for a Pilgrimage than I would imagine. But then again, even by sea round-trip from Al-Andalus to Mecca would take literally years (see https://worldbuilding.stackexchange.com/a/98368/8834 - Vashu's comment about trip Rome - Egypt being on average 1,5 times a year).
Secondly, even if Muslim sphere of influence was leading in the world scholarly (hesitate to use science here, but to some extent I would use those interchangeably in context of the period), there is little evidence that points to the majority of "Islamic scholars" to be actually Muslim. To be fair, there is not much more that they were NOT, too.
Taking all that into account reality would be that majority of the pilgrims would be mostly of certain statute and heritage, which also implies certain limits on types of ideas to be spread.
Next, the decline (or, according to some sources, almost total disappearance) of scientific progress in Muslim world by XIVth century, universal ban on non-Muslims entering area around Mecca and other... developments... definitely had impact (see Toby Huff, "The Rise of Early Modern Science").
I would say that famous saying of caliph Umar on the Library of Alexandria in 642: "They will either contradict the Koran, in which case they are heresy, or they will agree with it, so they are superfluous" is well in line for usual Islamic conquest style. He also said something similar in 651 after conquest of Ctesiphon. Book burning is not uncommon in every society of that era (notable mention would be burning of Library of Constantinople in 1199 by Crusaders), but Muslim history, including their own religious/historical sources, is replete with burning of all kinds of books.
This is very interesting question. However, I would say that the scenario described in question by OP is not only implausible, but in fact would have the exact opposite influence on exchange and spread of ideas, as attested by history. I would even go out on a limb quite a bit further: the origin of the idea of exchange of ideas on occasion of Hajj would be very, very recent. As in less than a 50 years old? As such it would not be supported by as much of historical evidence as one would expect.
If we take into account some recent development in the studies of origins and early spread of Islam the theory has merit. A lot, actually. But not in the way one would think. Mecca may be the destination of the Hajj only since end of Xth century. And the first ever mention of Quran (let alone the actual book), and Hadith and Sunnah in any written and compiled form does not appear before 822 AD or so.
Mecca is in the Quran no more than twice (and even then two different names are used), but description is wrong. Location of the city of Mecca is isolated from all major trade routes of the time, terrain is quite inhospitable, so pilgrimage would be a serious challenge back then, and all that, of course, is quite different from what is in the Quran.
Now, if the Hajj was to be made to Petra/Jerusalem (as some research suggests), then I would say that OP's premise has a lot of merit. Why Petra and/or Jerusalem? Because Mecca is not mentioned in any document before late IXth century, and Petra is gone by the end of VIIIth, and Jerusalem for a while held the status of holy city for Islam. However, it was Petra, not Mecca, that was the destination for pagan pilgrimages of worshippers of, for example, Allat. From this bit the pagan ban on entering Mecca comes from (see Dan Gibson, "Quranic Geography" and Patricia Crone, Michael Cook "Hagarism: The Making of the Islamic World", Tom Holland "In the Shadow of the Sword").
But that would mean a lot of different things, not always in line with traditional views...