There is almost no direct historical evidence that openly-practicing Muslims were LIVING in the British Isles in the decades and centuries after the Norman Invasion. But I guess I'll start this post by highlighting the one prominent fringe hypothesis that would say otherwise (note that I mean hypothesis in a loose scientific sense here, as in a well-researched conclusion that has not gained widespread acceptance). The geneticist and demographer Donald Yates in The Early Jews & Muslims of England & Wales claims that he has discovered distinct pockets of Muslim patrilineage among UK citizens in Wales and Lincoln. According to Yates, these individuals possess the "right" North African genetic markers and surnames to surmise that they are descended from 9th-11th century Moorish merchants. However; Yates is the only resource out there making this claim and it is such a recent one (2014) that there hasn't been a lot of outside investigation by historians that would confirm this conjecture.
On the other hand, the historical record is full of examples of English-Islamic cultural exchanges going back as far as the 8th century. Note that we can't necessarily assume that these cultural exchanges were a product of Muslim traders coming to Britain (many Islamic ideas about science, trade, and jurisprudence came to Britain via contact with the Normans), but the influence was nevertheless significant. Examples include when Offa, the 8th century King of Mercia, ordered the minting of gold coins designed to resemble the dinars of the Abassid Caliphate. The reason for doing so was to apparently facilitate Mercian traders venturing to Muslim Spain and southern France. How about another example: Islamic geographers from the 9th century included Britain on their maps. Also, starting around the time of the Third Crusade, if not earlier, the Latin-translated works of highly-regarded Muslim scholars were also making their way to England and being studied by intellectuals of the period (see writings of Roger Bacon and Chaucer).
As far as hard population numbers and primary source documentation of a permanent Muslim presence in England, you won't get that until the Elizabethan period I'm afraid. For various reasons, these folks also tended to be native English who converted (save the Ottoman diplomatic delegation to the Court of Elizabeth), so it's doubtful they "stood out on the street". Good luck in your search.