Personally, I don't think anything ever went particularly "wrong" with India. They only fell behind the civilizations of Western Europe, not the rest of the world. So the proper question to ask here is what suddenly went right with heretofore backward Europe.
To my mind the answer to this question is clear: The printing press. Nearly overnight Europeans had access to several orders of magnitude more knowledge than they had before (and than anyone still relying on slow, error prone, and expensive hand-copying could possibly have). The discourse this allowed would have had a self-multiplying effect. The difference would be like somebody from the 1800's (or even the 1970's) trying to compete for knowledge with today's internet society. There's just no hope for them, no matter how smart they may be.