I can go into further details if requested, but "TL;DR" answer is:
After Luther agitated that "Jews didn't convert to Christianity because Catholics treated them badly, and would convert if you treat them better", Jews still didn't show any great willingness to convert.
Here's one supporting quote (context was Luther's refusal to intercede on Jews's behalf to stop their expulsion from Saxony etc... by Elector of Saxony John Frederick, Luther's prince):
": "... I would willingly do my best for your people but I will not contribute to your [Jewish] obstinacy by my own kind actions. You must find another intermediary with my good lord." (source: Luther’s 1536 letter to Rabbi Josel as cited by Gordon Rupp, Martin Luther and the Jews (London: The Council of Christians and Jews, 1972), 14.)
Just to clarify, since your question leads me to believe you didn't quite understand his views: Luther didn't believe in freedom of conscience in the modern libertarian sense. His opposition to the idea of involuntary conversion of the Jews was based on impracticality, NOT on a notion that trying to convert Jews was something evil because Jews have the right to be Jews.
A further influence in his thinking about Jews is listed in Wiki, so I'll just quote verbatim:
It is believed that Luther was influenced by Anton Margaritha's book Der gantze Jüdisch Glaub (The Whole Jewish Belief). Margaritha, a convert to Christianity who had become a Lutheran, published his antisemitic book in 1530 which was read by Luther in 1539. Margaritha's book was decisively discredited by Josel of Rosheim in a public debate in 1530 before Charles V and his court, resulting in Margaritha's expulsion from the Empire.