I've been told by history-loving friends and read in books about in early American history, when older men, sometimes in sixties or seventies, would take younger, often teenage, wives, without popular outcry or concern. Now, it seems to be looked down upon. See this Psychology Today article and the linked study for examples. When did this shift happen?
One side of the problem is covered in the answers and comments already given.
The other side - I would argue the biggest side - is that women not only live longer, but that they now usually work and earn money. Marriage is no longer the only respectable profession a woman can take. So there is much less pressure on them to a) marry early, and b) marry at all. A century ago, an unmarried 30 year old woman was a failure, just like an unemployed man of the same age. This no longer holds.
Besides that, the increase of life expectancy is due a decrease in infant mortality, rather than in teenage or young adult mortality. This means that not only the pressure for early marriage is lesser, but also the pressure for early motherhood. A century and a half ago, you could not expect having two or three children reaching adult age without giving birth to half a dozen or more. And the reproductive age span of women is relatively short; a woman aged of 30 has already gone over half of it, and consequently can be expected to bear only half the children a woman at the age of 15 could.
I cannot speak for Europe, but a quick search shows that the standardized age of consent laws all across the US were established in the early 1920's to what we know today, generally 16-18 depending on state.
The ages of consent were raised across the U.S. during the late 19th century and the early 20th century. By 1920 ages of consent generally rose to 16-18 and small adjustments to these laws occurred after 1920. As of 2015 the final state to raise its age of general consent was Hawaii, which changed it from 14 to 16 in 2001.
One could argue largely, especially in the US, that the laws of the land reflect the will of the people. So age of consent laws began to appear at the turn of the 19th century around 1880 which would reflect the evolving culture of the publics view on underage martial status. Bearing in mind that US law tends to lag behind the will of the general public, I would presume the mid 1800's is when things started to turn.
The reason it is looked down upon is because of feminist influence on culture. This goes hand in hand with women being employed themselves, there were massive shifts in the 1960s onward. However, whether that is good or bad, what should be considered right or wrong is not decided by what is currently the norm or what is looked down on or not.
Historically, women married older men without problems. And when you look at the kinds of romance literature that women read, it is clear that even post-modern women have a secret interest in men older then they. There is an immense social pressure not to admit it thought. And there is co-incidentally also a decline in women's happiness.
Lots of statistics which could be considered here, and nobody has done it comprehensively. If you look at the long-term relationship age difference you notice a very peculiar graph almost unseen anywhere else in natural statistics: a sharp drop off, a singularity, right where the man is younger than the woman. The same shape you see, by the way, when looking at income differences between husband and wife. This is very telling. It means that no matter what the proclaimed social "norm" may be, clearly couples are more agreeable to the woman being with an older man than the man with an older woman, and that is a sharp difference, not even a gradual one!
This makes a lot of sense when you look at the value of a human being. How do you measure that value? You measure it by looking how much other people would give in resources for the person, how much money and time and interest they spend. In general, throughout history and to this day, men are expendable, this is why men make 95% of workplace accidents, military draft is for men only, even civilian was casualties are multiple times higher for men than women, and when the Titanic sinks it's women and children first. If half of the population must die, let it be men, for 1 man alone can father a complete replacement with 1000s of women, but each woman can only have 4-8 offspring, some have more but some have none. So this is why the value of men is about 1:10 of that of women. Naturally. By the kind of rational economy that is observable in nature.
Now if you look at the natural value of a man vs. a woman over their respective life-spans, you notice that a woman has her best years in the short span between 16 and 26. Men, however, enter their best years after 30. This is a huge difference in the value of an individual human being. The man will have to build his career in the time until 30, and before that, a successful man is extremely busy doing that, and also still finding his self-esteem. The 21 year old freshman is hardly considered a great catch. For this reason also, we say that young men appear more immature than young women.
18 year old girls are hardly interested much in 18 year old boys, but go for those who already entered a different, higher, stage in life. A woman on the other hand glows at her 16th birthday and would already catch handsome marriage proposals if she was "on the market". Generally men value a young woman for her biological characteristics plus her character, but hardly care much about her professional viability. Whereas men, in contrast, are not viable to women if they do not demonstrate a successful work ethics.
To make a good satisfying match, it is best if both partners are near their peak of their sexual marketplace value and if their value curves do not diverge too much over time. This means a woman in her early 20s and a man in his mid 30s make the best match. Initially you can have a good match when the current sexual value of both partners is about equal, but for longevity of the relationship that bet can be dangerous. Women who married a man her age find themselves traded in for newer models after reaching her 40s when the man reaches a peak in his sex value that he never before thought was possible. Women clearly have the upper hand in their youth, but men have the upper hand at middle age (ignoring for a moment high divorce rates initiated by women near the middle age also).
When the husband is much older another divergence is common. As time goes on, and women enter their 30s, they have an increased libido, while men in their later 40s generally experience a decrease in libido and sexual performance. But 30/40 is still very compatible. Older men tend to lose sexual attractiveness to women if they fail to take care of themselves quite similar to older women losing attractiveness to men for the same reason.
However, the drop of attractiveness for women is often sharper as for men, even if both take fairly good care of their health. What is gray hair and facial expression lines in a man for a woman is by far not the same as grey hair and wrinkles on a woman to a man. One may lament this fact, but it makes biological sense, as the value of a man lies in his performance of providing resources to the upbringing of offspring, whereas the value of a female is her capacity to bear offspring. Wrinkles and grey hair are directly linked physiologically with the hormonal change leading into menopause.
This means that as long as the man continues to perform, his greater age will be less of a problem to his wife than a younger husband would see his older aging wife favorably. These are insurmountable facts.
This is why this age difference can be observed in any cultures that are natural, modern, but not yet post-modern. The post modern cultures have some innovations that greatly distort this natural relationship, and more common today in post-modern cultures does not mean better in the sense of human progress. The best, again biological, indicator of this is the decline in population that belongs to the post-modern culture. As the native population in countries currently still defined by post-modern culture fails to reach their replacement fertility rates, immigration by people with pre-modern and only modern, but not post-modern cultures is sought to make up for the population decline and its economical consequences. This deal is fraught with problems, but all politics aside, it is clear that biologically the post-modern culture with its great shift in sex-relationships is inferior to pre-post-modern cultures and is being replaced.
Some theories are saying that decline in population growth is a good thing, others believe it is not. Some believe that pre-post-modern immigrants are prone to assimilate to post-modern culture of their host country, and some statistics indicate a certain fertility decline, but be that as it may, a fertility decline is an objective measure of the success of relationship norms. Longevity of marital relationships is another. And out-of-wedlock birth rate / single motherhood yet another. All are related, and by all indicators the post-modern norms stand by these objective measures as unfavorable. Even if you look at individual happiness, the one characteristic that the more individuum-focused era of post-modernism should perform better, is shown to probably be no better if not worse.
So, we can summarize, the answer to "when" this shift happened is in the 1960s when societal norms changed. But that these norms are not rated favorably by demographic outcomes, and hence, the greater age difference is biologically favored as well as explainable by models of sexual economy.