My question is whether tobacco and marijuana were ever competing products in Europe. It occurred to me that marijuana is an Old World plant, whereas tobacco is a New World plant, and therefore had to be introduced after 1492—by which time I guess marijuana would already have been available in Europe. So it is curious that of two broadly similar products, tobacco was the one that caught on. (I've never tried marijuana, and I have never had a smoking habit, so I'm not sure whether the products are directly comparable, but the general idea of smoking a dried leaf seems similar.)
No. Although it is true that hemp was grown and harvested in Europe,
...is grown specifically for the industrial uses of its derived products. It is one of the fastest growing plants and was one of the first plants to be spun into usable fiber 10,000 years ago.
its(hemp) use as a drug would have been minimal:
Although cannabis as a drug and industrial hemp are both members of the species Cannabis sativa and contain the psychoactive component tetrahydrocannabinol (THC), they are distinct strains with unique biochemical compositions and uses. Hemp has lower concentrations of THC and higher concentrations of cannabidiol (CBD), which decreases or eliminates its psychoactive effects.
Cannibas was known and used predominately in Eastern and Middle Eastern cultures(from History of smoking):
Cannabis was common in the Middle East before the arrival of tobacco, and is known to have existed in at least 5000 BC. Early consumption of cannabis was a common social activity involving the type of water pipe called a hookah.
In general, the public activity of smoking was introduced to Europe on an industrial scale, from the colonies. The level of production by the revolution:
After war broke out in 1775, the General Assembly voted to cease sending tobacco to Britain. Many planters switched to growing food crops, especially wheat, to support troops. In the first year of the war, tobacco production dropped from 55 million pounds to 14.5 million
So, when tobacco became available, it would have had no competition from the limited use of cannabis in comparison.
--Update: Concerning the history of Tobacco Use.
I found an excellent discussion of The Antiquities of Tobacco from Gentleman's Magazine and Historical Review, Volume 7, published in 1859! The discussion lists first occurrences of tobacco use in Spain, Europe and England, and speaks of people being so surprised by smoke coming from people that
... they never seeing the like wondered at it and seeing the vapour come out of Tarlton's nose cryed out fire fire and threw a cup of wine in Tarlton's face.
showing that smoking was indeed uncommon before tobacco's arrival.
During the sixteenth and seventeenth centuries, in certain parts of Europe (most notably the Netherlands) hemp was widely grown as an industrial crop, but tobacco was a relatively expensive import. So the amount of tobacco smoked was probably small in comparison to cannabis (hemp) grown. Smoking in general was disreputable, possibly even more so in the case of tobacco since it was more exotic and poorly understood.
So why did tobacco come to dominate the market? Tobacco, like a lot of things, would have become much more cheap and abundant over the course of the nineteenth century due to colonialism, the steamship, etc. Then prohibition in the twentieth century made cannabis expensive and illegal.