Basically, they were perceived as representing women's work. This reason is clearly stated clear the main surviving Third Crusade narrative in Latin, Itinerarium Peregrinorum et Gesta Regis Ricardi.
The enthusiasm for the new pilgrimage was such that already it was not
a question of who had received the cross but of who had not yet done
so. A great many men sent each other wool and distaff, implying that
if they exempted themselves from this expedition they would only be
fit for women's work.
Quoted in Helen Nicholson, 'Women on the Third Crusade'. In Journal of Medieval History, Volume 23, 1997 - Issue 4
This view of women is also well illustrated in the British Library article Women in medieval society
Women often participated in vital cottage industries, such as brewing,
baking and manufacturing textiles. The most common symbol of the
peasant woman was the distaff – a tool used for spinning flax and
wool. Eve is often shown with a distaff, illustrating her duty to
perform manual labour after the fall from Paradise. An image often
seen in medieval art is a woman waving her distaff at a fox with a
goose in its jaws; sometimes, in satirical images, women are even
shown attacking their husbands with a distaff or some other domestic
implement.
Also, there's this from Working Women in the Middle Ages:
Women were heavily involved in the textile industry. For example,
"spinning remained almost wholly in the hands of women, together with
many of the finishing processes" (Gies & Gies, 1978, p.168).