Many drawings of peasants in the Middle Ages show the men with an oversized collar or short cape with a distinctive castellated margin.
Did this really exist, and if so, was the shape dictated by a some detail of its function?
If it did not exist or was uncommon, when did people start drawing it like it was standard attire?
The basic form of this garment is like the gugel, a hood that protects the head and also covers the shoulders. The precursors for these are Roman paenula or Alpine Kotze made from various types of wool.
It is not exclusive to medieval times, although this basic style became quite fashionable for a while during the later middle ages, first in the lower classes, then in the higher up strata of society.
A recent offering for this type of head and shoulder protection for hunters on sale now:
One of the oldest finds of these is the 'Orkney hood', an iron age clothing with already this type of a seemingly 'ornamental' hem.
Since it is of functionally the same type of a poncho, its advantages are keeping the shoulders protected and warm while allowing for good movement of the arms.
Further, the cloth used is quite water repellent and thus protecting against rain.
This is one minor aspect for the "castellation": when it rains this fringe channels a bit of the water and also allows for faster drying. The effect is not really big and certainly even more diminished for the squarish style depicted in the question.
are often trimmed with a fringe – originally a functional detail, to allow the garment to shed rain, and to dry faster when wet because the fringe acted as a series of wicks to disperse the water – or quills
(WP: Buckskins)
In this clichéd version the effect cannot be that great. If worn in that style in reality, this is an ornamental remnant with a functional past. Here, it is more likely a stereotypical convention for the drawings. Notice that in the first picture only one of three wears this hem. In other words: in this picture 'only Robin wears a hood', and here with some indentations, or dagging or in German lands the often quite complicated zaddeln.
That such a hem might have been used in real life occasionally isn't precluded by the prior statement. Fashion isn't always really concerned with practicality.
Today the most close resemblance for this garment, often without the hood, is often called pélerine (compare the pictures on different language Wikipedia pages), and in liturgical items one sees the resemblance to a chasuble and mozzetta. The German term Gugel in Normannic would be Cagoule, although the modern item under this name looks quite a bit different now and covers the arms with sleeves.
In terms of archaeological finds, the Swedish Bocksten Man, dated 1350–70, presents both possible variants at the same time: a gugel (hood) and the hoodless cloak, in that case worn as a quite semi-circular long mantle, albeit both with a simple undecorated hem, no fringe.
The medieval 'book on the hunt' (Livre de la chasse) is an illuminated manuscript from between 1387 and 1389. In some versions of it we find all these types pictured: with hood and without, with decorated hem or without.
Further development and names:
In the very late medieval and early renaissance the gugel/hood went mainly into three distinct developments. The face covering was now often worn on top of the head, making the rest for a floppy brimmed hat, ending in the 'chaperon'. The head covering itself was separated from the shoulder coverings and receded into a kind of Phrygian cap. The lower part of the gugel was since the 14th century known as the goller (Latin: collare, ) or partlet.
https://en.wikipedia.org/wiki/Paenula
https://de.wikipedia.org/wiki/Kotze_(Kleidung)
https://en.wikipedia.org/wiki/Orkney_Hood
https://fr.wikipedia.org/wiki/P%C3%A8lerine
https://www.museumhalland.se/bockstensmannen/
https://en.wikipedia.org/wiki/Bocksten_Man
https://en.wikipedia.org/wiki/Livre_de_chasse
https://en.wikipedia.org/wiki/Partlet
Chaperons and Liripipes
— Herbert Norris: "Medieval Costume and Fashion", 1999. Shows a relative timeline for dominant styles of the hood.
— Sarah Thursfield: "The Medieval Tailor's Assistant.
Making Common Garments 1200 -1500", 2001. Shows the most basic cuts and construction of these hoods.
— Mary G Houston: "Medieval Costume and Fashion in England and France. The 13th, 14th and 15th Centuries", 1939. (Black & white drawing from Livre de Chasse, p170.)
The hood and the 'distinctive castellated margin' mentioned by the OP are really two separate features, so this answer will focus on the decorative hem, which was called dagging.
A very informative and well sourced article on Fashion History Timeline. The main definition being:
Dagging (also “daggings”, “dagges”; adjective: “dagged”) is a decorative element added to the edges of garments and objects. It is created by cutting or slashing the fabric into different shapes, some simple, some elaborate.
The same article discussed how this was achieved:
Dagging is created by cutting or slashing the edges of fabric with shears or scissors. It also can be created by sewing pieces of fabric to the edges of fabric
Concerning when this feature was popularly used, the same article
places dagging in the Middle Ages and Renaissance, the broadest chronological range being from the 13th to the 16th century. This treatment, found everywhere on clothing in England and Western Europe
This indicates the fancy edge visible on the OP's modern images is a realistic portrayal of a medieval fashion feature (though they can be much more elaborate).
The above-linked article has much more information, including some early illustrations of dagged clothing as well as numerous sources listed if you wish for more information concerning this aspect of your question.
