I am writing a fiction short story roughly based off medieval times in Europe. In it I have a group of men who live in a castle and fight for the castle's lord. (They mainly protect the surrounding villages from a group of bandits.) If they existed in modern day times I'd refer to them as soldiers. They are much like traditional knights, only they're a whole group, they've never been knighted and aren't members of nobility. What might they be called? I'm looking for a term, from sometime in the medieval time period (I don't care when), that the surrounding villagers would call them by. Are the terms vavasseur or sergeant appropriate? Is there another better term?
As Carlos Martin has noted, these soldiers are men at arms. They might be armed with swords, bows or crossbows, spears, or other pole-arms (eg pikes or halberds) depending on precise period and geographic origin. Generally they would be responsible for their own arms and armour, but a wealthier lord (or captain for a mercenary rather than feudal levy) might choose to upgrade that in order to field a more impressive and effective force.
Groups numbering in the range 10-20 would be commanded by a sergeant, and might be termed a peloton (the French root of our modern platoon), troupe, or escadre (French root of modern squad) without being too anachronistic. Larger groups, composed of multiple pelotons, would be termed a company and commanded by a captain assisted by a lieutenant. Within a single levy, either mercenary or feudal, there would be some consistency in peloton size, but there need not be complete consistency between companies of different origin, other than the general pattern just described. In companies of several pelotons the most senior sergeant would be the sergeant major.
If the company is of such a size to fight as two separate wings, the captain and lieutenant would command one each. If large enough to deserve three wings, then the captain would command the centre with the lieutenant and sergeant-major each commanding a wing to his left and right. There might or might not be a tactical reserve. A mercenary company might be as large as two or three hundred men in the later part of the period.
This answer about living arrangements for a castle's Constable notes that the castles built in Wales by Edward I generally were designed to support, and be adequately defended by, a garrison of slightly more than 30 men.
The History section of this answer about early modern Spanish officer and enlisted ranks details some of the etymology for the various uses of major (ie Sergeant-Major, (Battalion-)Major, and Major-General) in modern ranks - enlisted, officer, and general - as well as of modern unit organization in the early modern era.
They are Men at arms, professional soldiers.
The lance fourni of a knight, the unit of soldiers a knight brought to battles or protected the fief, was formed by professional horse-riders, archers and/or spearmen. They were also called men at arms, so the sergeant/marischal in charge of your unit could be a knight or a common man (a yeoman if you heavily borrow from English history).
You actually have a fair amount of freedom here given the particular flavor you want to give your story. As others have said, the most common would be Men-at-arms or armsmen with a captain or a Master-at-arms (Master-at-arms was generally more in charge of training, but frequently commanded in battle as well) as the leader. If you want an English/Norse flavor, you could call them Huscarls (or Housecarls). The term "retainer" is also very generic but could work.
I would recommend picking a time-period and culture as a model and using that as a starting point. Otherwise, men-at-arms is likely the best bet.