Referring to the Black Book of Edward IV - it's drawn from his own household accounts, so the limits to the retainers allowed are the limits Edward himself set. Remember that Edward had only managed to get, hold, then regain the throne by force-of-arms but he was very well aware that the nobles had their own retainers and that it was possible to lose the throne again. Warwick had a huge number of retainers, well into the hundreds. The limitations on the numbers of retainers were an attempt to control the issue of lords having private armies as large as they could afford and attacking each other if they disagreed with something.
As for colours - lords had their own livery. For example, the House of York, in the person of the Duke of York (father to Edward IV and Richard III) used murray (a sort of deep burgundy-red) and blue. There might also be a sigil and a coat-of-arms. Richard III's personal emblem was the White Boar, but his brother Edward's was the "Sunne in Splendor" - a sort of star-burst. The lord himself would use his colours and his sigil. His employed retainers - the men employed directly by himself to be his armsmen - would probably wear his colours - so for example, a heavy padded jacket made up of four sections of cloth in two of the lord's colours, with the diagonal-opposites being in matching colours. They would also likely wear his sigil (eg White Boar) sewn onto their breast. During battle, they would start grouped together under a banner displaying the colours and/or the sigil. However, over time, the lower lords didn't always have a standing army - it was expensive. So they would either hire professional arms-men when they needed them or they would gather (volunteers or strong-armed) peasants in from the villages they controlled.
These peasants would have an arming/padded jack (heavyly padded jacket) if they were lucky. They would be very unlikely to have the lord's colours.
So in a small skirmish between two small lords, you have your two sigil banners and other than that, very little way of telling who fought for whom.
In larger battles, you might have a mix of peasants in their own gear/mercs who might have put the sigil on for ease and would have decent gear/liveried retainers.
Where your higher lord needs back-up (perhaps to bring his own strength up to a level required by HIS higher lord), he will send to the lower lords to provide their men. So you have groups of peasants under their manorial lord's banner standing in larger groups headed by the liveried retainers of the upper lord under HIS banner with the upper lord's own peasants in the mix.
Most Middle Ages battles were bloody messes, in part because of the difficulty in determining friend and foe. Not helped by confusion over banners: at the Battle of Barnet (wars of the Roses) in poor weather, the Earl of Oxford's men (Lancastrian) attacked the men of Lord Hastings (York) and chased them off the field. In the time it took Oxford to get his men back under control, the battle-line veered around. As he returned, he unknowingly came up behind his own side, right behind the position held by John Montagu. Montagu's men mistook Oxford's banner of a "streaming star" for Edward IV's "Sunne in Splendor" and attacked Oxford. As Montagu had been on Edward's side at one point, Oxford's men assumed Montagu had turned sides again. They called "treason" and panic spread through the Lancastrian side. Edward attacked, Montagu was killed and the Lancastrians were utterly defeated, including the death of Montagu's older brother the famous Earl of Warwick. This defeat, with the deaths of the two brothers, led almost directly to the following defeat of the Lancastrian Prince Edward and the death of Henry VI, leaving the future Henry VII as the only Lancastrian with any chance of the throne. So the fact that English troops fought without any ready identifiers, and the fact that knowing your enemy very often relied on whether or not you recognised their badge, had a very large bearing on the rulership of England. Had Edward IV lost at Barnet, Henry VI would probably have been put back on the throne and Henry VII would perhaps never have ruled, as Henry VI had a son and heir.
