In age-of-sail fleet actions, the primary use of frigates (and smaller vessels) was to relay messages (usually in the form of flag signals) between the flagships and the rest of the fleet. They usually set themselves some distance from the main 'line' of battle where they could see and be seen by the ships of the line.
A secondary purpose was to act as tugs and rescue vessels for the ships of the line when they were dismasted or even sinking during or after the battle. In the case of a sinking or burning vessel, it was not unknown for frigates to act to save the sailors regardless of whose side they were on (as they were then considered non-combatants). As well as saving their own ships, it was also possible for them to take possession of enemy prizes when their ships of the line were unable to do so.
Given these roles, it's not surprising that the unwritten rule of not firing on the smaller ships arose. It was a great benefit for both sides to have ongoing signalling through the battle and, likewise, useful for both sides to have a source of rescue available if needed.
It should be noted that throughout the Napoleonic Wars (which edged towards total warfare), as frigates had grown considerably larger and more powerful, observance of this rule was starting to wane considerably.
As frigates came to be seen as players rather than onlookers in a fleet engagement, the old conventions surrounding their immunity also broke down; indeed by the 1790s it was already a custom more honoured in the breach than in the observance.
Source: Frigates of the Napoleonic Wars, R. Gardner (Chatham, 2006), pg 159-162.