This would have varied by region and culture.
Dueling has its roots in trial by combat. Presumably, if you refused such a duel, you were conceding your wrongness in the case, and the penalty would be whatever the judge decreed based on the crime and whether you were the accuser or accused.
The church frowned on duels, and strong centralized authority made it easier to pass laws against duels. Dramas of the period record excuses that gentlemen were permitted to make; appeal to the law was one of them, unsuitability of the occasion might be another. It does not seem that refusing carried any penalty; indeed, accepting might have been considered to stain one's character more than declining!
Early Modern America
By the 17th century, duels were all about defending perceived slights to your honor. If one refused a challenge to a duel, the challenger would post about it in the newspapers. This sounds fairly mild in our day and age, but such an insult would have ruined a man's social standing and - for a politician - career prospects.
Interestingly, one could refuse if the duiel's cause was frivolous, there was a great difference in age between combatants, or if one did not consider the challenger to be a gentleman which was a mortal insult. Given that duels in America were considered to be the better alternative to feuds between families, one imagines that an insult like this would lead to such a feud. This would also be an insult to the accuser's seconds, since it would imply that they were acting on behalf of a man of low social standing.
Interestingly, a public refusal to duel by war veteran Joe Wise may have led to the decline of dueling's popularity in the USA. Wise was known to be honorable, so his refusal did not tarnish his reputation, and the threat of dishonor for refusing was shown to be toothless.