The hand gesture showing the middle fingers together has been variously described as a ‘W’ or ‘pseudo-zygodactylous gesture’ or the ‘El Greco gesture’.
It seems to have originated in late renaissance or Mannerism period from 1520 to the late 17th century, and was subsequently adopted by many artists in later periods. El Greco was not the first to use this hand gesture, but two of his paintings are among the best known examples: The Disrobing of Christ (1579) and The Nobleman with his Hand on his Chest (c.1580).
The Disrobing of Christ (1579)
It was widely used by many artists in portraits of both male and female subjects, secular and non-secular, in the 16th, 17th and 18th centuries, most commonly (but not always) with the hand (either left or right) on the chest. Its meaning is uncertain as
there are no art manuals that describe this specific gesture or what
it signified,
Consequently,
its meaning and latent symbolism must be deduced from secondary
sources and then can only be theoretically supposed.
Several theories have indeed been proposed. There may also have been practical and / or stylistic reasons for using it (e.g. showing the elegance of the fingers). Among later artists who used this gesture was Anthony van Dyke, who influenced Peter Lely (he also inherited many of van Dyke’s patrons and "was the most technically proficient painter in England after the death of Van Dyck" in 1641). Lely himself used the gesture in a number of paintings (see, for example, Portrait of a gentleman in black and one of the Windsor Beauties, Elizabeth Hamilton).
Among the theories proposed, a couple are plausible explanations for the gesture of the right hand as used in the painting of James, Duke of York, (later James II of England / James VII of Scotland): an indication of sincerità (sincerity, honesty or frankness) or a vow or an oath, most likely to God or possibly to his brother and his king, Charles II. We should also consider the possibility that the gesture of the left hand (with the finger pointing down) may be directly related. Pointing down is commonly interpreted as representing the subject’s
...Higher spirit reaching down for the soul. The implication being
that the person is not yet on the path and is being 'searched for', as
they are in a state of innocence [or ignorance if you prefer].
Thus, if the gestures of the hands are directly related, the gesture of the right hand (sincerity) may be emphasizing or reinforcing that of the left. This interpretation would seem more likely if the portrait was done around the time James converted to Catholicism (1668 or 1669). Unfortunately, the date of Lely’s painting is uncertain; the Art UK website, a cultural education charity which “represents a collaboration between over 3,200 British institutions”, gives a date range of 1650–1675. We can narrow this down a bit to 1661 - 75 as (for practical reasons) it is highly unlikely that the portrait was done until after the restoration, and Lely was appointed the Principal Painter in Ordinary by Charles II in 1661. Also, Lely was popular with James and his first wife, Anne Hyde; they commissioned many pictures from the artist, but only from the early 1660s onwards.
Ultimately, though, we are unlikely to ever know for certain what Lely or his subject intended. Lely produced hundreds of paintings in his lifetime, many of which were actually completed by artists he employed in his workshop. Lely often only did the initial sketch and painted the face, leaving other details to his 'team' (which partly accounts for the sometimes variable quality of his work). In the case of such an important patron as James, Lely quite possibly did all the work himself but I've found no evidence that he explained the portrait (and I doubt there is any to be found as we can't even date this painting).
Other sources
M. Kirby Talley, 'Extracts from the Executors Account-Book of Sir Peter Lely, 1679-1691: An Account of the Contents of Sir Peter's Studio' (The Burlington Magazine Vol. 120, No. 908, Nov., 1978)
Museo Nacional del Prado
Sir Peter Lely (British, 1618–1680)
Norbert Schneider, 'The Portrait' (2002)
Sir Peter Lely (Soest, Westphalia 1618-1680 London)