Apparently, Yoshioka Matashichiro was the 12 year old son of Seijūrō:
This second defeat was a complete loss of face for the Yoshioka family and their school. Looking for revenge, the school challenged Musashi again. This time, 12 year old Yoshioka Matashichiro, the oldest son of Seijuro was chosen as the leader of the challenge. However, because of his age, he was to be a figurehead only and others from the school would act as his proxy. The place chosen for the duel was near the Ichijoji Temple at the base of the Sagarimatsu tree in northeastern Kyoto. The Yoshioka Family assembled a force of archers, musketeers and swordsmen in an all-out, go-for-broke bid to kill Musashi in the hopes of regaining their prestige. It is thought as many as 72 from the Yoshioka school assembled for the duel against Musashi.This time however, Musashi arrived early. It is said he killed Matashichiro, and also killed as many as one half of the opponents before escaping.
After this, the Yoshioka family continued with their fencing school, although they had lost all of their prestige and the school was never the same again as a result. In 1614, they supported Hideyoshi Hideyori at the Winter Campaign of the Seige of Osaka Castle. After this, the Yoshioka family closed the fencing school and started a business dying silk for kimonos. Today, the Yoshioka family is one of the most sought after and prestigious silk dyers in Kyoto. Musashi continued with his duels after this. In 1614, ironically he also fought with Toyotomi Hideyori at both the Winter and Summer Campaigns of the Osaka Castle Seige. There is much more of the story to tell regarding Musashi in his later life, but we will save that for a later time.
Source: Musashi Miyamoto in Kyoto - Samurai Tours
I realize the source of the quote is a bit... random, but most of its details seem to check out.
Also, the Wikipedia article on Yoshioka-ryū mentions that Vagabond isn't exactly historically accurate in regards to Musashi's feud with the Yoshioka family:
The Takehiko Inoue manga Vagabond has a variation on the feud between Musashi and the Yoshioka. While the Yoshioka apparently never connected Musashi to Munisai (thus the family feud being a non-issue), Musashi had sought them out as a way to leapfrog his way to fame, slaying a few disciples at the dōjō before Seijūrō arrived and exposed him as lacking. Nevertheless he continued to challenge and dueled Denshichirō, but their first encounter ended prematurely on account of the Yoshioka dōjō being inadvertently set aflame, so Denshichirō told Musashi to grow stronger and then face him a year later.