The total land area of Japan is around 146,000 square miles. 20% of that works out around 29,000 square miles or 18.6 million acres.
The population of Tokugawa japan was around 30 million people. 80% of that is 24 million people. This gives each farmer roughly 3/4s of an acre.
The basic unit of land in Japan was the cho, which was roughly 2.5 acres. This produced roughly 10 koku. A "koku" was the amount of rice needed to feed one person for one year. Simple math gives 4 koku an acre, and therefore 3 koku for 3/4s of an acre. In other words, in theory at least, each person has enough land to grow rice for three.
Thus, if 80% of the population is working 20% of the land in Tokugawa Japan, then they are easily growing enough to feed themselves plus the remaining 20% who are not farmers.
Now of course that's the ideal case, and bad weather, war and other mishaps are going to negatively affect that to the point where in actual fact there were periods of starvation and food related unrest. But it means that "80% of townspeople worked on only 20% of the whole land" is perfectly believable.