Obviously this depends on how many defenders there were inside the castle. A castle garrisoned by a single person could probably live reasonably well off the chickens that might be in the bailey, for instance. He could even start a vegetable garden or some such.
Realistically, no castle could hope to produce enough food to sustain a reasonably-sized garrison, and I doubt that anyone would have wanted to try. In fact, the whole point of a castle is so that you don't have to defend all your farms and forests and pastures.
Building and maintaining a castle is not cheap. The expense could be justified because you would gather (ideally) everything of value (treasures, food, your lives) inside one in the event of war. That, however, leaves little of any value in the countryside, apart from the land itself. Since even the best raiders in the world can't possibly take your lands with them, it simply doesn't make a whole lot of economic sense to protect them with expensive castle-like fortifications.
Even if we suppose that cost is no object, it is still militarily unsound to encircle so much land because doing so vastly lengthens your line of defence. In a normal castle, the garrison could relatively easily react to an assault and rally to it. However, if the garrison is spread across miles, then the besiegers could concentrate its forces on one specific spot. The defenders there would easily be overwhelmed long before the rest of the garrison could reinforce them.
Thing is, a castle typically relied on a larger territory than its garrison could defend in the first place. Enlarging the fortifications to cover those farms would also increase the number of defenders needed. Now, you are correct that the enclosed area could increase faster than the length of the enclosing walls. So theoretically, you could enlarge your castle until the balance tips in your favour.
But it's more likely that your "castle" would end up looking like this:
Note: During times of siege, livestock were often sheltered within the castle grounds, and slaughtered as needed for food. This was not exactly a sustainable source of food, however, nor could it feed an entire garrison for any length of time.