There are two assumptions that need to be clarified.
- What is the attacker's strategic intent?
- What time are you talking about?
If the attacker wants to possess the territory defended by the castle, then "going around" isn't an option. "Going around" only makes sense if the attacker wants to control territory beyond the castle.
This also assumes that the defender isn't stupid—the defender has placed the castle to inhibit the attacker. Either the castle is placed to control a strategic point (See @user5088's answer), or the castle is part of a network (English castles on the border of Wales, or English fortifications on the southern coast to protect against Napoleon).
If the defender is clever, the castle is positioned in a way so that "going around" the castle exposes the supply line, and if the defender is really clever, then the castle is situated so that the attacker cannot avoid the castle.
The original castles (motte and bailey) were really only intended to protect the inhabitants of the local region. When raiders attacked, the population retreated within the castle with their goods. The attacker certainly could go around.
Later castles were designed as strategic reserves. Everything valuable (people, crops, animals, chattel, etc.) were withdrawn within the castle. That means that the attacker can't just go around the castle, the attacker has to go around the entire cultivated area—because the defender has laid waste to the territory. If you can't bring it in the castle, burn it. Foraging is very expensive—even up until the US Revolutionary war, foraging determined whether the army could survive in the field.
The attacker's only option was to carry all supplies (food, ammunition, replacement stocks & stores, animals, feed for draft animals, etc.). The supply train consumes supplies as well as carrying them—every mile that the defender can extend the supply lines is more expensive for the attacker. And every mile that the defender can force the supply train to travel makes it exponentially more expensive to protect that supply train against sallies by the defender.