'Ghost walls' is a concept that is used in archaeology. But maybe not just so fanciful as in these historical fiction books:
Meanwhile, trial trenches at the north end of the adjacent long and narrow meadow—on the surface of which, when ploughed, stray finds of Roman pottery and coins had often been made—revealed a well-defined layer of Roman building debris associated with ‘ghost-walls’, or foundation-trenches from which all masonry had been removed by stone-robbers. Below this was a stratum of dark occupation-earth with an abundance of pottery indicating the presence of an extensive Early Iron Age settlement.
The Oxford Archaeology Dictionary refers to this as:
It seems reasonable to assume that the fancy name gives rise to imagination if you hear it.
Some books mention 'ghost fences' en passant in the way described in the question:
Alistair Moffat: "The Borders: A History of the Borders from Ealiest Times", Birlinn, 2011.
Alistair Moffat: "The Sea Kingdoms: The History of Celtic Britain and Ireland", Birlinn, 2011
And a few more. Seemingly with a claim to be real science.
But, as you can see, this concept seems to be a pet peeve of one man, outside of fiction and academia.